September 11, 2016

Tor @ DarkScience

For completeness I’m going to write this post as if you know nothing about me or Tor. If you know about DarkScience and Tor you can safely skip to here.

Tor #

[Tor], or more formerly [“The Onion Router”] is a method of decentralised VPN, it’s more commonly referred to as a “Anonymity network” and that would be very apt. The design of Tor is an encrypted mesh network operating over the top of the regular internet, there are intermediaries which have no concept of the data, where it came from or where it’s headed- only it’s next hop in the chain. There are entry points who only know where the data came from but not where it’s going or what it is, and more famously there are exits which take traffic originating from somewhere in the tor network and allow it to enter the public internet.

Tor, typically is rather slow, and it’s possible to address servers and services without ever leaving tor. These services typically have addresses that look like domains with the [.onion] TLD.

DarkScience #

I help run DarkScience, in fact, as of writing I am the majority machine holder and domain registree of the darkscience.net and drk.sc domains.

DarkScience is an IRC network for the technically literate, it’s sole purpose is to be a social group for those who care about privacy in an increasingly open and non-private world. And also because: nostalgia.

to quote the manifesto:

“DarkScience is the embodiment of the philosophy that centralised communication can still be open, private, and for the benefit of all people and remain that way, unfettered by greed.”

The operations staff at darkscience have always erred on the side of privacy where possible, removing all sa*.so modules to prevent oper abuse or impersonation, forcing TLS on the IRCd and using a signed cert from a widely trusted CA, forcing the services machine to be hidden, hardened and separate from anything user facing and of course running a Tor hidden service.

Due to the nature of Tor we never had a problem with people connecting via Tor over the internet to access darkscience chat systems, this was rarely abused (although, it was abused) by people evading bans and I speculate that it was the quality of the community that kept abuse to a minimum.

Desynchronisation #

These projects share similar principles, but Tor is too prone to abuse and can affect service quality. I’m going to talk about problems we faced and how we solved them.

Problem 1: Users evading bans #

If you have infinite usernames, nicknames and hostnames- you basically cannot be banned on IRC, there is no effective way to enforce a ban against a user who effectively leverages Tor.

Problem 1 Resolution: effective use of channel modes. #

Channel mode +r will stop all but registered accounts from joining, typically ban evaders will not spend disproportionate amounts of time generating accounts only to have them banned within seconds. So this has proven effective. Educating channel owners is the only effective solution here.

Problem 2: Connectivity struggles. #

People who connect via Tor have absolutely appalling connection times, typically timing out multiple times an hour- causing spam in less active channels and often interrupting conversation with the person using Tor.

Problem 2 Resolution: connect via the hidden service. #

Hidden service connections typically work around this by routing differently when nodes go offline and come online.

The latency is still appalling for the Tor user but this is in all ways superior to connecting to IRC via exit nodes.

To fix user behaviour we tried adding an auto-kline to the popular [m_dnsbl.so] but the message about accessing the hidden service was not available, so we modified it to be a 5-minute gline and so far this has shown a 5x increase in hidden service traffic.

Problem 3: Poke and run. #

DarkScience has been undergoing some form of automated bot scanning which has been aggressive enough to trick our connection handlers into thinking we’re being attacked- but not enough to cause our firewall to detect it as an attack and drop it- the happy middle ground where they can cause denial of service but not get caught doing so.

When our connection handlers start to feel paranoid, IRC operators get messages like this:

<celadon.darkscience.net> ANNOUNCEMENT: Connection throttle activated
<celadon.darkscience.net> ANNOUNCEMENT: Connection throttle deactivated
<celadon.darkscience.net> ANNOUNCEMENT: Connection throttle activated
<celadon.darkscience.net> ANNOUNCEMENT: Connection throttle deactivated
<celadon.darkscience.net> ANNOUNCEMENT: Connection throttle activated
<celadon.darkscience.net> ANNOUNCEMENT: Connection throttle deactivated

You cant see but these are 15 minutes or so apart, which means our service in East Coast USA is flapping constantly.

Since we don’t get much info about why from those messages we have to check the packet headers ourselves;

there’s a neat trick you can do with TCP dump to check for connections which do not fully handshake:

sudo tcpdump "tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 and port 6697"

The output is as follows:

23:57:39.696065 IP 191.96.249.110.52056 > celadon.darkscience.net.6697: Flags [S], seq 2069861709, win 14600, options [mss 1460,sackOK,TS val 1876446201 ecr 0,nop,wscale 7], length 0
23:57:40.197290 IP edwardsnowden1.torservers.net.46446 > celadon.darkscience.net.6697: Flags [S], seq 912483580, win 29200, options [mss 1460,sackOK,TS val 154420171 ecr 0,nop,wscale 7], length 0
23:57:40.695172 IP 191.96.249.110.52056 > celadon.darkscience.net.6697: Flags [S], seq 2069861709, win 14600, options [mss 1460,sackOK,TS val 1876447201 ecr 0,nop,wscale 7], length 0
23:57:41.194685 IP edwardsnowden1.torservers.net.46446 > celadon.darkscience.net.6697: Flags [S], seq 912483580, win 29200, options [mss 1460,sackOK,TS val 154420421 ecr 0,nop,wscale 7], length 0
23:57:42.696197 IP 191.96.249.110.52056 > celadon.darkscience.net.6697: Flags [S], seq 2069861709, win 14600, options [mss 1460,sackOK,TS val 1876449201 ecr 0,nop,wscale 7], length 0
23:57:43.198726 IP edwardsnowden1.torservers.net.46446 > celadon.darkscience.net.6697: Flags [S], seq 912483580, win 29200, options [mss 1460,sackOK,TS val 154420922 ecr 0,nop,wscale 7], length 0
23:57:46.695189 IP 191.96.249.110.52056 > celadon.darkscience.net.6697: Flags [S], seq 2069861709, win 14600, options [mss 1460,sackOK,TS val 1876453201 ecr 0,nop,wscale 7], length 0
23:57:47.206776 IP edwardsnowden1.torservers.net.46446 > celadon.darkscience.net.6697: Flags [S], seq 912483580, win 29200, options [mss 1460,sackOK,TS val 154421924 ecr 0,nop,wscale 7], length 0
23:57:49.619985 IP 85.31.101.98.static.nano.lv.58992 > celadon.darkscience.net.6697: Flags [S], seq 2318391510, win 13100, options [mss 1310,sackOK,TS val 1661078256 ecr 0,nop,wscale 7], length 0

As you can see, we’re undergoing some kind of bizarre SYN flood, which is very slow because it’s hitting us via Tor.

Problem 3 Resolution: Make the firewall smarter. #

We can’t modify the IRCd to be smart enough to handle this, but luckily the Tor project itself makes it easy to tag Tor nodes. (for this exact reason I believe)

The URL https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=45.63.7.122&port=6697 contains a list of Tor Exits that can connect to the IP specified (this IP is celadon’s IPv4)

This list can be directly parsed by [pf] as a table which we can add a rule to block!

I added this to celadons /etc/pf.conf:

table <tor> persist file "/etc/torlist.txt"
block quick from <tor>

and created a script to download and update the rules:

wget 'https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=45.63.7.122&port=6697' -O /etc/torlist.txt
pfctl -t tor -T replace -f /etc/torlist.txt

Setting the script on a 20 minute cron seems to keep things updated enough, hopefully we’re not causing much load on Tor’s webserver.

FWIW when it comes to Tor, we were actually getting spammed a lot, since I loaded these rules 2 days ago:

# pfctl -vvsTables
-pa-r-- tor
        Addresses:   953
        Cleared:     Fri Sep  9 13:30:00 2016
        References:  [ Anchors: 0                  Rules: 1                  ]
        Evaluations: [ NoMatch: 56271              Match: 165696             ]
        In/Block:    [ Packets: 165696             Bytes: 9659808            ]
        In/Pass:     [ Packets: 0                  Bytes: 0                  ]
        In/XPass:    [ Packets: 0                  Bytes: 0                  ]
        Out/Block:   [ Packets: 0                  Bytes: 0                  ]
        Out/Pass:    [ Packets: 0                  Bytes: 0                  ]
        Out/XPass:   [ Packets: 0                  Bytes: 0                  ]

that’s more than 3 times our normal traffic that was originating from Tor.

and while I disagree with the British parliaments desire to outlaw Tor I can see their point a little bit.

[Tor]: https://en.wikipedia.org/wiki/Tor_(anonymity_network)
[“The Onion Router”]: https://www.torproject.org/docs/faq#WhyCalledTor
[.onion]: https://en.wikipedia.org/wiki/.onion
[m_dnsbl.so]: https://wiki.inspircd.org/Modules/2.0/dnsbl
[pf]: https://www.openbsd.org/faq/pf/tables.html

Kudos