DNSFS, store files in others DNS resolver caches

01-13 00:25

DNSFS. Store your files in others DNS resolver caches

A while ago I did a blog post about how long DNS resolvers hold results in cache for, using RIPE Atlas probes testing against their default resolvers (in a lot of cases, the DNS cache on their modem/router).

That showed that some resolvers will hold DNS cache entries for a whole week if asked to ( https://blog.benjojo.co.uk/post/dns-resolvers-ttl-lasts-over-one-week ), and I joked at the end that one could use this for file storage.

Well, I could not stop thinking about doing this. There are surely a lot of open DNS resolvers out on the internet, that are just asking to be used for storing random things in them. Think of it. Possibly tens of gigabytes of cache space that could be used!

This is not the first time something like this has been done, Erik Ekman made PingFS , a file system that stores data in the internet itself .

This works because inside every ping packet is a section of data that must be sent back to the system that sent the ping, called the data payload:

Because you can put up to 1400-ish bytes in this payload, and pings take time to come back, you can use the speed of light in fiber as actual storage.

Now obviously this is not a great idea for long term data storage, since you have to keep transmitting and receiving the same packets over and over again, plus the internet gives no promise that the packet won’t be dropped at any time, and if that happens then the data is lost.

However. DNS has caches. It has caches everywhere .

This means that the DNSFS looks a lot of the same as PingFS, but once a query is sent it should be cached in the resolver, meaning you don’t have to keep sending packets to keep the data alive!

Resolver strategy

For this to work we need a lot of open DNS resolvers. Technically DNS resolvers (except the official ones that a ISP gives out) should be firewalled off from the public internet because they are a DDoS reflection risk , but a lot of devices out there ship with bad default configuration that allows their built in DNS resolvers to be reachable from outside the LAN.

The more open DNS resolvers there are, the more redundancy (or storage space) we have.

For this, we need to scan the whole internet for resolvers. This is a slightly daunting task, however when you take into account the ranges on the internet that are not routable and ranges of those who do not want to be scanned , it amounts to about 3,969,658,877 IP addresses.

In addition to that we are looking for open resolvers, this means that the DNS server on the other end must be able to look up public domain names, most DNS servers are setup to be authoritative for a single domain name, and can’t be used by us for file storage.

Getting a list of DNS resolvers

For this, I am using Robert Graham’s masscan to send DNS queries to all applicable IP addresses on the internet.

However this command has a problem, I am looking for open resolvers, not just things that will reply to port 53 on UDP.

My solution is to use a great feature of the linux kernel called BPF filters ( you can read a great article about BPF filters and their use to filter traffic on the Cloudflare blog ). You can use them with iptables to drop any traffic you don’t want, but programmatically! One BPF rule can do a whole chain worth of work.

I managed to write a tcpdump filter that only matched the DNS responses that I wanted (ones with a single results inside them).

tcpdump -ni eth0 port 53 and udp and ip[35] != 0x01 and net 185.230.223.69/32

I then compiled it to a raw BPF rule using a small helper program :

root@xxxx:~/masscan# ./bpf-gen  RAW 'port 53 and udp and ip[35] != 0x01 and net 185.230.223.69/32'
25,48 0 0 0,84 0 0 240,21 21 0 96,48 0 0 0,84 0 0 240,21 0 18 64,48 0 0 9,21 16 0 132,21 15 0 6,21 0 14 17,40 0 0 6,69 12 0 8191,177 0 0 0,72 0 0 0,21 2 0 53,72 0 0 2,21 0 7 53,48 0 0 35,21 5 0 1,32 0 0 12,21 2 0 3118915397,32 0 0 16,21 0 1 3118915397,6 0 0 65535,6 0 0 0

and then inserted it into IPTables:

root@xxxx:~/masscan# iptables -I INPUT -m bpf --bytecode "25,48 0 0 0,84 0 0 240,21 21 0 96,48 0 0 0,84 0 0 240,21 0 18 64,48 0 0 9,21 16 0 132,21 15 0 6,21 0 14 17,40 0 0 6,69 12 0 8191,177 0 0 0,72 0 0 0,21 2 0 53,72 0 0 2,21 0 7 53,48 0 0 35,21 5 0 1,32 0 0 12,21 2 0 3118915397,32 0 0 16,21 0 1 3118915397,6 0 0 65535,6 0 0 0" -j DROP

Now masscan will only see and then log results I am interested in. No need to do a 2nd pass to qualify servers.

After waiting about 24 hours for the scan to complete I got only two abuse notices! One was automated and very formal, the other not so much:

I think this guy misunderstands that a NOC abuse contact isn't for sending that kind of abuse to pic.twitter.com/t0ZIbgKyai

— Ben Cox (@Benjojo12) November 17, 2017

At the end I was left with 3,878,086 open DNS resolvers, from all ranges and places in the world. Visualised nicely from Cloudflare’s DNS traffic analytics:

Basically, where there are Cloudflare data centers, there are open DNS resolvers.

The country breakdown for open resolvers is as follows:

ben@metropolis:~/Documents/dnsfs/bulk-mmlookup$ pv ../uniqiplist.txt | bulk-mmlookup | awk '{$1 = ""; for (i=2; i<NF; i++) printf $i " "; print $NF}' | sort | uniq -c | sort -n | tac
52.9MiB 0:01:03 [ 850KiB/s] [=======================================>] 100%            
1498094 China
 285749 United States
 233549 Republic of Korea
 168979 Russia
 167145 Brazil
 153170 Taiwan
 142655 India
  83581 Italy
  76894 Turkey
  69300 Poland
  62542 Philippines
  53266 Indonesia
  46055 Japan
  43331 Romania
  40434 Bulgaria
  39548 Australia
  36099 Iran
  32996 Canada
  29971 South Africa

And ISP:

ben@metropolis:~/Documents/dnsfs/bulk-mmlookup$ pv ../uniqiplist.txt | bulk-mmlookup -type isp | awk '{$1 = ""; for (i=2; i<NF; i++) printf $i " "; print $NF}' | sort | uniq -c | sort -n  | tac
52.9MiB 0:00:15 [3.37MiB/s] [=======================================>] 100%            
 830615 4134 No.31,Jin-rong Street
 355713 4837 CHINA UNICOM China169 Backbone
 146755 4766 Korea Telecom
 140772 3462 Data Communication Business Group
  86075 4812 China Telecom (Group)
  67874 9829 National Internet Backbone
  62325 209 Qwest Communications Company, LLC
  59514 9121 Turk Telekom
  56682 3269 Telecom Italia
  55965 9299 Philippine Long Distance Telephone Company
  54555 4847 China Networks Inter-Exchange
  50841 12389 PJSC Rostelecom
  48674 5617 Orange Polska Spolka Akcyjna
  47039 4808 China Unicom Beijing Province Network
  40641 26599 TELEFÔNICA BRASIL S.A
  35171 8866 Vivacom
  33075 5650 Frontier Communications of America, Inc.
  31921 9318 SK Broadband Co Ltd
  30181 8708 RCS & RDS
  29821 9808 Guangdong Mobile Communication Co.Ltd.
  28777 17974 PT Telekomunikasi Indonesia
  28731 7738 Telemar Norte Leste S.A.
  25634 701 MCI Communications Services, Inc. d/b/a Verizon Business
  23598 35819 Bayanat Al-Oula For Network Services
  23427 26615 Tim Celular S.A.
  23322 42610 PJSC Rostelecom
  22488 7018 AT&T Services, Inc.
  17009 4713 NTT Communications Corporation
  14707 12880 Information Technology Company (ITC)
  14259 24560 Bharti Airtel Ltd., Telemedia Services
  13989 14420 CORPORACION NACIONAL DE TELECOMUNICACIONES - CNT EP
  13462 7303 Telecom Argentina S.A.
  12069 6713 Itissalat Al-MAGHRIB
  11865 13999 Mega Cable, S.A. de C.V.
  11439 45899 VNPT Corp
  10711 7922 Comcast Cable Communications, LLC
  10256 28006 CORPORACION NACIONAL DE TELECOMUNICACIONES - CNT EP
   9993 45595 Pakistan Telecom Company Limited
   8896 4739 Internode Pty Ltd

However, this data is kind of meaningless. Because all it is really showing is the biggest ISP and countries. So let’s try looking at the open resolvers per internet user in a country:

Country Open Resolvers Internet users People per resolver
Saint Kitts and Nevis 729 37210 51.0
Grenada 771 41675 54.1
Marshall Islands 158 10709 67.8
Bulgaria 40434 4155050 102.8
Belize 1063 165014 155.2
Republic of Korea 233549 43274132 185.3
Bermuda 308 60047 195.0
Maldives 968 198071 204.6
Guam 601 124717 207.5
Antigua and Barbuda 262 60306 230.2
Ecuador 28923 7055575 243.9
Romania 43331 11236186 259.3
Hong Kong 18698 5442101 291.1
Barbados 739 228717 309.5
Guyana 954 305007 319.7
Cayman Islands 136 45038 331.2
Seychelles 158 56168 355.5
Liechtenstein 100 36183 361.8
Tunisia 13733 5472618 398.5
Poland 69300 27922152 402.9
Georgia 5078 2104906 414.5
Malta 787 334056 424.5
Andorra 155 66728 430.5
Moldova 4396 1946111 442.7
Italy 83581 39211518 469.1
China 1498094 721434547 481.6
Gabon 358 182309 509.2

Or, if we strip the countries with less than 1 million internet users:

Country Open Resolvers Internet users People per resolver
Bulgaria 40434 4155050 102.8
Republic of Korea 233549 43274132 185.3
Ecuador 28923 7055575 243.9
Romania 43331 11236186 259.3
Hong Kong 18698 5442101 291.1
Tunisia 13733 5472618 398.5
Poland 69300 27922152 402.9
Georgia 5078 2104906 414.5
Moldova 4396 1946111 442.7
Italy 83581 39211518 469.1
China 1498094 721434547 481.6
Australia 39548 20679490 522.9
Turkey 76894 46196720 600.8
Russia 168979 102258256 605.2
Singapore 7552 4699204 622.2
Bolivia 7184 4478400 623.4
Panama 2673 1803261 674.6
Ukraine 27718 19678089 709.9
Philippines 62542 44478808 711.2
Kuwait 4493 3202110 712.7
Lebanon 6333 4545007 717.7
Costa Rica 3486 2738500 785.6
Saudi Arabia 25837 20813695 805.6
Brazil 167145 139111185 832.3
Sweden 10872 9169705 843.4
Uruguay 2654 2238991 843.6
Dominican Republic 6476 5513852 851.4
Morocco 23189 20068556 865.4
Armenia 1743 1510906 866.8

You can find the data yourself here oras a CSV here

How long do resolvers hold cache for?

Before testing this, I waited 10 days to allow for all of the resolvers who could be on dynamic IP addresses to change and become unusable. This waiting lost me 37.9% of my IP list.

RIPE Atlas did a decent amount of work to show that a non dismissable percentage of Atlas probes (normally on residential connections) change the public IP address once a day:

After that, I replicated the old method, a very long TTL (in this case, 68 years) and a TXT record containing the current unix timestamp of the DNS server.

The initial query rate on my DNS server was interesting, showing a large initial “rush”, likely due to some DNS resolvers having multiple levels of caches (and thus those upper caches warming up)

I then queried every server in the list every hour for about a few days using a simple tool link

Then refactored some code on the RIPE atlas post to work with my dataset. This showed that 18% of resolvers can hold items in cache for about a day:

ben@metropolis:~/Documents/dnsfs$ cat retention.csv | awk -F ',' '{print $1}' | grep -v time | ./mm -p -b 6
Values min:0.00 avg:470.30 med=56.00 max:1778.00 dev:635.18 count:2387821
Values:
 value |-------------------------------------------------- percent
     0 |************************************************** 45.28%
     1 |                                                    0.29%
     6 |                                                **  2.08%
    36 |                                        **********  9.70%
   216 |                        ************************** 24.27%
  1296 |                              ******************** 18.38%

ben@metropolis:~/Documents/dnsfs$ cat retention.csv | awk -F ',' '{print $1}' | grep -v time | ./mm -b 6
Values min:0.00 avg:470.30 med=56.00 max:1778.00 dev:635.18 count:2387821
Values:
 value |-------------------------------------------------- count
     0 |************************************************** 1081109
     1 |                                                   6816
     6 |                                                ** 49721
    36 |                                        ********** 231737
   216 |                        ************************** 579613
  1296 |                              ******************** 438825

The next task is to make a basic guess on how big the cache is in these 400k open resolvers. To assist in this guesswork, I queried version.bind on every one of these resolvers, filtered down to the major brands of DNS server, and ended up with the following:

The default cache configurations for these devices are as follows:

Brand Cache Size
DNSMasq 150 Entries
BIND 10MB worth
PowerDNS 2MB worth
Microsoft Unlimited???
ZyWall Unknown
Nominum Unknown

At this point I figured a maximum of 9 TXT records, each 250 character base64 string long (187 bytes ish) would be reasonable.

This means we get approximately (3 * 438825 * 187) = 250~ MB. Assuming we replicate to 3 different resolvers for each TXT record this should also prove “stable” enough to store files for at least a day.

Building the file system

As much as I like FUSE I found that due to modern assumptions of desktop linux it’s impractical to use it for high latency backends, which DNSFS would be since its data rate is very low (but still faster than a floppy disk!). So I chose a simple HTTP interface to upload and fetch files to and from the internet.

The DNSFS code is a relatively simple system, every file uploaded is split into 180 byte chunks, and those chunks are “set” inside caches by querying the DNSFS node via the public resolver for a TXT record. After a few seconds the data is removed from DNSFS memory and the data is no longer on the client computer.

To spread the storage load, the request filename and chunk number is hashed, then modulated over the resolver list to ensure fair (ish) spread.

Demo

For the demo, I will store one of my previous blog posts in the internet itself!

As you can see, my file has made quite a name for itself.

As always, you can find the code to my madness on my github here: https://github.com/benjojo/dnsfs

And you can follow me on Twitter here for micro doses of madness like this: https://twitter.com/benjojo12

原文链接:https://blog.benjojo.co.uk/post/dns-filesystem-true-cloud-storage-dnsfs?utm_source=tuicool&utm_medium=referral
标签: Linux命令
© 2014 TuiCode, Inc.