1. Harness Your Internet Activity

2. AAAA Deep Dive DNS-OARC, Buenos Aires March 2016 Ralf Weber

3. Geoff Houston talk at RIPE –DNS doesn’t use IPv6 Our default configuration at least didn't –DNS should use IPv6 What would be the impact? Find the state of IPv6 transport in the long tail –Alexa Top 1M isn’t long enough! –I’m not set up to do Geoff’s neat ad network trick! –I am set up to gather anonymized resolver data 3 Motivation for this talk

4. 4 How Nominum Gets Data Customer Resolvers Receivers Hadoop HDFS Receivers Kafka Hadoop Loader n x 100B queries/day stats 600 cores 8T RAM n x Pbytes storage stats

5. Unique Name Query-Type Tuples –We do daily rollups so a day looked like a natural choice –Raw Data 1,152,389,150 (1.15 Billion ) To much to run and analyze from –Only used data that has been queried more than once 602,661,609 (602 million) Still a lot –Remove known PRSD and DNS tunnels 135,919,893 (135 million ) 5 Getting a test data set

6. 135,919,893 Unique tuples 125,889,174 Unique names 27,466,881 Core domains Query Type distribution –108,509,872 A –11,663,222 AAAA –46,350 SPF –7,140 A6 –1,178 DNSKEY –12 HINFO –3 TLSA 6 What is in the test data set

7. 7 Test Setup

8. Use a couple of dnsperf to run the queries simultaneously against the hosts –Every host gets 1000qps –Timeout is 60 seconds as every query is cold cache – dnsperf -d allq.new -Q 1000 -q 100000 -t 60 -S 1 –s IP –Test ran for nearly 38 hours over a weekend 8 Test Run

9. 9 Result error codes

10. 10 Result timings

11. 11 Questions asked

12. 12 Servers talked to

13. 13 Question answered IPv4IPv6 IPv4 then 6IPv6 then 4 UDP Ok UDP Timeout TCP OK TCP Timout 0.08% 0.07% 0.18% 0.07%

14. 14 Question answered per Protocol IPv4 IPv6 UDP Ok UDP Timeout TCP OK TCP Timout IPv4 then 6 IPv6 then 4 0.04% 0.09% 0.06% 0.08%

15. ip | timeout | ok -----------------+---------+--------- 199.7.91.13 | 1196211 | 3617819 24.85 173.245.58.107 | 1030401 | 4836467 17.56 192.36.148.17 | 903004 | 5691109 13.69 192.43.172.30 | 886747 | 4528054 16.38 173.245.59.78 | 814280 | 3996683 16.93 192.42.93.30 | 771203 | 4071314 15.93 173.245.58.93 | 763090 | 2895030 20.86 192.35.51.30 | 739739 | 7579568 8.89 140.205.228.52 | 691765 | 5384750 11.38 173.245.59.202 | 689620 | 1936437 26.26 140.205.228.51 | 681882 | 3773220 15.31 192.41.162.30 | 669132 | 5223907 11.35 192.5.4.1 | 643232 | 300601 68.15 199.212.0.53 | 600026 | 282748 67.97 192.5.6.30 | 581040 | 2814187 17.11 192.31.80.30 | 565395 | 3942251 12.54 192.26.92.30 | 548059 | 3210573 14.58 192.134.0.49 | 541920 | 206272 72.43 192.58.128.30 | 534831 | 2557051 17.30 202.12.29.59 | 533246 | 89832 85.58 202.12.28.140 | 520510 | 123352 80.84 193.0.9.5 | 516087 | 173760 74.81 192.55.83.30 | 508246 | 5255387 8.82 15 Timeout offenders IPv4

16. ip | timeout | ok -----------------+---------+--------- 199.7.91.13 | 1196211 | 3617819 24.85d.root-servers.net. 173.245.58.107 | 1030401 | 4836467 17.56dina.ns.cloudflare.com. 192.36.148.17 | 903004 | 5691109 13.69i.root-servers.net. 192.43.172.30 | 886747 | 4528054 16.38i.gtld-servers.net. 173.245.59.78 | 814280 | 3996683 16.93buck.ns.cloudflare.com. 192.42.93.30 | 771203 | 4071314 15.93g.gtld-servers.net. 173.245.58.93 | 763090 | 2895030 20.86dee.ns.cloudflare.com. 192.35.51.30 | 739739 | 7579568 8.89f.gtld-servers.net. 140.205.228.52 | 691765 | 5384750 11.38 173.245.59.202 | 689620 | 1936437 26.26marek.ns.cloudflare.com. 140.205.228.51 | 681882 | 3773220 15.31 192.41.162.30 | 669132 | 5223907 11.35l.gtld-servers.net. 192.5.4.1 | 643232 | 300601 68.15sns-pb.isc.org. 199.212.0.53 | 600026 | 282748 67.97tinnie.arin.net. 192.5.6.30 | 581040 | 2814187 17.11a.gtld-servers.net. 192.31.80.30 | 565395 | 3942251 12.54d.gtld-servers.net. 192.26.92.30 | 548059 | 3210573 14.58c.gtld-servers.net. 192.134.0.49 | 541920 | 206272 72.43ns3.nic.fr. 192.58.128.30 | 534831 | 2557051 17.30j.root-servers.net. 202.12.29.59 | 533246 | 89832 85.58cumin.apnic.net. 202.12.28.140 | 520510 | 123352 80.84sec3.apnic.net. 193.0.9.5 | 516087 | 173760 74.81pri.authdns.ripe.net. 192.55.83.30 | 508246 | 5255387 8.82m.gtld-servers.net. 16 Timeout offenders IPv4

17. ip | timeout | ok ----------------------------------------+---------+---------- 2001:503:a83e::2:30 | 3455836 | 10516057 24.73 2001:503:231d::2:30 | 2975407 | 7492273 28.42 2001:500:2d::d | 1294969 | 6207712 17.26 2001:7fe::53 | 1127203 | 6256220 15.27 2400:cb00:2049:1::adf5:3a6b | 1029654 | 5197966 16.53 192.42.93.30 | 1007175 | 3539741 22.15 2400:cb00:2049:1::adf5:3a5d | 829338 | 4002273 17.16 2001:500:2e::1 | 801598 | 558786 58.92 140.205.228.52 | 787870 | 5046117 13.50 192.35.51.30 | 763128 | 3177282 19.37 2400:cb00:2049:1::adf5:3bca | 728956 | 3707200 16.43 2400:cb00:2049:1::adf5:3b4e | 707564 | 3561658 16.57 140.205.228.51 | 682317 | 3276224 17.24 2001:500:13::c7d4:35 | 638586 | 521112 55.06 2001:12f8:4::10 | 615008 | 1130405 35.24 2001:dc0:1:0:4777::140 | 598651 | 153675 79.57 192.41.162.30 | 594018 | 3497933 14.52 192.43.172.30 | 565599 | 1973125 22.28 192.31.80.30 | 561970 | 2648000 17.51 2001:dc0:2001:a:4608::59 | 542729 | 121656 81.69 192.55.83.30 | 503993 | 2258980 18.24 2001:67c:e0::5 | 491423 | 128349 79.29 192.5.6.30 | 467610 | 2696959 14.78 2001:660:3006:1::1:1 | 451779 | 19991 95.76 17 Timeout offenders IPv6 then IPv4

18. ip | timeout | ok ----------------------------------------+---------+---------- 2001:503:a83e::2:30 | 3455836 | 10516057 24.73 a.gtld-servers.net. 2001:503:231d::2:30 | 2975407 | 7492273 28.42 b.gtld-servers.net. 2001:500:2d::d | 1294969 | 6207712 17.26 d.root-servers.net. 2001:7fe::53 | 1127203 | 6256220 15.27 i.root-servers.net. 2400:cb00:2049:1::adf5:3a6b | 1029654 | 5197966 16.53 dina.ns.cloudflare.com. 192.42.93.30 | 1007175 | 3539741 22.15 g.gtld-servers.net. 2400:cb00:2049:1::adf5:3a5d | 829338 | 4002273 17.16 dee.ns.cloudflare.com. 2001:500:2e::1 | 801598 | 558786 58.92 sns-pb.isc.org. 140.205.228.52 | 787870 | 5046117 13.50 192.35.51.30 | 763128 | 3177282 19.37 f.gtld-servers.net. 2400:cb00:2049:1::adf5:3bca | 728956 | 3707200 16.43 marek.ns.cloudflare.com. 2400:cb00:2049:1::adf5:3b4e | 707564 | 3561658 16.57 buck.ns.cloudflare.com. 140.205.228.51 | 682317 | 3276224 17.24 2001:500:13::c7d4:35 | 638586 | 521112 55.06 tinnie.arin.net. 2001:12f8:4::10 | 615008 | 1130405 35.24 d.dns.br. 2001:dc0:1:0:4777::140 | 598651 | 153675 79.57 sec3.apnic.net. 192.41.162.30 | 594018 | 3497933 14.52 l.gtld-servers.net. 192.43.172.30 | 565599 | 1973125 22.28 i.gtld-servers.net. 192.31.80.30 | 561970 | 2648000 17.51 d.gtld-servers.net. 2001:dc0:2001:a:4608::59 | 542729 | 121656 81.69 sec1.apnic.net. 192.55.83.30 | 503993 | 2258980 18.24 m.gtld-servers.net. 2001:67c:e0::5 | 491423 | 128349 79.29 pri.authdns.ripe.net. 192.5.6.30 | 467610 | 2696959 14.78 a.gtld-servers.net. 2001:660:3006:1::1:1 | 451779 | 19991 95.76 ns3.nic.fr. 18 Timeout offenders IPv6 then IPv4

19. Servers that timeout are regular server that usually answer good I guess we see RRL in action Seems that people are not switching to TCP Good that DNS scales horizontally 6000 – 8000 qps is not much traffic outbound –Rule of thumb is 5 – 10% of inbound gets send out –Resolvers can easily do a couple of 100k qps inbound –Does this affect normal operation (another talk…) Maybe do a second test.... 19 Looking into timeouts

20. Found another DNS tunnel in the dataset –Put it on our list –Removed queries (~500k) First test –All servers were asking the same at the same time –Total of 6000 qps Second Test –Offset start time by 30 minutes for each test run –Lowered qps to 800 per test (total 4800 qps) Test now ran over 48 hours 20 Second test…

21. 21 Result error codes Test 2

22. 22 Result timings Test 2

23. 23 Questions asked Test 2

24. 24 Servers talked to Test 2

25. 25 Question answered Test 2 IPv4IPv6 IPv4 then 6IPv6 then 4 UDP Ok UDP Timeout TCP OK TCP Timout 0.03% 0.06% 0.03%

26. 26 Question answered per Protocol Test 2 IPv4 IPv6 UDP Ok UDP Timeout TCP OK TCP Timout IPv4 then 6 IPv6 then 4 0.06% 0.03% 0.001% 0.03%

27. Second test had less servers not answering Overall answers were faster and better The more baskets you have the better Still wonder if the low auth qps has an impact on production servers –Payload is different –At least cold cache could see the same problem 27 Analysis

28. Turning on IPv6 as additional transport has only good effects –More baskets –More resilliency Should be enabled by default –Latest Cacheserve version has (IPv4 then IPv6) 28 Summary