1. The Challenges of DNS
Resolution in China
Tim Hale, Solutions Engineer

2. Why China? “
In 2008 China became the largest population on the Internet.
As of July 2016, 730,723,960 people (53.2% of the country's total population) were Internet users.
Every tech company I have worked for has had a presence in China. Why is that?
It’s usually to facilitate others tapping into this huge market.
In 2008 China became the largest population on the Internet. As of July 2016, 730,723,960 people (53.2% of the country's total population) were Internet users
"The World Factbook — Central Intelligence Agency".

3. A Different Internet: A different strategy
Frequent Congestion
The Great Firewall
10 backbone access points
2 dominant, government- controlled ISPs
China Unicom
China Telecom
IP blocking
Keyword filtering
DNS tampering
The Chinese Internet looks very different from what the Internet looks like in the west.
In China, Internet service is primarily dominated by two ISPs (China Telecom and China Unicom) that are underdeveloped in some areas and often congested.
On top of that, network access from China to the rest of the world is restricted through a small number of choke points.
In addition, the Great Firewall heavily filters traffic both at the borders and within China, using a wide range of methods including IP blocking, DNS tampering and hijacking and deep packet inspection.
Needless to say, high latency and packet loss are issues much more common in China than in the US or Europe. When monitoring network performance to, from or within China, you’ll need to work with different baselines in mind and understand the common issues that may arise, so you can distinguish typical underperformance from atypical outages.

4. ThousandEyes

5. Frequent Congestion Packet loss Outside China
We’ll first look at an example where low DNS performance results more likely from congested networks that frequently see high levels of packet loss and latency, rather than from direct censorship.
Our example comes from a DNS server test for GitHub’s DNS records, which are served out of Route 53, AWS’s DNS service.
DNS traffic traveling across China’s borders to AWS DNS servers must overcome great obstacles, including significant levels of delay (around 200 ms) within the China Telecom network, likely as a result of congestion, stringent filtering, or both.
As a result of high packet loss and high latency conditions, many of the China agents could not reach the Route 53 DNS name servers at all.
The China agents that were able to reach AWS’s name servers received the correct mappings, though not without resolution times much higher than even Hong Kong, which is geographically close but located outside the Great Firewall.

6. Lower latency Less packet loss Inside China
In contrast, looking at a DNS server test for the AWS S3 China service, we see that performance is far better if DNS records are hosted within China.
In this case, the DNS records for S3 China are hosted within an ISP in Beijing.
Packet loss and latency are greatly improved relative to traffic traveling across the Great Firewall, but issues still frequently arise.

7. nytimes.com ns1.p24. dynect.net Uncensored?
Next, we’ll look at data that clearly shows the effect of censorship on DNS performance. This example comes from a DNS Server test that sends test traffic from all of our China agents to all name servers serving up the A record for nytimes.com.
The New York Times has long been blocked by the Great Firewall, and we’ll see what the repercussions of that block look like for DNS in the next example.
The nytimes.com A record is hosted in 6 name servers, 4 of which are external name servers in Dyn, and 2 of which are internal name servers maintained by the New York Times. We’ll first look at the Path Visualization showing each hop that test traces take from a selection of US and China agents to the Dyn name server ns1.p24.dynect.net. We see that traces from both US and China agents are able to reach the anycast Dyn name server, and traces originating in China successfully cross the Great Firewall into nodes in NTT in Frankfurt and Tokyo.

8. nytimes.com dns-plx.ewr1. nytimes.com Censored
However, the Path Visualization looks drastically different when we look at those same agents targeting the New York Times’ internal name server in Newark, dns-plx.ewr1.nytimes.com.
The US agents are able to look up the correct IP address for the name server ( ) and reach the name server in Newark with no issue.
In contrast, the China agents fail: on the far right, we see a variety of IP addresses that have been returned for the DNS lookup for dns-plx.ewr1.nytimes.com.
Once the agents received the spoofed DNS reply, they then sent traces off to these blocked IP addresses. The traces were then of course terminated within Chinese ISPs that practice IP blocking, which entails blackholing traffic destined for blacklisted IP addresses like the ones we see below. This manifests itself in the Path Visualization as terminating traces in the networks of a number of major Chinese ISPs, including China Telecom and China Unicom.
Traces to the Dyn name servers go unhindered, while traces to the New York Times internal name servers are consistently blocked. This data is strong evidence of censorship — the New York Times name servers are blocked likely because they only host their own DNS records, which are deemed to be sensitive, while the Dyn name servers host a variety of services’ records (both sensitive and otherwise) and should not be blocked on the name server level.

9. Cache poisoning Keyword based hijacking DNS Tampering
Routers hijack DNS requests containing sensitive keywords by injecting forged DNS replies. DNS servers along the route will also cache the tampered DNS responses.

10. DNS Tampering Suspiciously Low latency IPs returned are from other
blocked
destinations
However, blocking name servers’ hostnames is not the only DNS trick at play here. The Great Firewall also employs cache poisoning and DNS hijacking to cover all its bases. Even though it can’t entirely block Dyn’s name servers, it can stop sensitive records from being served up by Dyn’s name servers in other ways.
We can see this when we look at the Server Metrics view that shows the mappings for the nytimes.com A record that each agent received from the first New York Times internal name server we looked at previously. While the US agents all receive the same set of four IP addresses beginning with , the China agents each get a mapping to a single IP address, each of which belongs to a known blocked service like Facebook, similar to the mappings we saw earlier for the New York Times’ name server’s hostname.
The DNS records for nytimes.com that the China agents received are obviously spoofed, but can we tell more about how the tampering was done?
Resolution time gives us some clues — notice that the resolution times for the China agents range from 7-27 ms, while resolution times for the US agents range from ms. Keep in mind the target name server is located in Newark. It’s impossible that the China agents are able to achieve resolution times lower than even US agents that are geographically much closer to the destination.
These impossibly low resolution times indicate that the spoofed mapping was served from somewhere within the Great Firewall, likely via local cache poisoning, which is more lightweight and generally much quicker than keyword-based DNS hijacking.

11. What to Expect Incorrect mappings DNS poisoning and hijacking
Volatile conditions with high latency and loss
Evolving censorship mechanisms (keyword filtering, IP blocking)
Frequent congestion, especially when crossing the GFW
Diurnal patterns in latency and loss for outbound traffic
It should now be clear that DNS can fail in a number of ways within China’s borders; packets can be easily lost on frequently congested networks, or intentionally dropped as a result of censorship techniques. These conditions can change at the drop of a hat — with the political climate, new and evolving censorship policies and even with times of the day.

12. What to Do About It
Continuously monitor DNS tests and alerts to check if records:
Are always available
Have the correct mappings
Are served up quickly
Benchmark performance metrics like latency, packet loss
Adjust your expectations and alerts accordingly
It’s important to remember that application delivery completely relies on the availability of accurate DNS records. To keep abreast of any changes that affect important records, use DNS Server and Trace tests to continuously monitor the state of your DNS records, including their availability, accuracy and resolution time.

13. Thank You