On Friday, October 21, 2016, there was a major distributed denial of service (DDoS) attack that took down major U.S. company websites, including Twitter, Paypal, The New York Times, Box, Netflix and more. The attack targeted managed DNS provider Dyn Inc., which hosts the authoritative DNS for these popular domains. The attack originated from a large number of compromised IoT devices, including internet-connected cameras, routers and digital video recorders.
Around the world, communications service providers (CSPs) and subscribers were affected by the attack, making it virtually impossible to reach these popular websites for several hours. Although CSPs weren't targeted directly, they were still affected since the outages drove additional caching DNS traffic caused by the errors from failed DNS requests. This spike in traffic slowed overall network performance, likely driving up customer support call volumes from unhappy subscribers.
The attacks highlighted the easily overlooked--yet vital--role that DNS plays on the internet. A lone attacker was able to prevent hundreds of millions of internet users from accessing their favorite sites by targeting a single managed DNS provider. Given the growth in IoT devices, the scale and frequency of these types of attacks are likely to increase. Without question, CSPs must be prepared for the unfortunate day when their DNS--or one of their subscribers--is the intended target of an attack, so as to preserve both network and brand integrity.
A few key steps CSPs can take to prepare for similar attacks in the future are outlined below.
1. Monitor DNS carefully
The website failures during the recent DDoS attack caused a surge in "SERVFAIL" errors as subscriber queries to these popular domains generated error responses. The chart below shows a surge in SERVFAIL errors from the attack, taken from a sample of Nominum, now part of Akamai, CSP customers around the world. The yellow line represents the ration of "SERVFAIL" errors to total responses, which peaked at a remarkable 30%+ of traffic on the day of the attack.
The ratio of DNS responses by response code to overall DNS responses (RC2 = SERVFAIL)
CSPs must have tools in place to monitor these common errors so they can quickly drill down and see the top clients and domains generating the errors. With proper monitoring and tools, Network Operations can identify root causes within minutes, at which point they can isolate the issue and provide accurate details to call center personnel about the sites affected, as well as to subscribers with misconfigured devices.
2. Design DNS architecture for internet storms
When evaluating DNS software, network teams tend to look only at queries per second (QPS) as an indication of reliability, but these metrics can be misleading. Instead, network teams must evaluate how the DNS performs on the worst days when traffic patterns are highly unusual. Common DNS implementations have very simple rules that don't differentiate between legitimate and attack traffic. In the case of the latest attack, when the authoritative DNS servers were unable to respond to queries, the querying servers continued to flood the authoritative servers, waiting hopelessly for a response. This overwhelms the DNS server and slows DNS responses to all queries--both legitimate and malicious traffic--creating a major "traffic jam," which can bring the internet to a halt.
Nominum's, now part of Akamai, CacheServe, on the other hand, handled these errors smoothly, largely due to its "success-based rate limiting" feature. Success-based rate-limiting automatically detects non-responding authoritative DNS servers and immediately slows queries to these servers, substantially reducing attack traffic to the target sites, preserving the integrity of the network and ensuring the lowest possible latency for all queries.
3. Consider partnering with a secondary authoritative DNS & anti-DDoS vendor
Given the massive scale of attacks taking place today, it is difficult for CSPs to provision enough authoritative DNS capacity to address the biggest attacks on their own. There is now a mature industry with hosted authoritative DNS and anti-DDoS services that can be deployed to complement a service provider's authoritative DNS. Such services can be easily and securely configured to handle queries when the CSP's authoritative service becomes overwhelmed.
4. Enforce security best practices whenever possible
A significant portion of these attacks come from DVRs, webcams and other connected consumer devices, whose poorly configured security credentials allow them to be easily compromised. Any device managed directly by a service provider should follow strict security best practices. Such best practices require highly secure passwords before allowing the device to connect, use secure protocols such as HTTPs whenever possible and design devices to receive automated remote security updates without requiring user action.
5. Prioritize IoT security
There are now billions of connected IoT devices, most of which aren't controlled directly by the CSP, meaning there is only so much a service provider can do to enforce good security best practices. Many of these devices are inexpensive and don't offer strong security protections. In fact, Dyn reported that more than 10 million devices were used in this latest attack against them; additionally, Nominum has been tracking exponential growth in compromised IoT devices since the source code was released in early October.
Unfortunately, Nominum anticipates more IoT-based attacks in the near future. Our Data Science team has been monitoring malicious DNS queries from the Mirai botnet to these same domains and other popular domains for several weeks. While the exact reason for this activity remains unknown, we suspect it was used as a test for executing larger DNS-based or other types of attacks such as cache poisoning.
DNS is a great place to invest in IoT security since compromised IoT devices are using DNS for legitimate purposes such as checking for software updates and malicious communications, including command and control and DNS-based DDoS attacks.
Last week's attack was a wake-up call that put a spotlight on the importance of DNS, and the impact of IoT-based attacks on the internet and on CSPs. CSP security and operations teams should use this as an opportunity to evaluate their preparedness for an attack on their DNS, as well as on broader IoT-based attacks that originate on their network. Nominum is investing heavily in this area, designing products that work to prevent malicious attacks on DNS and IoT devices.