Akamai Diversity
Home > Carrier & Network

Recently in Carrier & Network Category

Distribution: What good is it?

You'll recall from Part I that I describe a scenario where the London node of a network has become overloaded in a DDoS attack. Now you may wonder why if all the users in that scenario going to London are having problems, what good is distribution? Some of you may have already noticed the benefit I glossed over earlier. In our example with a congested London node, the users in San Jose, Tokyo, Sydney, etc. are all unaffected.

This is great news. Not only does distribution make it more likely no node will be overwhelmed, but if one is, there should be lots of others which are not. This minimizes the damage as not all users will suffer during a failure.

At this point you should start feeling sorry for the poor sods going to the London node. Let's see if we can do anything about that.

Overwhelmed nodes: Any way to avoid user pain?

I have tried to convince you that some node or link will get overwhelmed no matter what. But even if you do not believe my logic, the empirical evidence is clear: This has happened, to every CDN. Yes even including Akamai. < I can hear the audible gasps from here. >

If you combine the fact that every CDN has had nodes overwhelmed, and what I said above about users going to the overwhelmed node suffering, the logic seems to say that attacks can and do harm users. Luckily, there is a way to escape this seemingly inevitable fate: Do not send users to overwhelmed nodes.

If only that were as simple as it sounds.

Anycast: How granular is BGP?

Most CDNs use anycast to direct users, either through anycasting their name servers or anycasting their web servers. BGP is a crude tool, lacking granularity and precise control.

Going back to our London example, if a CDN wanted to move traffic off the London node, it has to change something in BGP. If the CDN is anycasting its name server, chances are all it can do is direct traffic away for entire networks. You cannot tell a network "send your east London users to this node's name server, your west London users to Frankfurt's name server" with BGP. Moreover, unless the CDN has multiple prefixes with different name servers in each, it cannot say "send traffic for Customer A to Frankfurt and traffic for Customer B to Amsterdam."

If the CDN is anycasting its web servers, there might be slightly more flexibility. It is possible to send users to London for some web server addresses and Paris for other web server addresses. However, you can still only direct users by network, not sub-group.

Furthermore, many networks require peering partners to do what is called announce consistently. This forces a CDN to announce the exact same thing in BGP to a network in every point where they peer. Without the ability to modify BGP announcements per node, a CDN cannot affect where traffic flows.

Finally, some things in BGP are not black and white. A CDN can remove reachability information, e.g. "you cannot get to web server XYZ in London." But anything short of that, such as "please use Madrid first, and come to London if Madrid is down," is purely a suggestion. The network receiving the BGP announcement is allowed to listen to or ignore any hints provided by the CDN. This means you can say "please use Madrid first, then London" and the peer network might say "no, I'm going to London first." There is nothing the CDN can do other than remove London as a choice completely.

Now, imagine trying to mitigate a massive attack across multiple networks and multiple nodes when the tools you have involve hints which might be ignored or the ability to move traffic from whole network or CDN nodes at once, plus the million other details I did not cover.

Yeah, I don't want to think about it either.

Akamai Mapping: Does it use BGP?

Fortunately for Akamai, we do not use BGP to map users to web servers. Akamai's Mapping System can and does notice overwhelmed nodes in seconds and directs users, regardless of their ISP or internal BGP preferences, to other nodes.

Akamai has many ways of finding problem nodes and fixing them. We send probes out from each node, as well as probes into the nodes. And if that were not enough, we track TCP stats on the node which gives us telemetry on production traffic to real users. Node gets overwhelmed, traffic is moved seconds later automatically. Human involvement is neither required or preferred - people cannot move as fast as computers.

Moreover, Akamai's system is based on DNS, not BGP. We can, and frequently do, direct "east London users to Node A and west London users to Node B" from the same network. Or even "east London users to Node A for Customer Z and west London users to Node B for Customer Y" from the same or multiple networks.

This means even if an attack takes out one of our nodes, the collateral damage is minimal and very short.

On top of that, Akamai has the most traffic of any CDN. By some estimates, we have nearly as much as all other CDNs combined. Having double-digit terabits of outbound traffic means we have to have a lot more than a few hundred Gbps of inbound capacity.

Summary

All these things together make not just serving at the edge, but serving at the edge Akamai style, a great way to fight DDoS.


Patrick Gilmore is a Chief Network Architect at Akamai

Akamai's Chief Security Officer Andy Ellis recently commented on large DDoS attacks and how "size" can be misleading. In that post, Andy notes if you have more than 300 Gbps of ingress capacity, then a 300 Gbps attack is not going to hurt you too much.

He's right of course. However, total ingress capacity is only part of the equation. Also important are the type of attacks you're facing and your "ingress" configuration. I'd like to dig a little deeper into these two topics, and explain how a widely distributed infrastructure is useful for both improving performance and mitigating attacks.

Not surprisingly, I've used a generic CDN example to set the stage, but most of the concepts here apply to any large backbone network with many peering and transit links. Because we are talking about CDNs, we should first ask why CDNs push traffic to the "edge", and even before that, what is the "edge"?

Why serve at the edge?

On the Internet, the "edge" usually refers to where end users ("eyeballs") are located. It is not a place separate from the rest of the internet. In fact, frequently the edge and what most people consider the core are physically a few feet from each other. Topology matters here, not geography.

The reason CDNs want to serve at the edge is it improves performance. Much has been written about this, so I shan't bore you here. The important thing to realize is all CDNs distribute traffic. However, when CDNs were invented, the distribution was not designed to mitigate attacks, it just worked out that way.

And it worked out well. Let's see why.


Ingress capacity: How much is enough?

To set the stage further, we are going to discuss a "typical" DDoS (as if there were such a thing!) and possible mitigation strategies, not a specific attack.

The first and most obvious mitigation strategy is what Andy mentioned in his post: Have enough ingress capacity to accept the traffic, then drop or filter the attack packets. This begs the question of what "ingress capacity" means. If you have a single pipe to the Internet, making that pipe bigger is the only answer. While plausible, that would be very difficult, and very, very expensive to do with the size of attacks seen on the 'Net today.

Now, suppose you have many ingress points, such as a CDN with multiple links and nodes. Do you need to ensure each and every point is large enough for the worst-case DDoS scenario? Doing so would be insanely expensive and, frankly, nearly impossible. Not every point on the Internet can support connections into the 100s of Gbps.

Fortunately, the first 'D' in DDoS is "Distributed", meaning the source of a DDoS is not a single location or network, but spread widely. And because of the way the Internet works, different attack sources will typically hit different CDN nodes and/or links.

The chance of a distributed attack hitting all the same node is very, very small. Of course, the opposite holds as well - the chances of an attack having a perfect spread over all points is essentially nil. As such, you cannot just divide the attack by the number of nodes and assume that amount is the maximum required bandwidth per node to survive an attack. How much more capacity is needed per node depends on the exact situation, and it cannot be predicted in advance. This leads us to our next topic.

Node capacity: How much is enough?

Trying to size each node properly for an attack is an art, not a science. A black art. A black art that is guaranteed to fail at some point.

Of course, everyone still tries. They attempt to estimate what attack capacity is needed based on things like where a node is, what customers are on the system, how much connectivity costs, and several other factors. For instance, a node in Düsseldorf serving a small DSL network exclusively probably does not need as much attack capacity as a large node in Silicone Valley serving several Tier-1 providers combined. Engineers pray to the network gods, sacrifice a router and maybe a switch or two in their names, make a plan, implement it, and... pray some more.

But pray as they might, the plan will fail. Sooner or later, some attack will not fit the estimates, and a node will be overwhelmed with attack traffic. Don't let this get you down if you are making your own attack plan, remember the same is true for everyone. Not only is it impossible to predict how large an attack is going to be, but as mentioned above, it is also impossible to predict what percent of the attack will hit each node. Worse, since unused attack capacity is wasted money - a lot of wasted money - CFOs tend to push for less rather than more, making the plan that much more likely to fail.

The problem with an overloaded node is it doesn't matter how many nodes you have, if one is overloaded, any traffic going to that node will be affected. This means if your link to London is overwhelmed with attack traffic, it doesn't matter how many nodes you have in Tokyo, Sydney, San Jose, etc., your users going to the London node are suffering.

As a result, while CFOs push for less, engineers push for more.

In Part II, I'll cover in greater detail why a distributed infrastructure, such as the Akamai Intelligent Platform, is ideal for mitigating even the largest of DDoS attacks.

 

Patrick Gilmore is a Chief Network Architect at Akamai

This week KT and Akamai announced an important CDN deal. As the leading telecommunications company in South Korea, KT is another important strategic partner for Akamai as we continue to innovate and evolve how our CDN technology and business can bring benefits to telecom networks. In this instance, the basis for the CDN deployment is a Managed Operator CDN that leverages technology from our Aura Network Solutions group.

Looking at this partnership, it brings many mutual benefits to both companies. First, it provides KT with greater network efficiency for all the Akamai global content that gets served within South Korea. Korea has always been a leading edge broadband market with some of the fastest speeds in the world, and this relationship provides an even faster Internet experience, which certainly benefits our customers as well. Second, KT will be able to have their own CDN based on market-leading technology from Akamai, and sell these services to their regional customers bringing benefits to content owners, web properties, and enterprises alike. And finally, Akamai receives the opportunity to leverage KT's experienced sales organization to increase the overall market opportunity for our services in the region. It is further evidence of our commitment to involve the network service providers in the value chain of CDN in a way that best leverages their assets and ours.

Here is a quote from Heekyung Song, SVP for their Enterprise IT BU. "The era has come where culture and digital content are leading the market. Our main priority is to ensure access to content from any device, at anytime, anywhere. Through this partnership with Akamai, KT will provide a CDN platform specializing in media delivery, web performance and security so companies can focus on developing quality content and web applications without concerns about delivery."

Here is a great picture capturing the enthusiasm of the joint team responsible for the partnership.
 korea blog.png



 


There's been a lot of talk about the recent very large DDoS attack against Spamhaus. Although I was quoted in some articles about it, I want to be clear that the attacks did not affect or involve Akamai or our customers. However, we have been the object of similar attacks in the past, and Akamai has a vested interest in making the Internet better - safer, more reliable and higher performing.

Unfortunately, there are some common... let's call them "misconfigurations" on the Internet which make these types of attacks both easier and much more destructive than they should be.

The one most talked about recently is DNS Amplification. This has been discussed many times in many places so I won't go into a great deal of detail. Very generally, a miscreant can send a query (a very small amount of data) to a DNS server and the server will send 100 or more times as much data in response to the victim. It is the equivalent of a stranger sending a postcard to a store and you get a gigantic catalog in return.

Now imagine getting 1000s or even millions of gigantic catalogs delivered to your house - per second. You couldn't even leave your house. What's more, if they send enough, the catalogs could jam your whole street or even neighborhood, causing a significant amount of collateral damage.

ISPs can stop this by ensuring their DNS servers only answer queries from users inside their own network. For example, if I run Patrick's Network, I would ignore any queries from users in Mike's Network. Answering those would be a waste of my resources, but more importantly, it can be used to abuse Mike's network. Even though locking down DNS servers is a good idea, many, many ISPs do not do this. There are currently upwards of 20 million DNS servers configured to answer queries from anyone on the Internet.

While this situation isn't ideal and ISPs should lock down their DNS servers, it is not the root cause of what the problem. The problem is actually source address spoofing. The reason the store sends you a catalog is because the return address on the postcard is yours, not the miscreant's. The store doesn't know you did not ask for the catalog - just the opposite, because yours is the return address, the store - or in the case of the Internet, the server - thinks it is doing you a favor by delivering the catalog.

Combining these two situations can create a couple of serious problems for the Internet. First, it allows a miscreant to get a massive "bang for their buck". A little attack traffic can be amplified 100X. Second, and in some ways more importantly, it hides the true source of the attack. When the victim analyzes the attack traffic, they only see the misconfigured DNS servers' addresses; they do not know the miscreant's address. Either of these reasons is sufficient to require stopping this, but the two combined can be disastrous.

The rules on the Internet (as much as the Internet has rules) state ISPs should check all the packets leaving their network to ensure the source addresses are from their own network. Put another way, ISPs are supposed to stop any postcards which have a fake return address. This is codified in something called a Best Common Practices document, BCP-38.

This problem is not new, and all major router vendors built simple ways to implement BCP-38 into their equipment years ago. However, many ISPs do not set the configuration, allowing spoofed packets to leave their network.

There are several reasons an ISP may not configure their network to disallow spoofing. Let's discuss a few of the most common:

* Lack of Knowledge
A lot of ISPs simply do not know they are supposed to do this, or understand the consequences of not doing it. This is why I am beating the drum, and working to get people to implement BCP-38. It's an important piece of Internet hygiene. To be clear, this is not a silver bullet. Configuring BCP-38 will not stop all attacks on the Internet, but it will help,

* Time & Effort
It takes time and effort to implement BCP-38. ISPs are businesses and they have a strong motive to be profitable. Implementing BCP-38 does not have clear profit or revenue behind it. A CFO may think the time an engineer spends implementing BCP-38 is time that could have been spent doing something to make money. I personally believe this is a false choice. Not configuring BCP-38 opens the ISP to abuse, and not just through DNS amplification. As important, we're dealing with shared fate - if no one does it, everyone is vulnerable; while if everyone does it, no one is vulnerable. A true accounting of the costs will likely show implementing BCP-38 is actually good for the bottom line.

* Risk to Revenue
Configuring filters means the risk of misconfiguring filters, which could in turn lead to filtering legitimate customer traffic. The larger a network, the more likely this scenario becomes. Businesses try hard not to upset their customers. Misconfiguration is a real threat and ISPs need to be careful of it. But everything an ISP does carries risk. This is why ISPs have process and procedure in place to help evaluate and mitigate these potential risks. In addition, it is possible to tie the outbound filters to inbound routing. In that case, the only way to break the outbound filter is to break the inbound routing, which means the customer would not be connected anyway.

In summary, while it may seem like there are good reasons not to implement BCP-38, there really are not. Yet there are many important reasons to implement it. Akamai strongly urges all ISPs to implement BCP-38 as quickly as possible.

 

Patrick Gilmore is a Chief Network Architect at Akamai

Swisscom and Akamai Team Up

Today, Swisscom and Akamai made a very exciting announcement; we've teamed up to provide solutions that will improve the web and cloud experience for both enterprises and consumers in Switzerland.
 
Swisscom has over 6.2 million mobile customers, 791,000 Swisscom TV customers and around 1.7 million broadband connections. And with this partnership, Swisscom will act as a preferred 'go to' market partner for Akamai in Switzerland, reselling and integrating the complete Akamai services portfolio into its solutions to create value-added services for its customers.
 
Urs Schaeppi, CEO Swisscom (Switzerland) said it best..."This strategic partnership is very interesting for us. With this collaboration, we combine innovative Akamai services with our strong brand, deep customer understanding and local support. The new services will address the growing expectations of enterprise customers for high-performing delivery of digital content, video and cloud applications. It will support the explosive growth in devices, content and traffic, all while helping to defend customers against an increasing flood of security attacks. This not only enables an optimum online experience for Internet users but also supports our corporate promise to be a trustworthy partner in the digital world."
 
You can find more details on this partnership and the full release on our web site.

Devices versus Services: What Really Matters?

Having spent my career in the IT world, where I've participated in the design of high-level services, including CDN-related technologies, I now find myself helping to make a case for why network operators should consider CDNs as a core element of their network infrastructure. One of the factors that make this interesting is a difference in perspective between the IT view of the world and the operator world-view. I don't claim to be unique in witnessing this clash of perspectives since it's happening more generally as network operators consider adopting cloud technologies, but I would claim that CDNs are at the bleeding edge of traditional IT technology pushing deep into operator access networks.

In trying to put my finger on whether there's something fundamentally different in how these two communities build and operate systems, I keep coming back to the following distinction. Network operators think in terms of appliances and devices, and how they can be architected to build a network, whereas the IT perspective decouples hardware and software (treating the former as commodity), and focuses on how the software can be architected to provide a global service. This device/appliance versus software/service distinction then permeates the language: one talks about managing individual devices and the other talks about managing the service as a whole; one talks about appliance performance and the other talks about aggregate service performance; one talks about device reliability and the other talks about service-level reliability. Of course the network operator also thinks about network-wide behavior and IT people also think about per-server behavior, but their respective viewpoints start at opposite ends from each other.

CDN World Summit Asia - Will you be there?

CDN Asia begins next week and Akamai's Aura Network Solutions team will be there in full force. It's only natural since the event draws a "who's who" across the industry including CDN service providers (like Akamai), telcos and ISP's, OTT service providers, cable and satellite operators, content owners, etc.
 
We'll be using the event to talk more about what the company is doing in the region, our acquisition of Verivue, our partnership with Orange and our alliance with AT&T. It's pretty clear that a lot has been happening! All of which, benefits our customers. 
 
In addition, Akamai's Mick Scully, VP and GM of the Carrier Products Division, will participate in the event's conference program by presenting "The Convergence of Network, Cloud, and CDN to Build & Differentiate Telecom Services." Mick's session promises to be really enlightening. I'm sure he'd love to see you in the audience on January 29th at 12:20 pm local Singapore time.
 
If you're planning to be at the summit and want to learn more about how Akamai is partnering with international operators to lower their network costs and drive additional revenues, please come by and talk with us. Visitors to the summit can also see demonstrations of the Aura Network Solutions Operator CDN in the Akamai meeting room.
 
The CDN Summit takes place on January 29 & 30, 2013 at Marina Bay Sands in Singapore. You can find more details and registration information at: http://cdnworldsummitasia.com.

Tara Bartley is a Senior Product Marketing Manager at Akamai 

Last week, Akamai announced a partnership with AT&T to address the growing expectations of customers for more efficient content routing and better delivery of digital content, video and Web applications.  Addressing the explosive growth in devices, content and traffic, this partnership demonstrates several ways that network operators can leverage CDN capabilities to optimize the online experience for their customers and end-users.

Below is a bit more insight into how carriers can leverage CDN services. 

There are three primary reasons that Network Service Providers consider CDN technology:  1) Traffic Offload; 2) Network Performance; 3) Revenues.  Not surprisingly, those are the same exact reasons why Akamai partners with Network Service Providers. The challenge has always been aligning in a way where industry costs decrease, performance increases, and everyone sells more. And at the same time, provide online content owners and enterprises great performance and end users a better Internet experience. 

Most conversations with network operators start with the need for caching to help address the massive scale to meet the demands of digital media. How can an operator leverage caching in their network to provide the greatest network efficiency, what traffic should they cache, and how does it integrate with the network?  Many operators have different approaches to how this technology should be deployed, and Akamai now has several options, including recently acquired technology from Verivue

It helps that Akamai can aggregate 1000s of customers, and bring 100s of Gbps, or even Tbps of traffic to the table on day 1. Caching obviously produces network efficiency and performance benefits to enterprise customers and end users.  The impact of distributed caching should not be understated, and it helps the Internet run efficiently for all parties.

So if caching deployments closer to the edge of the operator's network help offload traffic and improve performance, where does revenue fit in? While the revenue impact may not be instant, the ability to enable better QoE for an operator's subscribers enhances retention and service satisfaction.  In addition, Akamai views enterprise channels as an important aspect of our long term growth strategy, and access to large enterprise sales teams is definitely compelling. 

At the same time, Network Service Providers are looking to enhance their existing enterprise portfolio and differentiate their network, hosting and Cloud services. Operator CDN has been an attempt at adding value to the network, but CDN services have evolved and a simple delivery service, while certainly needed, is no longer sufficient. Enterprises want services that not only deliver content, but improve service quality, simplify workflow, reach an array of devices, and provide global reach and scale. This revenue potential is part of the synergy that mutually drives network operators and Akamai to create partnerships.

Akamai is thrilled by our relationship with AT&T. While there is a lot of work to do, it brings the potential for added scale, performance, and growth on both sides of the relationship. It is also another proof point in the evolution of our ongoing relationships with Network Service Providers.

Frank Childs is Director of Product Marketing for Akamai's Aura Network Solutions.
1 2 3 >>