Akamai Diversity
Home > March 2013

March 2013 Archives

SSL, could this be your Achilles heel?

Have you embraced SSL/TLS to protect the sensitive parts of your website and used client certificates to authenticate connecting parties? If so, this new layer of security may expose you to a whole new set of threats - Distributed Denial of Service (DDoS) and application layer attacks over SSL.

Many people still think DDoS is all about volume - at Akamai we're saying (and seeing) that resource starvation attacks will become more and more prevalent as attackers realize that they can do the same amount or more damage with less computing resources - and to defend against it, you as a security professional need to think differently.

The reality is that a small, but well-organized, DDoS attack that attacks both encrypted and unencrypted web content can easily exceed 3-4Gbs of sustained DDoS traffic. This volume of traffic will knock most organizations off the air, and even if it is not a volumetric DDoS attack, attacks at the application layer can easily consume back-end resources without starving your network bandwidth.

For reference, the largest DDoS attack ever recorded was 124Gbps against a US government website in July 2009.

An emerging trend for attackers is to attack certain SSL handshake functions creating a resource starvation condition. A server will typically use 15x more computing resources in the SSL negotiation than the attacking system. This in turn provides the attacker with excellent economical advantage with force multiplication.

Live Video Madness

Turner Sports has released a trove of statistics on video streaming from the opening week of this year's NCAA Division I Men's Basketball Championship, which indicate healthy growth in live video streaming for the event across both broadband and mobile. Turner delivered what it called a record 36.6 million live video streams during the opening week of NCAA March Madness Live, doubling the 18.3 million live streams it delivered over the full course of last year's tournament.

The 36.6 million streams represented 10 million hours of live video, up 198% over the 2102 tournament. Unique broadband visitors watching live video increased 161% over last year to 4.2 million, while unique mobile app viewers rose 121% to 2.6 million.

Engagement went up as well, with viewers watching 105 minutes of live video on broadband and 61 minutes on mobile, a rise of 12% and 42%, respectively, over 2012. Interestingly, mobile streaming appeared to increase as viewers left their offices, having made up an average 46% of all live video streams on Thursday and Friday and 60% over the weekend.

Turner Sports also listed the five most popular games based on live video streams from last week:

  • Valparaiso vs. Michigan State - 1,844,000
  • Bucknell vs. Butler - 1,784,000
  • Mississippi vs. Wisconsin - 1,778,000
  • Albany vs. Duke - 1,488,000
  • Davidson vs. Marquette - 1,487,000

Four of the top five games started between noon and 1 PM on the East Coast. While it would seem to indicate that much of the viewing took place at work, employers can take at least some solace in the fact that employees in the eastern part of the U.S. were generally doing so during lunch hours. 

Somewhat surprising is that none of those top four games were necessarily nailbiters, with the average margin of victory being just over 11 points. The only close game - Marquette edging Davidson by one point - was fifth on the list and didn't start until 3:10 PM ET.

With the tournament field now whittled down to 12 after last night and the remaining games and the next round extending through the weekend, it will be interesting to observe how the combination of higher stakes but with fewer teams and games not being played during general working hours in the U.S. will affect streaming rates and viewing patterns.

Chris Nicholson is a senior public relations manager for Akamai.

The 300 Gbps attack this week against SpamHaus certainly seems epic.  But how big is it, really? When we think about an attack an Akamai, we think about three things: the attacker's capacity, their leverage, and the target's capacity. And when we think about leverage, it's really comprised of two smaller pieces: how much cost efficiency the attacker expects to get, and how the target's resilience mitigates it.

300 Gbps isn't that bad when it's restricted to reflected DNS traffic - if you have enough capacity to ingest the packets, they're pretty trivial to drop, and, until your network cards fill up, are less effective than a SYN flood. So why would an attacker resort to such an inefficient attack?  The attacker likely doesn't have 300 Gbps in their botnet - they probably have somewhere in the range of 3 to 60 Gbps. Attacks through DNS resolvers are amplified - so the attacker can create a larger attack than they might have otherwise, at the cost of reducing their leverage.

In comparison the BroBot botnets are routinely tossing around 30 Gbps attacks, with peaks upwards of 80 Gbps.   Because they're willing to sacrifice their hosts, they have a wider range of attacks available to them. Commonly, they send HTTPS request floods - requiring their targets to negotiate full SSL connections, parse an HTTP request, and determine whether they'll deliver a reply or not. BroBot could certainly throw around a bit more bandwidth with DNS reflection - but against most of their targets, it would have less effect than some of their current tactics.

It's hard to compare the two. If you have less than 60 Gbps of raw bandwidth lying around, they're both the same (you'll succumb either way). If you have more than 60 and less than 300 Gbps, BroBot is more palatable, although you need a lot more CPU to handle it. But above 300Gbps of bandwidth? The attack on SpamHaus is much, much easier to deal with.

Andy Ellis is Akamai's Chief Security Officer

Cross posted at csoandy.com

There's been a lot of talk about the recent very large DDoS attack against Spamhaus. Although I was quoted in some articles about it, I want to be clear that the attacks did not affect or involve Akamai or our customers. However, we have been the object of similar attacks in the past, and Akamai has a vested interest in making the Internet better - safer, more reliable and higher performing.

Unfortunately, there are some common... let's call them "misconfigurations" on the Internet which make these types of attacks both easier and much more destructive than they should be.

The one most talked about recently is DNS Amplification. This has been discussed many times in many places so I won't go into a great deal of detail. Very generally, a miscreant can send a query (a very small amount of data) to a DNS server and the server will send 100 or more times as much data in response to the victim. It is the equivalent of a stranger sending a postcard to a store and you get a gigantic catalog in return.

Now imagine getting 1000s or even millions of gigantic catalogs delivered to your house - per second. You couldn't even leave your house. What's more, if they send enough, the catalogs could jam your whole street or even neighborhood, causing a significant amount of collateral damage.

ISPs can stop this by ensuring their DNS servers only answer queries from users inside their own network. For example, if I run Patrick's Network, I would ignore any queries from users in Mike's Network. Answering those would be a waste of my resources, but more importantly, it can be used to abuse Mike's network. Even though locking down DNS servers is a good idea, many, many ISPs do not do this. There are currently upwards of 20 million DNS servers configured to answer queries from anyone on the Internet.

While this situation isn't ideal and ISPs should lock down their DNS servers, it is not the root cause of what the problem. The problem is actually source address spoofing. The reason the store sends you a catalog is because the return address on the postcard is yours, not the miscreant's. The store doesn't know you did not ask for the catalog - just the opposite, because yours is the return address, the store - or in the case of the Internet, the server - thinks it is doing you a favor by delivering the catalog.

Combining these two situations can create a couple of serious problems for the Internet. First, it allows a miscreant to get a massive "bang for their buck". A little attack traffic can be amplified 100X. Second, and in some ways more importantly, it hides the true source of the attack. When the victim analyzes the attack traffic, they only see the misconfigured DNS servers' addresses; they do not know the miscreant's address. Either of these reasons is sufficient to require stopping this, but the two combined can be disastrous.

The rules on the Internet (as much as the Internet has rules) state ISPs should check all the packets leaving their network to ensure the source addresses are from their own network. Put another way, ISPs are supposed to stop any postcards which have a fake return address. This is codified in something called a Best Common Practices document, BCP-38.

This problem is not new, and all major router vendors built simple ways to implement BCP-38 into their equipment years ago. However, many ISPs do not set the configuration, allowing spoofed packets to leave their network.

There are several reasons an ISP may not configure their network to disallow spoofing. Let's discuss a few of the most common:

* Lack of Knowledge
A lot of ISPs simply do not know they are supposed to do this, or understand the consequences of not doing it. This is why I am beating the drum, and working to get people to implement BCP-38. It's an important piece of Internet hygiene. To be clear, this is not a silver bullet. Configuring BCP-38 will not stop all attacks on the Internet, but it will help,

* Time & Effort
It takes time and effort to implement BCP-38. ISPs are businesses and they have a strong motive to be profitable. Implementing BCP-38 does not have clear profit or revenue behind it. A CFO may think the time an engineer spends implementing BCP-38 is time that could have been spent doing something to make money. I personally believe this is a false choice. Not configuring BCP-38 opens the ISP to abuse, and not just through DNS amplification. As important, we're dealing with shared fate - if no one does it, everyone is vulnerable; while if everyone does it, no one is vulnerable. A true accounting of the costs will likely show implementing BCP-38 is actually good for the bottom line.

* Risk to Revenue
Configuring filters means the risk of misconfiguring filters, which could in turn lead to filtering legitimate customer traffic. The larger a network, the more likely this scenario becomes. Businesses try hard not to upset their customers. Misconfiguration is a real threat and ISPs need to be careful of it. But everything an ISP does carries risk. This is why ISPs have process and procedure in place to help evaluate and mitigate these potential risks. In addition, it is possible to tie the outbound filters to inbound routing. In that case, the only way to break the outbound filter is to break the inbound routing, which means the customer would not be connected anyway.

In summary, while it may seem like there are good reasons not to implement BCP-38, there really are not. Yet there are many important reasons to implement it. Akamai strongly urges all ISPs to implement BCP-38 as quickly as possible.


Patrick Gilmore is a Chief Network Architect at Akamai

Intelligent User Mapping in the Cloud

A DNS-based, load-balancer solution allows companies to distribute application load between global datacenters or cloud providers. This solution, when combined with a Content Delivery Network (CDN), provides benefits such as improved load distribution, faster load times, and high content availability.  These benefits result from the user requests being directed to the optimal (often nearby) server. Latency and packet-loss are therefore minimized due to a shorter physical distance traveled.

When the client DNS resolver queries the IP address of a hostname, it recursively queries the CDN's DNS servers for the answer. The CDN's DNS server examines the IP address of the client name server, and returns the server that is optimal for that name server.

$ host www.example.com
www.example.com.    3353    IN    CNAME    www.example.com.edgesuite.net.
www.example.com.edgesuite.net. 530 IN    CNAME    a123.gi3.akamai.net.
a123.gi3.akamai.net.    20    IN    A
a123.gi3.akamai.net.    20    IN    A

In the above example, the last two DNS queries are performed by the CDN's DNS. Ideally, the CDN name server would map users based on the end-user's IP address, rather than by the IP address of the client name server. However, the client IP is not visible to a CDN's DNS server as limited by query structure. (RFC 1035). The choice of the local name server and its geographic proximity relative to the client become a critical factor in the quality of service a CDN provides.

I want to take the time to thank everyone who connected with Akamai at the RSA® Conference! Whether you just dropped by to pick up some shwag, or got to spend time discussing our Kona Security Solutions, I appreciate that you took the time in your busy schedule to engage with us.

As Akamai's Chief Security Officer, I also get a lot of vendors reaching out to me after a conference, so I apologize for joining that deluge; but I'd like to ask for a simple favor (or four):
    •    If you saw my keynote, let me know if the ideas resonated with you, either via email (aellis@akamai.com) or Twitter (@csoandy). You can read more of my thoughts on those subjects at http://www.csoandy.com/.
    •    If you picked up one of our penguins, please read George the Penguin's story at http://www.securitypenguin.com/. Share a picture of your penguin with @SecurityPenguin on Twitter; George likes to know how his minions are doing!
    •    If you received a copy of Bruce Schneier's book Liars & Outliers, after you've read it, start discussing it - either by blogging about it, or sharing it with your coworkers. You can also check out our review.
    •    I am participating in a webinar with my fellow Akamai security experts on March 28. We will discuss lessons learned at RSA Conference - key trends, takeaways, and best practices. If you are interested, make sure to register.

And, of course, if you'd like to learn more about Akamai's web security solutions, you can read the white papers, view copies of the demos, and see other info at http://akamai.com/html/ms/rsa_conference_2013.html. You can also contact your account team for more information. If you don't have an account team yet, you can call or chat with us - links to do so are available on http://www.akamai.com/.

Thanks, and may you Innovate Fearlessly.

Andy Ellis
, Chief Security Officer
 at Akamai

Akamai IO - Going Global

I'm writing this short post to update you on a significant change in Akamai IO's data source. Since its inception, IO's data was based on many websites, but most of those sites catered to a US audience. This means that the data set was biased - it included global traffic, but held a disproportionate amount of traffic from the US.

Starting February 16th, we've started using a new data stream, based on traffic from most Akamai customers - meaning a much more global distribution, enough to take away the entire bias. The new data set is also bigger, including over a billion requests each day, and uses a newer version of our device characterization engine.

Notable Changes
Looking at the data, you can see the market share has indeed changed.

For instance, the chart below shows IE's dominance fell from ~50% to ~40%, while Safari's share jumped to almost 10%. My guess is that this relates to the better sample even within North America, as opposed to the theory that Safari is just more popular outside the US.
Screen Shot 2013-03-14 at 3.02.21 PM.png

Another example is that Android's market share increased significantly to over 5% of the overall browsing traffic, and Opera's share more than doubled. Both browsers are known to be more popular outside the US, which is a likely explanation for this change.
Screen Shot 2013-03-14 at 3.02.54 PM.png

Then again, not everything changed. Chrome, Firefox and Mobile Safari maintained similar market shares, and even a global view doesn't help make Blackberry seem more dominant...

Screen Shot 2013-03-14 at 3.06.39 PM.png

This new global data set really highlights the need for a geographic split - showing the market shares by continent, country and perhaps even city. We're working on this feature, along with others, and hope to have it available by May.

Swisscom and Akamai Team Up

Today, Swisscom and Akamai made a very exciting announcement; we've teamed up to provide solutions that will improve the web and cloud experience for both enterprises and consumers in Switzerland.
Swisscom has over 6.2 million mobile customers, 791,000 Swisscom TV customers and around 1.7 million broadband connections. And with this partnership, Swisscom will act as a preferred 'go to' market partner for Akamai in Switzerland, reselling and integrating the complete Akamai services portfolio into its solutions to create value-added services for its customers.
Urs Schaeppi, CEO Swisscom (Switzerland) said it best..."This strategic partnership is very interesting for us. With this collaboration, we combine innovative Akamai services with our strong brand, deep customer understanding and local support. The new services will address the growing expectations of enterprise customers for high-performing delivery of digital content, video and cloud applications. It will support the explosive growth in devices, content and traffic, all while helping to defend customers against an increasing flood of security attacks. This not only enables an optimum online experience for Internet users but also supports our corporate promise to be a trustworthy partner in the digital world."
You can find more details on this partnership and the full release on our web site.

Springsteen Drives 'Astronomical' GRAMMY Live Views

Funny that the man who once grumbled about "57 Channels (And Nothin' On)" would later be responsible for an unprecedented spike in online video views, a term that didn't even exist when the song was released in 1992. Yet that's what happened when Bruce Springsteen fans flocked in record numbers to watch the online broadcast of the MusiCares Person of the Year Red Carpet, the pre-cursor to the star-studded tribute to Springsteen that took place the Friday prior to the 55th Annual GRAMMY Awards earlier this month. The red carpet arrivals were available for anyone to watch via GRAMMY Live, The Recording Academy's online broadcast experience, powered by Akamai and AEG Digital Media, which provided viewers around the world with access across multiple devices to exclusive, behind-the-scenes events surrounding the GRAMMYs.

The Person of the Year Springsteen Red Carpet event views were astronomical, giving us the highest traffic rates that we've seen in the four years of GRAMMY Live for that event," said Paul Madeira, Executive Producer of GRAMMY Live and Senior Director/Producer - Media Productions for The Recording Academy. "That's a testament to Bruce and his legions of passionate followers."

A Key Launch: Ad Integration Services

Last week, Akamai launched Sola Ad Integration Services, a cloud-based solution designed to enable targeted, dynamic ad insertion into online video streams on live, linear and VoD content. Why is this so important? The simple answer is a projected $4.14 Billion in ad spending in 2013 that's growing to $8.04 Billion by 2016 according to eMarketer. Reaching your audience when they're viewing their favorite video content across a dramatically growing range of connected devices is essential, and employing ad targeting is now a basic requirement. Delivering both conditions using the best video viewing quality, in a simplified workflow, on an open flexible platform at scale is central to participating in the ad industry's growth.

By developing Ad Integration Services within the Akamai network -- built upon the Akamai Intelligent Platform -- the solution offers reliability and scale matching content, bitrate matching, a simplified client, and many built in feature integrations not available anywhere else.

Akamai's Ad Integration Services solves the many challenges facing the advertising industry by delivering the capacity to monetize HLS connected devices like iPhones and iPads, which are harder to reach but amount to a huge portion of on-line Internet traffic. It delivers customized or personalized advertising that drives higher visitor engagement and ad revenue for publishers. By simplifying the client with ad stitching inside the network and building upon Sola Vision's Stream Packaging, the solution facilitates a better video ad quality that matches the video content for a smooth, TV-like viewing experience. The simplified workflow for content owners has a flexible and open ADS-agnostic platform, giving the choice of working with the Ad Decision Systems and OVP/CIS of their preference. In leveraging the Akamai Intelligent Platform, Akamai's Ad Integration Services is able to offer broadcast-level services with the availability, reliability and extensibility for quality performance at scale!