Akamai Diversity
Home > June 2013

June 2013 Archives

Authentication for TV Everywhere is a complex challenge to solve if consumers are to enjoy seamless, user-friendly experiences that enable high-quality, premium content to be viewed across devices. It's a challenge that Akamai is addressing through our Sola Vision Identity Services, which our partner, Brightcove, has integrated into its Video Cloud online video platform.
Recently, Corey Halverson, Akamai's senior director of media products, joined Jeff Whatcott, Brightcove's CMO, and Phil Costa, director of product management, to discuss the TV Everywhere opportunity and how the two companies are working together to help make it easier for pay TV operators and content owners to deploy and manage such services. The conversation is captured in the video below.

Identifying and mitigating unwanted bot traffic

All websites connected to the public Internet receive bot traffic on a daily basis.  A recent study shows that bots drive 16% of Internet traffic in the US, in Singapore this number reaches 56%. Should you be worried? Well, not necessarily. Not all bot traffic is bad, and some of it is even vital for the success of a web site. Web sites are also affected differently depending on the profile of the company, the value of its content and the popularity of the site.

 

Defining bots

What are the different types of bots?

  • White bots (good) like search engines (Google, Bing and Baidu) help drive more customers to the site and therefore increase revenue. They also help monitor the site availability and performances (Akamai site analyzer, Keynote, Gomez) as well as pro-actively look for vulnerabilities (Whitehat, Qualys)
  • Black bots (bad) send additional traffic to the site that may impact its availability and integrity. Bad bot traffic can drive customers away from the site, negatively impacts revenue and the web site reputation. For example Hackers trying to bring down a site with a DDoS attack or exposing / exploiting vulnerabilities. Competitors or other actors scrapping a site to harvest pricing information to be used for financial gains.
  • Grey bots (neutral) don't necessarily help drive more customers to the site, nor do they specifically seem to cause any arm. Their identity and intent is more difficult to define, they usually present characteristics of a bot but are usually non aggressive. Such traffic would only occasionally cause problem due to a sudden increase of the request rate.

 

Identifying bot traffic

Dealing with bot traffic can be challenging and pro-active measures should be taken to prevent any negative impact on the site. Monitoring bot activity is key. The one thing that all bots have in common is that they only request base HTML pages, which usually contain valuable information but are also more process intensive for the web server to generate. Bots generally never request any of the embedded objects (images, JavaScript, Cascaded style sheets) just because the client doesn't need to render the full page.

 

Now that we know how to find the bot traffic it is necessary to identify the different types of bots.

  • White bot traffic is usually predictable. It will have a specific header signature and will come from IPs belonging to the companies managing the bot. It is possible to control what these bots can request on the site through robot.txt or through the administration interface of the service managing the bot activity.
  • Black bots header signature will widely vary from exactly mimicking a genuine browser or search engine request to something that will present several anomalies with missing headers or atypical headers being present in the request. Black bots may also send requests at a higher rate.
  • Grey bot traffic can be more challenging to identify since it generally present the same characteristics as black bots.

 

In order to effectively identify bot activity it is necessary to implement and deploy a set of rules to look at the traffic from different perspective. Several features of the Kona Site Defender product can help:

  • The WAF application layer control feature consists of the mod_security core rules set and Akamai common rule set. Some of the rules are specifically designed to look for anomalies in the headers or look for know bot signature in the user-agent header value or combination of headers in the request.
  • The rules mentioned above can be complemented with several WAF custom rules to help identify specific header signature.
  • The WAF adaptive rate control feature can also be used to monitor excessive request rate from individual clients.
  • Lastly the User Validation Module (UVM) can be used to perform client side validation during extreme situation when none of the "traditional" methods seem to help.

 

Mitigating bot traffic

Once bot traffic is identified, the next step is to decide what to do with the black and grey bot traffic. You may decide to just monitor the traffic over time and only take action should the activity become too aggressive and represent a threat for the stability of the web site. You may decide to take action as soon as the activity is identified, regardless of the volume of traffic generated. The type of action taken may vary depending on your business needs:

  • Deny the traffic: this is the default but least elegant solution; client will receive a HTTP 4xx or 5xx response code. This will give the bot operator a clear indication that such action is not allowed on the site and that they've been identified by some security service or device. Bot operators could vary the format of the request and see if they can stay under the radar.
  • Serve alternate content: the content served could vary from a generic "site unavailable" page to something that looks like a real response but only containing generic data. This strategy may slow down the bot operator and keep them in the dark as too why they cannot access to the data they want.
  • Serve a cached / stale / static / version of the content: This is the best strategy of all but not always necessarily possible to implement, some content just cannot be cached or stored as static data on an alternate origin, because of compliance concerns or its dynamic nature.  It could potentially take the bot operator some time to realize the data they are getting is worthless, an attacker running a DDoS against the site would also get discourage and move on to a different target.


David Senecal is senior enterprise architect at Akamai.  Patrice Boffa is a director of global service delivery at Akamai.


Opera Browser Hacked

A note of caution for Akamai customers and anyone else using the Opera web browser: Hackers have broken into Opera's internal network. As a result, thousands of users have been the unfortunate recipients of malware.

According to a report on the E Hacking News site, the culprits were able to exploit an expired Opera code-signing certificate. "Cybercriminals used the certificate to send their malware and distributed the malicious software to thousands of Opera users through [the] automated update function," the news site reported. "The malware is currently detected by half of the antivirus engines used by the virus total scanner."

In the Opera blog, Sigbjørn Vik from the quality assurance department wrote:

On June 19th we uncovered, halted and contained a targeted attack on our internal network infrastructure. Our systems have been cleaned and there is no evidence of any user data being compromised. We are working with the relevant authorities to investigate its source and any potential further extent. We will let you know if there are any developments. It is possible that a few thousand Windows users, who were using Opera between 01.00 and 01.36 UTC on June 19th, may automatically have received and installed the malicious software. To be on the safe side, we will roll out a new version of Opera which will use a new code signing certificate.

The solution, he said, is to upgrade your browser to the latest version.

Walking on a Wire

Discovery Channel's nerve-wracking "Skywire Live with Nik Wallenda" last Sunday brought new meaning to the term, "cable TV." The live television and online broadcast featured tightrope walker Nik Wallenda, a.k.a. "The King of the Highwire," traversing a 1,400-wide section of the Grand Canyon some 1,500 feet in the air on a two-inch cable.

Skywire_Screen 2.jpg 

As pleased as Akamai was to help Discovery Channel deliver the live online video stream to viewers worldwide, we were far happier - and relieved - that Nik successfully completed the stunt without incident. Akamai worked with Discovery Channel to make streams from five different camera angles during the walk available to visitors to its Wired In multiplatform experience, which generated more than 2.1 million streams on Sunday and peaked at 322,000 concurrent streams.

Discovery Channel reported that the event was watched by 12.98 million total viewers and became the "#1 most social show across broadcast and cable in the U.S.," where it generated 1.3 million Tweets.

Chris Nicholson is a senior public relations manager at Akamai.

Experiencing Compliance From The Inside Out

One of the big educations I've been getting since joining Akamai's InfoSec group is what it's like to deal with the multiple tasks of compliance from within an organization. As a journalist, I always tackled the subject from the outside, where I'd ask a company which regulations they were bound by, and which security procedures they had adopted as a result.

Now I'm inside a publicly-traded corporation that is on the hook for all kinds of regulations, and a lot of the work going on around me is about making sure Akamai is on top of its compliance game. 

Two weeks before I officially started, I paid a visit to sit in on some meetings that were part of an audit the company was having done. It started with an overview CSO Andy Ellis gave auditors regarding the main components of our security program. There was also a meeting where representatives from human resources told the auditors about security training they give new employees and the follow-up training employees continue to receive. Having just gone through the training, I can tell you it is extensive. Another meeting dealt with Akamai's Edge Tokenization deployments.

Now I'm watching my colleagues work on the daily bits and pieces that go into our compliance upkeep.

I'm looking at compliance from two angles at all times: There's the internal compliance efforts, and then there are the products we sell to customers to help them with their efforts. I don't pretend to know everything yet. Indeed, it will take time to fully absorb everything. The greatest lesson so far for me is that the work is far deeper and far more complex than what I understood as the outsider looking in. 

I see a lot of heavy lifting ahead. But it's going to be fun.

compliance.jpg


Akamai InfoSec Has Become More Social

Akamai InfoSec has some new social networking accounts designed to keep the focus squarely on what our team is up to while inviting customers and the wider industry in to share ideas and ask us questions.

In the past week we've created new pages on Twitter, Facebook, LinkedIn and Google+ -- and we ask you all to "like," "follow" and add the team to your online social circles.

For now, these accounts are being used mostly to share the security blogging and research we do daily. But the goal is to be more accessible to customers and those who may be thinking of trying us out. 

We currently field a lot of customer questions using inside e-mail lists and other private communications. The new social networking pages will make it even easier to ask us questions and get answers. And since it's in a public setting, a wider audience will be able to take in the discussions and learn from them.

Private conversations will continue, of course. Many of our customer dealings must remain private for their protection. The social networking pages are for more general questions and answers.

Our new Twitter handle is @Akamai_Infosec.

Our page on Facebook is Akamai InfoSec.

On Google+ we are Akamai Sec.

On LinkedIn, we are the Akamai InfoSec group.

Social-Logos.jpg


The Unforeseen Risk of Shared Services DDoS

A DDoS attack targeted at one web site is bad enough. But what happens when that single attack poses the distinct possibility of doing even more damage than originally intended. The kind of collateral damage I'm talking about is very real when you take into account IT architectures reliant on shared services.

Shared services include anything that serves more than one application or set of users, for example:
- Network infrastructure
- Network bandwidth
- Market data and other sources of information.
- Domain name servers

And while shared services can benefit an organization by bringing down IT costs, creating resource efficiencies and shrinking the IT footprint, in the case of a DDoS attack, there can be significant disadvantages. An attack on an organization with a healthy amount of shared services has the capability to cause unforeseen outages across a wide number of applications, users, and geographies.

In this post I'll present three cases in which a DDoS attack impacted a shared service, knocking out applications far beyond the attack target. In each of these cases, the companies were not using Akamai to protect the systems under attack.

Blunting Attacks During Olympic-sized Events

InfoSec receives many questions from Akamai customers on a daily basis. Yesterday, someone asked if we had a case study on attack vectors against the 2012 London Olympics. The customer has a big event coming up, and wanted a picture of what they're up against -- and how they can defend against it all to keep their sites running smoothly. 

As it turns out, CSIRT Director Michael Smith wrote something on that very subject. 

As someone who covered security as a journalist for 10 years, I was always on guard for event-related attacks. Everyone has been a target at one time or another: The NHL, NFL, and non-sporting events like the Academy Award ceremonies. And, of course, there's the Olympics.

What follows is the full write-up Michael did in advance of the London games. I think you'll find it useful. If you're concerned about details you don't see covered below, let me know and we'll work to address what's on your mind. Remember, this was written before the 2012 Olympics, so it's in the future tense.

Online Attacks and Large Online Events

The upcoming Olympic Games, much like other widely publicized, international events, offer unique challenges for online security.  In the course of any given year, Akamai supports many of these online events including concerts, sporting competitions, elections, and other newsworthy happenings.  Because of this, we've had substantial visibility into the various ways the "bad guys" may try to take advantage of an online event for their own gain. As important, these events typically involve a variety of online components - from live streaming to commerce - that providing a significant amount of attack surfaces for the event's security staff to protect.

The primary concern when supporting a large event is that online resources may be built in a hurry and then receive a sudden influx of users.  As such, there are time and effort constraints to securing these websites and the infrastructure that carries them.  Usually as the security team for the event, you do not have a lot of historical Internet traffic to define what is "normal" so you have to rely on attack trends from other events and threat intelligence to detect any new techniques that specifically are targeting your event.

One thing you need to be prepared to defend against is Denial of Service (DoS) attacks, where the attacker disrupts the operation of an online service such as a livestream or website. Highly visible event websites are prime targets and a cleverly-conducted Distributed DoS attack looks like a flash mob of legitimate users that are coming to a website.

The high visibility for events such as the Olympics can also prompt defacement style attacks.  Because the event draws a large volume of website users, hacktivist groups wishing to propagate their messages can alter the event's website to display their message to a broad audience and to generate headlines that create awareness for their cause.

In a similar vein, most large events have a scheduling site or a storefront where they sell tickets, memorabilia, or other services.  These can be prime targets for data exfiltration for anything from email addresses to passwords to credit card information to VIP contact information.

Data breaches can also lead to inappropriate information disclosure. Although not a big fear for a real-time event such as the Olympics but for events with a predetermined outcome such as awards ceremonies, attackers can access the results before they are officially released - this can lead to significant audience loss and loss of revenue. The loss of revenue could happen as a result of actual content theft where attackers make a copy of the event content available on their own website or on portable media.

Significant interest in an event may make associated online assets a possible target for distributors of malware. In this situation, attackers would alter the website in a non-obvious, non-visible manner to serve hooks to malicious content that runs on the users' computer and installs other software such as viruses, keyloggers, and the Zeus banking trojan.

And unfortunately, the event organizers and their online assets are not always the sole target. Event audiences can also be targets. Vehicles could include phishing, spam, and malware email where attackers seek a wide variety of goals such as stealing information from the user's computer, implanting viruses on the user's computer, and conducting outright scams involving selling counterfeit tickets, VIP passes, and fraudulent "discount tickets" to unsuspecting consumers.

Overall, the trick to keeping online events as safe as possible is to understand your potential adversary based on previous trends and current capabilities and understand how they're most likely to attack, the motivation for the attack, and countermeasures that you can implement. Doing so will help you apply the right defenses to the right assets and have a successful event.

london-summer-olympics-2012.jpg


 

Blogs From Akamai's InfoSec Team (Updated)

Akamai's InfoSec team does a lot of blogging, both on the company site and in personal, security-oriented blogs where they offer opinions that are theirs and not always their employer's. What follows is a directory of who is blogging and where. I'll update the list as more examples come to my attention, but for now I hope you'll check out these sites. In a future post, I'll point you to InfoSec staff on Twitter and other social networks.

"Liquid Matrix" is overseen by Akamai Security Evangelist Dave Lewis. A cast of talented security professionals contribute podcasts, features, etc.

"The Security Penguin," written by George, the Penguin of Awesomeness and spokesman for Akamai InfoSec.

"Andy Ellis > Protecting a Better Internet," written by Akamai's chief security officer. His most recent post dealt with the complexities of DNS reflection defense.

"Zen of security," by John Ellis, Akamai's enterprise security director for Asia Pacific and Japan. He also blogs for CSO.

"The Guerilla CISO," by Akamai CSIRT Director Michael Smith, known in the blog as "rybolov." This is a group blog he is in charge of. Topics range from the strategic (cyberwar, pending legislation, and public policy) through the operational (NISTs Framework for FISMA) to the tactical (penetration testing, forensics, vulnerability scanning, and security engineering). 

Akamai Security Evangelist Martin McKeay has two sites that rose to popularity long before he joined the team. There's the page for his "Network Security Podcast" and his "Network Security Blog."

Akamai Chief Security Architect Brian Sniffen has a site called "Sniffen Packets," which extends beyond security into such topics as travel and religion.

Akamai Senior Systems Engineer Larry Cashdollar has a site called "Vapid Labs Security Research." It's not necessarily a blog. In fact, the page takes you to a stream of code. Larry explains: "I wrote the web server running there in C when I was experimenting with 'attack aware' ideas in the late 90's.  Embedded in the fake public pgp block are links to security vulnerability advisories I've written and exploits. If you try hitting a link like http://vapid.dhs.org/;id>/tmp/p; it will log it as an attack and display a funny message."

Josh Corman, our director of security intelligence, has one called "Cognitive Dissidents." Josh takes the philosophical approach here, tackling issues of consequence that are often poorly understood and/or obfuscated by FUD. One of the standouts for me was a series of posts he authored with Brian Martin of Attrition.org on "building a better Anonymous."

Then there's the blog of Akamai security researcher Christian Ternus, "Adversarial Thinking." He'll soon be writing in the Akamai Blog as well, and his latest post about InfoSec's "jerk" problem is a must read.

I'll end for now with my own blog, The OCD Diaries. It's not a security blog, but I do occasionally cover issues affecting the InfoSec community -- including job-induced depression and how we humans talk to each other, for better or worse.
It has been a year since I last wrote about World IPv6 Launch and our measurements of IPv6 adoption at the time. Since then, we've seen continued momentum around increased IPv6 adoption on multiple fronts, with more IPv6 end users as well as with more content becoming available over IPv6. The net result of this is that IPv6 traffic on Akamai's global platform has increased to be over 250% of its June 2012 level, and we are now delivering around 10 billion requests per day over IPv6, up from around 3.4 billion requests per day at this time last year. Over the course of a given week, Akamai is now seeing between 200 million and 300 million unique IPv6 addresses contact our network.

In addition, we have observed continued momentum in IPv6 adoption by our customers, with over twice as many customers delivering IPv6-enabled properties from our platform as we had at World IPv6 Launch last June.

Akamai also continues to increase the footprint of our IPv6 network deployment as more of our network partners make IPv6 connectivity available. We now have IPv6 enabled in 64 countries and over 800 network locations around the globe.

IPV6_Image1.jpg
IPv6 Requests/Day on Akamai from June 2012 to June 2013

While IPv4 is still the dominant protocol on the Internet and will be for years to come, IPv6 adoption continues to move forward, especially in the mobile space and in some parts of the world. We explore this in more detail below.

'InfoSec's Jerk Problem,' By Christian Ternus

I wanted to take a moment to flag a post from another blog that's well worth your time, especially if you want to get a better understanding of the security industry culture. It's from Akamai InfoSec's own Christian Ternus. The subject is something any industry can relate to -- the so-called "jerk problem."

An excerpt:

Put bluntly: to others, we're jerks.

If you don't think this is a problem, you can stop reading here.

The dysfunctional tale of Bob and Alice

Imagine this. Developer Bob just received an email from your Infosec department, subjectImportant Security Update. He sighs, thinking of the possibilities: a request to rotate his password, or a new rule? Maybe it's a dressing-down for having violated some policy, a demand for extra work to patch a system, or yet another hair-on-fire security update he doesn't really see the need for. His manager is on his case: he's been putting in long hours on the next rev of the backend but library incompatibilities and inconsistent APIs have ruined his week, and he's way behind schedule. He shelves the security update - he doesn't have time to deal with it, and most things coming out of Infosec are just sound and fury anyway - and, thinking how nice it would be if his team actually got the resources it needed, continues to code. He'll get to it later. Promise.

Meanwhile, you, Security Researcher Alice, are trying not to panic. You've seen the latest Rails vulnerability disclosure, and you know it's just a matter of hours before your exposed system gets hit. You remember what happened to Github and Heroku, and you're not anxious to make the front page of Hacker News (again?!). If only Bob would answer his email! You know he's at work - what's happening? The face of your boss the last time your software got exploited appears in your mind, and you cringe, dreading an unpleasant meeting ahead. You fume for several minutes, cursing all developers everywhere, but no response is forthcoming. Angrily, you stand up and march over to his cube, ready to give him a piece of your mind.

Pause. What's going on here, and what's about to happen?

Read the full post HERE.

Akamai at IRCE 2013

Two weeks ago, a large number of eCommerce professionals converged in Chicago for IRCE 2013.  I was one of the presenters to talk about "Building m-commerce: Different approaches, different outcomes."

There were plenty of instances throughout the conference where the growing mobile traffic and revenue numbers highlighted the importance of delivering fast, quality mobile experiences to consumers.  As most of us know - fast, quality mobile experiences are better for business.

 

IRCE 1.png

 

IRCE 2.png

Yet as most of us also know, delivering fast, quality mobile experiences isn't exactly easy.  The challenges associated with mobile performance are well documented (browsers, network, devices, etc.).  For those who are interested in diving a little deeper into the topic, I recommend watching Ilya Grigorik's Google I/O talk - Mobile Performance from the Radio Up: Battery, Latency and Bandwidth Optimization.

The fact remains shoppers expect fast, quality mobile Web and app experiences and they generally don't know or care about the technological challenges associated with delivering them.  In addition, as we start to think about the latest techniques to engage mobile users such as Responsive Web Design (RWD) these challenges become even greater.  Responsive Web Design is a Web development approach that suggests Web pages should respond to the context in which they're loaded (primarily screen size) and change their user interface accordingly.

So what does delivering large, complex pages to mobile devices mean from an end-user's perspective?  Below is a snapshot of the experience of an end-user visiting a US retailer's RWD site's home page on a variety of different devices/networks.  The conclusion is obvious.  The delivery of a relatively small 700KB site to a mobile device, over wireless networks, has resulted in serious performance shortcomings.

IRCE 3.png

 

Optimizing RWD site performance is not easy and requires considerable expertise and resources however.  As mentioned earlier, RWD pages contain the HTML required to display all versions of a Web site, including both mobile and desktop views.  CSS and JavaScript run in the browser and hide or modify the content to fit the screen size.  On smartphones, this often means the browser downloads the entire content needed to display the desktop site, only to have CSS/JS hide the vast majority of it.

The first step to deliver fast, quality RWD sites is to focus on the actual page and the associated objects delivered to the end-user.  As Web performance optimization guru Steve Souder likes to point out: "80-90 percent of end-user response time is spent on the frontend. Start there."

There are a variety of options available to developers looking to overcome the challenges associated with delivering heavy RWD sites.  To start with, move content as close to the end-user as possible (i.e. use a Content Delivery Network (CDN)) and leverage optimal delivery mechanisms such as SPDY that are particularly relevant for wireless networks.  Next, focus on the components of the RWD application; the HTML, images, JavaScript and CSS objects.  To deliver faster pages, focus on:

  • Reducing the number of requests
  • Reducing the number of bytes
  • Accelerating rendering

For a more detailed view of how to actually reduce the number of requests & bytes and accelerate rendering download Akamai's Front-End Optimization primer.

IRCE 4.png

Independent of your approach to engaging mobile users, it is always worth to remember the following:

  • Deliver consistent, fast, quality web and application experience
  • Adopt your customers' perspective
  • Optimize for mobile first
IRCE 5.png

Bug Bounty Programs A Turning Point For Microsoft

Here in Akamai's InfoSec department, we constantly remind employees and customers to keep up on all the latest security patchesin their environment. Since Windows is everywhere in the business world, it's particularly important to keep an eye on Microsoft's patching efforts.

This week, the software giant made a big move in the name of vulnerability management, unleashing bug bounty programs that will likely lead to many more security patches in the future. Katie Moussouris, a senior security strategist with Microsoft, announced the initiative in a Microsoft blog post and on the podcast of Akamai InfoSec strategist Martin McKeay. She wrote in the blog post:

Today is an inflection point for Microsoft, as well as the security industry. For the first time ever, Microsoft is offering direct cash payouts in exchange for reporting certain types of vulnerabilities and exploitation techniques. We are making this shift in order to learn about these issues earlier and to increase the win-win between Microsoft's customers and the security researcher community.

Full details for the new bounty programs and a fantastic technical deep-dive by our esteemed panel of judges (headed by Matt Miller and David Ross) can be found on SRD's blog.

In short, we are offering cash payouts for the following programs:

  • Mitigation Bypass Bounty - Microsoft will pay up to $100,000 USD for truly novel exploitation techniques against protections built into the latest version of our operating system (Windows 8.1 Preview). Learning about new exploitation techniques earlier helps Microsoft improve security by leaps, instead of one vulnerability at a time. This is an ongoing program and not tied to any event or contest.
  • BlueHat Bonus for Defense - Microsoft will pay up to $50,000 USD for defensive ideas that accompany a qualifying Mitigation Bypass Bounty submission. Doing so highlights our continued support of defense and provides a way for the research community to help protect over a billion computer systems worldwide from vulnerabilities that may not have even been discovered.
  • IE11 Preview Bug Bounty - Microsoft will pay up to $11,000 USD for critical vulnerabilities that affect IE 11 Preview on Windows 8.1 Preview. The entry period for this program will be the first 30 days of the IE 11 Preview period. Learning about critical vulnerabilities in IE as early as possible during the public preview will help Microsoft deliver the most secure version of IE to our customers.
As Martin noted in his podcast, that's a lot of money for those who rise to the challenge. 
I congratulate Katie and her colleagues for making this happen. It's a big turning point for the software giant. I remember covering flaws, malware and patches impacting Microsoft a decade ago. Back then, the folks in Redmond balked whenever a researcher took the liberty of taking new flaw findings public. Now Microsoft is encouraging people to take their best shots and find breaks in the armor. That means more vulnerabilities will be discovered and fixed, and we'll all be more secure as a result.
 
Below: A group photo of those who worked on the bug bounty programs.
 


L-R: David Seidman, Gerardo di Giacomo, Mark Oram (via avatar), Mike Reavey, Dustin Childs, Leah Lease, Rob Chapman, Neil Sikka, Jacqueline Lodwig, Brandon Caldwell, Katie Moussouris, Nate Jones, Sweety Chauhan, Emily Anderson, Claudette Hatcher, Cynthia Sandwick, Stephen Finnegan, Manuel Caballero, Ben Richeson, Elias Bachaalany, David Ross, Cristian Craioveanu, Ken Johnson, Mario Heiderich, Jonathan Ness. Not pictured: Christine Aguirre, Danielle Alyias, Michal Chmielewski, Chengyun Chu, Jules Cohen, Bruce Dang, Jessica Dash, Richard van Eeden, Michelle Gayral, Cristin Goodwin, Angela Gunn, Joe Gura, Dean Hachamovitch, Chris Hale, Kyle Henderson, Forbes Higman, Andrew Howard, Kostya Kortchinsky, Jane Liles, Matt Miller, William Peteroy, Georgeo Pulikkathara, Rob Roberts, Matt Thomlinson, David Wheeler, Chris Williams. Behind the camera: Jerry Bryant.

On the Road with Aura Network Solutions

As previously posted here, Akamai's Frank Childs recently presented at the CDN Summit in NYC alongside Charter Communication's Kreig DuBose for a session titled "Deploying and Operator CDN to Enhance Customer Experience." Frank spoke about our Aura Network Solutions and Kreig explained his decision to select Aura, the results of the implementation and next steps. If you're interested in seeing the presentation, I've inluded the video below...

 

After the CDN Summit it was back on the road for the Cable Show in Washington, DC. The event was the perfect place to see new advances in interactive video applications, breakthrough technologies that are changing the way people communicate online, and multi-screen content delivery strategies, among others. And of course we heard from industry leaders talking about their investment and product development priorities for 2014 and beyond. While I spent some time celebrity spotting - MC Hammer, Ricky Schroder and JLo, to name just a few - our very own Kris Alexander presented in what is known as "Imagine Park" in the center of the exhibit floor. To a packed crowd, Kris showcased Akamai's "Hyperconnected Living Room Experience", explaining how the second screen trend is likely to evolve and what is possible in the world of synchronized experiences and companions apps.

Here is a video of his presentation.

 

As you might imagine, The Cable Show folks do a remarkable job of capturing all of the show's presentations online. Take a look at their web site to view more sessions at: https://2013.thecableshow.com

Tara Bartley is a Senior Product Marketing Manager at Akamai

DNS reflection defense

Recently, DDoS attacks have spiked up well past 100 Gbps several times. A common move used by adversaries is the DNS reflection attack, a category of Distributed, Reflected Denial of Service (DRDos) attack. To understand how to defend against it, it helps to understand how it works.

 

How DNS works

At the heart of the Domain Name System are two categories of name server: the authoritative name server, which is responsible for providing authoritative answers to specific queries (like use5.akam.net, which is one of the authoritative name servers for the csoandy.com domain), and the recursive name server, which is responsible for answering any question asked by a client. Recursive name servers (located in ISPs, corporations, and data centers around the world) query the appropriate authoritative name servers around the Internet, and return an answer to the querying client. An open resolver is a category of resolver that will answer recursive queries from any client, not just those local to them. Because DNS requests are fairly small and lightweight, DNS primarily uses the Universal Datagram Protocol (UDP), a stateless messaging system. Since UDP requests can be sent in a single packet, the source address are easily forgeable with any address desired by the true sender.


DNS reflection

A DNS reflection attack takes advantage of three things: the forgeability of UDP source addresses, the availability of open resolvers, and the asymmetry of DNS requests and responses. To conduct an attack, an adversary sends a set of DNS queries to open resolvers, altering the source address on their requests to be those of their chosen target. The requests are designed to have much larger responses (often, using an ANY request, a 64 byte request yields a 512-byte response), thus resulting in the recursive name servers sending about 8 times as much traffic at the target as they themselves received. A DNS reflection attack can directly use authoritative name servers, but it requires more preparation and research, making requests specific to the scope of each DNS authority used.

Eliminating DNS reflection attacks

An ideal solution would obviously be to eliminate this type of attack, rather than every target needing to defend themselves. Unfortunately, that's challenging, as it requires significant changes by infrastructure providers across the Internet.

BCP38

No discussion of defending against DRDoS style attacks is complete without a nod to BCP38. These attacks only work because an adversary, when sending forged packets, has no routers upstream filtering based on the source address. There is rare need to permit an ISP user to send packets claiming to originate in another ISP; if BCP38 were adopted and implemented in a widespread fashion, DRDoS would be eliminated as an adversarial capability. That's sadly unlikely, as BCP38 enters its 14th year; the complexity and edge cases are significant.

The open resolvers

While a few enterprises have made providing an open resolver into a business (OpenDNS, GoogleDNS), many open resolvers are either historical accidents, or resulting from incorrect configuration. Even MIT has turned off open recursion on its high-profile name servers.
Barring that, recursive name servers should implement rate limiting, especially on infrequent request types, to reduce the multiplication of traffic that adversaries can gain out of them.

Self-defense

Until ISPs and resolver operators implement controls to limit how large attacks can become, attack targets must defend themselves. Sometimes, attacks are targeted at infrastructure (like routers and name servers), but most often they are being targeted at high-profile websites operated by financial services firms, government agencies, retail companies, or whoever has caught the eye of the attacker this week.
An operator of a high-profile web property can take steps to defend their front door. The first step, of course, should be to find their front door; and to understand what infrastructure it relies on. And then they can evaluate their defenses.

Capacity

The first line of defense is always capacity. Without enough bandwidth at the front of your defenses, nothing else matters. This needs to be measurable both in raw bandwidth, as well as in packets per second, because hardware often has much lower bandwidth capacity as packet sizes shrink. Unfortunately, robust capacity is now measurable in the 300+ gigabits per second, well beyond the resources of the average datacenter. However, attacks in the 3-10 gigabit per second range are still common, and well within the range of existing datacenter defenses.

Filtering

For systems that aren't DNS servers themselves, filtering out DNS traffic as far upstream as possible is a good solution, but certainly at a border firewall. One caveat - web servers often need to make DNS queries themselves, so ensure that they have a path to do so. In general, the principal of "filter out the unexpected" is a good filtering strategy.

DNS server protection

Since DNS servers have to process incoming requests (an authoritative name server has to respond to all of the recursive resolvers around the Internet, for instance), merely filtering DNS traffic upstream isn't an option. So what is perceived as a network problem by non-DNS servers becomes an application problem for the DNS server. Defenses may no longer be simple "block this" strategies; rather, defense can take advantage of application tools to provide different defenses.

Redundancy

While the total number of authoritative DNS server IP addresses for a given domain is limited (while 13 should fit into the 512-byte DNS response packet, generally, 8 is a reasonable number), many systems use nowhere near the limit. Servers should be diversified, located in multiple networks and geographies, ensuring that attacks against two name servers aren't traveling across the same links.

Anycast

Since requests come in via UDP, anycasting (the practice of having servers responding on the same IP address from multiple locations on the internet) is quite practical. Done at small scale (two to five locations), this can provide significant increases in capacity, as well as resilience to localized physical outages. However, DNS also lends itself to architectures with hundreds of name server locations sprinkled throughout the internet, each localized to only provide service to a small region of the Internet (possibly even to a single network). Adversaries outside these localities have no ability to target the sprinkled name servers, which continue to provide high quality support to nearby end users. 

Segregation

Based on Akamai's experience running popular authoritative name servers, 95% of all DNS traffic originates from under a million popular name server IP addresses (to get 99% requires just under 2 million IP addresses). Given that the total IPv4 address space is around 4.3 billion IP addresses, name servers can be segregated; a smaller number to handle the "unpopular" name servers, and a larger amount to handle the popular name servers. Attacks that reflect of unpopular open resolvers thus don't consume the application resources providing quality of service to the popular name service.

Response handling

Authoritative name servers should primarily see requests, not responses. Therefore, they should be able to isolate, process, and discard response packets quickly, minimizing impact to resources engaged in replying to requests. This isolation can also apply to less frequent types of request, such that when a server is under attack, it can devote resources to requests that are more likely to provide value.

Rate Limiting

Traffic from any name server should be monitored to see if it exceeds reasonable thresholds, and, if so, aggressively managed. If a name server typically sends a few requests per minute, having name servers not answer most requests from a name servers requesting dozens of time per second (these thresholds can and should be dynamic). This works because of the built in fault tolerance of DNS; if a requesting name server doesn't see a quick response, it will send another request, often to a different authoritative name server (and deprioritizing the failed name server for future requests).

As attacks grow past the current few hundred gigabit-per-second up to terabit-per-second attacks, robust architectures will be increasingly necessary to maintains a presence on the Internet in the face of adversarial action.

 
crossposted at csoandy.com

clickfrenzy.jpeg

Last year's Click Frenzy online sale in Australia, modeled after the hugely successful Black Friday sale in the US, attracted a huge amount of media attention for all the wrong reasons. Within minutes of the sale going live on November 20 last year, the website buckled under the strain of unanticipated traffic volumes. A number of consumers, excited at the prospect of grabbing a great online bargain, were left empty handed and disappointed.  The media's reaction was swift, and merciless. 

Fast-forward six months and the latest Click Frenzy sale was completed without any technical issues at all. Click Frenzy needed an industry leading solution, and as such enlisted Akamai.

Following their initial sale and the subsequent technical issues they encountered, Click Frenzy wanted to understand how they could handle the sudden bursts in traffic volumes that characterizes their online business model.  For Click Frenzy, it was absolutely imperative that their next big sale performed seamlessly to restore consumer and retailer confidence.  Further technical issues had to be avoided at all costs. 

We approached Click Frenzy and explained how Akamai's intelligent platform - specifically Akamai's AQUA & KONA solutions - enables traffic to be securely offloaded to distributed computing resources, thereby alleviating the burden on the their data centre.  This model allows businesses, such as Click Frenzy, to manage traffic spikes that would be unheard of for most conventional online retailers. 

Following Click Frenzy's 24-hour sale last month, the Akamai platform managed 169 million requests, with a peak of 29,722 requests per second. Total traffic volume exceeded 3TB.  This article, which features an interview I did following the sale, provides a great overview of some of the positive outcomes Click Frenzy has enjoyed since its rollout of the Akamai intelligent platform.

Although the Click Frenzy site itself performed flawlessly, some technical issues were still evident on the part of some retailers involved in the promotion since their websites - not utilizing Akamai's platform - were unable to cope with the traffic being directed at them by Click Frenzy.

For retailers, having too many customers can be a nice problem to have, but customers can be unforgiving if they are running into the same issues time and again.  This is particularly true for online where a customer can 'enter' another store with a simple click of a mouse. 

A robust technology platform that enables customers to access the information they need and purchase in an efficient manner is absolutely essential in today's super-competitive online retail environment.  And retailers don't want to be bogged down by technology. They just want it to work, so they can focus on their core business and what they know best - retailing.  For our friends at Click Frenzy, that's exactly what they're doing now and we look forward to many more successful online 'frenzies', and maybe picking up a bargain or two ourselves along the way. 

 

Ian Teague is a regional sales manager at Akamai.

Who Is George Penguin?

One of the more challenging tasks as the new guy in Akamai's InfoSec department is getting to know George Penguin. He's our mascot and ambassador of good will. His likeness is everywhere in the office, most notably in the form of soft, stuffed toys that dominate the workspace like an invasion of the tribbles from "Star Trek."

I met George long before starting this job, and I admit that I've had a little fun at his expense. During the RSA conference in San Francisco last February, I acquired a stuffed George and stuck him in the side pocket of an unsuspecting colleague, who spent the night bouncing from one vendor party to the next with no clue that a penguin's head was bouncing up and down on the side of his leg.
As this department's storyteller, I can't do that sort of thing anymore. I have to play nice with George and keep him happy. Akamai CSO Andy Ellis absolutely adores George, and failing to get on the flightless waterfowl's good side could prove career limiting.
The first time I met George, he looked familiar. Duh, you're probably thinking. Everyone knows what a penguin looks like. But the fluffiness of this guy was something distinctive that stuck in my mind like a thorn. So I did some digging and remembered: I had run into his likeness dozens of times during family trips to the New England Aquarium. He was always in the gift shop, sold in stuffed animal form and in a smaller, rubber version. My youngest son Duncan had one of the latter. His name was Bucky, and he brought the child tremendous joy until he got old and worn out, at which point his rubber butt fell off.
It turns out one of the stuffed penguins was purchased by an Akamai employee during a team outing, and she was allowed to make the purchase as a business expense. That meant he had to be put to work.
And so Akamai's InfoSec emissary was born.
The little dude even has his own Twitter account (@SecurityPenguin), LinkedIn page.
Here's how he describes himself on LinkedIn:
"I am a highly motivated information security professional, looking to promote awareness of security practices. In my role as the Penguin of Awesome, I promote and recognize practices that promote and raise awareness of Information Security. I am assigned in 1-week rotations to shadow staff who have helped make Akamai a more Security-aware place to work, so that I may learn from them and make sure that their peers know how awesome they are."
 
He even has some LinkedIn recommendations. Akamai InfoSec CSIRT Director Michael Smith wrote, "GTP is hands-down the most awesome dictator that I have ever had the opportunity to work for. Just the other day I asked him 'George, I'm having a problem getting the sales reps to say no to customer audits, would it help if I showed up at meetings with a crowbar and threatened them physically?' He nibbled on his herring lunch and nodded. Such genius, such drive, such vision!"
There are pictures on the wall of team members with George. The photo op is something that comes your way in recognition of a job done well. My mug isn't up there yet, but it's something I covet. 
Still, as popular as he is around here, there's something mysterious about George. There's a lot we don't know about him. There are rumors that he has a nemesis out there, someone dedicated to trouncing on the InfoSec principals we hold most dear.
I do have 20 years of reporting experience under my belt, and I intend to use those skills to peel back the layers of mystery.
Stay tuned.

Lessons From Akamai InfoSec Training

Though I've written about InfoSec for the past decade, I've still had my moments of shame. There was the time last year when I fell for one of the oldest social engineering tricks in the book, clicking the link on a direct Twitter message where someone I worked with asked if I'd seen the nasty post someone wrote about me. The co-worker's Twitter account had been hijacked and similar messages were sent to his contacts. The second I clicked the link, I knew I had just done something stupid.

It was a similar story a few years back when I clicked the link to a sci-fi site I received by email from someone masquerading as an old friend. Five-hundred pieces of malware downloaded onto my laptop that day, mainly the stuff that makes adware for pornography and pump-and-dump stock scams pop up all over the screen. I spent several hours cleaning up the mess, and the folks at the office had a good long laugh at my expense. 

In both cases, I hadn't had security training at the companies that employed me, though as a writer of security stories I should have known better. In terms of company security awareness, we received security warnings when an attack was making the rounds, but never a lesson on basic best practices.

On my first day at Akamai, nearly an hour of orientation was dedicated to the subject. I had written about the importance of security training for employees many times over the years, but this was the first time I received it.

Security training in the business world isn't something you can do with a one-size-fits-all mindset. Different companies have different needs, and Akamai is no exception. We dealt with specifics I won't discuss here. But a lot of the directions were pretty basic and applicable in any company and industry.

For example:

--We are told it's fine to use the IM app of our choice to communicate with friends and family. But for any internal, work-related communications, we must use a separate, specific IM tool -- one that has added protection around it.

--We have a routine schedule of pushing out security patches for various programs, and we will occasionally see a box appear on screen asking us to press a button to install new updates. In the training session, it's made clear that we have to pay attention and heed the call to update when called upon.

--Our passwords have to be complex and ironclad. To make sure it is, Akamai has an automated program that tries to crack employee passwords every 24 hours. If yours is penetrated, you get a message telling you to come up with a new one.

--If you walk away from your desk, you must lock your screen. On my second day, I walked away with several applications running on my machine, and I returned to find a sticky note on the monitor that said, "Screen savers FTW." That was also the day I got my first cable-locking device.

--Speaking of sticky notes, another directive we get is to never leave around notes with our passwords and ID authentication questions written on them. 

--There are physical security rules to heed as well. One is to never use something to keep a door hoisted open. No key card, no access.

There are many more details that go into our program, but those are good examples of the basics -- items other companies would benefit from adopting.

I'll have more to say on training and awareness in future posts.

A History of Akamai InfoSec Storytelling

As part of my new role as Akamai's security storyteller, I've been digging around in search of all the press coverage this group has gotten over the years. I'm finding that many articles and blog posts came from me, particularly what I wrote in my last job as managing editor of CSO Magazine.

You could say my coming here was destiny, based on how easily I focused on Akamai InfoSec research as a journalist. Most recently, I wrote about two presentations from SOURCE Boston 2013. One, by Senior Security Architect Eric Kobrin, was an analysis of the BroBot DDoS attacks that have targeted the banking sector. 

The other talk, by researcher Christian Ternus, was about Akamai's Adversarial Resilience program. The goal: better protect Akamai's customers by thinking like those who attack them. "At Akamai the attack surface is huge," Ternus said. "As the bad guys attack our customers, we are constantly being tested to see if our systems are good enough. What's needed then is resilience -- the ability to adapt. Our job is to think and act like the adversary to make Akamai safer."

Looking further back, as a journalist I usually gravitated toward Akamai's InfoSec team for perspective and raw data on the biggest DDoS attacks and pretty much any story concerning cloud and application security.

There was this inside look at what it's like for Akamai to deal head-on with incoming DDoS attacks against customers.

And there was this report -- I didn't write it but did assign it -- throwing cold water on the notion that hacktivists were the chief culprits in the banking attacks.

Indeed, I've often come knocking when I wanted to measure the real impact of attacks against the hype I'd be seeing elsewhere in the media. The realities have often been less dramatic than reported.

Now that I've tossed my reporter's hat on the shelf to collect dust, expect a much deeper focus from me on the raw detail that comes out of a company that, at last check, handled tens of billions of daily Web interactions for 90 of the top 100 online U.S. retailers, 29 of the top 30 global media and entertainment companies, nine of the top 10 world banks, and all branches of the U.S. military. 

This is going to be both fun and informative.

And it won't take long to ramp things up. In hindsight, I've been telling Akamai security stories all along.

Microsoft's June Patch Load

A reminder to Akamai customers and the larger InfoSec community that Microsoft has released its security update for June. Below are the specific bulletins. Click the link on each to get full details.

You'll want to get a fix on which of these are most important to your organization and install them as soon as possible.

MS13-047
Cumulative Security Update for Internet Explorer (critical)

MS13-048
Vulnerability in Windows Kernel Could Allow Information Disclosure (important)

MS13-049
Vulnerability in Kernel-Mode Driver Could Allow Denial of Service (important)

MS13-050
Vulnerability in Windows Print Spooler Components (important)

MS13-051
Vulnerability in Microsoft Office (important)

Rafael Nadal's unprecedented eighth French Open championship wasn't the only record set at Roland Garros last week. France Télévisions' live webcast of number one-seeded Nadal's semifinal victory over second-seed Novak Djokovic on Friday, June 8th, set new peak concurrent viewer and traffic records for the event on the Akamai Intelligent Platform. 

More than 126,000 viewers were simultaneously watching the four-hour match at one point, driving traffic to peak at 155 Gbps. Both figures mark new highs for France Télévisions and its delivery of the live online video broadcast of the French Open over Akamai.

Nadal.jpg(Photo: Susan Mullane, USA TODAY Sports)

France Télévisions used Akamai's Sola Media Solutions to live stream every match from Roland Garros to viewers in France and French territories watching on the website and through tablet and smartphone apps. Through a "multi-court" feature, viewers could choose to watch any match of their choice either live or on-demand. 

Supported by the Akamai Intelligent Platform, Sola Media Solutions enable the economic online distribution of content to the broadest possible array of devices, delivering rich, high-definition quality video to any device at broadcast-size audiences.

Chris Nicholson is a senior public relations manager at Akamai.

Let Me Tell You Some Akamai Security Stories

Allow me to introduce myself: My name is Bill Brenner, Akamai's newly-minted security storyteller (my title is actually senior program manager, but my role here will be quite different from that of other people with similar titles).

I'm a journalist by trade, with two decades of newspaper reporting and editing behind me. I've spent the last decade writing about information security, a path that brought me here. I'm not a security technician. But I do know how to get the real technicians to help me understand what they're working on. I remember how lost I felt the first time I read a security advisory about a buffer overflow vulnerability in a long-since-forgotten piece of software. Within a month or two, I was able to look at advisories and make instant sense of them. Because I asked the people in the trenches to explain it to me like I was a toddler -- and they happily obliged.

A decade later, I'm experienced in the art of seeing the bigger infosec picture and how all the smaller puzzle pieces fit in. I know how to get practitioners to explain threats and defenses in a way that's accessible to the public. And that's what I'm here to do for Akamai.

I'll routinely blog about security events making news headlines, as I have in previous blogs. I'll offer attitudes and opinions and will be forceful about it. That's my style, and it helped get me here. But most of the time I'll be telling you about what's happening in Akamai's InfoSec department. It's a powerful security operation, but the public still doesn't consider that part of the equation when they hear the Akamai name. They think of the cloud and the speed with which Internet services are delivered to them. The threats to those services and the ways Akamai defends against them? Not so much.

I'll be using my journalistic and editorial skills to extract the details from InfoSec staff and explain it to you in ways that make sense. I'll use this blogging account, but I'll also make sure you see the blogging of others on the InfoSec team. There will also be many slideshows, videos and articles to help tell the story. I'll make sure you get to know the different players on this team, what they do and what they're working on. I'll help write research reports and work with the team to give you infographics that make things easier to digest and act upon. And there will be podcast recordings of my conversations with the different players.

I'm already blown away by the heart, talent and grit of this team. You will be, too.

I'm thrilled to be here, and eager to get started.