Akamai Diversity
Home > Andy Ellis

Recently by Andy Ellis

SSL is dead, long live TLS

An attack affectionately known as "POODLE" (Padding Oracle On Downgraded Legacy Encryption), should put a stake in the heart of SSL, and move the world forward to TLS. There are two interesting vulnerabilities: POODLE, and the SSL/TLS versioning fallback mechanism. Both of these vulnerabilities are discussed in detail in the initial disclosure.


POODLE is a chosen-plaintext attack similar in effect to BREACH; an adversary who can trigger requests from an end user can extract secrets from the sessions (in this case, encrypted cookie values). This happens because the padding on SSLv3 block ciphers (to fill out a request to a full block size) is not verifiable - it isn't covered by the message authentication code. This allows an adversary to alter the final block in ways that will slowly leak information (based on whether their alteration survives verification or not, leaking information about *which* bytes are interesting). Thomas Pornin independently discovered this, and published at StackExchange.

Shellshock Update

The Shellshock vulnerability, originally announced as one critical issue in bash that allowed an adversary to execute arbitrary code, has grown from one vulnerability to six in the last week. For background on Shellshock, we've collected an overview and list of the vulnerabilities; for some history on Akamai's initial responses, read our original blog post
 Shellshock raised a lot of questions among our customers, peers, auditors, and prospects. This post addresses some of the most frequently asked questions, and provides an update on how Akamai is handling its operations during this industry-wide event.
Are Akamai production servers vulnerable?  What is the status of Akamai mitigation?
Akamai's HTTP and HTTPS edge servers never exposed any vulnerability to any of the six currently available CVEs, including the original ShellShock vulnerability.  Our SSH services (including NetStorage) were vulnerable post-authentication, but we quickly converted those to use alternate shells. Akamai did not use bash in processing end-user requests on almost any service. We did use bash in other applications that support our operations and customers, such as our report generation tools. We switched shells immediately on all applications that had operated via bash and are deploying a new version of bash that disables function exporting. 
Akamai's Director of Adversarial Resilience, Eric Kobrin, released a patch for bash that disables the Shellshock-vulnerable export_function field. His code has aggregated additional upstream patches as available, meaning that if you enable function import using his code, the same behaviors and protections available from the HEAD of the bash git tree are also available. His patch is available for public review, use, and critique.
We do not believe at this time that there is any customer or end user exposure on Akamai systems as a result of Shellshock.
What about Akamai's internal and non-production systems?
Akamai has a prioritized list of critical systems, integrated across production, testing, staging, and enterprise environments. Every identified critical system has had one or more of the following steps applied:
  • Verify that the system/application is not using bash (if so, we disabled the vulnerable feature in bash or switched shells);
  • Test that the disabled feature/new shell operates seamlessly with the application (if not, we repeated with alternate shells);
  • Accept upstream patches for all software/applications where available (this is an ongoing process, as vendors provide updates to their patches); and
  • Review/Audit system/application performance to update non-administrative access and disable non-critical functions.
Can we detect if someone has attempted to exploit ShellShock? Has Akamai been attacked?
Because the ShellShock Vulnerability is a Remote Code Execution vulnerability at the command shell, there are many possible exploits available using the ShellShock vulnerability. Customers behind our Web Application Firewall (WAF) can enable our new custom rules to prevent exploits using legacy CGI systems and other application-level exploits. These WAF rules protect against exploits of four of the six current vulnerabilities, all that apply to our customers' layer seven applications.
However, because ShellShock was likely present for decades in bash, we do not expect to be able to find definitive evidence -- or lack thereof -- of exploits.
There have been news reports indicating that Akamai was a target of a recent ShellShock-related BotNet attack. (See information about WopBot). Akamai did observe DDOS commands being sent to a IRC-controlled botnet to attack us, although the scale of the attack was insufficient to trigger an incident or need for remediation. Akamai was not compromised, nor were its customers inconvenienced.  We receive numerous attacks on a daily basis with little or no impact to our customers or the services we provide. 
Akamai's Cloud Security Research team has published an analysis of some attack traffic that Akamai has seen across its customers for Shellshock. As the authors note in that article, the kinds of payloads being delivered using the ShellShock vulnerability have been incredibly creative, with Akamai's researchers seeing more than 20,000 unique payloads. This creativity, coupled with the ease of the ShellShock vulnerability, is one of the many reasons that Akamai is keeping a close eye on all of the associated CVEs and continuing to update its systems and developing better protections for its customers, including custom WAF rules.
Where can I find updates on Akamai's WAF rules?
Information about our WAF rules can be found on our security site
How will Akamai communicate updates?
We will maintain this blog with interesting news from Akamai. 
As the list of CVEs and implications of ShellShock expand, we do our best to only deliver verified information, sacrificing frequency of updates for accuracy.
Akamai is maintaining additional materials for the public on its security site at , including a running tally of the bash-related vulnerabilities.
If you have questions that aren't addressed by one of these vehicles, please feel free to contact your account team.

Environment Bashing

[UPDATE: 9/25/2014 11:30AM]

Akamai is aware that the fix to CVE-2014-6271 did not completely address the critical vulnerability in the Bourne Again Shell (bash). This deficiency is documented in CVE-2014-7169. The new vulnerability presents an unusually complex threat landscape as it is an industry-wide risk.

Akamai systems and internal Akamai control systems have been or are being urgently patched or otherwise mitigated in prioritized order of criticality.

Akamai has developed an emergency patch for today's vulnerability which makes function forwarding conditional on the compile-time switch "FUNCTION_EXPORT". We're using it for systems that can't be switched to the Almquist Shell. In the hope that it's useful to others, and for public review, we've posted that patch at https://bit.ly/ShellShockPatch.



About CVE-2014-6271
Today, CVE-2014-6271 was made public after its discovery last week by Stephane Chazelas. This vulnerability in bash allows an adversary who can pass commands to bash to execute arbitrary code.  As bash is a common shell for evaluating and executing commands from other programs, this vulnerability may affect many applications that evaluate user input, and call other applications via a shell.  This is not an Akamai-specific issue; this impacts any system that uses a vulnerable bash.
Akamai has validated the existence of the vulnerability in bash, and confirmed its presence in bash for an extended period of time.   We have also verified that this vulnerability is exposed in ssh---but only to authenticated sessions. Web applications like cgi-scripts may be vulnerable based on a number of factors; including calling other applications through a shell, or evaluating sections of code through a shell.  
There are several functional mitigations for this vulnerability: upgrading to a new version of bash, replacing bash with an alternate shell, limiting access to vulnerable services, or filtering inputs to vulnerable services.  Akamai has created a WAF rule to filter this exploit; see "For Web Applications" below for details.
Customer Mitigations
Systems under our customers' control which might be impacted include not only vulnerable web applications, but also servers which expose bash in various ways.  System owners should apply an updated bash with a fix for this vulnerability as expeditiously as possible.  In the meantime, there are several workarounds that may assist.
For SSH servers:  Evaluate the users who have access to critical systems, removing non-administrative users until the systems are patched.
For Web Applications: CGI functionality which makes calls to a shell can be disabled entirely as a short term measure; alternately WAF mitigations can be deployed.  The following Akamai WAF protections will assist customers that have purchased the Kona Web Application Firewall or Kona Site Defender services:
  • All customers of Akamai's web acceleration services (Ion, Dynamic Site Acceleration, EdgeSuite, Object Delivery, and similar) are protected against some attacks using this vulnerability by Akamai's HTTP normalization.  Attack text in Host headers, HTTP version numbers, and similar will be silently discarded by the Akamai Platform.  
  • Use of the "Site Shield" feature can further enhance the HTTP normalization protection by removing attackers' ability to connect directly to a customer web server.
  • Customers who use the new Kona Rule Set are protected against some attacks using this vulnerability if they enable the Command Injection risk scoring group.  This scoring group is being extended today to provide more comprehensive coverage by incorporating specific rule triggers for this vulnerability.  
  • We also have a custom WAF rule which will provide a targeted defense against this specific vulnerability.  This WAF rule operates by filtering on the four-byte attack string '() {' in the request header or body . It can be implemented to cover both headers and query strings (for maximum safety), or headers only (in the event that query strings generate too many false positives).  The custom rule is available to customers by contacting their account team.  It will be available for self service today.
  • This custom WAF rule will shortly be available for self-service provisioning for non-KRS customers.
Akamai Mitigations
Public-facing Akamai systems and internal Akamai control systems have been or are being urgently patched or otherwise mitigated in prioritized order of criticality. 
For many of our critical systems, Akamai first switched away from using bash to another shell, to protect those systems.  We did not make a universal switch, as the alternate shell was not completely feature-compatible with bash (not all of our systems would continue to operate with this change).
For other systems which could tolerate the downtime, we disabled those systems pending receiving an updated bash. For Akamai web applications which couldn't take an alternate shell and couldn't be disabled, we blocked many Akamai-owned CGI scripts at the Akamai Edge, until we developed and deployed the WAF rules above.
Do you have any evidence of system compromises?
No.  And unfortunately, this isn't "No, we have evidence that there were no compromises;" rather, "we don't have evidence that spans the lifetime of this vulnerability."  We doubt many people do - and this leaves system owners in the uncomfortable position of not knowing what, if any, compromises might have happened. 
The OpenSSL Project today disclosed new vulnerabilities in the widely-used OpenSSL library.  These are vulnerabilities that can potentially impact OpenSSL clients and servers worldwide.
The most interesting is the ChangeCipherSpec Injection, which would enable a man-in-the-middle attack to force weaker ciphers into a communication stream.  
Akamai SSL services (both Secure Content Delivery and Secure Object Delivery) have been patched for this vulnerability. The other vulnerabilities are relatively uninteresting for our environment - we don't support DTLS, and we don't enable SSL_MODE_RELEASE_BUFFERS.
Despite the time and inconvenience caused to the industry by Heartbleed, its impact does provide some impetus for examining the underlying certificate hierarchy. (As an historical example, in the wake of CA certificate misissuances, the industry looked at one set of flaws: how any one of the many trusted CAs can issue certificates for any site, even if the owner of that site hasn't requested them to do so; that link is also a quick primer on the certificate hierarchy.)
Three years later, one outcome of the uncertainty around Heartbleed - that any certificate on an OpenSSL server *might* have been compromised - is the mass revocation of thousands of otherwise valid certificates.  But, as Adam Langley has pointed out, the revocation process hasn't really worked well for years, and it isn't about to start working any better now.
Revocation is Hard
The core of the problem is that revocation wasn't designed for an epochal event like this; it's never really had the scalability to deal with more than a small number of actively revoked certificates.  The original revocation model was organized around each CA publishing a certificate revocation list (CRL): the list of all non-expired certs the CA would like to revoke.  In theory, a user's browser should download the CRL before trusting the certificate presented to it, and check that the presented certificate isn't on the CRL.  In practice, most don't.  Partly because HTTPS isn't really a standalone protocol: it is the HTTP protocol tunneled over the TLS protocol.  The signaling between these two protocols is limited, and so the revocation check must happen inside the TLS startup, making it a performance challenge for the web, as a browser waits for a CA response before it continues communicating with a web server.
CRLs are a problem not only for the browser, which has to pull the entire CRL when it visits a website, but also for the CA, which has to deliver the entire CRL when a user visits one site.  This led to the development of the online certificate status protocol (OCSP).  OCSP allows a browser to ask a CA "Is this specific cert still good?" and get an answer "That certificate is still good (and you may cache this message for 60 minutes)."  Unfortunately, while OCSP is a huge step forward from CRLs, it still leaves in place the need to not only trust *all* of the possible CAs, but also make a real-time call to one during the initial HTTPS connection.  As Adam notes, the closest thing we have in the near term to operationally "revocable" certs might be OCSP-Must-Staple, in which the OCSP response (signed by the CA) is actually sent to the browser from the HTTPS server alongside the server's certificate.
One Possible Future
A different option entirely might be to move to DANE (DNSSEC Assertion of Named Entities).  In DANE, an enterprise places a record which specifies the exact certificate (or set of certificates, or CA which can issue certificates) which is valid for a  given hostname into its DNS zone file.  This record is then signed with DNSSEC, and a client would then only trust that specific certificate for that hostname. (This is similar to, but slightly more scalable than, Google's certificate pinning initiative.)
DANE puts more trust into the DNSSEC hierarchy, but removes all trust from the CA hierarchy.  That might be the right tradeoff.  Either way, the current system doesn't work and, as Heartbleed has made evident, doesn't meet the web's current or future needs.
(Footnote:  No conversation made herein around Certificate Transparency, or HSTS, both of which are somewhat orthogonal to this problem.)
This entry crossposted at www.csoandy.com.

Heartbleed: A History

In the interest of providing an update to the community on Akamai's work to address issues around the Heartbleed vulnerability, we've put together this outline as a brief summary:
  • Akamai, like all users of OpenSSL, was vulnerable to Heartbleed.
  • Akamai disabled TLS heartbeat functionality before the Heartbleed vulnerability was publicly disclosed.
  • In addition, Akamai went on to evaluate whether Akamai's unique secure memory arena may have provided SSL key protection during the vulnerability window when we had been vulnerable; it would not have.
  • Akamai is reissuing customer SSL certificates, due to the original Heartbleed vulnerability. 
More detailed information is below.

Heartbleed Update (v3)

Over the weekend, an independent security researcher contacted Akamai about some defects in the software we use for memory allocation around SSL keys.  We discussed Friday how we believed this had provided our SSL keys with protection against Heartbleed and had contributed the code back to the community.  The code that we had contributed back was, as we noted, not a full patch, but would be a starting point for improving the openssl codebase.
In short: we had a bug.  An RSA key has 6 critical values; our code would only attempt to protect 3 parts of the secret key, but does not protect 3 others.  In particular, we only try to protect d, p, and q, but not d mod (p-1), d mod (q-1), or q^{-1} mod p.  These intermediate extra values (the Chinese Remainder Theorem, or CRT, values) are calculated at key-generation time as a performance improvement. As the CRT values were not stored in the secure memory area, the possibility exists that these critical values for the SSL keys could have been exposed to an adversary exploiting the Heartbleed vulnerability.  Given any CRT value, it is possible to calculate all 6 critical values.
As a result, we have begun the process of rotating all customer SSL keys/certificates.  Some of these certificates will quickly rotate; some require extra validation with the certificate authorities and may take longer. 
In parallel, we are evaluating the other claims made by the researcher, to understand what actions we can take to improve our customer protection.

Heartbleed Update

Update 2014-04-13: Our beliefs in our protection were incorrect; update here.
Today, we provided more information to our customers around the research we've done into the Heartbleed vulnerability.  As our analysis may inform the research efforts of the industry at large, we are providing it here. 
Summary: Akamai patched the announced Heartbleed vulnerability prior to its public announcement.  We, like all users of OpenSSL, could have exposed passwords or session cookies transiting our network from August 2012 through 4 April 2014.  Our custom memory allocator protected against nearly every circumstance by which Heartbleed could have leaked SSL keys.  There is one very narrow window through which 4 Akamai server clusters had a vulnerable release for 9 days in March 2013.  For the small number of customers potentially affected, we are pro-actively rotating certificates.
All certs issued on or after 1 April 2013 are certainly safe.
Please read below for more details on this issue.

Update 2014-04-11: Updated information on our later analysis here.

We're getting a lot of questions about the OpenSSL Heartbleed fix. What follows are the most commonly asked questions, with our answers.

The Heartbleed bug affects a heartbeat functionality within the TLS/DTLS portion of the library. It allows the attacker to -- silently and without raising alarms -- dump portions of the servers memory to the client. This can allow the attacker to walk through the memory space of the server, possibly dumping private SSL keys and certainly exposing important secrets.

All versions of the OpenSSL library between 1.0.1 and 1.0.1f contain the Heartbleed bug and should be updated to 1.0.1g as soon as possible. (The vulnerability researchers have posted their analysis, and an excellent analysis is up on Sean Cassidy's blog.

Whither HSMs

Hardware Security Modules (HSMs) are physical devices attached or embedded in another computer to handle various cryptographic functions.  HSMs are supposed to provide both physical and logical protection of the cryptographic material stored on the HSM while handling cryptographic functions for the computer to which they are attached.
As websites move to the cloud, are HSMs the right way to achieve our goals?  
Before we talk about goals, it is useful to consider a basic model for talking about them.  Our Safety team often uses the following model to consider whether a system is safe:
What are the goals we are trying to achieve? (Or, in Leveson's STPA hazard-oriented view, what are the accidents/losses which you wish to prevent?)
What are the adversaries we wish to defeat?
What are the powers available to those adversaries? What *moves* are available to them?
And finally, what controls inhibit adversaries' use of their powers, thus  protecting our goals?
Our hazards (or unacceptable losses) are:
An adversary can operate a webserver that pretends to be ours;
An adversary can decrypt SSL traffic; and
An adversary can conduct a man-in-the-middle attack on our SSL website.
In the protection of SSL certificates in the cloud, it would seem that our goals are two-fold:
Keep the private key *secret* from third parties; and
Prevent unauthorized and undetected use of the key in cryptographic functions. While SSL certificate revocation is a weak control (many browsers do not check for revocation), it is that which generally constrains this goal to both unauthorized *and* undetected; a detected adversary can be dealt with through revocation.
I could argue that the first is a special case of the second, except that I want to distinguish between "cryptographic functions over the valid lifetime of the certificate" and "cryptographic functions after the certificate is supposed to be gone."
As an aside, I could also argue that these goals are insufficient; after all, except for doing man in the middle attacks, *any* SSL certificate signed by any of the many certificate authorities in the browser store would enable an adversary to cause the first of the losses.  HSMs don't really help with that problem.
Given that caveat, what are the interesting adversaries?  I propose four "interesting" adversaries, mostly defined by their powers:
The adversary who has remotely compromised a server;
The adversary who has taken physical control of a server which is still online;
The adversary who has taken physical control of a server at end of life; and
The adversary who has been given administrative access to a system.
The moves available to these adversaries are clear:
Copy key material (anyone with administrative access);
Change which key material or SSL configuration we'll use (thus downgrading the integrity of legitimate connections)
Escalate privileges to administrative access (anyone with physical or remote access); and
Make API calls to execute cryptographic functions (anyone with administrative access).
What controls will affect these adversaries?
Use of an HSM will inhibit the copying of keying material;
Use of revocation will reduce the exposure of copied keying material;
System-integrated physical security (systems that evaluate their own cameras and cabinets, for instance) inhibit escalation from physical access to administrative access;
Auditing systems inhibits adversary privilege escalation;
Encrypting keying material, and only providing decrypted versions to audited, online systems inhibits adversaries with physical control of systems.
What I find interesting is that for systems outside the physical purview of a company, HSMs may have a subtle flaw: since HSMs must provide an API to be of use, *that API remains exposed to an adversary who has taken possession of an HSM*.  This may be a minor issue if an HSM is in a server in a "secure" facility, it becomes significant in distributed data centers.  On the contrary, the control system which includes tightly coupled local physical security, auditing, and software encryption may strike a different balance: slightly less stringent security against an adversary who can gain administrative access (after all, they can likely copy the keys), in exchange for greater security against adversaries who have physical access.
This isn't to say that this is the only way to assemble a control system to protect SSL keys; merely that a reflexive jump to an HSM-based solution may not actually meet the security goals that many companies might have.
(Full disclosure: I'm the primary inventor of Akamai's SSL content delivery network, which has incorporated software-based key management for over a decade.)
<< 1 2