Akamai Diversity

The Akamai Blog

Heartbleed Update

Update 2014-04-13: Our beliefs in our protection were incorrect; update here.
Today, we provided more information to our customers around the research we've done into the Heartbleed vulnerability.  As our analysis may inform the research efforts of the industry at large, we are providing it here. 
Summary: Akamai patched the announced Heartbleed vulnerability prior to its public announcement.  We, like all users of OpenSSL, could have exposed passwords or session cookies transiting our network from August 2012 through 4 April 2014.  Our custom memory allocator protected against nearly every circumstance by which Heartbleed could have leaked SSL keys.  There is one very narrow window through which 4 Akamai server clusters had a vulnerable release for 9 days in March 2013.  For the small number of customers potentially affected, we are pro-actively rotating certificates.
All certs issued on or after 1 April 2013 are certainly safe.
Please read below for more details on this issue.
You may have heard of the openssl vulnerability released in the last few days.  As a reminder, the Heartbleed vulnerability occurs when an adversary sends a TLS heartbeat message, which contains both a message, and a purported size of the message.  A server allocates an unitialized memory block based on the actual size of the message, and stores it there.  This memory block is at least 16 kB in size, and may contain information from previous uses of it. The server then allocates a response buffer based on the purported size of the message, and then blindly copies memory space from the receive buffer area into the response buffer.  If the response buffer is larger than the receive buffer, this means that other memory is copied (and sent back to the attacker). Most server software runs for a long time, re-using memory for whatever purpose comes up--so that other memory, as well as the uninitialized memory, could contain almost anything. Examples shared publicly have shown passwords, session cookies, form values--very sensitive data, all processed on the heap by that server process.
If you have a certificate in Akamai's Secure Content Delivery Network, we wanted to give you an update on our investigations. Our blog post on Tuesday described our then-current understanding of the protections that the Akamai platform had around SSL keys in the presence of the Heartbleed vulnerability:
Akamai uses a custom secure memory allocation scheme, controlling which portions of memory each library can use, and more specifically controlling the storage location of SSL keys and other secrets. Our initial assessment suggests that this memory allocation provided better protection than the ordinary OPENSSL_malloc memory allocator. At this time, we don't believe it possible to overflow a buffer from the ordinary heap to the special custom key heap. We are continuing to investigate.
In our initial assessment, we did not believe that customer private keys could have been leaked as a result of this vulnerability; although session information (cookies, logins) could certainly have been exposed to an active adversary.
Akamai uses a custom memory allocator with a separate heap for SSL private keys. We replace the OPENSSL_malloc call with our own secure_malloc. Our edge server software sets up two heaps. Whenever we're allocating memory to hold an SSL private key, we use the special "secure heap." When we're allocating memory to process any other data, we use the "normal heap." The secure heap is a single contiguous region of memory set up with the Linux mmap system call, very early in the server process's life. Our normal heap is a set of many regions of memory managed by the tcmalloc library--which also uses the mmap system call.
Our version of tcmalloc only allocates zones of 1 MB or more at a time. There should never be a hole of 1 MB or more near the secure heap: that space is full of other long-term allocations, like library code. Given that design, we did not see an opportunity for accident or random chance to put a buggy SSL buffer near the secure heap.
To test this belief, our engineers have rebuilt every Akamai configuration since August 2012, when we upgraded to openssl-1.0.1. This included every kernel, hardware, and edge server software combination; and then a careful inspection of the ways in which memory was allocated, to see if any non-long-term memory allocations might border on our secure heap.  Most of the configurations were proven safe; but we found one configuration that was not - there was an available memory block in range of the secure key store.  
This less safe configuration was active on our network for nine days in March 2013, in four Akamai server clusters.  Further, only certificates stored in the first 48 KB of the multi-megabyte secure key store had a possibility of exposure.  Unfortunately, we cannot at this time perfectly identify which certificates would have been in that space. Around 435 still-active customer keys were available to those servers.  While we believe that the likelihood of compromise is vanishingly small, and the number of exposed certificates tiny, we cannot rule out that it could have happened.  We will therefore proceed with rotating customer SSL keys that had even this marginal elevation in risk.  Some of these rotations will happen smoothly, when working with our CA partners with robust APIs.  Other rotations will involve CAs which require human interaction; our account teams will engage with customers as needed to facilitate validation and rotation.
We've tried to address some of the frequent questions we heard from our customers below.
* What should I do?
You should certainly evaluate your own infrastructure and risks. Because anything protected in an SSL connection could have been leaked to an adversary, consider rotating login credentials, session cookies, and long term secrets.  SSL keys held in your origin infrastructure should be considered in this analysis.
* How did Akamai fix this so quickly, and why didn't you tell your customers in advance?
As a courtesy to us, we were notified shortly before public disclosure, which gave us enough time to patch our systems.  We were asked not to publicly disclose the vulnerability, as doing so would have shortened the window of opportunity for others to fix their systems.  Once we were notified, our incident management process governed patching, testing, and deploying the fix to our network safely.
* Do you have any evidence of a data breach?
No.  And unfortunately, this isn't "No, we have evidence that there was no breach of data;" rather, "we have no evidence at all."  We doubt many people do - and this leaves data holders in the uncomfortable position of not knowing what, if any, data breaches might have happened.  Sites using Akamai were not measurably safer - or less safe - than sites not using Akamai.

1 Comment

Cloudflare's Heartbleed challange cracked: private server keys stolen.