Akamai Diversity

The Akamai Blog

Heartbleed: A History

In the interest of providing an update to the community on Akamai's work to address issues around the Heartbleed vulnerability, we've put together this outline as a brief summary:
  • Akamai, like all users of OpenSSL, was vulnerable to Heartbleed.
  • Akamai disabled TLS heartbeat functionality before the Heartbleed vulnerability was publicly disclosed.
  • In addition, Akamai went on to evaluate whether Akamai's unique secure memory arena may have provided SSL key protection during the vulnerability window when we had been vulnerable; it would not have.
  • Akamai is reissuing customer SSL certificates, due to the original Heartbleed vulnerability. 
More detailed information is below.
A Heartbleed Primer
Heartbleed is a bug in the TLS heartbeat implementation where an adversary sends a request to be echoed back; and specifies a length of the response to be echoed.  Because the length to be echoed back isn't checked against the length of the inbound request, a server can respond with information that happened to be in memory: up to 64KB of it per request.
There are really two distinct ways in which memory is exposed.  The first exposure reveals the contents of OpenSSL buffers. OpenSSL manages its own memory space for requests and replies, and aggressively reuses them without clearing them. In and of itself, this is not a bug, but it does aggravate the impact of Heartbleed.  You log in to a web server, which puts your username and password inside a chunk of OpenSSL buffer memory; then a Heartbleed attack comes in, and that request is assigned the same chunk of memory.  Because the attacker only sends a small amount of data, it only overwrites the first few bytes of that chunk of memory; the rest of the memory is now available to the attacker.  That chunk of memory is at least 16KB in size.
The second memory exposure occurs when openssl reads past that 16KB buffer into the additional 48k of memory that follows it. That 48k chunk of memory isn't necessarily owned by OpenSSL; it might belong to other code running in the same process.  So, OpenSSL first copies the 16KB chunk of memory (the first bug), and then it copies whatever happens to be in the next 48KB.
This Heartbleed vulnerability impacted everyone using openssl, including Akamai. Because memory was exposed in two different fashions, not only could parts of previous user sessions be exposed, other parts of the webserver's memory were also potentially exposable.  In August 2012, Akamai upgraded to the version of openssl that contained the heartbeat functionality, and became vulnerable to Heartbleed attacks.
About Patching
Akamai learned about Heartbleed slightly ahead of public notice of the vulnerability. On April 4th, as recommended, we patched our Secure Content Delivery Network by disabling heartbeats.  On April 5th, we patched our core HTTP content delivery network.
About Akamai's key protection contribution
In addition to updating openssl as recommended, Akamai also evaluated whether we may have had the advantage of additional protections in place between August 2012 and April 2014 as a result of our unique openssl implementation. If so, we might have uniquely mitigated the worst of the vulnerability for Akamai.  In doing so, we took a long look at the secure memory arena we had added to the Akamai system in 2001. We suspected that some of the special characteristics of the Akamai implementation, beyond applying the standard recommended patch, might have already mitigated the worst of the vulnerability. 
Imagine that Heartbleed is a vulnerability in the locks in everyone's homes. So in your own home, a lock doesn't really work correctly, and attackers can freely wander about your house.  This vulnerability not only gives an attacker access to your valuables, it also means they can pick up your housekey. Then you realize that you always put your housekey into a safe built into the foundation, so it probably wasn't stolen.   You still fix the locks, but you think you don't also need to change the keys.
Unfortunately, it turns out we did need to change the keys after all - just like everyone else using openssl.   We submitted our secure memory allocator to the community for review and, unfortunately, it did include bugs, which Willem Pinckaers helpfully pointed out to us.  This meant that while we had correctly fixed the locks themselves, we needed to be rekeying them as well.
About Rotation
We acknowledged the need to rotate all customer certificates on the evening of April 13th, and began rotating the SSL keys on the 14th.
Patch synchronicity for Akamai represents an interesting challenge.  Due to network partitions, some of our edge servers operate autonomously for extended periods. Usually, as we are performing software upgrades, we will identify machines running older software as these network partitions clear up. Our NOCC will be alerted, triggering a fresh installation on those machines. After some careful consideration, we decided to blacklist all Akamai servers in the Secure Content Delivery Network if we hadn't recorded that they had positively received updated software.   We feel this is a prudent step to prevent outdated machines from having any access to certificates that have been rotated.
We reissued some certificates before it might have been safe to do so.  Out of concern regarding synchronicity with our blacklisting of servers, those are being re-reissued.  
After we rotate customer certificates, we expect that many of them will be rotating their passwords - but those are individual decisions for each customer.
The future of our memory allocator
We are currently reviewing a revised version of our secure memory arena with some external researchers and developers.  Once we are more confident that it more closely achieves its goals, we will contribute this code to the community. We also plan to evaluate how we can better collaborate and support the open source community.