Akamai Diversity

The Akamai Blog

On memory overflow and responses

On February 23, 2017, Cloudflare released information on a bug that was disclosed by Google security researcher, Tavis Ormandy, in their content delivery network. The bug potentially exposed sensitive customer data to the Internet. Approximately 1 in every 3,300,000 HTTP requests may have contained potentially sensitive information.  This information would normally be stored and cached by users and search engines as part of normal website sessions.  This bug is similar to Heartbleed, in that uninitialized memory was accidentally being sent along with regular data. Unlike Heartbleed, which required malicious requests, this bug was in Cloudflare's HTML parser code, which means that sensitive data could be sent as part of normal client requests.

The Bug
The bug occurred in Cloudflare's custom HTML parser. It would likely be triggered when their content delivery network would receive malformed HTML from their customers' sites. In those cases, random other data from memory, including from other HTTP or HTTPS requests, could be returned as part of the data transmitted to the end-user. Cloudflare did not use their HTML parser on all responses. It was primarily used for implementing their email obfuscation feature. Because the bug is in Cloudflare's bespoke software, sites that do not transmit data through Cloudflare are not directly affected.

What data was potentially exposed?
Samples of client responses for sites using the content delivery network analyzed by Google employees contained encryption keys, cookies, passwords, POST data, and HTTPS requests for many major Cloudflare customers. Because the data was returned in unrelated HTTP responses, the extent of compromised information is unclear. Cloudflare has been working with search engines to clear cached data as quickly as possible.

Is Akamai Impacted?
Akamai's content delivery network runs software that is fundamentally different than Cloudflare's. The HTML parser used by Cloudflare was custom code developed internally and does not affect Akamai or other service providers. However out of an abundance of caution, we are reviewing our systems to look for similar problems or possible bugs. If a similar vulnerability is discovered, it will be handled per Akamai's incident management process.

What should I do?
If your site transmits any data through Cloudflare, you may have been affected. This includes sites which primarily use Akamai but rely on third-party inclusions from Cloudflare-fronted sites or which use third-party APIs behind Cloudflare. If you're using Cloudflare for any part of your content delivery and want to know more, Google's Project Zero disclosed the details of vulnerability here: https://bugs.chromium.org/p/project-zero/issues/detail?id=1139.

Cloudflare's response
Cloudflare responded rapidly and repaired the damage caused by this bug. Any development of complex systems for processing data across the Internet is susceptible to behaviour that causes unexpected responses. It is important for service providers to respond in a rapid and transparent fashion in order to retain customer trust.

Further Reading
In addition to the Google disclosure linked above, additional material is available at:
Cloudflare's Public Blog Post: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
Ryan Lackey's Guide on how to deal with Cloudbleed: https://medium.com/@octal/cloudbleed-how-to-deal-with-it-150e907fd165#.5qpn3vwlz
Akamai's Vulnerability and Patch Management Process: https://blogs.akamai.com/2016/08/vulnerability-management-at-akamai.html
Akamai's Incident Management Process: https://www.akamai.com/us/en/our-thinking/information-security/compliance/information-security-incidents.jsp