Akamai Diversity

The Akamai Blog

On memory overflow and responses

On February 23, 2017, Cloudflare released information on a bug that was disclosed by Google security researcher, Tavis Ormandy, in their content delivery network. The bug potentially exposed sensitive customer data to the Internet. Approximately 1 in every 3,300,000 HTTP requests may have contained potentially sensitive information.  This information would normally be stored and cached by users and search engines as part of normal website sessions.  This bug is similar to Heartbleed, in that uninitialized memory was accidentally being sent along with regular data. Unlike Heartbleed, which required malicious requests, this bug was in Cloudflare's HTML parser code, which means that sensitive data could be sent as part of normal client requests.

The Bug
The bug occurred in Cloudflare's custom HTML parser. It would likely be triggered when their content delivery network would receive malformed HTML from their customers' sites. In those cases, random other data from memory, including from other HTTP or HTTPS requests, could be returned as part of the data transmitted to the end-user. Cloudflare did not use their HTML parser on all responses. It was primarily used for implementing their email obfuscation feature. Because the bug is in Cloudflare's bespoke software, sites that do not transmit data through Cloudflare are not directly affected.

What data was potentially exposed?
Samples of client responses for sites using the content delivery network analyzed by Google employees contained encryption keys, cookies, passwords, POST data, and HTTPS requests for many major Cloudflare customers. Because the data was returned in unrelated HTTP responses, the extent of compromised information is unclear. Cloudflare has been working with search engines to clear cached data as quickly as possible.

Is Akamai Impacted?
Akamai's content delivery network runs software that is fundamentally different than Cloudflare's. The HTML parser used by Cloudflare was custom code developed internally and does not affect Akamai or other service providers. However out of an abundance of caution, we are reviewing our systems to look for similar problems or possible bugs. If a similar vulnerability is discovered, it will be handled per Akamai's incident management process.

What should I do?
If your site transmits any data through Cloudflare, you may have been affected. This includes sites which primarily use Akamai but rely on third-party inclusions from Cloudflare-fronted sites or which use third-party APIs behind Cloudflare. If you're using Cloudflare for any part of your content delivery and want to know more, Google's Project Zero disclosed the details of vulnerability here: https://bugs.chromium.org/p/project-zero/issues/detail?id=1139.

Cloudflare's response
Cloudflare responded rapidly and repaired the damage caused by this bug. Any development of complex systems for processing data across the Internet is susceptible to behaviour that causes unexpected responses. It is important for service providers to respond in a rapid and transparent fashion in order to retain customer trust.

Further Reading
In addition to the Google disclosure linked above, additional material is available at:
Cloudflare's Public Blog Post: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
Ryan Lackey's Guide on how to deal with Cloudbleed: https://medium.com/@octal/cloudbleed-how-to-deal-with-it-150e907fd165#.5qpn3vwlz
Akamai's Vulnerability and Patch Management Process: https://blogs.akamai.com/2016/08/vulnerability-management-at-akamai.html
Akamai's Incident Management Process: https://www.akamai.com/us/en/our-thinking/information-security/compliance/information-security-incidents.jsp

6 Comments

Hi Larry,

I've found over the years, and recurring more and more often, these types of issues are directly related to the coding style of software engineers and the lack of adequate security policies and reviews in software engineering departments. I have seen less and less attention paid to security built into software with up and coming software engineers, and this has been a concern to me.

As you and I learned many years ago, security is all too often the last thing considered when developing a system or part of a system. After all, security costs money with no real visible return on the investment - until a system is hacked and money is lost. I've seen this time and time again over the years. On the up side it keeps you and I employed. :)

Too often engineers (especially in C/C++ languages) declare variables in functions/classes/methods without initializing them. This is especially troublesome when it comes to allocating larger chunks of memory that might contain names, addresses, credit card info., etc. They design the code and write in on-the-fly, often without the thought of security. Most often, software engineers have no background or understanding of security in the software they write and how such mistakes (as uninitialized variables, buffer overflows, etc.) can be used to compromise a system.

Another problem is a lack of destroying (erasing and re-initializing) memory once it is no longer needed by the system. Especially with data such as passwords, card numbers, etc. - this data, once used, should be directly removed from memory, but many systems do not do this (as many engineers do not understand the reasons for doing it).

It's important for corporations, especially in this new age of "The Internet of Things", to design and develop their systems with security as a top priority, and not as an after thought. Engineers developing such systems need to have training in software security and what kinds of things to look for and avoid when developing their systems. Executives must realize that system security is only as good as the underlying software it is based upon and, even though the cost of development may be more up front, the loss that will be imposed later on is far greater.

Thank you for sharing the info

thanks so much for sharing this

thanks so much for sharing this

I think this is wonderful I truly appreciate the information shared in this post I am going to bookmark this..

I think this is wonderful I truly appreciate the information shared in this post I am going to bookmark this..

Leave a comment