Akamai Diversity

The Akamai Blog

Vulnerability Management at Akamai

Akamai's InfoSec team has been documenting security vulnerabilities on this blog for years, and we have long evangelized the security benefits of keeping companies' systems current by applying the latest patches.  Today, management of vulnerabilities and application of the latest patches has become a critical component of any InfoSec program.

Our customers are increasingly required not just to prove that their systems are properly patched, but also that the systems of their service providers such as Akamai are likewise properly patched.  Akamai has always worked to identify and manage the risk of vulnerabilities as we become aware of them.

Akamai's production environment code base is highly bespoke, minimizing the number of impacting vulnerabilities and patches to be applied from public CVEs and software updates. Our enterprise environment does include more commercially-available software packages, and has a separate vulnerability and patch management program that does not impact customer data that traverses Akamai's Intelligent Platform.

Akamai has always been open about our process to identify and mitigate vulnerabilities. We make a version of our process available to customers under NDA.  Our customers and their auditors have requested time-specific service level agreements (SLA) in which we commit to patch relevant vulnerabilities within an agreed upon time frame.  

This post discusses Akamai's vulnerability process in a bit more detail, to help our customers understand Akamai's commitment to timely resolution of vulnerabilities in our unique, global network, and to explain why we are reluctant to commit to specific time-based SLAs in this area.  

Akamai's Intelligent Platform is Designed to Mitigate Risks from Vulnerabilities

Akamai's Intelligent Platform, our network of over 216,000 globally distributed servers, is the basis for our CDN and most of our cloud security solutions. It is built from the ground up with security in mind.  We strive to ensure the availability of our network by building our servers ourselves from commodity parts.  We use our own custom version of Linux that restricts how the servers may be used and strips out unnecessary functionalities. Each Akamai Edge server is designed and configured as a bastion host.  This dramatically reduces the chance of our servers being exploited by adversaries.  

Since we strip out the unnecessary features, the vast majority of reported vulnerabilities on other platforms simply do not apply to our network.  

That same architecture complicates matters when it comes to patching those few that do apply.  We first must analyze the systems and networks across our platform to determine whether a vulnerability will have any impact on the limited functionality supported by our platform.  If we are impacted by the vulnerability, the process of patching the necessary code requires more time-consuming integration into our unique code base.  

As with any software release, we want to ensure that vulnerability patches solve the problem they are intended to solve without compromising the performance or safety of our platform.  Whenever possible, we ensure that vulnerability patches follow our standard Software Development Life Cycle process, to ensure that the necessary testing and quality assurance procedures are followed.  Rather than rushing to install a patch on its own, therefore, we prefer to incorporate it into a regularly scheduled software release.  

When a critical vulnerability arises, we utilize our Incident Management Process to escalate the matter and ensure that patches and other remediations are in place rapidly, while giving us the ability to back out changes, or incorporate further fixes into our next release.  

Layers of Defense Against Vulnerabilities Mitigate the Need for Patching SLAs

Akamai's vulnerability management process relies on several layers of defenses, in addition to patching.  This often gives us the necessary flexibility to ensure that we can patch our systems responsibly while minimizing the risk during that process.

1.  Vulnerability Management Process: Akamai manages risk to its deployed networks on a continuous basis.

a.  Akamai continually operates a parser that automatically analyses the daily CVE reports and publicly available patches from sources related to any open source and commercial software used by Akamai. The parser compares against code Bill of Material (BoM) files and opens Change Requests (CRs) within our tracking system for any patches that need to be applied.

b.  Security risks undergo a qualitative risk assessment, which evaluates the potential severity of a risk, as well as an assessment of attackers that could instantiate a risk. Based on these two factors, the vulnerability will receive a classification that indicates the severity of the risk. Additionally, risks caused by software defects are also scored using the Common Vulnerability Scoring System (CVSS) - the CVSS system has a higher focus on likelihood than on potential damage.

c.  Vulnerabilities rated as "Critical" or "High" - those that may result in disclosure, alteration, or destruction of sensitive data become formal security incidents managed through Akamai's Incident Management Process., This ensures that the appropriate staffing is available to address the matter, and allows for exceptions to the normal software release process.  Additionally, the incident process requires that the appropriate subject matter experts are available to determine whether and how the vulnerability applies to Akamai's platform, the risks inherent in each system, and to determine the appropriate time frame to fix them.

d.  "Moderate" or "low" rated vulnerabilities are evaluated and to the extent feasible, are patched with the next regularly scheduled software releases.  

e.  Akamai's FedRAMP compliance provides an additional layer of oversight and protection to all of our customers.  We undergo continuous monitoring and monthly reporting of our security controls, including vulnerability management.  We are required to patch critical and high vulnerabilities within 30 days, and we have a goal to patch other vulnerabilities within 180 days.  This entire process, and any exceptions or delays in patching or remediation, is reviewed by our FedRAMP assessor, and our FedRAMP compliance depends on our assessor being satisfied that we are appropriately addressing those vulnerabilities that apply to us.

2.  In addition to patching vulnerabilities, each server on Akamai's Intelligent Platform is protected by additional rules and health checks that ensure that known attacks, known IP blacklists, and other potential exploits are filtered out.  These systems include our Web Application Firewall (WAF), our internal Forcefield port hardening solution and the audit servers on our Secure CDN, which perform randomized and request-triggered audits of active server processes to ensure that the network is operating correctly and has not been compromised.  These systems may be updated in near real time, and can often remediate the impact of vulnerabilities immediately, well before a patch can be deployed.  

3.  Akamai's Intelligent Platform is managed 24/7 by our global Network Operations Command Center (NOCC), and all of our systems are designed to alert the NOCC of any anomalous or undesirable activity on network.  Upon receiving an alert, the NOCC may immediately suspend a server from the network for service or investigation and/or delete all secrets from the server.  This allows us to take vulnerable servers offline while investigating or implementing a patch, while keeping our platform as a whole running, and without disrupting service to our customers.  

With all these layers of defense, Akamai is confident that its Incident Management Process ensures that critical vulnerabilities, and those that cannot be blocked by other means are patched immediately, while protecting our customers from the impact of those less critical vulnerabilities before a patch is in place.  This allows us to perform any necessary patches, on timelines subject to third party review, while ensuring that upgrades to our platform are safe and effective.

 

*Evidence of Akamai's FedRAMP Compliance can be found here, but FedRAMP regulations do not permit service providers to share the underlying documentation directly with customers.  Federal agencies may request the documents by following the instructions on the page linked to here.

Leave a comment