Akamai Diversity
Home > Web Security > Planning for the end of 2016: a leap second and the end-of-support for SHA-1 TLS certificates

Planning for the end of 2016: a leap second and the end-of-support for SHA-1 TLS certificates

New Year's Eve is typically in the depth of end-of-year change freezes for most IT organizations. At the end of 2016, however, two major events will be occurring right at the end of the year: a leap second and the final end of browser support for SHA-1 TLS certificates.  Both of these changes have the potential to break software systems and applications.  Significant preparation, planning, and testing ahead-of-time can significantly reduce the risk for both.

The Leap Second

At the stroke of midnight UTC on New Year's Eve, an extra second will be introduced, effectively causing the last second of the day to last for two seconds.  Time will advance from: 2016-12-31 23:59:58 UTC to 23:59:59 to 23:59:60 before reaching 2017-01-01 00:00:00 UTC.  This will apply earlier in other time zones.  For example, the US East Coast will observe 6:59:60 PM.  Since this change has resulted in significant software bugs across the world during the past two leap seconds in both 2012 and 2015, either significant testing or some other mitigation strategy is required to shield software systems and devices from leap second bugs.  

As we did during the leap second in 2015 and as Google pioneered in 2008 and 2012, Akamai will again be implementing a "leap smear" where we apply the leap second over the course of 24 hours rather than all at once.  We believe this is significantly safer overall, but during the peak of the leap smear it does mean that Akamai edge server clocks may appear up to a second off relative to origin server clocks, for origin servers not implementing a leap smear or applying a different leap smear curve.

This difference has some risk of causing issues by itself (although relying on sub-second time synchronization between Akamai servers and origins is generally discouraged as it may result in problems during normal operations), but the overall risks are significantly lower than introducing an additional second right on New Year's Eve.

SHA-1 TLS Certificates

When the TLS/SSL certificates used to identify web sites are signed by certificate authorities (CAs), the signatures use a cryptographic "signature algorithm" that includes a cryptographic hash function. The end of 2016 marks the end of support for the SHA-1 hash function in most major browsers as a signature algorithm for leaf and intermediate certificates. This is because cryptographic weaknesses in the SHA-1 algorithm, combined with increases in computational power, make it viable for well-funded attackers to generate multiple certificates with the same SHA-1 hash but with different contents.  When a CA applies a SHA-1 RSA signature over what they have validated and what appears benign, they are thus also applying a signature to the alternate certificate that the attacker has prepared, which may impersonate a well-known website.

To address this risk, CAs were prohibited from issuing any SHA-1 certificates after January 1, 2016 as well as any expiring after December 31, 2016.  Many browsers (Chrome, Mozilla Firefox, Microsoft IE/Edge, and others) will also start rejecting certificates signed with SHA-1 early in 2017.  While some have exceptions for locally installed CA roots, Chrome plans to also present fatal network errors even in these cases.

The big risk of the retirement with SHA-1 certificates is that some clients out there (including some Symbian phones, Windows XP SP2, some old versions of Android, and many custom applications and embedded devices) do not know how to handle modern RSA SHA-256 certificates and return validation errors when encountering them.

Much of the web has already switched over to RSA SHA-256 certificates.  Since Akamai started offering SHA-256 certificates in early 2015, over 95% of customer certificates served on Akamai's Secure CDN have already switched over to using only RSA SHA-256 certificates.  

However, we do still periodically encounter customers who have a custom client or application break when their SHA-1 certificate rotates into a SHA-256 certificate.  This has typically involved a roll-back of the cert rotation, a scramble to make and release changes to the application, and then rolling-forwards with the rotation.  As SHA-1 certificates are no longer possible to provision, options here quickly become limited.

Some large web sites, as well as Akamai's shared certificate platform (with *.akamaihd.net and *.akamaized.net and a248.e.akamai.net), have been trying to stretch out SHA-1 support as far as safely possible.  On the Akamai shared certificate platform, we return an RSA SHA-256 certificate for clients using TLS 1.2 and indicating support with the signature_algorithms extension.  Even excluding clients which also send a TLS SNI extension (indirectly correlated with SHA-256 support), we still see significant numbers of handshakes completing and using SHA-1.  At the end of 2016, handing out SHA-1 will no longer be possible (as this would mean serving an expired or invalid certificate to clients which may or may not support SHA-256 but would very likely error on any expired certificate).

To avoid making the change to our shared certificate on New Year's Eve, we will be shutting off the SHA-1 certificate, and will always hand out an RSA SHA-256 or ECDSA SHA-256 certificate, on or around December 27.

Note that if your company has a local CA root signing certificates for internal sites, you should still make sure that SHA-1 certs are no longer in use, as at least Chrome has announced that they will start returning errors on these early next year.

A common question is whether SHA-1 root certificates will continue to work: the answer is yes, but only for signatures they perform with SHA-256.  This is because the risk exposure is around performing signatures over a hash function where two certificate inputs can be readily found that  hash to the same value.

Takeaways

To hopefully have a more relaxing New Year's Eve, it is suggested to:

  • Make sure you have a leap second plan: either performing complete testing on all software, systems, and infrastructure, or implementing a leap smear.

  • Make sure you have all sites on Akamai, internally, and elsewhere rotated over to using SHA-256 certificates before getting any further into end-of-year lock-downs.

  • Make sure applications or devices relying on Akamai's shared certificate can handle SHA-256 certificates when Akamai drops SHA-1 support.

It may be a very Western-centric perspective, but I'd also plead with folks to pick days for leap seconds and other major infrastructure change milestones that aren't on a New Year's Eve.

Here is to a Happy New Year's and wonderful 2017!

About the author: Erik Nygren is a Fellow and Chief Architect in the Akamai Platform. He spent New Year's Eve on December 31, 1999 in the Akamai NOCC just in case we'd missed something in our Y2K planning.

3 Comments

Are you smearing during the 24 hours before the leap second, the 24 hours after the leap second, or the 24 hours centered on the leap second? If you haven't decided yet, I recommend smearing on the 24 hours centered on the leap second, since that way your clock will differ from UTC by not more than 1/2 second.

Do you have a publicly accessible link to your article about the June 2015 leap second? I created an account on community.akamai.com but when I follow the link it still says:

Access to this place or content is restricted. If you think this is a mistake, please contact your administrator or the person who directed you here.

As the link above about the leap second in 2015 requires a login to access, here are its contents:

Akamai's Approach for June 30, 2015 Leap Second Event

Posted Jun 3, 2015

Background

The International Earth Rotation and Reference Systems Service (IERS) has announced that a leap second will be introduced on June 30, 2015. Leap seconds correct for irregularities in the earth's rotation, allowing timekeeping systems to avoid diverging from solar mean time (http://en.wikipedia.org/wiki/Leap_second).

On the day of a leap second, Internet time servers notify their clients of the upcoming event. In response, client machines adjust their clocks at the end of the day by inserting an extra second. This effectively stops time for one second, causing the final second of the day to last for two seconds. When this happens, any software that relies on fine-grained time information can be disrupted. As the industry has seen in the past, without proper planning/execution, leap second events can be disruptive in a variety of ways.

Akamai's Approach

To ensure proper operation of Akamai systems and services during the June 30 leap second event, Akamai intends to employ an approach known as a 'leap smear'. This approach was pioneered by Google engineers to ensure seamless operation of their systems during previous leap second events in 2008 and 2012 (http://googleblog.blogspot.com/2011/09/@time-technology-and-leaping-seconds@.html)

Rather than inserting an extra second at the end of the day, the leap smear approach applies the leap second gradually over the course of a day. This is accomplished by slightly increasing the duration of each second on the day of the leap event. Because time drifts slowly rather than stopping abruptly at the end of the day, software is unaware of the adjustment and continues to function normally throughout the event.

A positive side effect of the smear approach is that it will not (noticeably) interfere with response time/performance measures taken within the Akamai platform.

Questions?

If you have any questions, please comment below. Alternatively, please feel free to reach out to your Akamai Representative and/or Akamai Customer Care.

(Special credit/thanks to Jack Wadden, one of Akamai's Platform Infrastructure Engineering Manager's, for help in crafting this message)

Leave a comment