The past few years have seen a dramatic increase in client support for TLS SNI (a technology standard that makes HTTPS much more scaleable). While early 2014 saw fewer than 85% of HTTPS requests being sent by clients supporting TLS SNI, many Akamai customers today now see client TLS SNI usage exceeding 99%. This shift means that deploying SNI-only Web sites is now increasingly viable, with 31% of the Alexa top-100k hostnames with valid certificates for HTTPS only presenting those certificates when TLS SNI is sent by clients.
Historically, there have been two major scaling challenges with moving all Web traffic to HTTPS: the availability of server certificates and the technical issues facing TLS virtual hosting. Readily available automated Domain Validation (DV) certificates through providers such as Let's Encrypt are helping significantly with the former. TLS Server Name Indication (TLS SNI) is the most scalable solution for the latter, but until recently there was no way to rely on it without breaking a significant fraction of clients. This recent significant increase in client support means that the near universal use of TLS SNI may be within reach. The remaining set of clients that do not send TLS SNI will likely need to deploy fixes over the coming year or so if they wish to continue accessing all Web sites, especially with an increasing number of sites switching to HTTPS (now over half of page loads per Firefox telemetry).
For background, the Transport Layer Security (TLS) protocol underlies HTTPS and provides for authentication and encryption. When Netscape's SSLv3 protocol and its IETF-standardized successor TLS 1.0 were first developed, they lacked support for virtual hosting: when a "ClientHello" message is sent to a server, the client expects to receive a ServerHello in return with a security certificate identifying the server by name. If the name in the certificate doesn't match the name the client expects, a properly behaved client will fail to make a connection and return an error. In the case of virtual hosting where a single server handles tens, thousands, or millions of independent web sites, the server needs a way to know which certificate to return. Without any host name in the ClientHello, servers looking to serve HTTPS on port 443 have no option but to use distinct IP addresses for each certificate. For CDNs such as Akamai deploying in hundreds or thousands of locations, this can mean requiring hundreds or thousands of Virtual IP (VIP) addresses per certificate. With the exhaustion of freely available IPv4 addresses, this quickly becomes a fundamental scaling problem.
In 2003, the TLS Server Name Indication (TLS SNI) extension was introduced (in RFC 3546). This allows clients to include the name of the host they are trying to contact (the "server name") in the ClientHello message, allowing the server to select and respond with the proper certificate in the ServerHello. The SNI extension is thus routing information that enables virtual hosting, similar to the HTTP Host header that has been mandated ever since HTTP/1.1 in 1997. Load balancers in a data center are even able to use TLS SNI to steer traffic arriving on a shared IP address to the proper backend server.
IPv6 and HTTP/2 do have features to help, but they are not solutions by themselves. The HTTP/2 specification requires that clients send TLS SNI. Scaling VIPs isn't a problem for IPv6 because of the virtually unlimited address space, and Akamai made IPv6+IPv4 dual-stack the default for new customer configurations in 2016. However, many of the same clients that don't send TLS SNI also don't implement HTTP/2 or only have IPv4 connectivity.
This lack of universal adoption by clients constrained the practical use of TLS SNI. With widely deployed operating systems such as Windows XP and Android 2.2 not sending TLS SNI, using "SNI-only" HTTPS was breaking support for significant numbers of clients. In 2013, Akamai saw only 80% of HTTPS clients sending TLS SNI. Even as recently as April 2015, Akamai saw only 90% of HTTPS requests come from clients which sent TLS SNI. The decline in clients not sending TLS SNI is partially due to other changes in the TLS ecosystem also breaking those clients, such as the end-of-support of SHA-1 certificates.
This landscape has changed dramatically over the past two years. On average, we now see 98% of HTTPS requests come from clients that send TLS SNI. Since a large portion of non-SNI clients are custom applications used by individual customers, the median customer certificate configuration ("slot") sees around 99% of HTTPS requests from clients sending TLS SNI. Over 12% of customer slots see TLS SNI usage exceeding 99.9%, even when excluding SNI-only slots.
With the exception of China (which has significantly lower TLS SNI usage) and the United States (where many bots and agents not sending TLS SNI are deployed), client support of TLS SNI is consistently strong around the world. Excluding the US and China, TLS SNI usage exceeds 99.4% for the median Akamai customer slot when looking at client traffic for most continents and large countries. Even the U.S. still sees TLS SNI usage for the median customer slot around 98.8%.
Looking at the remaining User-Agents not sending TLS SNI to Akamai, they appear to fall into the following categories, almost all of which represent single-digit percentages of the remaining non-SNI clients:
Modern clients that are likely behind a TLS Man-in-the-Middle proxy (such as local parental control software or an enterprise traffic inspection appliance) which may be otherwise downgrading TLS connection security. US-CERT recently released this advisory on how poorly-implemented HTTPS Interception weakens TLS security and a recent paper with additional details, showing how some don't even check server certificate identity.
Remaining Windows XP (and even more ancient Windows) clients. These users are running a past-end-of-life operating system and many likely have security issues. It is even unclear how some of these still connect at all following the SHA-1 EOL, but it is possible that some of the remainder are also behind TLS Man-in-the-Middle proxies that do not check certificates.
Some bots and crawlers, including for some search engines. We have reached out to a number of those at the top of the list and some have plans to deploy TLS SNI support in the near future. It is likely that many of these also include a range of malicious and benign bots that are scanning the entire IPv4 address space for HTTPS servers.
Various clients and bots implemented on older WebKit forks, older versions of Java (version 6 and lower), older versions of curl, and older versions of python. Clients using newer versions of these libraries and languages may also have issues if they don't initialize TLS connections properly. These span the gamut from malware to headless browsers used for site performance analytics.
Custom applications (such as download managers, apps for some gaming consoles, some mobile clients). These tend to be isolated to individual customer sites.
While few Web sites were willing to switch to SNI-only and abandon 10% or 20% of their user-base, a much larger portion of sites may be willing to break 1% of their clients, especially if many of these are old or insecure clients, bots, or malware. With Let's Encrypt supporting more than 27 million active certificates in total, giving just a single IPv4 address to each certificate would require a massive 1.6 /8's worth of IPv4 address space (i.e., over 1/150 of all possible unicast IPv4 addresses). Akamai has been offering SNI-only HTTPS certificates as part of some products since late 2015 and some of our upcoming offerings will default to SNI-only HTTPS out-of-the-box with customers being able to add-on VIP-based certificates. Site operators are thus encouraged to future-proof by encouraging remaining clients within their ecosystem to implement TLS SNI.
The shift to SNI-only has already begun, especially among smaller sites. Today 75% of the Alexa top-1000 hostnames have valid HTTPS certificates, and of those roughly 11.5% are SNI-only (i.e., only presenting a valid HTTPS certificate when SNI is present). Of the Alexa top-100k hostnames, 55% have valid HTTPS certificates and 31% of those are SNI-only, although many of these are still available over HTTP as well.
As more sites switch to SNI-only, operators of the remaining non-SNI clients will continue to see increasing pressure to implement and deploy TLS SNI. This quickly becomes a virtuous circle, as a shrinking non-SNI client base will encourage increasing use of SNI-only by larger and larger sites over the coming years, especially as historically HTTP-only sites switch to HTTPS. As such, authors of client software such as bots, crawlers, API clients, and mobile apps should implement TLS SNI support as soon as possible if they haven't done so already to ensure they can continue to reach the increasing number of SNI-only HTTPS sites.
Some related links for further reading: