Now that Cyber Monday has passed, we've left the most daunting of the 2012 traffic peaks behind us. US retailers were largely stable and available this year, with few catastrophic site failures. In general, retailers are much more mature with their readiness planning, starting earlier, and conducting load tests and simulations right before peak.
For those of you tracking our Net Retail Traffic Index this past weekend, you may have noticed an odd spike of 11.5 million page views on the Sunday before Thanksgiving. A few savvy observers asked the very logical question - what's that spike? The biggest online retailers all engaged in "best case" load testing this past weekend.
Load testing is our number one recommendation for being prepared - you have to know your vulnerabilities. While this may seem obvious to many of you, it is surprising to hear the number of retailers who still attempt to squeak by without load testing before the holidays or peak events.
An Interview with Akamai's Load Testing Expert
Who better to speak with about Load Testing then Akamai's Load Testing expert, Paul Korenevsky. Fortunately Paul spoke with me last week, before he spent the entire weekend sleepless, onsite with his customer, watching the traffic and orders fly in. This is Paul (looking a lot more awake than he is this week).
[Lelah] Paul thanks for talking with me today as I know it's a particularly crazy time of year. Can you share what you do for Akamai?
[Paul] My primary responsibility is an Engagement Manager, which makes me the owner of the professional services team for our premium clients. My secondary responsibility is a SME [subject matter expert] for load testing best practices. We have defined such practices for load testing providers SOASTA and Keynote, and I am working to optimize those practices and expand the list of providers.
[Lelah] When a retailer runs load test, what are their primary goals?
[Paul] There are two goals. First they are trying to identify potential bottlenecks early. They will stress tests their obvious bottlenecks to understand how they will perform under load during their holiday peaks, and to tune where possible. Secondly they want to evaluate the performance degradation under load. The reality is that most retailers historically only got through to #1 but more and more they are starting to get to #2.
[Lelah] Which load testing providers does Akamai most commonly work with? How are they optimized to work with the Akamai cloud?
Most of the larger ecommerce companies offload not just static content but application processing onto Akamai. If they load test direct to origin, without Akamai, the traffic will not represent the load under real conditions.
Akamai initially didn't allow our customers to conduct load testing on our platform, without express consent of the Akamai Performance team, because of potential adverse impact to their end users, or to other customers. Before we formalized the load testing framework with SOASTA and Keynote, the load tests would require a good amount of preparation and each test would be subject to approval by the performance team. We ensure load testing providers are not firing all traffic from one or a limited number of locations to represent accurate traffic patterns from a wide variety of geographies. This ensure "real to life" caching and offload metrics, and performance measurement with Akamai's ability to route around internet latency.
SOASTA more recently has become prevalent in many of our larger retail customers. Last year we worked closely with them to make sure we trained their team and vice versa. They even made changes to their platform to ensure they could work well with us. The Akamai Performance team then signed off on a framework that is "pre-approved" for working with SOASTA and Akamai. The framework ensures full visibility with monitoring and reports on the Akamai side.
[Lelah] The click stream that is tested is essential for a valid load test. How do retailers identify the clickstreams through their site? How many are tested and how often?
[Paul] Retailers typically leverage analytics tools to identify common paths. They also identify high-risk problem areas based on data from previous performance monitoring tests. For example, features of checkout or checking out directly from a wish list. They then take these common and high-risk paths and then schedule them for load tests.
Pre-holiday they do testing with each release in lower environments. They start production testing in August and do it regularly up until holiday. Those who do it regularly and properly end up conducting between 8 and 20 tests.
The intention is to not take down the site. Scheduling the tests during consistently low traffic times is least likely to impact their end users. They include the real user traffic numbers in their load testing generator targets.
[Lelah] How do retailers set those load testing targets?
[Paul] Usually they take Thanksgiving night, Black Friday morning or Cyber Monday best case projections from the business. This year retailers were testing with the expectation of 40 - 60% traffic growth at peak. If they can exceed forecasted load then they stay up for the holidays. The hard part is sustaining traffic without performance degradation. The page performance targets under load are driven by their business team. They have alerts that fire if performance drops below a certain level.
[Lelah] What are the most common vulnerabilities found through load testing?
Normally its database and application server issues. We'll then work with them to see if we can provide offload for those vulnerable areas on Akamai. It also helps them determine how well their load-balancing infrastructure is working. We'll double check that their disaster recovery center (or other failover scenario) is operating appropriately. Often times a SPA [Shopper Prioritization Application - Akamai's traffic throttling solution] test is paired with a load test to evaluate when SPA should be used and how they can maintain optimal load and performance in their environment.
Retailers considering load tests through Akamai should make sure to contact their account rep to see if their load tests comfort with the pre-approved framework.