Stats. How geeky. And boring. I hardly remember anything from basic elementary math, let alone anything to do with statistical analysis at a college / university level. But it turns out that statistics can be incredibly useful in describing what's happening across a population, such as when you use Real User Monitoring (RUM) to get performance metrics (such as load times) across your user base.
Why is this important? Glad you asked. There is well-accepted evidence from some of the largest e-tailers including Amazon and Wal-Mart on the effect of conversions and sales revenues with a site's performance. Most of this data centers around median scores (in other words, the "average" performance of a page), where an increase (or decrease) in a page's median load times correlate to measurable changes in conversion rates. The reason why we use median instead of mean is because median is much more resistant to outliers that may markedly affect the "average" number, and hence distort what we perceive as the typical experience when visiting a page.
However, this is not the whole truth. If the median page load times decrease from 4 seconds to 3.5 seconds, the conversions will increase. But are these the users who you're making the most impact? Those 3.5 seconds users are impacted, it can also be that the users who are experiencing 10 or 12 seconds load times may now be experiencing 6 or 7 seconds, and these users will drive larger conversions and revenues beyond the users who are already performing well. If you can move your 60th, 70th and 80th percentile scores tighter around the median, you will inevitably increase the performance for users who matter most, and that's where the money lies, so to speak. So just as important as measuring the median or "typical" experience, we want to measure how consistently a visitor will get that experience.
Enter descriptive statistics.
Looking at consistency in performance in a RUM distribution
Variance: This is typically measured using Standard Deviation. The higher the value, the more inconsistent the metric is and that's bad for #WebPerf. You will want to drive this number lower whenever possible.
The problem with just using Standard Deviation is that it doesn't describe the whole story, since it is more relevant in a Gaussian (or normal) distribution, which is symmetrical. Under those conditions, variability is fairly constant across the population. Web performance distributions however, are inherently skewed and asymmetrical, so we need additional statistics to help quantify consistency.
Skewness: The skewness helps describe whether variability (consistency) increases or decreases over a population. A skewness with a positive number means you should see the "fat part" in the graph towards the left (or skewed right), and the mean is greater than the median. A negative number is obviously the reverse of the previous statement. When applied to RUM performance distributions, a positive skewness factor occurs when there is a greater frequency of data points with users experiencing shorter (and therefore better) timings in a tight range. Web performance distributions using RUM should typically be positively skewed, and the higher the number the better.
Kurtosis: Kurtosis tells you how "flat" or "spikey" a distribution is relative to a normal distribution. When it is "flatter" than a normal distribution, this number is negative, while a positive number means it is more "spikey". More precisely, the lower the number, the more the weight in the "tail" of the distribution away from the mean. From a #WebPerf standpoint, a higher number calculated for a RUM distribution means less data points at the fringes, which again generally means less variability and greater consistency. You therefore want your kurtosis to be generally a positive number, and the higher the better.
At Akamai, we evangelize the axiom that you can improve only what you measure. These metrics can greatly help organizations to understand more precisely the performance of a web page, and should be an integral part to addressing situational performance. In a follow up post, I will cover how to quickly export a frequency distribution table of RUM statistics from Akamai's LUNA portal, and then use Excel to calculate descriptive statistics in a matter of minutes.
Desmond Tam is a senior enterprise architect at Akamai.