Akamai Diversity

The Akamai Blog

Situational Performance Measurement with a Splash of RUM

Long gone are the days when you could gut-check website performance by running a straightforward synthetic backbone test on a simulated browser. The web has evolved, and the old ways of measuring performance do not line up with how end users interact with today's website.

The web -- and how websites perform in the real world -- are directly related to the various ways users interact with them. I covered this concept of situational performance and why it's important in an earlier post, "Web Performance: Why One size Doesn't fit all." A modern website has to work well on new browsers, old browsers, powerful desktop machines with great connectivity, and a long tail of mobile devices with a broad range of bandwidth situations.

If you care about performance, you have to measure it. If you want to measure it right, you absolutely must keep at least some of the different performance situations you expect to see in mind. Playing this out a bit, your once simple synthetic testing approach just got complex.

Flashlights and floodlights
A good synthetic monitoring solution is like a good flashlight: it gives you visibility to what you're pointing it at and that's just about it. The implication here is that if you want to know how your site performs for IE 9 users on a DSL connection, you can spin up a test and find out. Want to know your site performs for IE 9 users on a cable connection? That's a different test. IE 8? Different test. Chrome? Yup, more tests.

Modern synthetic testing tools like Keynote and Gomez have gotten very good at simulating the end user experience by measuring to the on load event with modern browsers on last mile networks. Despite these advances, the nature of synthetic testing means that monitoring even a small number of situations can quickly get out of control.

Following through with this analogy, if a synthetic monitoring solution is a flashlight, Real User Measurement (RUM for short) is a floodlight. With RUM, you can see how all of your users experience your content. Rather than having to pinpoint your important performance situations (and that's an entirely different can of worms on its own), you can aggregate across all of them and get a good sense of what real users are experiencing.

It is important to stress that RUM is not a replacement for synthetic testing, however. The two can, and should, be used side-by-side. Synthetic testing is key for identifying availability issues, performance issues like a slow third-party, or origin side issues like a misbehaving server.

Pairing RUM with Synthetic Testing in the Real World
Using both a synthetic testing solution and RUM technology under development at Akamai, we measured homepage load times for one of our customers. Synthetic testing gave us an easy to read median page load time graph:

Image 1.png

Things look good from here. Load times were consistently under 2.0 seconds. Given that this data is from last mile agents running a modern browser, this is phenomenal performance. Time to take a bow, our job here is done.

Or is it?
Looking at the median alone does not give any indication as to the distribution of the data. For that, we need to take a look at a load time histogram:

Image 2.png

This is a very compact, healthy distribution. Even so, it shows us something we didn't see before: load times slower than 3.0 seconds. About 21% of requests took longer than 3.0 seconds. This doesn't negate our impressive median load time, but at least now we know what the distribution looks like and what proportion of page loads were relatively slow.

Maybe now we're ready for a victory lap?

But wait, there's more!
Our synthetic test requests came from agents running IE 9 and using DSL connections. Although IE 9 currently wears the browser plurality crown (see below), IE 9 users on DSL connections only represent a small slice of the overall user experience pie. A representative slice, maybe, but still only a fraction of the overall user base.

Image 3.png

What about Chrome? Firefox? Faster connections? Slower ones? It's a jungle out there, and accounting for the long tail of situations is non-trivial.

What does this actually mean for performance measurement? During this test period we also collected RUM data from real users hitting the site. Taking a look at the same data and contrasting synthetic requests with real user requests gives us a slightly different picture:

Image 4.png

Looking at real user requests, 46% of page load times came in at over 3.0 seconds. Page loads over 3.0 seconds, which are not evident on the median load time report we started with, represented a full half of the aggregated end user experiences for this particular page.

Putting it all together, we can draw several conclusions:

  • Averages alone do not show the distribution of experiences or the proportion of users getting suboptimal performance
  • Synthetic monitoring is great but only takes a limited section of the user base into account

  • The synthetic modeling of IE 9 on DSL was faster than the overall set of aggregated RUM data for this particular customer, giving us skewed picture of page load time if we look at synthetic testing alone

Why you should care
Time is money. Reducing page load time drives revenue (see studies from JupiterResearch, Forrester Consulting, and Walmart).

To get a good understanding of how your site actually performs, add RUM to your performance measurement strategy. For the DIY inclined, Boomerang offers a solid, free option, but there are other solutions out there as well.

To understand how much performance matters, you'll want to correlate web performance with your business metrics. How does page load time affect conversion rate? What is the influence of page load time on bounce rate? If you use Google Analytics, for example, take a look at the Site Speed report to get a feel for the impact of performance on your business metrics.

Wrapping up
The web has evolved. Performance for modern websites is hard to measure effectively with the same techniques we relied on five years ago. Fortunately, web performance measurement has kept pace. Modern synthetic measurement tools do a great job of simulating the user experience, and RUM solutions can effectively measure performance across a highly fragmented, situational landscape.

If you care about website performance, measure it right.

Assaf Kremer is a product manager in the Web Experience Business Unit at Akamai


Assaf, great insights and solid advice. Site owners looking to understand how performance impacts their business definitely need to invest attention and effort to analyzing RUM data. I think the tipping point is nearly here, with real user performance having become almost universally attainable. The challenge will be developing the data structures, analytic conventions and methodologies to make it easy for customers to understand that data and apply it to operational decisions and continuous improvement. More work needed from the vendor community here!

One quick observation: I think the chart published here supporting your real user observations may not be the intended graphic.

Very interesting data and analogies. May I ask what solution was used to measure the synthetic data for your client?

Great catch! We have updated the posting with the correct image.

I completely agree with you, the challenging (and interesting) part of RUM is interpreting the data.

I'd be happy to have a conversation with you offline about that, please reach out to me at akremer@akamai.com.

we at rumanalytics are working on something interesting in this space.