Q1 2014 Site Performance Report

Posted by on May 15, 2014

May flowers are blooming, and we’re bringing you the Q1 2014 Site Performance Report. There are two significant changes in this report: the synthetic numbers are from Catchpoint instead of WebPagetest, and we’re going to start labeling our reports by quarter instead of by month going forward.

The backend numbers for this report follow the trend from December 2013 – performance is slightly up across the board. The front-end numbers are slightly up as well, primarily due to experiments and redesigns. Let’s dive into the data!

Server Side Performance

Here are the median and 95th percentile load times for signed in users on our core pages on Wednesday, April 23rd:

Server Side Performance

There was a small increase in both median and 95th percentile load times over the last three months across the board, with a larger jump on the homepage. We are currently running a few experiments on the homepage, one of which is significantly slower than other variants, which is bringing up the 95th percentile. While we understand that this may skew test results, we want to get preliminary results from the experiment before we spend engineering effort on optimizing this variant.

As for the small increases everywhere else, this has been a pattern over the last six months, and is largely due to new features adding a few milliseconds here and there, increased usage from other countries (translating the site has a performance cost), and overall added load on our infrastructure.  We expect to see a slow increase in load time for some period of time, followed by a significant dip as we upgrade or revamp pieces of our infrastructure that are suffering. As long as the increases aren’t massive this is a healthy oscillation, and optimizes for time spent on engineering tasks.

Synthetic Front-end Performance

Because of some implementation details with our private WebPagetest instance, the data we have for Q1 isn’t consistent and clean enough to provide a true comparison between the last report and this one.  The good news is that we also use Catchpoint to collect synthetic data, and we have data going back to well before the last report.  This enabled us to pull the data from mid-December and compare it to data from April, on the same days that we pulled the server side and RUM data.

Our Catchpoint tests are run with IE9 only, and they run from New York, London, Chicago, Seattle, and Miami every two hours.  The “Webpage Response” metric is defined as the time it took from the request being issued to receiving the last byte of the final element on the page.  Here is that data:

Synthetic Performance - Catchpoint

The increase on the homepage is somewhat expected due to the experiments we are running and the increase in the backend time. The search page also saw a large increase both Start Render and Webpage Response, but we are currently testing a completely revamped search results page, so this is also expected.  The listing page also had a modest jump in start render time, and again this is due to differences in experiments that were running in December vs. April.

Real User Front-end Performance

As always, these numbers come from mPulse, and are measured via JavaScript running in real users’ browsers:

Real User Monitoring

No big surprises here, we see the same bump on the homepage and the search results page that we did in the server side and synthetic numbers. Everything else is essentially neutral, and isn’t particularly exciting. In future reports we are going to consider breaking this data out by region, by mobile vs. desktop, or perhaps providing other percentiles outside of the median (which is the 50th percentile).

Conclusion

We are definitely in a stage of increasing backend load time and front-end volatility due to experiments and general feature development. The performance team has been spending the past few months focusing on some internal tools that we hope to open source soon, as well as running a number of experiments ourselves to try to find some large perf wins. You will be hearing more about these efforts in the coming months, and hopefully some of them will influence future performance reports!

Finally, our whole team will be at Velocity Santa Clara this coming June, and Lara, Seth, and I are all giving talks.  Feel free to stop us in the hallways and say hi!

Posted by on May 15, 2014
Category: performance Tags: ,

Related Posts

3 Comments

As always, thanks very much for your openness and thought leadership on WebPerf.

I’m curious, do you get some utility from CatchPoint that you don’t get from WebPageTest, or did you just switch to the CatchPoint dataset due to a problem with your WebPageTest instance?

I have most of my clients on a protocol with WPT (Synthetic) and Sosta (RUM), but have been interested in trying out Catchpoints Synthetic and RUM solutions.

Cheers,

-Jason

    Hi Jason,

    For the purposes of this report the two are fairly interchangeable, but when drilling into the data and trying to figure out what changed I think Catchpoint is a little better. It has built in graphs that you can segment by browser, location, page, etc., and it’s very easy to change what metric you are looking at. This is something we layered on top of WebPagetest with wpt-script and Splunk/Graphite, but having it integrated into the Catchpoint product (with one click to see waterfalls) is helpful.

    The other more meaningful difference is consistency. Given that our WebPagetest agents are small or medium instances on EC2, we see a decent amount of fluctuation in their performance. Catchpoint’s agents are much more stable over time, and thus give us more confidence that the trends shown in Catchpoint are accurate.

    Hopefully this helps!

    -Jonathan

I’ve been reading these for some time now and they’re always great so thanks – great to see you sharing this stuff publicly for the greater good. Also good luck with the new book – it’s on the reading list.

I just have a question on the first section about server-side perf. Do you find that looking at your web-logs ever comes up with a surprising result and is that the point at which you then either debug ad-hoc or are you using a classic APM tool for code-level performance? For example when you say this:

“…is largely due to new features adding a few milliseconds here and there, increased usage from other countries (translating the site has a performance cost)”

How are you digging in and getting that confidence? Are you using or have you considered supplementing your efforts with what Gartner would call “Deep Dive Component Monitoring”?

-Rich