September 2013 Site Performance Report

Posted by on October 14, 2013

As we enter the fourth quarter of 2013, it’s time for another site performance report about how we did in Q3. Our last report highlighted the big performance boost we saw from upgrading to PHP 5.4, and this report will examine a general front-end slowdown that we saw over the last few months.

Server Side Performance

Here are the median and 95th percentile load times for signed in users on our core pages on Wednesday, September 18th:

On the server side we saw a modest decrease on most pages, with some pages (e.g. the profile page) seeing a slight increase in load time. As we have mentioned in past reports, we are not overly worried about the performance of our core pages, so the main thing we are looking for here is to avoid a regression. We managed to achieve this goal, and bought ourselves a little extra time on a few pages through some minor code changes. This section isn’t very exciting, but in this case no news is good news.

Synthetic Front-end Performance

The news here is a mixed bag. As usual, we are using our private instance of WebPagetest to get synthetic measurements of front-end load time. We use a DSL connection and test with IE8, IE9, Firefox, and Chrome. Here is the data, and all of the numbers are medians over a 24 hour period:

On the plus side, we saw a pretty significant decrease in start render time almost across the board, but document complete time increased everywhere, and increased dramatically on the listing page. Both of these global effects can be explained by rolling out deferred JavaScript everywhere – something we mentioned back in our June report. At that time we had only done it on the shop page, and since then we have put it on all pages by default. This explains the decrease in start render time on all pages except for the shop page. The profile page also had a slight uptick in start render time, and we are planning on investigating that.

One of the side effects of deferring JavaScript is that document complete time tends to rise, especially in IE8. This is an acceptable tradeoff for us, since we care more about optimizing start render, and getting content in front of the user as quickly as possible. We’re also not convinced that a rise in document complete time will have a negative impact on business metrics, and we are running tests to figure that out now.

The massive increase in document complete time on the listing page is due to the rollout of a page redesign, which is much heavier and includes a large number of web fonts. We are currently setting up a test to measure the impact of web fonts on customer engagement, and looking for ways to reduce page weight on the listing page. While document complete isn’t a perfect metric, 8 seconds is extremely high, so this bears looking into. That said, we A/B tested engagement on the old page and the new, and all of the business metrics we monitor are dramatically better with the new version of the listing page. This puts further doubt on the impact of document complete on customer behavior, and illustrates that performance is not the only thing influencing engagement – design and usability obviously play a big role.

Real User Front-end Performance

These numbers come from mPulse, and are measured via JavaScript running in real users’ browsers:

The effect here mirrors what we saw on the synthetic side – a general upward trend, with a larger spike on the listing page. These numbers are for the “page load” event in mPulse, which is effectively the onload event. As Steve Souders and others have pointed out, onload is not a great metric, so we are looking for better numbers to measure on the real user side of things. Unfortunately there isn’t a clear replacement at this point, so we are stuck with onload for now.

Conclusion

Things continue to look good on the server side, but we are slipping on the front-end. Partly this has to do with imperfect measurement tools, and partly it has to do with an upward trend in page weight that is occurring all across the web – and Etsy is no exception. Retina images, web fonts, responsive CSS, new JavaScript libraries, and every other byte of content that we serve continue to provide challenges for the performance team. As we continue to get hard data on how much page weight impacts performance (at least on mobile), we can make a more compelling case for justifying every byte that we serve.

You can follow Jonathan on Twitter at @jonathanklein

Posted by on October 14, 2013
Category: performance Tags: ,

Related Posts

3 Comments

Hey Jonathan,

you mentioned abiut retina image. Do you guys have any experiment result with WebP so far to mitigate the image size? If not, what is the bottleneck that prevents you from using it? The only problem we have encountered so far is due to our CDN provider that couldn’t cache different version of image based on header Vary.

Huan

    We don’t currently have any data of our own on WebP. There are two main blockers for us – Firefox support, and the fact that we have a giant working set of images, and we would need to converted hundreds of millions of images to WebP if we wanted to use it site wide. That said, we are actively building and testing a service that will be able to do this on the fly, so we will have more flexibility with image formats in the future.

Not related to the blog post but I noticed a typo in the blog template under the Related Posts section where it should be “posted by” instead of “psted by” (and since this typo is in the template, it shows up on all blog posts). Sorry, couldn’t figure out the right person notify 🙂