June 2013 Site Performance Report

Posted by Jonathan Klein | Filed under performance

The second quarter of 2013 has come to a close, and it’s time for another site performance report. In our last update we had modest improvements across the board from code changes and removing a module from the homepage. The story this time is the huge win we saw from upgrading to PHP 5.4.

Server Side Performance:

Here are the median and 95th percentile load times for signed in users on our core pages on Wednesday, June 26th:

Server Side Performance

As you can see, we had a significant drop in load time across all pages, at both the median and the 95th percentile.  As I mentioned above, this was primarily due to upgrading all of our webservers to PHP 5.4.  One of the core enhancements in PHP 5.4 was improved performance and reduced memory consumption, and this really worked out well for us at Etsy:

PHP 5.4 Upgrade

In addition to improved performance, the reduced memory usage in PHP 5.4 allows each of our servers to handle more traffic, so we gained capacity while serving every request more quickly – not a bad deal for a software upgrade. To be clear, this wasn’t a trivial process; it took us months to make sure that we could safely upgrade without breaking anything, and to roll out the upgrade on a gradual basis, fixing things along the way. Our engineers put in a huge amount of effort to get this done, and it paid off in a big way. The transition went flawlessly, and now we get to reap the benefits. If you are a PHP shop and you aren’t on 5.4, I strongly recommend starting the upgrade process.

The other big improvement we had on the server side was due to a change in the way we load translation files. Etsy has buyers and sellers in many countries, and the site is currently available in 9 different languages. Until recently we were loading our translations from JSON files on every request, which could take up to 100ms for the larger languages. By changing these to PHP arrays and caching them with OPcache, this time dropped to ~20ms. This improvement only shows up in the 95th percentile numbers, since the majority of our traffic is in US english which doesn’t have this overhead.

Front-end Performance:

As usual, we are using our private instance of WebPagetest to get synthetic measurements of front-end load time. We use a DSL connection and test with IE8, IE9, Firefox, and Chrome. Here is the data:

WebPagetest Performance

Overall, the time to start render is pretty stable, with a decent decrease on the Shop page. This is largely due to a change we made to move all of our JavaScript to the bottom of the page. Even though this is one of the core rules for fast websites, like many websites we had some bootstrap files in the <head>, and some additional JavaScript sprinkled throughout the document. We moved everything to the bottom of the page as a test, and we saw an impressive improvement in start render time. We are now in the process of rolling this out to all of our other pages.

You will also notice that the document complete times jumped on the listing page and the shop page. For the shop page, we think this is also because we moved the JavaScript. In IE8 specifically we saw a large jump in document complete time after this change. We are thinking about how best to accomodate those users given the diminishing market share of that browser version, but luckily this problem does not appear in later versions of IE. For the listing page, we are rolling out a new version of it that is currently undergoing testing, and we are actively working on tuning its performance. Hopefully that number will come down significantly for the next report.

We are also introducing Real User Monitoring (RUM) data to this report on the front-end.  This data is collected with LogNormal/mPulse, and looks at the latest version of Chrome only, since it implements the Navigation Timing spec and is our most popular browser.  Median page load time in seconds is below (baseline is excluded because real users don’t hit that page):

mPulse Performance

The numbers are pretty consistent from six months ago, except for a slight improvement on the homepage and degradation on the search results page. The improvement is expected is part because of the server side improvements we saw above, and the slowdown on search is likely due to front-end changes on that page (new JS files, another row of search ads, etc.). We haven’t really had front-end performance in our sights over the past quarter, so we’re happy to see that things have remained fairly stable.

Conclusion:

Once again the news in this report is predominantly good. Server side times continue to get better without much work from the performance team – we simply sit back and let hardware and software upgrades do our work for us :-). Front-end performance is pretty stable across our major pages, even though we haven’t spent a ton of time working on them. Our JS deferral strategies are paying off in the time to start render metrics, which is a huge benefit to all of our users. Over the next quarter we are going to continue to focus on improving our slowest pages (typically not pages that appear on this report) and look into bringing down the document complete time for the Listing page. We’re also kicking off a project to investigate how the use of progressive JPEGs impacts customer behavior on Etsy.com. We’ll post the results of that study on CodeAsCraft once we have them!

You can follow Jonathan on Twitter at @jonathanklein


9 responses to June 2013 Site Performance Report

  • jsnrkd says:

    Are you guys using the Apache LogFormat directive to get your server side numbers and then crunching with Splunk? If so, could you share the LogFormat options you’ve chosen?

    • Yup, that’s exactly right. Here is our current log format:

      LogFormat “%{X-Forwarded-For}i %{True-Client-IP}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %{etsy_shop_id}n %{PHPSESSID}C %{etsy_uaid}n %V %{etsy_ab_selections}n %{etsy_request_uuid}n %{etsy_user_id}n %{etsy_api_consumer_key}n %{etsy_api_method_name}n %{php_memory_usage_bytes}n %{php_time_microsec}n %D %{X-Secure}i %{etsy_request_locale}n %{etsy_page_type}n %{display_mode}n %{X-CDN-Provider}i %{php_utime_microsec}n %{php_stime_microsec}n” combined

  • tgabi333 says:

    Nice backend improvements. What about php5.5? Switching from 5.4 to 5.5 will be much easier than 5.3 to 5.4.

    • We don’t have any plans to move to PHP 5.5 yet, although I’m sure we will start thinking about it in a few months. The performance improvement from 5.4 –> 5.5 is much more modest than the one from 5.3 –> 5.4.

  • Andrew says:

    Very nice performance improvements here. You mention incorporating some RUM data next time, Cedexis provides great free RUM performance measurements. Let me know if you have a minute to chat.

  • Thomas Genin says:

    Hi,
    Thank you for the post.
    What were the major pain points during the upgrade of PHP?

    • Dan Rowe says:

      One of the pain points for us was actually less about php 5.4 itself and more about the amount of different server roles running php in our environment. It took time and effort to validate the upgrade path for every php extension + php configuration combination. For roles like our primary web servers, we got a large number of servers upgraded quickly, but for the roles that we only have one or two servers in, it took just as much time.

  • visseraj says:

    What process did you use to upgrade to PHP 5.4? Compile it your selves or use existing packages?

  • Leave a Response

    Recent Posts

    About

    Etsy At Etsy, our mission is to enable people to make a living making things, and to reconnect makers with buyers. The engineers who make Etsy make our living with a craft we love: software. This is where we'll write about our craft and our collective experience building and running the world's most vibrant handmade marketplace.

    Code as Craft is proudly powered by WordPress.com VIP and the SubtleFlux theme.

    © Copyright 2014 Etsy