Atomic deploys at Etsy

Posted by rlerdorf | Filed under infrastructure

A key part of Continuous Integration is being able to deploy quickly, safely and with minimal impact to production traffic. Sites use various deploy automation tools like Capistrano, Fabric and a large number of homegrown rsync-based ones. At Etsy we use a tool we built and open-sourced called Deployinator.

What all these tools have in common is that they get files onto multiple servers and are able to run commands on those servers. What ends up varying a lot is what those commands are. Do you clear your caches, graceful your web server, prime your caches, or even stagger your deploys to groups of servers at a time and remove them from your load balancer while they are being updated? There are good reasons to do all of those things, but the less you have to do, the better. It should be possible to atomically update a running server without restarting it and without clearing any caches.

The problem with deploying new code to a running server is quite simple to understand. A request that starts on one version of the code might access other files during the request and if those files are updated to a new version during the request you end up with strange side effects. For example, in PHP you might autoload files on demand when instantiating objects and if the code has changed mid-request the caller and the implementation could easily get out of synch. At Etsy, it was quite normal to have to split significant code changes over 3 deploys before implementing atomic deploys. One deploy to push the new files. A second deploy to push the changes to make use of the new code and a final deploy to clean up any outdated code that isn’t needed anymore.

The problem is simple enough, and the solution is actually quite simple as well. We just need to make sure that we can run concurrent requests on two versions of our code. While the problem statement is simple, actually making it possible to run different versions of the code concurrently isn’t necessarily easy.

When trying to address this problem in the past, I’ve made use of PHP’s realpath cache and APC (a PHP opcode cache, which uses inodes as keys). During a deploy the realpath cache retains the inodes from the previous version, and the opcode cache would retain the actual code from the previous version. This means that requests that are currently in progress during a deploy can continue to use the previous version’s code as they finish. WePloy is an implementation of this approach which works quite well.

With PHP 5.5 there is a new opcode cache called Opcache (which is also available for PHP 5.2-5.4). This cache is not inode-based. It uses realpaths as the cache keys, so the inode trick isn’t going to work anymore. Relying on getting the inodes from a cache also isn’t a terribly robust way of handling the problem because there are still a couple of tiny race windows related to new processes with empty caches starting up at exactly the wrong time. It is also too PHP-oriented in that it relies on very specific PHP behaviour.

Instead of relying on PHP-specific caching, we took a new look at this problem and decided to push the main responsibility to the web server itself. The base characteristic of any atomic deploy mechanism is that existing requests need to continue executing as if nothing has changed. The new code should only be visible to new requests. In order to accomplish this in a generic manner we need two document roots that we toggle between and a new request needs to know which docroot it should use. We wrote a simple Apache module that calls realpath() on the configured document root. This allows us to make the document root a symlink which we can toggle between two directories. The Apache module sets the document root to this resolved path for the request so even if a deploy happens in the middle of the request and the symlink is changed to point at another directory the current request will not be affected. This avoids any tricky signaling or IPC mechanisms someone might otherwise use to inform the web server that it should switch its document root. Such mechanisms are also not request-aware so a server with multiple virtual hosts would really complicate such methods. By simply communicating the docroot change to Apache via a symlink swap we simplify this and also fit right into how existing deploy tools tend to work.

We called this new Apache module mod_realdoc.

If you look at the code closely you will see that we are hooking into Apache first thing in the post_read_request hook. This is run as soon as Apache is finished reading the request from the client. So, from this point on in the request, the document root will be set to the target of the symlink and not the symlink itself. Another thing you will notice is that the result of the realpath() is cached. You can control the stat frequency with the RealpathEvery Apache configuration directive. We have it set to 2s here.

Note that since we have two separate docroots and our opcode cache is realpath-based, we have to have enough space for two complete copies of our site in the cache. By having two docroots and alternating between them on successive deploys we reuse entries that haven’t changed across two deploys and avoid “thundering herd” cache issues on normal deploys.

If you understand things so far and have managed to compile and install mod_realdoc you should be able to simply deploy to a second directory and when the directory is fully populated just flip the docroot symlink to point to it. Don’t forget to flip the symlink atomically by creating a temporary one and renaming it with “mv -T“. Your deploys will now be atomic for simple PHP, CGI, static files and any other technology that makes use of the docroot as provided by Apache.

However, you will likely have a bit more work to do for more complex scenarios. You need to make sure that nothing during your request uses the absolute path to the document_root symlink. For example, if you configure Apache’s DOCUMENT_ROOT for your site to be /var/www/site/htdocs and then you have /var/www/site be a symlink to alternatingly /var/www/A and /var/www/B you need to check your code for any hardcoded instances of /var/www/site/htdocs. This includes your PHP include_path setting. One way of doing this is to set your include_path as the very first thing you do if you have a front controller in your application. You can use something like this:

ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include');

That means once mod_realdoc has resolved /var/www/site/htdocs to /var/www/A/htdocs your include_path will be /var/www/A/htdocs/../include for the remainder of this PHP request and even if the symlink is switched to /var/www/B halfway through the request it won’t be visible to this request.

At Etsy we don’t actually have a front controller where we could easily make this app-level ini_set() call, so we wrote a little PHP extension to do it for us. It is called incpath.

This extension is quite simple. It has three ini settings. incpath.docroot_sapi_list specifies which SAPIs should get the docroot from the SAPI itself. incpath.realpath_sapi_list lists the SAPIs which should do the realpath() call natively. When the extension does the realpath() itself it is essentially a PHP version of the mod_realpath module resolving the symlink in the extension itself. And finally, incpath.search_replace_pattern specifies the string to replace in the existing include_path. It is easier to understand with an example. At Etsy we have it configured something like this:

incpath.docroot_sapi_list = apache2handler
incpath.realpath_sapi_list = cli
incpath.search_replace_pattern = /var/www/site/htdocs

This means that when running PHP under Apache we will get the document root from Apache (apache2handler) and we will look for “/var/www/site/htdocs” in the include_path and replace it with the document root we got from Apache. For cli we will do the realpath() in the extension and use that to substitute into the include_path. Our PHP configuration then has the include_path set to:

/var/www/site/htdocs/../include:.

which the incpath extension will modify to be either /var/www/A/htdocs/../include or /var/www/B/htdocs/../include.

This include_path substitution is done in the RINIT PHP request hook which runs at the beginning of every request before any PHP code has run. The original include_path is restored at the end of the request in the RSHUTDOWN PHP hook. You can, of course, specify different search_replace_pattern values for different virtual hosts and everything should work fine. You can also skip this extension entirely and do it at the app-level or even through PHP’s auto_prepend functionality.

Some caveats. First and foremost this is about getting atomicity for a single web request. This will not address multi-request atomicity issues. For example, if you have a request that triggers AJAX requests back to the server, the initial request and the AJAX request may be handled by different versions of the code. It also doesn’t address changes to shared resources. If you change your database schema in some incompatible way such that the current and the new version of the code cannot run concurrently then this won’t help you. Any shared resources need to stay compatible across your deploy versions. For static assets this means you need proper asset versioning to guarantee that you aren’t accessing incompatible js/css/images.

If you are currently using Apache and a symlink-swapping deploy tool like Capistrano, then mod_realdoc should come in handy for you, and it is likely to let you remove a graceful restart from your deploy procedure. For non-Apache, like nginx, it shouldn’t be all that tricky to write a similar plugin which does the realpath() call and fixates the document root at the top of a request insulating that request from a mid-request symlink change. If you use this Apache module or write your own for another server, please let us know in the comments.


15 responses to Atomic deploys at Etsy

  • dynom says:

    I believe you can do this quite easily in Nginx also. By simply changing the configuration to a new document-root and reload Nginx with -HUP. This will start new workers with the new configuration, while the old workers (with the old configuration) slowly die when they finished serving their request.

    This would be quite easy to do with Capistrano, for example. I’ve not tried it, it’s just hypothetical.

    • rlerdorf says:

      Yes, I just tested this with nginx and php-fpm with Opcache. An nginx config reload will not reset the opcode cache so it looks like this approach might be even easier with nginx.

  • Peter Mescalchin says:

    I’m guessing you could also recreate the nginx/php-fpm behaviour above with Apache if required by swapping out mod_php for mod_fcgid and using php-fpm instead. Would also avoid blowing away your “warm” Opcode cache on change of doc-root.

  • Ivan Kurnosov says:

    Having a redundant and nice infrastructure (according to the posts in this blog) – why don’t you just roll changes to half of the servers and switch them on load balancer?

  • rlerdorf says:

    Peter, yes, that is probably true.

    Ivan, that is rather invasive though. It means removing half the servers from the production pool 30 to 40 times every day. The goal here was to be able to push safely to all the running production servers with minimal impact.

  • manu08 says:

    This is one thing I love about deploying in AWS. I can avoid this complexity by spinning up a new 40 servers behind a new lb, deploy the code, prime the cache, then swap DNS to point to the new lb. After a few sanity checks I can just throw the old environment away forever.

  • Daniel Schlenzig says:

    For nginx you do not even need a module/patch, you can simply use the (undocumented) variable $realpath_root in your configuration, e.g.:

    location ~ \.php$ {
    root ;
    fastcgi_pass …;
    fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
    fastcgi_param DOCUMENT_ROOT $realpath_root;

    }

    Just be sure to remove SCRIPT_FILENAME and DOCUMENT_ROOT from your default fastcgi_params configuration file. The only problem might by that the call to realpath() does not get cached.

  • Manu08,

    You do that 40 times a day? Seems like you’re wasting a lot of resources for something that is solved fairly simply. Deploying software isn’t hard, so why blow away your entire infrastructure several times a day.

    Pass.

  • xiongchiamiov says:

    Wouldn’t you be able to do the same thing with Apache as DYNOM suggests with nginx – that is, changing the docroot of the config and then -HUPing the workers?

  • rlerdorf says:

    XIONGCHIAMIOV, no, when you HUP Apache it re-initializes all the modules which looks like a fresh startup to them which means you lose your shared memory segments and thus your opcode cache.

  • […] issue can be handled in many ways, Rasmus Lerdorf has posted a quite detailed description how they handle deploys atomically at Etsy by using a symlinked document root directory and a custom Apache module and PHP extension to […]

  • titpetric says:

    One more reason to stick with nginx for me. I’ve used with success it in the same way (-HUP with configuration changes). I assume that using lighttpd or apache with -HUP would work in much the same way, as long as you’re using fastcgi/php-fpm and not mod_php/sapi, avoiding clearing shared memory segments/opcodes. I don’t suppose this is a valid benchmark today, SAPI vs. FastCGI? Any plans to migrate to nginx? :)

    • rlerdorf says:

      No, no plans to move to nginx at this point. The one drawback with the built-in nginx mechanism is that you can’t cache the realpath() result, so you take the hit on every request.

      • titpetric says:

        If you’re modifying the configuration to actually point at site/A and site/B before you issue -HUP, you avoid any realpath calls because you don’t use $realpath_root. Confirmed by looking at nginx code, realpath isn’t called anywhere else.

  • Leave a Response

    Recent Posts

    About

    Etsy At Etsy, our mission is to enable people to make a living making things, and to reconnect makers with buyers. The engineers who make Etsy make our living with a craft we love: software. This is where we'll write about our craft and our collective experience building and running the world's most vibrant handmade marketplace.

    Code as Craft is proudly powered by WordPress.com VIP and the SubtleFlux theme.

    © Copyright 2014 Etsy