mctop – a tool for analyzing memcache get traffic

Posted by on December 13, 2012

Here at Etsy we (ab)use our memcache infrastructure pretty heavily as a caching layer between our applications and our database tiers. We functionally partition our memcache instances into small pools and overall it works fabulously well. We have however suffered occasionally from what we call “hot keys”.

What is a “Hot” key?

A “hot key” is a single key hashed to an individual memcache instance with a
very high get rate, often being called once for every page view. For the most part network bandwidth across all memcache instances within a pool is relatively balanced. These hot keys, however, contribute a significant additional amount of egress network traffic and have the potential to saturate the available network bandwidth of the interface.

Image

The graph above is an example of a recent hot key issue. The graph y-axis represents bytes per second inbound and outbound of memcached01’s network interface.

As we hit peak traffic, memcached01’s network interface was completely saturated at approximately 960Mbps (it’s a 1Gbps NIC). This has a particularly nasty impact to get latency:

Image

As we began to push past 800Mbps outbound, 90th percentile get request latency jumped from 5ms to 35ms. Once the NIC was saturated latency spiked to over 200ms.

Diagnosing the Issue

This wasn’t the first time a hot key had been responsible for unsually high network bandwidth utilization so this was our first line of investigation. Comparatively memcached01’s bandwidth utilization was significantly higher than the other servers in the pool.

Diagnosing which key was causing problems was a slow process, our troubleshooting process took the following steps:

  1. Take a brief 60 second packet capture of the egress network traffic from memcached01
  2. Using the tshark (wireshark’s awesome command line cousin) extract the key and response size from the memcache VALUE responses in captured packet data.
  3. Post process the tshark output to aggregate counts, estimate requests per second and calculate the estimated bandwidth per key.
  4. Sort that list by bandwidth then further investigate that key.

Once the potentially offending key is found we’d repeat this process from a couple of client machines to validate this as the offending key. Once the key was confirmed engineers would look at alternate approaches to handling the data contained in the key.

In this particular case, we were able to disable some backend code that was utilizing that key with no user facing impact and relieve the network pressure.

Overal this diagnostic process is quite manual and time intensive. 60 seconds of packet capture at 900Mbps generates close to 6GB of packet data for tshark to process, and if this process needs to be repeated on multiple machines the pain is also multiplied.

Welcome mctop!

Given this wasn’t a new issue for us I decided to have a crack at building a small tool to allow us to interactively inspect in-real time, the request rate and estimated bandwidth use by key. The end result is the tool “mctop” we’re open sourcing today.

Inspired by “top”, mctop passively sniffs the network traffic passing in and out of a server’s network interface and tracks the responses to memcache get commands. The output is presented on the terminal and allows sorting by total calls, requests/sec and bandwidth. This gives us an instantaneous view of our memcache get traffic.

Image

mctop is now available as a gem on rubygems.org and the source is available over at github.

Patches welcome, we hope you find it useful!

10 Comments

Awesome. Nice drill down into a problem with a (new) alternate solution. Code as Craft continues to inspire!

[…] Etsy Open Sources mctop Etsy’s tool for tracking responses to memcache get commands is now open. […]

Nice example of the value of monitoring keys in Memcached. Have you tried out the sFlow instrumented branch of Memcached?

https://github.com/sflow/memcached

The sFlow instrumentation lets you continuously monitor cluster-wide top keys / missed keys and send metrics to Ganglia.

http://blog.sflow.com/search/label/Memcache

Why only a 1Gbps network and interface? Seems awfully low for a production network.

    We grow our infrastructure as needed. We’re currently evaluating 10Gbps for the memcache network. We didn’t have this before as we simple didn’t need it!

We’re using this great little tool at Wikia.com as well and it’s given our Ops team a new avenue to monitor and improve site performance.

Cheers,
WW

Nice tool!

Are there any ways to monitor “set” traffic?

Would adding a random (within a small range) postfix to the key help? This would spread the key over more servers in your cluster and you would benefit from aggregate bandwidth. Unless you have intelligent key invalidation (as opposed to simple expiration-based invalidation) this approach would actually improve the performance of hot-key retrieval without adding complexity to the code.