Is It A Bird? Is It A Plane? No, It’s Supergrep!
Etsy parties have come to earn quite the boisterous reputation — there may or may not have been kegs dragged up to the roof on a particularly balmy occasion, not to mention cadaver eating contests and mariachi bands. Thus, one of the greatest survival skills I have come to hone here at HQ is the art of selective hearing. Being able to focus in on the subtle wit of your coworker in the midst of a rowdy congo line is the only way you will leave at the end of the evening with your sanity (and self-respect) intact. Luckily for us, eliminating the noise in our data using technology is a little bit easier.
After a long week two years ago with a number of site incidents, our then-CTO Chad Dickerson sent an email to our all-engineering mailing list about a new tool:
For a few of the incidents we’ve had in the past week, we’ve found
obvious trails in the error logs leading directly to the source of the
problem. I talked to a bunch of folks and what I generally heard was
that the logs were noisy, which is technically true, but. . . . .
Through the magic of grep, you can get past the noise. I did some
digging through the error logs today to better understand what was in
there and wrote a simple ugly grep that kills all the stuff that
repeats a lot (not that we don’t need to fix it).
And thus supergrep was born. The original was a simple grep -v command line bash script to make our logs easier to parse, but the tool has evolved into a communication mechanism for the entire team to view our logs in real time over a web browser. Since then, we have cleaned up many of the noise problems that obscured our logs, while iterating the tool into something easily accessible by everyone on team. It’s a simple node app that everyone on the team can access as they are pushing code, to see if new errors appear. In addition we’ve linked the errors to the exact line of code that the error is happening on for easier analysis. Its yet another change-awareness tool in our kit that allows us to push code continuously and safely to production.
We’ve open sourced the code on our github repository, which you can find here: https://github.com/etsy/supergrep