The Product Hacking Ecosystem

Posted by on January 4, 2012

Most product ideas are shitty, yet we spend the majority of our lives working on them.

As a product hacker, you’ll be working on a constant stream of ideas that excite you to the point of obsession; staying up late writing code, thinking about it every waking and non-waking minute. We’ve all admitted that a minority of our ideas will turn into something that will have the impact we dream of, but we don’t let that truth prevent us from being excited that this next thing might be the one. Some have admitted this and accepted that they’re a junky who’s only going to get that fix from a great feature once in a long while. Although I admit that I’m a junky, I haven’t yet become a fatalist.

Web Operations people speak about measuring their work by the Mean Time Between Failures (MTBF). For product hackers, we should be thinking in terms of minimizing Mean Time Between Wins (MTBW). Because it’s difficult to know which ideas are going to blossom into that great feature, a nice proxy for MTBW is Mean Time to Bad Idea Detection (MTTBID).

By building out an ecosystem for you and your team that allows bad ideas to be detected quickly, you can spend your time iterating on the great ideas and shipping your wins quickly while the shitty ideas die a meaningless death somewhere in a pile of other shitty ideas.

The best hackers I know are impatient. As soon as you get an exciting result, you’re going to be talking about it with whoever will listen. An ecosystem of tools that are just there providing a source of truth that everyone can understand and agree with is like having a posse of hardened thugs at your back at all times. Instead of excitement going sour when people who haven’t seen the light are doubting you, you can all agree on whats actually going on. If the numbers you care about are getting better, then great. If your product isn’t something that can be measured easily, or is a long term bet, you can show that the numbers you care about aren’t getting worse and show that its safe to push on into the wilderness.

Here are some things we’ve learned about how to build that ecosystem.

Make Tools for Failing Fast

Ideas can fail at any level of scrutiny. Some ideas don’t pan out when looked at under a microscope. Others don’t work out when talking about it over a drink. If it survives to the point of being shown to users, it can fail when you’re looking at it through a telescope and you’re just not seeing the response you hoped for. We spent some time trying to improve the quality and performance of our relevance sorting algorithm for search results before we made relevance-ordering the site-wide default. During the four month period where we did this work, we were able to get thirty experiments completed. Of those, eleven were real wins that made it into the final product.

At Etsy, the birth of every idea is the simplest possible implementation that permits experimentation. To give ourselves immediate feedback on the effects of search algorithm changes we created a tool that let us see the new ranking and all of the information we need to understand why a listing is ranked the way it is. The tool let us see this new ranking the moment our search server finished compiling, allowing for rapid iteration on tricky edge-cases, and the ability to quickly detect and kill bad components.

We created a tool that runs a sample of popular and long-tail queries through a new algorithm and displays as much information as can be determined without real people being involved; an estimated percent of changed search results over the universe of all queries, a list of the most strongly affected queries, a list of the most strongly affected Etsy shops, etc..

We created tooling for running side-by-side studies where real users were asked to rate which set of search results they preferred for a given query. When a feature was ready to be launched as an A-B test, we were able to see a set of visualizations explaining how our change was performing relative to the standard algorithm.

What a Search AB Test Looks Like What a site-wide AB test looks like

The best part is that we don’t think about these tools while building new products and running experiments. We come up with ideas, implement them, and if they do well we ship them. Our conversations are about the product, the code we write is for the product and our shitty ideas are executed on the spot and sloppily buried in shallow graves, as they deserve and as is our wont.

Make Tools that Make Process Disappear

Edward Tufte introduced the concept of “chart junk”; the distracting stuff on a visualization that isn’t saying anything about the data. Marshall McLuhan made a compelling case that “The medium is the message” implying that the vehicle through which you perceive something impacts your understanding of it. Just because your paying clients won’t see your internal tooling doesn’t give you license to slap together an ill considered tool. The medium is the message, and your tools are your medium. Working Memory is limited and people are busy. Decisions are worse when getting the answer to a question about your product requires that you lose track of what you asked or why it’s important. Decisions are even worse if you never get a chance to ask questions and get answers. Products designed with fewer poor decisions are less shitty than products designed with more poor decisions. GNU wouldn’t exist without GDB.

 Etsy Search Query Analysis Tool  Solr Search Query Analysis Tool
Our Non-Shitty Search Query Analysis Tool Solr’s Shitty Query Analysis Tool

It’s really important to our business that we return great results when people are doing searches on Etsy. It turns out we’re super lazy and if there are any barriers in the way of us asking “why is this item showing up for this query”, we’re just not going to ask the question and it’s not going to get fixed. Our query analysis tool (pictured on the left) helps reduce that barrier to getting an answer.

The best information about your product is going to come from real users. Unfortunately, its often painful to get your products out in to the real world. Having completed an iteration of a product, you’re filled with excitement and fear. You’re hoping you got it all right, but if you didn’t, you’re ready to fix it because you know every intimate detail of your new creation. This state of excitement and readiness is the last thing you want to let go of. Continuous deployment, the practice of pushing your code live the moment its ready, is absolutely essential for product hackers.

If you need to wait any non-trivial amount of time between completing something and seeing how well it’s performing, you’re not going to be working on that project by the time you get your answer. When you do get your answer, you’re not only going to have to refresh your memory on what you had been working on, but you’re going to have to do the same on whatever else you had started working on. Asking your team to work with patience and discipline has never worked and never will work. Build an ecosystem where doing the right thing is the easiest thing. Build an ecosystem where making great decisions is the easiest thing. Build an ecosystem where the lazy, excitable and impatient really shine.

9 Comments

Andrew — awesome! I love the part about spending the effort to modify the UI to give internal users extremely strong visual indicators about how and why the product is working a particular way. Thanks.

    It seems as if people often make the mistake of thinking that if you don’t have an answer immediately available (and understandable) the moment its asked that once someone goes and computes the answer they’ll be able to pick up exactly where they left off.

    The reality seems to be that the value of any analytics used in decision making is multiplied by 1/t where t starts counting up the moment the question is posed. To me, this implies that we need to solve the very difficult task of being able to answer as broad a range of questions as possible in as little time as possible. Thats where the UI challenge comes in.

Too many developers and managers linger in the cozy realm of internal demos, QA testing and staging environments, doing nothing but buffering up the pain to come. They might get better at keeping deadlines and staying on budget this way, but they’re missing out on the learning opportunities we all need to create valuable products. Your perspective is important because – I hope – it might make them think about what a smarter competitor might achieve with continuous deployments and experimentation.

Continuous deployment can be dangerous sometimes if you release deployments with dependencies on earlier deployments. You can find yourself far down a “shitty” path with a mess of deployments to unravel. This is spoken as one who has done such and ended up having to detangle the mess in the wee hours 🙂

I’ve found I prefer fewer, but more robust and well-tested deployments and limited dependencies.

I really like how this post highlights the importance of the right technology, people and process to enable continuous delivery.

@TULA What sort of time and code-scale are you talking about? Combining very small and very frequent deployments with appropriate testing and feature flags (very important) help avoid the need to unravel. This is somewhat theoretical though, I’d be interested to know if the folk at etsy, flickr et al have [ever|often] had to deal with wee hours.

Continuous deployment has been working well for us at Blurb on one of our projects. With a build system that demands constant stability and includes robust automated testing, the build is easily & frequently updated and has fewer wee-hour de-tanglings than ever.

I will say that I feel a greater rush to check in code under the continuous model. Sitting in a corner for 2 weeks without a code commit feels more problematic. This has led to breaking down features/bug fixes into smaller pieces though, to make them easier to check in quickly. Coupling the continuous deployment with an agile development schedule now feels almost necessary.

Thanks. “Mean Time to Bad Idea Detection (MTTBID)” has enlightened my day.

My favorite part of the article was the section title “Make Tools that Make Process Disappear” – this is one of the critical components that make continuous delivery processes work.

@Andrew – any way you can make those screenshots bigger, so we can see a bit more about what exactly is on them and what exactly you have done? Thanks!