Etsy’s Device Lab

Posted by on August 9, 2013

At the end of 2012, Etsy CEO Chad Dickerson noted that mobile visits grew 244% year over year and represented more than 25% of all traffic. Halfway through 2013, this metric continues to steadily increase, with mobile web and app traffic accounting for about 40% of visits to Etsy.

As this traffic grows, our designers and developers need to be able to test how Etsy works on more devices, screen sizes, and operating systems. The Mobile and Corporate IT teams built a mobile device testing lab to help with this. Originally a gray cabinet with devices, now a handcrafted shelf with labels and power, we’ve begun encouraging product development teams to come and use it!

We’ve learned a ton so far from building this lab, and we’re eager to share these lessons with others as more companies consider creating their own testing suite.

Why build a testing lab?

team-testingMany designers and developers at Etsy have iPhones or very recent Android smartphones. Mobile web work was historically tested on an employee’s phone or in their desktop browser using spoofed user agents, which is great for day-to-day development, but we need to test out work on a wider range of real, physical devices that represent what our visitors are using when they arrive at Etsy.com. Small bugs and edge cases were being missed during development, and we weren’t necessarily testing on devices that our members are actually using.

We decided to build the device lab with devices that drive the most traffic to the site, as well as some older devices that we know can be problematic to develop for (particularly older Androids and Blackberries). There are a ton of articles on how to choose devices for your lab; see the end of this article for links!

“Testing your designs in the device lab is great. I had no idea how many strange aspect ratios were out there on Android devices. Being an Apple only mobile device user, the device lab was an invaluable resource for testing my designs across unfamiliar operating systems and hardware.” – Jason Huff, Designer

This has helped our teams not just test new projects, but also go back and find things to fix in already-shipped projects:

“The mobile device lab really is a phenomenal resource. While it’s easy to check features on your own phone or tablet, it’s easy to forget how many other devices are out there and being used by our members. When we went up to the lab for the first time, our team discovered a number of device specific scenarios that we hadn’t come across during our previous testing, and I’m so glad these were surfaced” – Triona Fritsch, Product Manager

Unexpectedly, the device lab has also been a brainstorming opportunity for teams who are new to mobile development. As people go through and test their existing projects, they also see how they could have reframed their approach from the beginning, and they’re inspired to do more mobile-first work.

“[The device lab] also gave us some new insight into scenarios that we should consider during the initial design phases for future mobile projects.” – Triona Fritsch, Product Manager

Setup

2013-08-06 13.05.28 copyWe currently have 34 devices on this custom-made bookshelf (made by Reade Bryan who works on Etsy’s Facilities team). See the end of this article for more resources on selecting devices. Here’s our breakdown:

The bookshelf was designed to have different-height openings for our range of device sizes. Each bookshelf has a lip on the front so that we have the option of running cables behind it and putting labels on the front of it.

The bottom shelf holds all our power setup as well as custom built crates from Etsy seller EmmersonWoodworks. These crates hold cables for testers to borrow if they’d like to take devices back to their desk, as well as screen wipes, labels and other miscellaneous items that we need nearby.

standsWe purchased a variety of stands from Etsy sellers, including PhoneStandz, DonorStand, woodworksRD and Fenton Creek WoodWorks. The stands ranged in material, size, and slot width – some older phones need slightly bigger stands since they’re thicker than newer phones. We typically stand phones upside-down in their stands to charge because the cables rarely sit nicely in the stands while charging.

We also use library cards for signing out devices that leave the room, which we purchased from jardindepapier. The library cards are color-coded by operating system (as is the washi tape from InTheClear we used to hold them down, label the front of the shelves and the back of each device). On the front of the library card pockets we include:

This way, testers can make sure they’re checking out a range of devices, screen sizes, and OS versions. We also developed a getting started guide that outlines device traffic, known quirks like overlays and fixed elements, and how to get set up with Adobe Edge Inspect.

Each device gets a cable drop to hold its cable in place so they don’t fall behind the shelves, a library card with its information, Adobe Edge Inspect installed, an asset tag, and a stand that fits it. We also install a background image that reminds testers of our two device lab rules:

  1. Please sign out devices
  2. Don’t upgrade the OS or apps

It does take some time to get each device set up in the lab, but the organization is really worth it to keep it easy for testers to use a device, sign it out, and plug it back in.

Major Lessons (So Far)

Power

Power has been the biggest challenge for us. As a Certified B Corporation, we want to track energy consumption and make sure we’ve set up the device lab in an optimal way. We’re measuring the amount of power we use, and we’ve also set the devices up on a power strip with a timer so they only charge for a few hours each day. We’re still finding the right balance.

Power was further complicated when we realized that most USB hubs are not set up for devices with lots of power draw (like iPads). We had to learn the hard way that when you plug too many devices into a hub at once, the whole hub can stop working. We are now using a USB hub that can handle up to 32 devices, which maxes out at 16 Amps (at 500mA per port). There is a safety protection device within the circuit board that prevents a higher charge going to each port. This is ideal for lower powered devices (most of the ones we have in the lab) but is not suitable for larger, more power hungry devices like the iPad or other tablets. Tablets generally require a higher charging level so they will charge very slowly, or not power the device at all. This is why we have some of the lab’s devices running directly into a power strip. The other nice thing about this hub is a fault indicator light which helps identify problematic devices; devices which consume excess current are also disabled, leaving non-faulty devices charging.

Label all the things

iphone4Any web content editor will tell you that people don’t read, they scan. When a designer or developer comes to use the device lab, we want to make it so easy to use that they don’t have to think. Need to grab a cable to take back to your desk? Try the crate that says “BORROW THESE”. Need an iPhone 4? We’ve labeled the front of the bookcase where it lives and the back of the device so that it’s easy to match up later. Need to find the Blackberries among the devices? Look for the blue library cards (and blue washi tape to hold the cards down). The color-coding of library cards and tape was intentional to easily group Androids (green), iOS devices (manila), Kindles (yellow), Windows (pink), Blackberries (blue) and a Firefox phone (orange).

We wrapped washi tape around each of the USB cables to borrow so that, should they walk away to someone’s desk and forgotten, they’ll be relatively easy to spot later. We’ve also added a sliver of washi tape on or near the power button for each device so that it’s easy to find — this tends to be the most challenging part of using the device lab!

Testing on VMs

We have an awesome Security Team at Etsy who has been invaluable during the device lab setup. Recently, they helped install security certificates across devices to make it easier to test on VMs. Previously, testers would have to individually accept certificates device-by-device, which really hampered testing and development, especially when they were trying to tackle a lot of testing at once with Adobe Edge Inspect.

Adobe Edge Inspect

bucky-testingSo far, this tool has been really handy for testing on lots of devices at once. We’re using a multi-device subscription (free Adobe Edge allows you to test on only one device at a time) and installed the Adobe Edge app on all Androids, Kindles and iOS devices in the lab. It’s really awesome to feel like a puppet master and control all of those devices at once, but what’s really handy is capturing screenshots from all those devices at the same time so you can check them out on your laptop and audit them later.

We’ve come a long way with our device testing lab, and I’m excited to keep iterating on it and making it even easier to test with. In the future, we’ll be adding more automated testing to the devices, easier tracking of devices (including signing them out), and more analysis of use. As mobile traffic continues to increase, we’re eager to make sure that it’s incredibly easy for designers and developers to test their work and experience what our members do when they use Etsy from a mobile device.

before-after

More reading:

You can follow Lara, engineering manager for the mobile web team, on Twitter at @laraswanson

Posted by on August 9, 2013
Category: mobile

16 Comments

Great article! I’m trying to begin purchasing used mobile phones so I can set up my own test lab. The biggest challenge is finding older devices in good working order that are running the OS I’m interested in. I’ve been looking on mobilekarma, swappa, craigslist, and eBay with spotty success. Do you have any suggestions on finding and buying used phones at a reasonable price? Thanks!

Thanks for publishing this, it’s been very valuable. Up to now most articles on testing across mobile devices have just left it at the number, but here to learn about your issues with power, the tools you use to supplement your testing, and how you encourage staff to test is very useful. Thank you.

CM.

Robert – when we purchased used phones, they were off of eBay. It definitely takes some work to find legitimate ones, but it ended up being so much better than purchasing them new!

Looks really good, I especially like the whole library card idea — can imagine that devices would be scattered all over at any given point otherwise. Bought solutions I’ve seen for teaching often include all power needs, slots for tons of devices, and are secure (can lock them) but something custom like this of course allows for much more flexibility.

I would imagine that for some devices/versions you may end up needing more than one of each, would be interesting to see how you solve that too.

Thanks for sharing! We have a “virtual lab”, where lots of kind employees will hand over their devices for an hour a week for testing (or sometimes even test themselves!).

The biggest problem we’ve had is finding a suitable way to expose our internal, test, servers to our devices without compromising security. I’d be interested in hearing how Etsy does that!

Thanks for sharing your insights, Lara!

I’d like to contribute the following two links to your list that should definitely not be missing:

http://opendevicelab.com – A registry of all existing Open Device Labs (ODL) worldwide; Easily find the lab next to you …

http://lab-up.org – The non-profit organization behind opendevicelab.com, helping people around the work establishing Open Device Labs; You can find lots of useful resources there as well …

Have you considered powering the lab with a self sustainable solar system? I built a solar/battery system last year that saved my bacon when Sandy knocked my power offline for 5 days. I was able to charge all of my family’s devices, some laptops and provide some LED lighting.

Have you considered powering the lab with a self sustainable solar system? I built a solar/battery system last year that saved my bacon when Sandy knocked my power offline for 5 days. I was able to charge all of my family’s devices, some laptops and provide some LED lighting.

That is awesome! I’d love to build something identical to this. Mobile visits will only continue to grow as more and more people get browse capable phones.

Any update on how this is working out for you guys? New learnings or process/hardware changes?

How do you track who checked out a device or borrowed a cable? Do you use software to do this?

Hi! This article is fantastic! I’m wondering if you would mind explaining more about Adobe Edge and whether or not you still use it/how you use it?

Thanks!

[…] is a great post by ETSY on the subject of device labs and this image rings true to me. This is what we need to transition […]

[…] is a great post by ETSY on the subject of device labs and this image rings true to me. This is what we need to transition […]

[…] is a great post by ETSY on the subject of device labs and this image rings true to me. This is what we need to transition […]

[…] top devices that are using your apps, or driving traffic to your site, and focus there. This was a primary selection criteria of Etsy, in the formation of their mobile lab. We at testmunk consider an in-house lab of 30 to 40 devices […]