By Lara Hogan

Aug 9, 2013

At the end of 2012, Etsy CEO Chad Dickerson noted that mobile visits grew 244% year over year and represented more than 25% of all traffic. Halfway through 2013, this metric continues to steadily increase, with mobile web and app traffic accounting for about 40% of visits to Etsy.

As this traffic grows, our designers and developers need to be able to test how Etsy works on more devices, screen sizes, and operating systems. The Mobile and Corporate IT teams built a mobile device testing lab to help with this. Originally a gray cabinet with devices, now a handcrafted shelf with labels and power, we've begun encouraging product development teams to come and use it!

We've learned a ton so far from building this lab, and we're eager to share these lessons with others as more companies consider creating their own testing suite.

Why build a testing lab?

Many designers and developers at Etsy have iPhones or very recent Android smartphones. Mobile web work was historically tested on an employee's phone or in their desktop browser using spoofed user agents, which is great for day-to-day development, but we need to test out work on a wider range of real, physical devices that represent what our visitors are using when they arrive at Etsy.com. Small bugs and edge cases were being missed during development, and we weren't necessarily testing on devices that our members are actually using.

We decided to build the device lab with devices that drive the most traffic to the site, as well as some older devices that we know can be problematic to develop for (particularly older Androids and Blackberries). There are a ton of articles on how to choose devices for your lab; see the end of this article for links!

"Testing your designs in the device lab is great. I had no idea how many strange aspect ratios were out there on Android devices. Being an Apple only mobile device user, the device lab was an invaluable resource for testing my designs across unfamiliar operating systems and hardware." - Jason Huff, Designer

This has helped our teams not just test new projects, but also go back and find things to fix in already-shipped projects:

"The mobile device lab really is a phenomenal resource. While it's easy to check features on your own phone or tablet, it's easy to forget how many other devices are out there and being used by our members. When we went up to the lab for the first time, our team discovered a number of device specific scenarios that we hadn't come across during our previous testing, and I'm so glad these were surfaced" - Triona Fritsch, Product Manager

Unexpectedly, the device lab has also been a brainstorming opportunity for teams who are new to mobile development. As people go through and test their existing projects, they also see how they could have reframed their approach from the beginning, and they're inspired to do more mobile-first work.

"[The device lab] also gave us some new insight into scenarios that we should consider during the initial design phases for future mobile projects." - Triona Fritsch, Product Manager

Setup

We currently have 34 devices on this custom-made bookshelf (made by Reade Bryan who works on Etsy's Facilities team). See the end of this article for more resources on selecting devices. Here's our breakdown:

• 12 Android phones (a range of operating system versions and major manufacturers)
• 3 Android tablets (a range of screen resolutions)
• 2 Blackberry phones (one older phone, one BB10)
• 1 Firefox phone (developer preview)
• 5 iPads (including 1 mini, different generations)
• 4 iPhones/iPods (one upgraded to iOS7)
• 3 Kindle Fires (two are HD, and we have a range of screen resolutions)
• 1 Windows Phone
• 1 Windows Surface
• 1 Google Glass
• 1 Chromebook Pixel

The bookshelf was designed to have different-height openings for our range of device sizes. Each bookshelf has a lip on the front so that we have the option of running cables behind it and putting labels on the front of it.

The bottom shelf holds all our power setup as well as custom built crates from Etsy seller EmmersonWoodworks. These crates hold cables for testers to borrow if they'd like to take devices back to their desk, as well as screen wipes, labels and other miscellaneous items that we need nearby.

We purchased a variety of stands from Etsy sellers, including PhoneStandz, DonorStand, woodworksRD and Fenton Creek WoodWorks. The stands ranged in material, size, and slot width - some older phones need slightly bigger stands since they're thicker than newer phones. We typically stand phones upside-down in their stands to charge because the cables rarely sit nicely in the stands while charging.

We also use library cards for signing out devices that leave the room, which we purchased from jardindepapier. The library cards are color-coded by operating system (as is the washi tape from InTheClear we used to hold them down, label the front of the shelves and the back of each device). On the front of the library card pockets we include:

• device name and number
• screen resolution
• pixel density
• operating system version

This way, testers can make sure they're checking out a range of devices, screen sizes, and OS versions. We also developed a getting started guide that outlines device traffic, known quirks like overlays and fixed elements, and how to get set up with Adobe Edge Inspect.

Each device gets a cable drop to hold its cable in place so they don't fall behind the shelves, a library card with its information, Adobe Edge Inspect installed, an asset tag, and a stand that fits it. We also install a background image that reminds testers of our two device lab rules:

1. Please sign out devices
2. Don't upgrade the OS or apps

It does take some time to get each device set up in the lab, but the organization is really worth it to keep it easy for testers to use a device, sign it out, and plug it back in.

Major Lessons (So Far)

Power

Power has been the biggest challenge for us. As a Certified B Corporation, we want to track energy consumption and make sure we've set up the device lab in an optimal way. We're measuring the amount of power we use, and we've also set the devices up on a power strip with a timer so they only charge for a few hours each day. We're still finding the right balance.

Power was further complicated when we realized that most USB hubs are not set up for devices with lots of power draw (like iPads). We had to learn the hard way that when you plug too many devices into a hub at once, the whole hub can stop working. We are now using a USB hub that can handle up to 32 devices, which maxes out at 16 Amps (at 500mA per port). There is a safety protection device within the circuit board that prevents a higher charge going to each port. This is ideal for lower powered devices (most of the ones we have in the lab) but is not suitable for larger, more power hungry devices like the iPad or other tablets. Tablets generally require a higher charging level so they will charge very slowly, or not power the device at all. This is why we have some of the lab’s devices running directly into a power strip. The other nice thing about this hub is a fault indicator light which helps identify problematic devices; devices which consume excess current are also disabled, leaving non-faulty devices charging.

Label all the things

Any web content editor will tell you that people don’t read, they scan. When a designer or developer comes to use the device lab, we want to make it so easy to use that they don’t have to think. Need to grab a cable to take back to your desk? Try the crate that says “BORROW THESE”. Need an iPhone 4? We’ve labeled the front of the bookcase where it lives and the back of the device so that it’s easy to match up later. Need to find the Blackberries among the devices? Look for the blue library cards (and blue washi tape to hold the cards down). The color-coding of library cards and tape was intentional to easily group Androids (green), iOS devices (manila), Kindles (yellow), Windows (pink), Blackberries (blue) and a Firefox phone (orange).

We wrapped washi tape around each of the USB cables to borrow so that, should they walk away to someone’s desk and forgotten, they’ll be relatively easy to spot later. We’ve also added a sliver of washi tape on or near the power button for each device so that it’s easy to find -- this tends to be the most challenging part of using the device lab!

Testing on VMs

We have an awesome Security Team at Etsy who has been invaluable during the device lab setup. Recently, they helped install security certificates across devices to make it easier to test on VMs. Previously, testers would have to individually accept certificates device-by-device, which really hampered testing and development, especially when they were trying to tackle a lot of testing at once with Adobe Edge Inspect.

Adobe Edge Inspect

So far, this tool has been really handy for testing on lots of devices at once. We're using a multi-device subscription (free Adobe Edge allows you to test on only one device at a time) and installed the Adobe Edge app on all Androids, Kindles and iOS devices in the lab. It's really awesome to feel like a puppet master and control all of those devices at once, but what's really handy is capturing screenshots from all those devices at the same time so you can check them out on your laptop and audit them later.

We've come a long way with our device testing lab, and I'm excited to keep iterating on it and making it even easier to test with. In the future, we'll be adding more automated testing to the devices, easier tracking of devices (including signing them out), and more analysis of use. As mobile traffic continues to increase, we're eager to make sure that it's incredibly easy for designers and developers to test their work and experience what our members do when they use Etsy from a mobile device.

More reading:

• Strategies for choosing test devices
• How to build a device lab, with a thorough first section on choosing and acquiring devices
• Mobile testing for dummies
• Establishing an open device lab

You can follow Lara, engineering manager for the mobile web team, on Twitter at @laraswanson