Optimizing Meta Descriptions, H1s and Title Tags: Lessons from Multivariate SEO Testing at Etsy

Posted by , and on January 25, 2017

We recently ran a successful split-testing title tag experiment to improve our search engine optimization (SEO), the results and methodology of which we shared in a previous Code as Craft post. In this post, we wanted to share some of our further learnings from our SEO testing. We decided to double down on the success of our previous experiment by running a series of further SEO experiments, one of which included changes to our:

Screen Shot 2017-01-25 at 10.42.39 AM

We found three surprising results:

Some notes on our methodology

For full details on the setup and methodology of our SEO split testing methodology, please see the previous Code as Craft post referenced above.

For this particular test, we used six unique treatments and two control groups. The two control groups remained consistent with each other before and after the experiment.

control_vs_control_annotated

To derive the exact estimated causal changes effected in visits, we most often used Causal Impact modeling as implemented in the CausalImpact package by Google to standardize the test buckets against the control buckets. In some cases, the difference in differences method was used because it was more reliable in estimating effect sizes when working with strong seasonality swings.

This experiment was also impacted by strong seasonality and external event effects related to the holidays, sports events and the US elections. The statistical modeling in our final experiment analysis was adjusted for these effects to ensure accurate measurement of the causal effects of the test variants.

Learnings

Takeaway #1: Short Title Tags Win Again

The results of this experiment aligned with the findings from our previous SEO title tag experiments, where it appeared shorter title tags drove more visits. We have so far validated our hypothesis that shorter title tags perform better in title tags in multiple separate experiments including many different variations and now feel quite confident that shorter title tags perform better (as measured by organic traffic) for Etsy. We hypothesize that this effect could be taking place through a number of different causal mechanisms:

  1.  Lower Levenshtein distance and/or higher percentage match to target search queries rewarded by Google’s search algorithm and thereby improving Etsy’s rankings in Google search results. Per Wikipedia: “the Levenshtein distance between two words is the minimum number of single-character edits (i.e. insertions, deletions or substitutions) required to change one word into the other.”
  2.  Shorter title tags, consisting only of the target search keyword, appear more relevant/enticing to search users

Takeaway #2: Meta Descriptions Matter

We found that changes in the meta description of a page can lead to statistically significant changes to visits. It appeared that longer and descriptive meta descriptions performed better and that conversely, shorter, terse meta descriptions performed worse. We hypothesize that longer meta descriptions might perform better via two possible causal mechanisms:

  1.  Longer meta descriptions take up more real estate in a search results page, improving CTRs
  2.  Longer meta descriptions give an appearance of more authority or more content, improving CTRs

Takeaway #3: H1s matter

We found that an H1 change can have a statistically significant impact on organic search traffic. However, changes in the H1 section of a page appear to interact with changes in title tags in hard to predict ways. For example, in this experiment, a title tag change in a certain variant increased visits. However, when the title tag change was combined with an H1 change, the positive effect of the title tag change was dulled, even though an H1 change by itself in a different variant led to slight increases in visits. This highlights the importance of SEO testing before rolling out even seemingly minor changes to Etsy pages.

Accounting For Unexpected Events

The Donald Trump Effect

One example of an event we had to control for (among a number of others) was the “Donald Trump effect”. We observed a large bucket skew on November 9 and 10, 2016 in one of our test groups. Upon investigation, it was found that the skew was due to large increases (+2000% to +5180%) in daily visits to pages related to “Donald Trump” the day after the US Presidential elections. Although the spikes in traffic to these pages were short lived, lasting only several days, they nevertheless did have the potential to unduly bias or reduce the statistical significance of the results of our experiment. These pages were therefore removed and controlled for when conducting the causal impact analyses for this experiment.

trump_outlier

Conclusion

Our SEO experiments illustrate the importance of running SEO tests before making changes to a web page and continue to offer surprising results. However, it is important to note that the results of our experiments are only true for Etsy and do not necessarily reflect best practices for sites generally. We would therefore encourage everyone to discover the strategy that works best for their website through rigorous SEO testing.


Related Posts

18 Comments

Your meta information is the creative behind the click-through. The title and description are very important. Not only for micro search platforms such as Etsy, but also search engines such as Google and Bing.

I make note to know exactly how much of the meta title and description actually show up in the results of a search.

For instance, here is a list of character counts for three of the major players:
Google Title (69), Description (156)
Yahoo Title (72), Description (161)
Bing Title (65), Description (150)

Knowing this, it is important to go no higher than 65 in the title, and 150 in the description; otherwise you will cut off some of your meta info on a few of the other players. Remember, they all call the same meta info.

I really enjoyed this article. Keep optimizing my friend 😉

    It’s not about characters for Google anymore but pixel width. And it’s closer to 55 per title and 150 per description. 🙂

Wow. Thanks for this. I have always said that Title Tags and Descriptions are the most important elements. Most people would say that Descriptions have no effect but I have never believed that.
Even though Google has ‘gone semantic’ Incorporating the primary keywords in the title tag is essential. Using every character available in the description and writing compelling content and yes, using keywords, makes people more likely to click. After all, you are directly responding to their search query.
Your experience with H1’s is interesting were the keywords in the title tag and H1 the same or different?

    Hi David, the keywords in the H1 and the title tag were more or less the same. Would definitely be interesting to test a slight variation in them!

This was really fascinating. I have to admit when I first read “testing” worry came over me. For about three months my Etsy sales were no longer thriving. However, if I am understanding these tests and potential site changes, the down time for experimentation will very much be worth the end results to bring in new buyers to Etsy.com from organic google search. Organic search is where my websites do best. I couldn’t quite figure out why the past couple of months I still saw my Etsy shops that generally rank in the 1 & 2 positions on google for what I sell doing so poorly where I had previously shown steady growth since I started selling solely on Etsy in 2011. (I did have a small drop in 2012 another election year go figure!)

I am really pleased the Etsy developers are finding the shorter “spit it out” titles are winning. I’m not a fan of the word salad titles. Slowly as I’ve had time to make changes to some listings that stopped selling, by creating less “word salad / spammy” titles and making a few listings titles shorter, I’m happy to say they are increasing in views.

This would be a welcomed change for me and I’m sure for shoppers and Google too.
(I’ve read 2017 Googles focus is on users “intent” and not so much keyword stuffed titles).

I’m hoping that if a permanent change is coming soon someone could help me understand how to better utilize my Etsy shop tags for the change. I understand the less than 55 characters titles are doing better and it is certainly better for shoppers to read short, concise product titles.

To the developers, I thank you for your hard work. I was blown away seeing these statistics for the Donald trump effect.

Take Care,
Kelly Neddo
Featherswholesale/UniqueFeathers/SolKat on Etsy.

Appreciate! Great results! Thank you for sharing!

Etsy’s Title Tag starts with Etsy.com. What do you think of the reason that made them starting their Title with Etsy.com rather than just “Etsy”?

    Hi Sam,

    Many of our titles end with either Etsy.com or Etsy. Historically there wasn’t much rhyme or reason to it but it is something we would like to experiment on.

Multi-variate testing in SEO? Yes please, really dig these types of tests. The whole “meta description” is useless thing was something I was ascribing to a while. For those of us not quite as technical with experience in R, could you perform perhaps a “loose” A/B test using Google Search Console? aka, getting a visits to clicks ratio for a specific blog post, changing the meta description, and then seeing the results a month later?

Checked out your link for this: https://google.github.io/CausalImpact/CausalImpact.html

And it is currently going over my head to implement!

    Hi Tim, you could certainly try to use Search Console CTR data to validate the experiment but it likely wouldn’t change the need to perform statistical analysis on the results. Ultimately, you’ll need to be able to compare your control group to your treatment group using some sort of statistical analysis.

Very Interesting! ==> But so we don’t get ahead of ourselves, all of this is assuming there are no adverse issues associated with: site structure, linking profile, duplicate content and any other factors that might negatively impact ranking… Right? LOL!

    Hi Johnny,

    This particular experiment did not make any changes to our site structure, link profiles or raise duplicate content concerns but you’re definitely right that those are important SEO considerations in general.

“Short Title Tags Win Again” – no, you’re wrong again, that’s wrong perspective! Google is trying to find out how could it make it better for the users, and this is a short-term victory, I’m talking about the short titles.

The sizes won’t matter in the long run. The requests that are going to win the search queries are the ones that satisfy the user’s intent.

🙂

    Hi Kiril,

    In this context the shorter title tags did win (and we’ve seen this result in a few of our experiments). However, you’re correct that Google is looking to reward sites that provide the best user experience, not those that game the system.

[…] Good SEO experiments are rare. This one is concise and interesting from the team at Etsy. Read it here…  […]

Hello!

So, you proved the main thing: Title & H1 influences on your position at SERP.
Meta Description influences on CTR at SERP with your page.
That’s cool!

Another question, how will it influence on your results if you remove “Etsy.com |” from the start position in your title? Can you try to count this? Thanks 🙂

    Hi Nick, – it’s possible that this will be tested in the future :).

Really great content…thank you a lot, guys!