Reducing The Roots of Some Evil

Posted by on July 16, 2013

High profile compromises of SSL Certificate Authorities have become increasingly common over the past several years, and although there are some interesting advancements in this area coming down the line (such as TACK.io), CAs currently present a unique source of risk for a number of organizations. In May of this year, we decided to see what actions we could take to reduce our exposure to compromised or malicious Certificate Authorities and we hope our research will be of use to other groups as well. Our goal was simple: We wanted to determine what CAs the browsers across our organization actually use, and remove the rest from being trusted. While this wouldn’t prevent a user from clicking through a certificate warning from a now untrusted CA, our goal was simply to reduce the number of CAs that could silently MITM the browser traffic of people who work at Etsy with no certificate warnings at all. We felt this would provide us with a practical way to reduce our exposure to compromised and/or malicious CAs that we didn’t even use in the first place.

Which CAs?

The first question we had to answer was how do we build a comprehensive list of what CAs our systems actually use? Arguably we could guess that certain CAs shouldn’t be on our systems by name alone (we’re looking at you, “AAA Certificate Services”) however we wanted to gather real data we could analyze and make our decisions from.

Enter CAWatch

After debating ideas such as pushing browser plugins to record CAs seen by the browser, we decided to implement a simple solution called CAWatch. CAWatch sits at the main network egress point of our Brooklyn HQ and, unsurprisingly, watches for CA certificates crossing the wire. Whenever a CA is observed, CAWatch logs the name of the CA and the time it was seen. In order to ensure there was no privacy impact, no information about the site being visited is recorded.

Our first step in creating CAWatch was to obtain a list of CA certificates currently trusted by our standard laptop/desktop endpoints so that we knew what to watch for. On most modern desktops, there are two sources of these certificates: the OS key storage mechanism, and the collection of keys baked in to Firefox. To acquire the CAs built in to OSX one can simply use the Keychain Access utility and export all System Roots. However, Firefox does not natively have support for exporting all the CA certs (although they can be exported manually one by one) so we used a Firefox plugin called Export All Certificates to obtain them all in bulk. Unfortunately, the certificates are then exported in DER format when we needed them in PEM format. Luckily they can be easily converted from DER to PEM with the following command:

mkdir output; ls *.der | xargs -L 1 -I {} openssl x509 -in {} -inform DER -out 'output/{}' -outform PEM; cat output/*.der > certs.pem; rm -rf ./output/

Make sure to use a freshly installed Firefox profile separate from normal use, otherwise any CA certificates you have made exceptions for will end up in the output.

Our Results

During the two months we’ve had CAWatch in operation, we’ve seen only 61 unique CA certificates cross the wire. This accounts for slightly less than 29% of the 212 total CA certificates installed by default in our standard build, and means that we could significantly reduce our exposure by removing a large number of unused CAs.

Of the 61 observed, below are the 23 CA’s we saw more than 0.01% of the time:
21.29% EQUIFAX SECURE CERTIFICATE AUTHORITY
10.37% ENTRUST.NET SECURE SERVER CERTIFICATION AUTHORITY
10.07% DIGICERT HIGH ASSURANCE EV ROOT CA
8.97% GO DADDY CLASS 2 CERTIFICATION AUTHORITY
7.91% GEOTRUST GLOBAL CA
7.23% ADDTRUST EXTERNAL CA ROOT
6.48% HTTP://WWW.VALICERT.COM/
6.04% GTE CYBERTRUST GLOBAL ROOT
4.45% VERISIGN CLASS 3 PUBLIC PRIMARY CERTIFICATION AUTHORITY - G5
4.08% CLASS 3 PUBLIC PRIMARY CERTIFICATION AUTHORITY
3.82% BALTIMORE CYBERTRUST ROOT
3.22% CLASS 3 PUBLIC PRIMARY CERTIFICATION AUTHORITY - G2
1.37% THAWTE PRIMARY ROOT CA
1.36% THAWTE PREMIUM SERVER CA
1.33% ENTRUST.NET CERTIFICATION AUTHORITY (2048)
0.65% GLOBALSIGN ROOT CA
0.47% STARTCOM CERTIFICATION AUTHORITY
0.22% COMODO CERTIFICATION AUTHORITY
0.20% STARFIELD CLASS 2 CERTIFICATION AUTHORITY
0.16% UTN-USERFIRST-HARDWARE
0.11% NETWORK SOLUTIONS CERTIFICATE AUTHORITY
0.09% UTN - DATACORP SGC
0.05% AMERICA ONLINE ROOT CERTIFICATION AUTHORITY 1
[The 38 CAs which had < 0.01% traffic have been edited out for brevity]

Removing Unused Certificates

Although we’re waiting on a few more months of data to begin removing certificates en masse, we’ve begun the process on a few test machines. Removing unused certificates from the OS X Keychain is quite straightforward and can be done via the Common Name of the certificate or its SHA1 fingerprint with the following commands:

/usr/bin/security delete-certificate -Z {SHA1 Fingerprint} /System/Library/Keychains/SystemRootCertificates.keychain
or
/usr/bin/security delete-certificate -t {Common Name} /System/Library/Keychains/SystemRootCertificates.keychain

Conclusions

Based on our first two months of data we have removed a number of unused CA certificates from some pilot systems to test the effects, and will run CAWatch for a full six months to build up a more comprehensive view of what CAs are in active use. Over the coming months, as we confirm there are no issues, we’ll reduce the certificates installed on endpoints across the organization. While this approach certainly doesn’t solve all risks associated with the Certificate Authority system, we feel that it provides some practical steps for reducing overall exposure. We hope that by sharing our results and methodologies from this research it will help other organizations interested in similarly limiting their exposure to compromised CAs.

Additional information on this and other security research will be discussed in a talk entitled Attack-Driven Defense at the upcoming Nordic Security Conference.

This post was co-authored by Zane Lackey (@zanelackey) and Omar Ahmed (@ancat) of the Etsy Security Team

Posted by on July 16, 2013
Category: engineering, security

4 Comments

This CAWatch, will you be making it opensource anytime soon?

    The current iteration of CAWatch is much more specific to our environment than you might expect, so unfortunately open sourcing it at this stage wouldn’t be of much value. That being said, feel free to drop me an email and we’re happy to share the details of it with anyone who’s interested in setting up something similar.

How do you (or will you) handle removing the CAs from every employee system, and ensure they stay removed and consistent over time? E.g. new laptops, upgrades, new CAs added into OS updates, browser updates, etc

    Good question! We’ll be relying on our endpoint management IT systems for a lot of this work. We’re fairly certain there will be some gotchas as we go from changing just a few pilot systems to the entire enterprise, and our plan is to do a follow up post on these challenges and any solutions we come up with.