Java: Not Even Once

Posted by on March 18, 2013

Note: This post was co-written by Avleen Vig (@avleen) and Zane Lackey (@zanelackey).

In mid-January of this year we started an initiative to remove Java browser plugins from all employee systems at Etsy, as we feel this is a best practice to be striving towards. To that end, we wanted to discuss the challenges we encountered when removing Java browser plugins in the hope that it will help other organizations with the removal process.

The first question we needed to answer before removing Java was “Who actually needs Java in their browser?” Most organizations face the unfortunate reality that some number of internal systems use Java applets, and that there are groups which use these systems on a daily or weekly basis. In our case, our operations team needed Java to be able to access a group of internal network appliances, such as IPMI console interfaces. Once we identified these requirements, we disabled Java browser plugins across the rest of the organization and set about trying to engineer a safer way for the team to access these network appliances.

Initially, we looked at three approaches:

  1. Installing the Java plugin in a dedicated browser/browser profile and asking the team to use it to only access the network appliances.
  2. Writing a wrapper script to copy the Java plugin into the plugins directory, launch a dedicated browser, then remove the Java plugin when the browser closes.
  3. Using nssecurity to whitelist the hosts that could instantiate Java in the browser.

However, all of these approaches didn’t fulfill our design goals that the approach be safe by default, be easy to maintain, and ideally wouldn’t require the Java browser plugin on the teams laptops at all.

We realized the only way to approach the situation that met our requirements would be to have Java installed in a dedicated, controlled, and isolated environment.

This model is similar to a bastion host, or “jump box”. We opted to use NoMachine as the remote desktop protocol because of the increased performance and usability over low latency links. We have operations engineers located in multiple countries and we also occasionally need to use 3G/4G mobile services to diagnose problems, so this was critically important.

The installation method we followed for CentOS 6 and FreeNX was:

  1. Install the required packages. All of these come from the standard CentOS repos:
    yum install -y jre freenx gnome-desktop gnome-session 
        gnome-panel nautilus firefox fre

    (After installation, be sure to replace the provided SSH keys: https://help.ubuntu.com/community/FreeNX#Using_custom_SSH_keys)

  2. Create a symlink from the Java browser plugin, to the plugins directory used by Firefox:
    ln -s /usr/java/latest/lib/amd64/libnpjp2.so
    /usr/lib64/mozilla/plugins/libnpjp2.so
  3. Now create
    /etc/nxserver/node.conf

    with these contents:
    ENABLE_PASSDB_AUTHENTICATION="1"
    NX_LOG_LEVEL=7
    NX_LOGFILE=/var/log/nx/nxserver.log
    SESSION_LOG_CLEAN=0
    COMMAND_MD5SUM="md5sum"

FreeNX is now configured!

Each of your users who wants to use the new system needs to do the following steps once to add their user to the NX authentication database:

(NOTE: These steps below follow the Ubuntu FreeNX instructions of using a shared account/SSH key for access to the jump system. In this circumstance the risk was accepted as the network appliances that are the end target also use a shared account, so no additional risk was introduced. Obviously different circumstances will have different security requirements.)

  1. Copy /etc/nxserver/client.id_dsa.key to your local machine and save it as ~/.ssh/nx_dsa
  2. On the jump host, run sudo nxserver --adduser <username>
    This adds your account to the nxserver database.  Note: at this step it will also add a key to your .ssh/authorized_keys2 file, if you manage this in your configuration management system it will get overwritten, so you should add the key there.
  3. On the server again, run sudo nxserver --passwd <username>
    This sets your password in the nxserver database
  4. Download the “NoMachine Player v4” from: http://www.nomachine.com/preview/download-package.php
  5. Start the player, click “New connection”.
    1. Name: jumpbox
    2. Host: jump.your.domain
    3. Select: Use the NoMachine login, then click the ... button
    4. Check Use an alternate key and point it to ~/<username>/.ssh/nx_dsa
    5. Press the X twice and then click the jumpbox connection you see.
    6. Enter your login details.
    7. Click Create a new session
    8. Click Create a new GNOME virtual desktop
    9. You should get a gnome desktop, with a firefox icon.

Using the jump system approach, we’re now able to firewall off these hosts from Internet browsing, we can re-image them on a frequent basis, and we have a very small number of hosts that need to receive Java updates. This approach also allows us to configure our endpoint management software to nuke Java from orbit if the browser plugin ever shows up on employee systems. While we generally abhor single points of failure in our infrastructure, we felt comfortable in this case because the use of configuration management meant that the environment could be quickly rebuilt if needed. Finally, this approach could also be used to isolate other inherently risky technologies when they are still required by certain groups.

In closing, we genuinely hope this helps other organizations facing similar challenges on the road to removing Java browser plugins from the enterprise.

Posted by on March 18, 2013
Category: infrastructure, operations, security

10 Comments

I’m curious, did you also contact the vendor of your network appliances about moving away from the Java plugin as a means of managing their appliance?

    Hi Ted,

    We actually didn’t, for two good reasons:
    1. We were heavily focused on securing the existing environment 🙂
    2. Java is required for the out-of-band console access on HP, Dell, Supermicro and other servers. Hopefully one day we’ll be able to get rid of this requirement for java there all together, but right now it’s pretty embedded (figuratively, and literally in the IPMI hardware on the servers).

[…] a post to its programming and infrastructure blog Monday, e-commerce startup Etsy detailed its new approach to the endless stream of new bugs Java in the […]

@Avleenetsy

If I remember correctly, you can actually SSH to HP’s OOB console access and if needed you can enter the command textcons to get a TTY.

    Indeed you can! That mostly works, but there’re still a bunch of times it’s inconsistent.
    Eg: You need to get into the BIOS to set it to use text mode, and that often still doesn’t work once the OS has booted up.
    And the IPMI systems from other vendors don’t support a TEXTCONS equivalent on their servers. But yes TEXTCONS is nice a lot of times.
    It’s definitely a start! More like this.

How are you handling the nearly-as-numerous Flash vulnerabilities and updates?

[…] Java: Not Even Once […]

[…] de Andy Greenberg alertando de las vulnerabilidades de Java, al mismo tiempo que la solución de la compañía Etsy de tener instalado la JVM en unas pocas de máquinas sobre las cuales todos los accesos son […]

[…] Favorite Post – Java: Not Even Once […]

Just as additional security measure: most modern browsers have “Click to play” for plugins. This also includes flash plugin but it had many security vulns too. Also it increases battery time =)