SSL Offloading

For the first time I actually have an opinion contrary to Michael Farnum, lol.

In his Computerworld article he discusses his concern for certain acceleration devices that decrypt the SSL traffic and inspect it before moving it on to the end target web server. I happen to deploy just such a device and initially had identical concerns. However, when I thought more about it I realized it isn’t that big a deal

Here are Michael’s first concerns:

I also see a HUGE [his emphasis] privacy concern for users. What if there is some personal data being decrypted? What about credit card numbers? While the vendor showed me how these concerns could be reduced (by different policies that stop certain traffic from being decrypted), it still has me worried about abuse by some disgruntled or just plain evil IT guy.

If the device is decrypting sensitive data, the device should be ‘classified’ with the same sensitivity as the web servers ultimately receiving the decrypted data. In that case the security team or the same team that deploys the web servers should be the ones to deploy and manage the accelerator. In other words, who ever already has access to systems that house the decrypted data should be responsible for the accelerator. If they don’t have the expertise to deploy the device, it should fall on the security team.

Second is the issue of a very tempting hacking target for the baddies out there. If this thing is decrypting traffic, it makes sense that the bad guys would want to see what it is decrypting.

Also a valid point. However, any security shop worth their salt won’t permit any connections to the device itself from the public Intarweb. Instead, only the VIPS (virtual IPs) of the websites are accessible from the public and any administration of the accelerator must come from inside the protected LAN. This will greatly reduce the attack surface of the accelerator itself. Also, the more likely target will be the web server, as I imagine its hard to 1) determine there is indeed an accelerator there in the first place and 2) what make and model the accelerator is and 3) why attack an intermediary host when you can probably more easily attack the end host?

But it also means that the device has the keys to the kingdom. While it is arguably easier to lock down a device than a server, it still gives the bad guy a target to focus on

Either way the bad guys have a target and either way the decrypting SSL cert is likely on a host exposed to the Internet, which is why anything exposed needs to be hardened and well-protected.

Not only have you given the attacker another place to focus, you have also given him MORE choices for his attack.

This is exactly why you don’t allow access to the device itself from the public Internet. You only permit access to the VIPS (virtual IPs), which significantly reduces the attack surface. Also, since the accelerator is more than likely a *nix flavor, you’ve probably doubled the knowledge requirement for the hacker, not to mention the fact that most attempts to discern the OS of the appliance will likely reveal the OS of the web server on the back end (if they are attacking the VIPS).

Also, a very, VERY cool feature of some of these devices is the ability to replace strings of text with something else. For example, the device can take the string “cmd.exe” and replace it with “cmds.exe”. Tasty! Some of these devices also perform basic firewall and NAT functions as well, adding another hurdle for the bad guys to jump.

[these devices] are quite often being adopted by the network team. If the company does it right, the security team is going to have to sign off on this puppy. But that is too often not the case, and this is a huge concern because these devices can go in without a true test of their claims of being secure. I am not saying network people are stupid. I am saying that their main concern is not usually security (though everyone should be concerned with security first, but I digress).

Well said. I’m wrangling with a vendor right now because I’m trying to convince network teams and other service teams to do things securely but we’re having problems compromising, namely because network guys are packet movers and security guys are packet filterers.

Alan Shimel commented on the article by saying “The SSL data was supposed to be encrypted all the way to the real “end.” I have a semantic argument for that. Wherever the data owner wants the data decrypted is the real end. Alan goes on to say “When it is decrypted before that, you are introducing risk into the equation.” That is correct but its a “Captain Obvious” statement. A firewall itself introduces risk because its also an attackable device. The art of security, however, is to manage risk and weight the risk against the reward and that isn’t strictly security’s job, but also the job of the data owner, asset owners, etc. There are times when you have to accept a risk. In fact by placing a web server on the Internet, you’ve accepted the risk of a web-based attack against that server, though hopefully you have well placed mitigators.

When managed well these devices can actually increase the security of the web farm based on the ‘replace text’ feature, high availability feature, and the other points I’ve made above.

One more point: Alan used the word ‘hijack’ on his blog in response to Michael’s article. Again, I have a semantic argument that it isn’t a hijack if the data owner approves it and provides the SSL cert for decryption. Also, Alan mentions transmitting in clear text from the accelerator to the web server. While that is a correct statement, the web farm is going to be in the same data center as the accelerator and there will likely be only one, maybe two switches between the two devices. If someone can sniff that traffic, you have physical security problems.

For more information about this topic

I tend to agree with you. I think it might be important to make sure readers (this might be more for Farnum than your article) are aware which direction SSL decryption is dealing with: egress to external connections or ingress to web servers you own.

You can terminate SSL at the hardware device. In fact, that’s the whole point of it. You’re not only offloading the SSL decryption/encryption burden from the web server (typically the most intensive native actions on a web server) but you’re also saving money in certificate licensing. One cert is oftimes needed per web server…or per hardware SSL terminator. So one load-balancer or SSL terminator for 10 web servers still just means 1 for-pay cert.

And, yes, while Alan has a point that you now have unencrypted communication between device and web servers, you can also encrypt that communication using your own internal self-signed certs if you want. In fact, that’s what we do on our NLB (of course, we lose the gains we made on web server load, but we’re ok with that).

You’re absolutely correct. Even if that internal traffic is unencrypted from device to web servers, if someone is able to sniff that, you have other more important problems.

On a side note, even if you hack an SSL terminating device, I’d be curious if anyone has yet been able to hijack that device in a way that will leak the unencrypted information inside the device (i.e. before it hits the wire on the internal side). I’d be impressed if someone has done that… (And you’re right to tackle that by saying it’s not much different than hijacking the web server anyway…)