When talking with Check Point customers, a common request I hear is for the ability to “decrypt” SSH traffic, see inside of said traffic, and make security decisions based on what it finds, including blocking tunneling. In my last post, I explain how the SSH Inspection feature provided by a couple of Check Point’s competitors actually works and why you might want to think twice about using it. If you don’t believe my analysis, check out the following video, which shows how these SSH decryption features can be used against you:

And in case you’re thinking I’m just picking on Palo Alto Networks, Fortinet’s SSH decryption has the same flaw:

There is clearly a need to control what happens with SSH connections. When used by the general user population, SSH is similar to an anonymizer like Tor or UltraSurf which hide the true intent of the encrypted traffic and serve no business purpose. How should you handle this requirement without comrpromising the security of SSH the way SSH Inspection does?

My general stance on allowing SSH still stands: SSH should only be permitted for specific individuals to specific hosts with a clear business need for this kind of access. That said, let’s assume you want to do more than just allow/block SSH access, prevent tunnelling, and log what your users do over SSH without compromizing the security of SSH in the process.

Many years ago, when I worked for Nokia, whenever I was sitting in the office and wanted to SSH out to the Internet, I had to go through an explicit SSH proxy server that operated on the following principle:

  1. You SSHed explicitly to the proxy
  2. You were prompted by the proxy for the desired destination IP and username to use on the remote server
  3. You logged into the remote SSH server

It was not possible to use port forwarding using this configuration. You also got some visual indication that the remote host key changed as the proxy would keep track of that the same way a regular SSH client would. This still didn’t allow authentication with an RSA key, but it is still an improvement.

Nokia’s SSH proxy was something that was home grown. You could do something similar with a jump server that authorized individuals could use to ssh to other hosts with full ability to use RSA keys and verify a remote server’s host key. You can prevent port forwarding and the like by disabling tunneling in the SSH daemon on the jump server. You could also set up whatever type of logging is necessary on this jump server to meet your requirements. The servers that could be connected to from this jump server would be allowed or blocked by your security gateway. Additional logging should be set up on the permitted servers as well.

If setting up a jump server is too much work, you still want to allow users to SSH to random hosts, and you’re just concerned that people might use SSH for things other than an interactive SSH session (e.g. tunneling), a way limit this is to implement QoS on all SSH connections. You would have to choose a limit that provides acceptable performance for interactive terminal sessions but makes tunneling other things very slow. On a Check Point Security Gateway, you can achieve this in the Application Control rulebase using the “limit” Action.

At the end of the day, it’s important to understand what your actual requirements are with respect to controlling and logging SSH traffic so the right controls can be put into place. The methods described herein balance security and usability far better than the SSH Inspection features some security gateways provide.

Disclaimer: While I did get some link love from Check Point, whom I work for, the thoughts herein are my own.

Edited to add video embeds and link to Check Point blog post on 13 Aug 2015

SSH is a wonderful tool for accessing remote systems via a CLI. It is encrypted, if set up properly, I can verify I am talking to the correct server using mutual key exchange and I can tunnel all kinds of stuff over it. If you’re so inclined, you can even use an SSH tunnel as a SOCKS proxy.

And therein lies the problem. SSH represents a potential way to bypass security controls, in much the same way as HTTPS. To mitigate this threat, security gateways can man-in-the-middle HTTPS and SSH to “see” inside the traffic and make further security decisions on it.

Fortinet has a feature called SSH Inspection that performs this man-in-the-middle on SSH. Palo Alto Networks calls their similar feature SSH Decryption. Throughout this post, I am going to refer to the general technology as SSH Inspection but my comments apply to both implementations.

Conceptually, SSH and HTTPS are man-in-the-middled in similar fashions even though the underlying protocols are very different. While SSH Inspection provides more visibility and control, there are some tradeoffs you should be aware of.

First, a brief explanation of what happens to web traffic when HTTPS is man-in-the-middled by a security gateway. For most web (looking) traffic, provided you can easily distribute a new Certificate Authority to client PCs, end users will be none the wiser their HTTPS is being inspected unless they check the “lock” on their browser to see what Certificate Authority signed the key of the remote server.

The security gateway ensures the connection between it and the destination site hasn’t been tampered with by validating the server certificate the same way a regular web browser would. The security gateway can even be configured to disallow connections to sites where this validation cannot take place. This gives a high degree of confidence that even though the security gateway is inspecting the connection, you are ultimately speaking with the legitimate destination server.

There are problems with this approach. If a specific application/site utilizes Certificate Pinning, then the security gateway-generated SSL certificate will be rejected because it is signed by a different Certificate Authority. Likewise, when client certificates are used for authentication, the process will fail through a gateway performing SSL Inspection via man-in-the-middle techniques as there is no way for the security gateway to perform client authentication on behalf of the user.

The good news is that the number of applications and webssites affected by these limitations are few and far between. Specific exceptions for these applications and sites can be included in the relevant SSL Inspection policy and those applications will work as before.

When SSH traffic is similarly inspected man-in-the-middle style, the exact same issues come up with SSH for the exact same reasons with a far greater impact. This is because, unlike HTTPS where client authentication is rare, mutual authentication of client and server is the norm with SSH.

When SSH Inspection is in use, end users are limited to password-based authentication for SSH connections as it is not possible for the security gateway to send the end user’s SSH key. Furthermore, you as an end user have no easy way to verify you are connecting to the correct host as the security gateway presents its own generated SSH host key instead of that of the remote server.

Why is that an issue? SSL/TLS utilizes certificate authorities, meaning some third party vouches for the veracity of a given certificate presented. In the case of man-in-the-middle for SSL, the firewall is, in essence, vouching for the veracity of the certificate as it is signing the certificate, and we trust the firewall, right?

SSH does not use certificate authorities, meaning that if the connection is being man-in-the-middled, we have no idea who is man-in-the-middling the connection, we just know the SSH host key changed. This might be ok if the firewall actually verifies the veracity of the SSH host key presented.

From what I can tell, neither Palo Alto Networks or Fortinet (two companies that have SSH Inspection features) provide a way to verify that a particular SSH host key hasn’t changed. As SSH does not make use of certificate authorities, there is no way to automatically verify that the key was changed legitimately. A human could do the verification if they could see the new host key coming from the remote SSH server. With SSH Inspection enabled, the user will never see that key, which leads to a potentially interesting attack vector:

And in case you’re thinking I’m just picking on Palo Alto Networks, Fortinet’s SSH decryption has the same flaw:

To summarize, when we man-in-the-middle SSH, we can only use password authentication and we lose the ability to verify we are connecting to the desired server. And what security benefits do we get from this degredation of SSH security? The ability to:

  • Prevent Port Forwarding (including “X11 Forwarding)
  • Block the use of ssh “commands” or “shells”
  • Possibly prevent file transfers provided they are done in a way the security gateway knows about (and there are many, many ways to do this over the CLI)
  • Possibly log commands issued over SSH

Personally, I don’t see the value of this SSH Inspection feature as it degregades SSH security unacceptably and provides little benefit above and beyond what you would get by blocking SSH entirely except for specific, trusted individuals accessing specific, known servers. If this is not feasible, I provide some suggestions for inspecting SSH connections securely in another blog post.

Disclaimer: I’m not sure what Check Point Software Technologies thinks about this issue as I didn’t ask. These thoughts are my own.

Edited to add video embed on 13 Aug 2015

From Palo Alto CEO: Beware the Internet of Things – and watch your car:

Meanwhile, corporate network security is already facing stiff challenges that have many experts saying that breaches are inevitable – something [Palo Alto Networks CEO Mark] McLaughlin isn’t willing to concede.

“It’s as if you and I would go home tonight and say to our families, ‘Somebody is going to break into the house, probably every night. They’re going to walk around, they may take stuff, take whatever they want, but they’re coming in any time they want to every day of the week, and there’s really nothing we can do about that, so we just have to be OK with that,’” he says. “Nobody’s OK with that. That’s sort of the equivalent.”

We can argue whether or not breaches are inevitable all day long. The real question is: when a breach happens, will you be ready?

People who espouse the “assume breach” mindset aren’t saying to be OK with it. What they’re saying is to make sure critical assets are protected and that when a breach occurs, the means are present to be alerted to it and contain it.

The reality is, many organizations aren’t adequately prepared for a breach and may not even know they’ve been breached.

What I regularly see in my customer engagements are flat networks beyond the perimeter and DMZ segments with no security controls in place between internal segments. The reflects the mindset that security devices will block 100% of all attacks, not letting anything malicious slip through. Which is a bit optimistic, and does not account for attacks that might not even go through the gateway.

In an “assume breach” mindset, you would design your network a bit differently, isolating user workstations from servers hosting critical applications with an enforcement point in the middle applying a strict access policy. The application servers themselves might be further segmented behind enforcement points of their own to make sure applications don’t unnecessarily talk to one another. The end user machines themselves would also have additional controls on them to provide protection when they are off network (as is often the case with laptops) and to protect data. All of this would be centrally managed all with appropriate logging, reporting, and alerting across the security infrastructure.

With all the enforcement points in place with a well designed policy, breaches, if they ever happen, can be spotted and contained quickly. With the addition of other security controls, further granularity of access control can be achieved.

If you don’t even know what your critical assets are, who should have access to them, have the ability to apply any sort of access policy, and report on activities (allowed or otherwise), you’ve got far bigger issues than any single security tool can solve.

While it’s not always a focus area of my PhoneBoy Speaks podcast, I do occasionally cover Information Security topics on my podcast. It happens often enough that I decided to create a dedicated RSS feed just for these topics: https://phoneboy.com/ps/infosec.xml

If you subscribe to the regular PhoneBoy Speaks feed, you don’t need to subscribe to this one as this is merely a subset of the episodes that appear on my regular feed. This is only for people who are interested only in the Information Security topics I cover and not some of the other stuff I prattle on about.

I know a few articles have been written about this topic already elsewhere. That said, I sometimes will do a blog post so if I need to find something again, I know it will exist on my own blog.

For a while I had been using WhiteHat Aviator, which is a fork of Google Chrome that has many privacy options enabled by default. Since, at the time of this writing, it is currently five major versions behind mainline Chrome, it’s gone Open Source, and there doesn’t appear to be any recent checkins on the GitHub project, and there will likely be some difficulty updating the existing browser, I’m guessing we probably won’t see an update unless someone undertakes a major effort.

With that in mind, I’m looking to try and configure Google Chrome as securely as possible. There is the Google Chrome Privacy Whitepaper and another independently maintained page on Privacy and Security Settings in Chrome. You could also use something like the Disconnect Search plugin. However, none of these pages solve one feature I really enjoyed about Aviator: starting up the browser in Incognito Mode. This does two important things:

  • Doesn’t log website browsing history at all

  • Clears all cookies on browser exit

However, Google Chrome does not offer an easy way to do this, but it can be done in two specific places: The shortcut you use to start Google Chrome on your Windows machine and the Windows registry. 

First, the shortcut. Find it on your computer, right-click on it and select Properties from the pulldown menu. Add a –incognito to the end of the Target field (note, it’s a space, two dashes, then incognito) and click Apply.

This will cause Chrome to start in Incognito Mode when launched from the shortcut. However, this will not cause web links you click on in other apps to also launch into the Incognito Mode. To do that, you need to edit the registry as follows:

  • Open Regedit (From the Start Menu, type regedit)

  • Find the following registry key: HKEY_CLASSES_ROOT\ChromeHTML\shell\open\command

  • Edit the registry string (double click on “(Default)”), adding a –incognito to the value data between the quote and the double hyphen and click Ok.

Congratulations, you’re now permanently incognito in Chrome. Note if you want to use some of your extensions in Chrome, you will need to manually enable them for Incognito Mode by clicking on the hamburger menu in the upper right corner of the browser window (the three vertical lines), selecting Settings, and then clicking on Extensions. Check the “Allow in Incognito” checkbox for each one.

While it’s all fine and good that I can configure this stuff by default, I really wish Google had a one-click option that just enabled all this stuff by default.

To do something similar on the Mac, try the Google Chrome Incognito app. For Linux, it will likely be a distribution specific answer, but this should work for recent versions of Ubuntu.