A blog post written by Greg Ferro, a.k.a. EtherealMind, had come to my attention on a review of ClusterXL in Check Point R77 based on the documentation. I'm familiar with Ferro from the Packet Pushers Podcast where he and Ethan Banks talk nerdy about networking. They occasionally dip into the security arena and when Check Point does come up, it's not in a positive light.

When I actually read Ferro's post about ClusterXL, I was not surprised that the end result was a negative review: not only of ClusterXL but of Check Point itself. What surprised me was how incomplete the review was, technically speaking, because I know how nerdy he can get.

Ferro only used one document for his paper review: the Check Point R77 Versions ClusterXL Admin Guide dated 28 July 2014. He could have easily asked the customer he supposedly did this review for to obtain additional documentation, such as the ClusterXL Advanced Technical Reference Guide in sk93306. Because he didn't have direct access to said documentation as an independent consultant, he chose not to use it.

What is ClusterXL anyway?

Even with the documentation Ferro chose to use, the more I read his Tech Notes Check Point Firewall ClusterXL in 2014 piece, the more I conclude his review was not very thorough. Clearly he read some of the documentation as there are quotes from it throughout the piece, but it's obvious he missed the part that defined what ClusterXL actually is:

ClusterXL is a Check Point software-based cluster solution for Security Gateway redundancy and Load Sharing. A ClusterXL Security Cluster contains identical Check Point Security Gateways.

• A High Availability Security Cluster ensures Security Gateway and VPN connection redundancy by providing transparent failover to a backup Security Gateway in the event of failure.
• A Load Sharing Security Cluster provides reliability and also increases performance, as all members are active

Ferro's definition of ClusterXL?

There seems to be two modes of active/active High Availability for CheckPoint Firewall one. Both of which are called ClusterXL.

Ferro then talks about the requirements for ClusterXL, which he gets mostly right: sync is a Layer 2 protocol, ClusterXL requires appropriate licenses (which every current Check Point appliance has, as do many current software-only firewall SKUs), and you should use NTP between members. Actually NTP is a good idea for reasons other than state sync (e.g. VPNs and SSL).

State Synchronization

Then Ferro comments on State Synchronization.

• by defaults all connection state is synchronised across the cluster.
• you can decide not to synchronise. Why ? Does this imply a performance issue in the devices, network, speed or other issues ? What would I not sync all state ?
• “Synchronization incurs a performance cost” – implies that CheckPoint performance remains a problem for “many customers

For the vast majority of Check Point customers, state sync does not have performance issues. Where issues have been observed is in environments where the connection rate is far higher than the throughput would suggest. You can reduce the load by disabling sync for services with connections that are transient in nature (e.g. DNS, short-lived HTTP connections) and can be done easily in SmartDashboard.

But that's not all he has to say about State Synchronization:

• Synchronisation Network requires elevated security profile, suggests CCP is insecure protocol, isolated network.
• Recommends using an isolated switch – this is stupidly impractical who the heck wants to waste money and power on a dedicated switch in the DMZ.
• Using a crossover cable is equally stupid and impractical. What are they thinking ?
• Appears that sync is ONLY supported on the lowest VLAN ID on an interface.

The Synchronization Network requires an elevated security profile: yes, this is stated in the documentation and is no different than every other major vendor. A dedicated switch or a crossover cable allows this elevated security profile to be maintained better than a dedicated VLAN, which by the way, if you want to use a dedicated VLAN, yes, this is supported; it's a common configuration in customer networks. As to whether a dedicated switch or crossover cable is "stupidly impractical," customers have asked for support for all of these options, which is why they are tested and supported options (and thus listed in the documentation as such).

Ferro's next complaint? You can't use more than one synchronization interface. This actually used to be a supported configuration once upon a time but in recent versions, the supported approach is to let the operating system handle interface redundancy via bonded interfaces (i.e. with LACP). I asked in the comments of the post why he felt using LACP was an inferior solution to multiple independent synchronization interfaces. No response.

ClusterXL Load Sharing

Ferro never once explicitly mentions the sort of configuration he was using as the basis for his paper review of ClusterXL. Based on the fact he only briefly mentions High Availability (or Active/Passive, as he said) and spends the majority of his time talking about Load Sharing (Active/Active), I can only assume that is the configuration he's most interested in, ignoring the fact that High Availability configurations are the far more common customer configuration.

Of the two ClusterXL modes that relate to Load Sharing, Ferro only talks about one: Load Sharing Multicast. This mode, according to Ferro, "should be avoided at all costs" because it involves implementing workarounds that are, in his words, "complex and hard to maintain/operate." These workarounds, static multicast mappings and/or disabling IGMP, are necessary because some networking equipment out there does not properly support RFCs or has issues with frame replication (needed for Multicast). 

ClusterXL has a mode that requires none of these workarounds: Load Sharing Unicast mode. This mode was only mentioned once, but no analysis of this mode was offered by Ferro.

Ferro's opinion about VPN and Load Sharing is that "there are many conditional statements about VPN connections" and "would have little confidence in running Load Sharing Multicast Mode and running VPN services on the same physical unit." I couldn't find the conditional statements he was referring to, but what I did find was one particular statement in relation to third party peers and load sharing (repeated in a couple of different ways):

The third-party peers, lacking the ability to store more than one set of SAs, cannot negotiate a VPN tunnel with multiple cluster members, and therefore the cluster member cannot complete the routing transaction.

The problem isn't Check Point, per-se, the problem is with interoperating with non-Check Point devices. The workaround: using Sticky Decision Function, which is listed in the documentation where this "conditional" statement is listed. This, along with the other so-called conditional statements he refers to are curiously absent from his paper review.

Performance and Cost

Ferro writes in several digs at Check Point's allegedly "punishingly expensive" products. In most of the competitive situations I've been engaged with during my tenure at Check Point, pricing compared to the competition is rarely a factor. That said, I have seen numerous situations where competitors have purposely specified lower-end (and thus cheaper) solutions that, in reality, will not meet the customer's stated requirements while maintaining the same security posture.

Ferro also states "forwarding performance is and price/performance is very poor" which is provably false. Independent third parties have verified Check Point's performance and pricing and have found Check Point in-line with the competition. It's also something being validated internally at Check Point on an ongoing basis.

Ferro is correct that performance decreases when additional security features are enabled, however this phenomenon is by no means unique to Check Point products. The expected performance for a given Check Point appliance is listed right on the data sheet along with the unrealistic "lab" numbers every networking vendor since the dawn of time has included on their datasheets. Unlike Check Point's competitors, the "real world" performance is evaluated with real-world traffic flows, multiple software blades (security functions), a real-world rulebase, and logging/NAT enabled. Your local Check Point account team can provide more refined appliance recommendations based on your specific requirements.

In Conclusion

As noted before, it's not really clear what use cases Ferro was evaluating ClusterXL against. Even his conclusion is non-specific about this:

Check Point doesn’t have strong solution for clustering and the weak documentation suggests that it’s not fit for critical use cases.

Perhaps Ferro does have a use case that ClusterXL would not be a good fit for. I'd love to know what those "critical use cases" are in more detail so his conclusion could be evaluated more thoroughly. 

Ferro's conclusion is based on an incomplete review of the available information. Only one document was reviewed when others exist, and I question how thoroughly even the one document was reviewed. Only one of the load sharing modes of ClusterXL was reviewed in any detail and he completely ignores the other mode as well as high availability configurations. Ferro also ignores the fact that practically all of Check Point's 100,000+ customers use ClusterXL successfully in their networks, which includes 100% of the Fortune 100 and Global 100 customers across many different industry verticals.

In short: Ferro's review of ClusterXL is incomplete and his conclusion is not supported by the facts.

Disclaimer: I work for Check Point Software Technologies. That said, the above are my own thoughts.

I decided that I'm going to let the phoneboy.net URL expire when it comes up for renewal in a couple of months. As a result of that, the URL for this blog (such that it is) will change to http://securitytheater.phoneboy.com (*Edit*: Now it's http://phoneboy.org, but the securitytheater URLs will redirect) Please update your RSS readers, bookmarks, and the like.

Back when I first got into IT and just started working with FireWall-1, Pointcast was a thing. For those who weren't around back in the mid to late 1990s, Pointcast had a very popular screensaver that displayed news and other information delivered periodically over the Internet to PCs. The problem was: it used an excessive amount of bandwidth on corporate networks, especially if more than a couple of people used it.

The result was, of course, corporations wanted to block access to Pointcast. The problem: how to do it. All we had in the mid 1990s was the traditional firewall which could control access based on IP and port. So we should be able to block the port or IPs it communicates with, right?

Pointcast used good old HTTP. Even back then, no one in their right mind would block HTTP. Of course, everything uses HTTP or HTTPS to communicate these days, and with a traditional firewall with the ability to control traffic only by IP or port, leaving HTTP or HTTPS wide open is tantamount to leaving the barn door open. 

Pointcast didn't exactly publish their list of servers, but users of the PhoneBoy FireWall-1 FAQ contributed a list of IPs plus a couple of other clever solutions to the problem, which I've made available after the break if you're curious.

Of course, with things like content delivery networks, Amazon Web Services, and a host of other ways to serve up an application to users that are available today, attempting to control access to these applications merely by port and IP address is crazy. 

Fortunately, there are a number of solutions to this problem. Check Point's solution is the Application Control Software Blade, which can allow/block access to an application regardless of the ports and destination IP users, and even limit the bandwidth these applications use. New applications or changes to existing applications are made available to the gateway periodically so you can see that you're users are using it and, when it kills you bandwidth or worse, you can block it. 

If only tools like App Control were available back in the day, security admins could have spent more time on more important issues rather than figuring out how to block Pointcast and other applications and I would have a few less FAQ entries on "how do I block X application."

There are a few ways to block access to Pointsec:

1. Deny HTTP Access to Pointcast Servers
2. Use the HTTP Security Server
3. Create a Dummy Host in your DNS/WINS

Deny HTTP Access to Pointcast Servers

To deny HTTP requests to the Pointcast HTTP server, deny access to the following machines:

205.228.184.31 through 205.228.184.60, inclusive. 
206.64.127.31 through 206.64.127.60, inclusive.

To minimize the number of network objects needed (since range objects aren't supported), create the objects as follows and put them into a group:

Create host 205.228.184.31 
Create network 205.228.184.32 with subnet mask 255.255.255.240 (include broadcast) 
Create network 205.228.184.48 with subnet mask 255.255.255.248 (include broadcast) 
Create network 205.228.184.56 with subnet mask 255.255.255.252 (include broadcast) 
Create host 205.228.184.60

Create host 206.64.127.31 
Create network 206.64.127.32 with subnet mask 255.255.255.240 (include broadcast) 
Create network 206.64.127.48 with subnet mask 255.255.255.248 (include broadcast) 
Create network 206.64.127.56 with subnet mask 255.255.255.252 (include broadcast) 
Create host 206.64.127.60

Deny HTTP traffic to these hosts.

Using HTTP Security Server

Thanks to Daniel Blander for this idea:

Create a URI resource that filters the following URLs:

http://*/FIDO*/*

This roughly translates to creating a Wildcard URI Resource with the following parameters:

Service: http 
Action: all 
Host: * 
Path: /FIDO* 
Query: *

You will want to use this URI resource in a rule that denies access.

Create a Dummy Host in your DNS/WINS

Thanks to Mark Syroka for this idea.

Create an entry in your DNS or WINS for the hostname PCNPROXY. Your clients will try and access whatever host resolves to this name if it exists. If you wish to use the PointCast Caching Manager, which is designed to Cache PointCast Requests and is available for free from http://www.pointcast.com/products/intranet/, your DNS/WINS entry would point to this machine. Otherwise, this entry can point to a non-existant machine or any machine that does not run a web server on port 80.

From Chris Hoff's (a.k.a. Beaker) NGFW = No Good For Workloads:

NGFW, as defined, is a campus and branch solution. Campus and Branch NGFW solves the “inside-out” problem — applying policy from a number of known/identified users on the “inside” to a potentially infinite number of applications and services “outside” the firewall, generally connected to the Internet. They function generally as forward proxies with various network insertion strategies.

If you look at the functionality Check Point and its various competitors provide, this is precisely what a large chunk of the "next generation" functionality is geared towards--protecting a number of known/identified users from the dangers they might encounter from a potentially infinite number of application and services. There are differences in how the different security solutions perform this task, as well as how well they perform, but that's their overall goal.

That is, as Beaker continues, very different from what a Data Center firewall needs to do:

Data Center NGFW is the inverse of the “inside-out” problem.  They solve the “outside-in” problem; applying policy from a potentially infinite number of unknown (or potentially unknown) users/clients on the “outside” to a nominally diminutive number of well-known applications and services “inside” the firewall that are exposed generally to the Internet.  They function generally as reverse proxies with various network insertion strategies.

In other words, we're not always sure who is coming in, but we know what they are going to and (hopefully) what applications and services they are going to connect to. 

What kinds of protection do you need in these scenarios? Usually very different. Can every next generation firewall provide just the right protection? 

First, let's take a step back and realize that the Data Center itself is very different from what it used to be a decade or two ago. Whereas we started with a number of servers hosting resources in one or two physical locations with users mostly in known physical locations, we now potentially have services, data, and users all over the place, with a mix of physical and virtual servers where traditional methods of segmentation and protection are not practical. 

The "core" of the enterprise network--where all the necessary resources ultimately connect together--is quickly becoming the Internet itself. How do you protect your resources in this reality?

We go back to one of the fundamental tenets of information security, our old friend segmentation. This means grouping together resources with like function and like information confidentiality levels, placing a enforcement point at the ingress/egress point where you can enforce the appropriate access control policy. The goal for that enforcement point? Let the authorized stuff in and keep the unauthorized and bad stuff out. 

Of course with virtualization, end user PCs, and mobile devices, the boundaries become more difficult to apply but with virtualized security solutions, integrated endpoint security on the end user PCs, trusted channels (VPNs), and secure containers on mobile devices, more is possible than you think. Check Point and other companies have various solutions for this. 

Once the network is segmented and enforcement points are in place, then you can decide what protections and policies should be applied. In some cases, like on User Segments, you want lots of protection as users could go anywhere on the Internet and unknowingly bring in some malware to run amok in your network or send company secrets to their Gmail account. For your data center? Maybe you just want to make sure authorized users can reach specific applications and you want to sanity check the traffic to make sure it's not malicious. Or maybe you just need a simple port-based firewall with low latency for a given app.

The idea of putting a firewall as the core of your network--especially a next generation one-- is silly, as Beaker rightfully points out. Really, your core should be a transit network with enforcement points--those things we typically call firewalls--at the ingress point of the various network segments. This way, just the right policy and just the right protections can be applied without applying them to traffic that doesn't need it. 

This is where I think Check Point's portfolio shines. In the Security Gateway space, the Software Blades architecture is flexible enough to allow you to be very granular about what protections are applied to a specific enforcement point, whether a physical gateway, or a virtual one either in a Check Point chassis (e.g. VSX) or in a VMware or Amazon Web Services environment. This means you can scan a random MS Word document from the Internet for malware on one gateway close to the users while not impeding the flow of traffic in and out of your Data Center that flows through a different Security Gateway. And yes, if you have a 5 microsecond transmission requirement, Check Point has a solution for that with the Security Acceleration Module in the 21000 series of appliances. 

Does an NGFW solve every problem? No, and anyone that tells you it will is flat out wrong. It's not always the right tool for the job, as Beaker points out:

Show me how a forward-proxy optimized [Campus & Branch] NGFW deals with a DDoS attack (assuming the pipe isn’t flooded in the first place.)  Show me how a forward-proxy optimized C&B NGFW deals with application level attacks manipulating business logic and webapp attack vectors across known-good or unknown inputs.

While an Enforcement Point needs to be hardened for DDoS--especially if it is exposed to the Internet--no Enforcement Point is going to completely mitigate a DDoS. There are a number of mitigation strategies that include on-premise DDoS-specific appliances as well as external services, which I know Check Point has advised customers to utilize in various scenarios as part of their Incident Response Services.

Likewise, business logic and webapp attack vectors are outside of the wheelhouse of all NGFWs. You still need to properly secure your web applications, even with an NGFW in place. In addition, there are dedicated, Web Application Firewalls for this purpose and if you've properly segmented your network, you can make sure only those resources are protected by them.

At the end of the day, a Next Generation Firewall, whether it is from Check Point or someone else, is not a panacea. It can be a powerful tool, but like all tools, it needs to be applied properly as part of a comprehensive security strategy that begins with proper segmentation and a well-defined policy. From there, you can apply just the right protection to just the right resources.

Disclaimer: It should be obvious from my last post I work for Check Point, but this is my own opinion.