Resisting Comparison

​From Sophos Blog: Thoughts on Comparative Testing

Cylance itself has acquired access to many other vendors’ products, including Sophos, and has been using them in its own competitive testing in public demos, in violation of end user licenses. In fact, Cylance just renewed its licenses for Sophos products through one of our partners. When Cylance acquires our software we don’t threaten the reseller. Note that despite our efforts, to date, Cylance has been unwilling to allow us to license its products.

As long as there has been a marketplace for products, vendors of products have always sought to acquire the competition’s products to understand if they are better and how. Likewise, third party analyst firms acquire products from a number of vendors in a space to compare and contrast them. No matter what vendors might try to do, including End User License Agreements to restrict product uses, these activities will continue unabated.

It seems silly to me that organizations deploy products to protect their critical assets without doing due diligence to make sure the products do what their marketing claims it does. That said, information security departments in companies of all sizes are understaffed and barely have the time to operate the tools they have, much less evaluate the efficacy of new tools.

A quality information security product should stand up to reputable third party scrutiny. Even if you don’t do a direct comparison yourself, there are plenty of analyst firms who do these sorts of comparative evaluations and publish their results (usually for a fee). While it’s impossible for vendors to participate in all third party testing and not all third party evaluations are created equal, a dearth of third party evaluations for a particular vendor’s products should be a huge red flag.

The one sort of scrutiny that no vendor can ignore is the scrutiny of the bad guys. They are guaranteed to find the product flaws you didn’t find in testing or didn’t find documented in the third party evaluation reports that you didn’t read.

Disclaimer: My employer, Check Point, also recently called out a competitor on their marketing claims. These views, however, are my own.