Security Software Certification

I’ve been working on cleaning up some of the low hanging vulnerabilities in major security products lately. For the last two weeks, I’ve been looking at Comodo Antivirus, and have developed a reasonable understanding of how the product works and what it does. Just installing the product, I immediately noticed a few simple problems. Things like:

• The default installation included and enabled a VNC server with weak authentication.
• The browser installed by default disabled the same origin policy.
• The scanning process didn’t enable ASLR.
• Incorrect ACL’s used throughout the product.

These issues are mostly fixed now, Comodo have been responsive and are taking all of these reports seriously. Identifying these problems didn’t require any skill, there are point-and-click tools that can identify some of these problems. Then, using techniques familiar to most security professionals, I went on to find critical memory corruption flaws.

After that, I used reverse engineering to find even more serious design flaws and logic errors. Comodo aren’t alone here, it’s the same story for all the major security software vendors.

Unfortunately, I’m not doing a thorough job.  I don't have access to source code, developer documentation, symbols,  and I can’t ask the developers questions about their code. A competent security consultancy bought on-site by the vendor could do a much better job.

Still, I’m trying to clean up some of the low hanging fruit that is endangering billions of users worldwide. I don’t think the antivirus industry is going to make even a token effort at resolving these issues unless their hand is forced.

While I've been working on this audit, triaging/analyzing hundreds of unique bugs, and writing vulnerability reports, I noticed that Comodo were busy working on certification from Verizon. This certification process isn’t free, and products have to follow guidelines from the certifier. It doesn’t take a genius to figure out that the testing methodology was most likely installing Comodo, then scanning a thumbdrive full of malware - but the fact is that vendors are willing to pay to make that happen and implement the guidelines they publish.

Verizon actually publish their methodology, and it’s about as ridiculous as you would expect. Requirements for “excellence” include “The Certification Candidate must include Administrative Functions to Enable and disable the Detection of Malware”, and “The Certification Candidate must demonstrate through On-Demand testing that it Detects Malware”.

These are the meaningless tests that antivirus vendors will actually scramble to pass. Perhaps the first step in improving the situation throughout the industry is making sure these certifications actually test something worthwhile.

There’s no need to reinvent the wheel here, why don’t these tests simply integrate some part of Microsoft’s SDL? Many of these tests can be automated, but their results would actually be useful. For example, "Product survives 24 hours of fuzzing with zero exceptions", "All processes and modules use minimum standards for mitigations", "An attack surface analyzer report identifies no new issues", and so on. Award bonus points in some ranking for using sandboxing, and maybe we'll see the first vendor actually implement that.

Something has to change soon. The next slammer or codered isn’t going to target IIS or MSSQL: the security of Microsoft products is in a different universe than it was a decade ago. All of the major security vendors are using ancient codebases with no awareness of modern security practices, it’s still hacking like it’s 1999.