Microsoft’s antivirus technologies—used in the consumer-oriented Microsoft Security Essentials (MSE) and business-focused Forefront Endpoint Protection products—have pretty much always performed poorly in formal AV tests. But after yet another drubbing, this time by AV-Test, the software giant is finally pushing back. And for good cause: Those tests prove nothing.
In its most recent test, AV-Test refused to grant MSE and Forefront Endpoint Protection its vaunted AV-Test Certified status because the products didn’t detect 28 of its zero-day malware samples, and 9 percent of its recent malware samples. It sounds damning. But Microsoft says these tests are flawed and do nothing to show the real-world effectiveness of its antivirus products.
“We take the protection of our customers very seriously, and the investments we make to do these reviews is an example of that commitment,” Microsoft’s Joe Blackbird writes in a post to the Malware Protection Center blog. “It is difficult for independent antimalware testing organizations to devise tests that are consistent with the real-world conditions that customers live in … We prioritize our work on customer impact.”
How far off are AV-Test’s results?
According to Microsoft, although AV-Test’s results indicated that Microsoft’s antivirus products detected only 72 percent of all “zero-day malware,” Microsoft knows from its telemetry data—from hundreds of millions of systems around the world—that fully 99.997 percent of its customers hit with any zero-day attack did not in fact encounter the malware samples tested in this test (basically a 100 percent success rate in the real world). AV-Test’s sample size was just 100 pieces of malware.
Related: Anti-Virus Vendors Prepare for War with Microsoft ... Again
Furthermore, though AV-Test’s results indicated that Microsoft’s antivirus products missed 9 percent of “recent malware,” the firm knows from telemetry that 94 percent of these missed malware samples were never encountered by any customers in the real world. So, in the real world, it’s possible that Microsoft antivirus products missed 9 percent of 6 percent of recent malware, although Microsoft doesn’t admit to that.
According to AV-Test, Microsoft scored just 1.5 out of a possible 6.0 for the protection component of its test, and it scored fully 0 out of 1.5 in both zero-day malware and recent malware protection. But according to Microsoft’s usage data, the missed samples affected just 0.003 percent of customers in the real world. Looking past telemetry, the firm used “retrospective analysis” to determine whether any customers had actually encountered malicious files it had missed. And it found that only 2 percent of these files existed across 0.003 percent of customers.
The conclusion: “The other 94 percent of the [AV-Test] samples don't represent what customers encounter,” Mr. Blackbird writes. “When we explicitly looked for these files, we could not find them on our customers’ machines … [But] we're committed to reducing our 0.0033 percent margin to zero.”
The message here is simple. You can conduct tests that prove almost anything. But in the real world, Microsoft’s MSE and Forefront Endpoint Protection products don’t just work, they work very, very well. And this isn’t based on anecdotal data—though this certainly mirrors my own usage and explains why I continue to recommend these products—it’s a fact.
See also, "Q: Where can I download real viruses to test my antivirus solutions?"
