I think the results of the test are even more important than they seem, considering previous reports that Microsoft plans to make Smartscreen a base part of Windows 8. This would extend parts of the protection to any executable hitting the file system. This would be big news.
Smartscreen in IE9 has 2 components: A URL reputation system and a file reputation system. The URL reputation system is similar in concept to the Google Safe Browsing API, used by Chrome, Firefox and Safari, but vastly superior in results. It picked up 92 percent of malware-serving sites. Safe Browsing never reached 30 percent in the tests and generally settled much lower.
For the 8 percent of sites that Smartscreen doesn't block, there's backup protection. Smartscreen tracks downloaded files (presumably by some hash like SHA-1) and a reputation for them. If the file is known to be good, it goes through. If it's known to be bad, it's blocked. If the system doesn't recognize it, the file throws up a warning:
This warning could be a bit clearer at the cost of brevity, but I think it's worth it: “Microsoft has not yet encountered this file. If you know this file is new and unusual and know that it is safe, you may proceed. If it doesn't make sense that Microsoft has not yet seen this file, you may wish not to execute it in the interests of your own safety.” I hope Microsoft submits such files to VirusTotal or some such service in order to share them with the rest of the AV community.
So back to Windows 8: At least some betas have included indications that this version of Windows will apply Smartscreen to any file, or at least any executable, that hits the file system. This would address one misplaced criticism in Smartscreen in IE9, that it only protects against the web vector. Of course, the web is how the vast majority of malware is distributed these days, but fix that route and attackers will move elsewhere, so Microsoft has to think ahead.
I've argued that Microsoft should open up Smartscreen to other apps the way Google opened up the Safe Browsing API; Firefox was using it long before there was a Google Chrome. But putting the system into Windows itself may make that less advantageous.
Another thing that Smartscreen doesn't do is protect against application vulnerabilities. If a site is not blocked and it exploits some browser vulnerability, Smartscreen doesn't block it. Of course if you're Microsoft you should patch the browser, and there are plenty of other defense-in-depth techniques, like ASLR and DEP, to limit the damage of vulnerabilities. I'd argue that Smartscreen plus timely patching is really good protection, even without an AV product.
We're always hearing about the coming obsolescence of antivirus software. Could this be it? A Win8 Smartscreen as I see it doesn't cover everything an anti-malware product should. For instance, if you're offline and copy a file in via a USB drive would you be at all protected? I don't know. It's getting there though.