McAfee has been forced to backtrack on claims that one of its products was misrepresented in a test of anti-phishing toolbars.

According to McAfee, last month's Microsoft-sponsored report by researchers 3Sharp, which rated the software as poor at detecting phishing websites, was unfair because the version of SiteAdvisor assessed had never been designed to perform this function.

The company also said 3Sharp had refused to remove SiteAdvisor from the study, despite its requests to do so, resulting in the product receiving an embarrassingly low score of only 3 out of a possible 200.

At the time of the tests, SiteAdvisor was described on the company website as having phishing as one of its features. It also had a degree of anti-phishing capability before the company was acquired by McAfee in April this year. But it now appears that McAfee quietly removed or scaled back this capability without telling the world, generating confusion over its abilities.

More recently, and not entirely coincidentally, McAfee launched a premium version of the software, SiteAdvisor Plus for US$24.99, which makes explicit claims to spot and block websites suspected of carrying out phishing. This has yet to be tested.

In the disputed study, Gone Phishing: Evaluating Anti-Phishing Tools for Window, 3Sharp tested the software against six other security toolbars from Internet Explorer, Mozilla, Netscape, eBay, Earthlink, GeoTrust, Google and Netcraft. Contentiously, in a report sponsored by Microsoft, top marks in the test went to Internet Explorer 7.0's anti-phishing capabilities, leaving SiteAdvisor at the bottom of the group.

In a blog post on the topic, Paul Robichaux of 3Sharp justified the inclusion of SiteAdvisor despite its awful performance by claiming the McAfee website mentioned the word "phishing" in a list of the product's features. McAfee's Shane Keats offered McAfee's position in his own blog on the same date, where he set out the company's unhappiness in detail.

Keats now admits that the website was changed to remove the word "phishing" after the 3Sharp test when it was realized that an old FAQ, dating from the days before McAfee acquired SiteAdvisor, had been left unchanged apparently in error. Prior to its inclusion in the McAfee product line, SiteAdvisor had featured an unspecified degree of anti-phishing protection, he said.

Why McAfee removed the anti-phishing may well be explained by the subsequent release of the paid-for version, which includes anti-phishing. Why it didn't tell anyone at the time is open to speculation.

Last week, a second study sponsored by Mozilla came to a slightly different conclusion from the 3Sharp analysis, rating Mozilla's own anti-phishing capabilities above those of Internet Explorer 7.0. No mention was made of SiteAdvisor.

A third, independent study from Carnegie Mellon, published this week, did test SiteAdvisor and also rated it as having zero anti-phishing abilities in a field where all products generated mediocre scores. However, the confusion as to the product's status and features clearly extended to this entirely separate team of researchers too.