An evaluation of 10 Anti-phishing tool bars.

In a recent study of all the major Anti-phishing tool bars conducted by the Carnegie Mellon University, they analysed the following 10 Anti-phishing tool bars:


1.Cloudmark Anti-Fraud Toolbar
2.EarthLink Toolbar
3.eBay Toolbar
4.GeoTrust TrustWatch Toolbar
5.Google Safe Browsing
6.McAfee SiteAdvisor
7.Microsoft Phishing Filter in Windows Internet Explorer 7
8.Netcraft Anti-Phishing Toolbar
9.Netscape Browser 8.1
10.SpoofGuard

The study was conducted by using verified phishing URLs and legitimate URLs to test the effectiveness of 10 popular antiphishing toolbars. And according to the study the Anti-phishing tool bars that have been studied leaves a lot to be desired.

Let me break up the study and give you the main points:

• SpoofGuard did a very good job at identifying fraudulent sites.At the same time, SpoofGuard incorrectly identified 38% of the legitimate URLs as phishing URLs. It would seem that such inaccuracies might nullify the benefits SpoofGuard offers in identifying phishing sites.

• The only toolbar tested that is known to make NO use of blacklists was SpoofGuard. While it was able to identify the majority of phishing sites using only heuristics, it still missed some phishing sites and it had a very high false positive rate.

• SpoofGuard could potentially be improved through the use of a whitelist, which would prevent the problems that occurred when phishing sites were visited before their corresponding legitimate sites. The whitelist would not necessarily need to be extremely large or updated frequently to be effective.

• The study found that three of the 10 toolbars, SpoofGuard, EarthLink and Netcraft, were able to identify over 75% of the phishing sites tested.

• EarthLink, Google, Netcraft, Cloudmark, and Internet Explorer 7 identified most fraudulent sites correctly and had few, if any, false positives, but they still missed more than 15% of fraudulent sites.

• The TrustWatch, eBay, and Netscape 8 toolbars could correctly identify less than half the fraudulent sites.

• McAfee SiteAdvisor did not correctly identify any fraudulent sites.

• The 10 toolbars that the researchers examined used a variety of methods for identifying fraudulent sites; however, it was able to exploit vulnerabilities in most of them.

• The experiments also suggest that there is no single technique that will always outperform others for identifying phishing web sites.

• Most of the tools tested used blacklists, but only half of them were able to identify the majority of phishing web sites.

• The researchers don’t know the size of the blacklists used by each toolbar, nor do they know what heuristics are used by any of the toolbars other than SpoofGuard.

• The researchers suspect that the toolbars that performed best use larger and more frequently updated black lists. They may also use heuristics that allow them to detect phishing sites that haven’t yet been put on the blacklist.

• Relying solely on heuristics requires that the software is designed with the foresight to prevent circumvention. In this study we were able to exploit both techniques, which leads us to believe that a combination of techniques is necessary

• The success of a blacklist relies on massive amounts of data being collected at frequent intervals.

• The study says that much more work needs to be done in this area from a technical standpoint. Yet even if it is possible to create a technically sound antiphishingtoolbar, it is still unclear as to whether or not this would be beneficial to users. Usability problemsplague all varieties of software, security software in particular. When using an anti-phishing toolbar, poorusability could mean the difference between correctly steering someone away from a phishing site and having them ignore the warnings only to become a victim of identity theft. Thus, we plan to further examine both the technical aspects of this domain as well as the human factors.

My conclusion:

Ok, after going through these facts I come to the conclusion that among the 10 tool bars tested EarthLink, Google, Netcraft, Cloudmark, and Internet Explorer 7 are the best among the 10 tested since they hardly gave any false positives and they identified 75% of the sites correctly. Spoofguard is also good but since it relies solely on heuristics it gave a lot of false positives. The other three tools viz:- TrustWatch, Ebay, and netscape 8.1, performed poorly. And am also so surprised at the fact that Mcafee's Site Advisor did not correctly identify any fraudulent sites.

Now I am waiting to see a study which includes Firefox 2 Anti-phishing and Opera 9.1 Real-time Anti- fraud technique. BUt firefox also use Google's Anti-phishing database and Opera uses database from GeoTrust. So may be that is why they were not included in the study since they included Google tool bar and GeoTrust's tool bar in the study. And if you look at the study while Google's Anti-phishing tool bar performed well, GeoTrust's TrustWatch performed badly. So if I go by the study it means among the popular three browsers, Viz:- Internet Explorer 7, Firefox 2 and Opera 9.1, GeoTrust which is used by Opera performed badly while the Anti-phishing tools in the other two browsers performed well. But am still waiting for a further study to make a strong conclusion about this aspect.

You can read the full report here (.PDF)

Update: Please look at the comment below by Shane Keats of Mcafee regarding Mcafee's Site Advisor.