Original URL: http://www.theregister.co.uk/2005/08/23/ncsa_google_yahoo_study/

NCSA edges away from Google-Yahoo! study

Garbage in, garbage out

By Andrew Orlowski

Posted in Financial News, 23rd August 2005 08:44 GMT

A widely discredited report that cast doubt on claims made for Yahoo!'s search engine is in even more trouble. The study appears to have been disowned by the university that published it, although a history professor originally credited as a co-author is continuing to host the material.

After Yahoo! boasted that it now used a much larger search index than Google, two students at the National Center for Supercomputing Applications at the University of Illinois conducted a study that suggested otherwise. Matthew Cheney and Mike Perry concluded that only in 3 per cent of test cases did Yahoo! return more results.

However the students' methodology came under attack almost immediately. Researchers including Seth Finkelstein and Jean Veronais drew some quite different conclusions. The NCSA researchers had used pairs of words at random picked from a word list, which skewed the results towards web pages consisting of word lists. In other words, Google was simply returning more junk than Yahoo!

(See My spam-filled search index is bigger than yours!).

Now the biohazard signs are being erected around the report, and the students have issued a qualification. Gone from the original are the University's logo, and the name of a Professor listed as a co-author alongside Cheney and Perry has disappeared.

Added is a disclaimer:

"The study was done outside the scope of any NCSA core projects. When first published online, staff at the NCSA noted several issues with the study, and some revisions have been made to the document to reflect several of these concerns."

Incredibly, the students insist they were right first time round, and have only removed one of silliest conclusions. They're re-running the data and say they expect to reach the same results they got the first time round.

Whatever.

The Professor of History and Sociology at the NCSA who hosted the original, Vernon Burton, continues to host the study, alongside an ever-lengthening list of qualifications.®