Original URL: https://www.theregister.com/2011/03/11/google_admits_search_algorithm_whitelists/
Google contradicts own counsel in face of antitrust probe
Admits existence of search algorithm whitelists
Posted in Legal, 11th March 2011 00:16 GMT
Google has admitted that it uses whitelists to manually override its search algorithms, more than a year after its European corporate counsel denied the existence of whitelists when defending the company against antitrust complaints in the EU.
According to Search Engine Land, Matt Cutts – the head of Google's webspam team – told a Silicon Valley conference that Google uses "exception lists" that prevent certain algorithms from affecting certain sites. There is no global whitelist, Cutts said, but for some algorithms, Google will make an exception for sites it believes have been wrongly demoted on its search pages.
Last February, when the world learned that EU antitrust regulators were investigating claims that Google unfairly manipulated its search results at the expense of competitors, Google European corporate counsel Julia Holtz told journalists in Brussels that the company doesn't "whitelist or blacklist anyone,” according to The Financial Times. And as late as December, Google spokesman Adam Kovacevich told The Register that the company did not use whitelists.
Whitelists play a big role in the Google antitrust complaint brought by Foundem, a UK-based vertical search engine. Foundem is one of three companies whose complaints are being formally investigated by the European Commission.
In 2006, Foundem essentially vanished from Google, after Mountain View introduced a new algorithm that demoted vertical search engines. But some vertical search engines were not affected by the new algorithms. They were apparently whitelisted – and even though Holtz and other Googlers denied the existence of whitelists, Foundem produced emails from Google's AdWords team indicating that whitelists are used on search-ad algorithms. The company was in discussions with the AdWords team because it had also been demoted in Google's search ad system.
Foundem fought its removal for more than three years, but it didn't return to Google's "organic" search until it went public with its story at the end of 2009. Foundem was granted an audience with Google's Search Quality Team, and the UK outfit has always said that it was then manually whitelisted.
Though Holtz also denied the existence of blacklists, Google clearly has blacklists in place as well – or at least something very similar to blacklists. Last month, Matt Cutts publically admitted that the company will manually demote sites. "We have confirmed that Google's webspam team is willing to take action manually – for example, if we get a spam report, off-topic porn, things like that," he said.
Google took one of these "manual actions" when it demoted JC Penney in its search ranking in mid-February.
On one level, it's hardly surprising that Google manually overrides its algorithms. As even Foundem says, manual interventions are an important part of any search engine. "Whether manual interventions are required to override something that the algorithms have got wrong or to override something that the algorithms have overlooked, such interventions are an essential part of the process," Foundem CEO Shivaun Raff has told us.
The trouble is that Google spent years refusing to acknowledge that these manual interventions exist – and, in some cases, outright denying them. Google spent nearly a decade telling the world that its search engine was completely objective, and it has only recently begun to freely admit that this is not the case, presumably as a result of the EU investigation.
Foundem's EU antitrust complaint takes aim at Google's Universal Search setup, which inserts links from other Google services – such as Google Maps and Google Product Search – into prominent positions on the company's main search-results pages. Foundem says the setup is transforming Google's search engine into an "immensely powerful marketing channel" for its own services. Like Foundem, Google Product Search is a price-comparison service.
But Foundem's complaint also takes issue with the way Google handles manual interventions, including how little information the company has provided about how they work. "The half of our Complaint detailing the anticompetitive power of Google's exclusionary penalties is as much about the systemic failures of Google's manual review process as it is about the legitimacy of the penalty algorithms themselves," Shivaun Raff has told us.
Last month, Cutts also revealed that if a site owner appeals its Google search ranking, making a "reconsideration request", Google won't examine the request unless the site has been manually demoted. Foundem made repeated reconsideration requests, but unbeknownst to Foundem, they were not considered because its site had been algorithmically demoted.
Raff encapsulates Foundem's EU complaint like this: "You have an overwhelmingly dominant search engine. If you add to that that search engine's ability to apply discriminatory penalties - they're discriminatory because some services are manually rendered immune through whitelists - and you add the ability of that search engine to preferentially insert its own services at or near the top of the search results, all of that adds up to an unparalleled and unassailable competitive advantage."
Google has apparently tried to settle the EC's investigation. But the EC has made it clear that the probe will go on.
According to Search Engine Land, Cutts did not use the term whitelist. He called them "exception lists." But the two terms mean the same thing. "It is clear to us that Google no longer likes to use the term 'whitelisting'," Foundem has told us in the past. "But this was their word, not ours. In the context of lifting our AdWord penalty, Google themselves called it “whitelisting', and in leaked quality reviewer guidelines from 2003, Google used the same word 'whitelisting” when talking about immunity from search penalties.
"So Google can’t claim they don’t understand what you mean by the term whitelisting. It also wouldn’t be ok for Google to claim, for example, that 'there is no such thing as whitelisting' simply because they now prefer to call it something else, such as manual removal of a demotion." ®