Google opens curtain on 'manual' search penalties
Whitelists still hidden in Mountain View dungeon
When the world learned that EU antitrust regulators were investigating claims that Google unfairly manipulated its search results at the expense of competitors, Google's European corporate counsel was unequivocal in her defense of the company. "We don’t whitelist or blacklist anyone,” Julia Holtz told journalists in Brussels last February, according to The Financial Times.
Holtz seemed to imply that Google does not manually override its search algorithms, that all sites are subject to the same mathematical rules. But Google does manually intervene, and this past week, for the first time, the company at least partially explained its rules for search penalties, including manual demotions.
"We have confirmed that Google's webspam team is willing to take action manually – for example, if we get a spam report, off-topic porn, things like that," Google's Matt Cutts said in a video posted to the web on Monday. He also discussed algorithmic penalties, and he said that whereas site owners could appeal manual interventions, they could not appeal the algorithmic variety. This was new information, and it provides some additional insight into the dispute under investigation by the EU.
The video arrived two days after a New York Times story detailing a campaign by American retailer JC Penney – or at least the search engine consulting firm hired by the retailer – to artificially boost the company's Google search-ranking by purchasing links to other sites across the web. The Times contacted Google about the JC Penney scheme, and the company immediately took action.
Thanks to the link-building campaign, JC Penney was the number-one Google search result on a wide variety of keywords, from "dresses" to "bedding" to "area rugs". But when Google intervened, its ranking on the keyword search "Samsonite carry on luggage" dropped from 1 to 71, and its "living room furniture" placement dropped from 1 to 68. Matt Cutts – who heads Google's webspam team – is quoted at length by The Times, and the story makes it clear that Google took what it calls "manual action".
Manual interventions are an important part of any search engine. Foundem, one of the EU complainants, says as much. "Whether manual interventions are required to override something that the algorithms have got wrong or to override something that the algorithms have overlooked, such interventions are an essential part of the process," Foundem CEO Shivaun Raff tells us. The problem is that for so many years, Google has largely avoided acknowledging that these interventions exist, and it has said almost nothing about how they work.
For years, Google told the world that its search engine was completely objective, and only now is the company beginning to freely explain that this is not the case.
In 2006, Foundem essentially vanished from Google's engine, after Mountain View took aim at vertical search engines. Foundem was not the victim of a "manual action". It was demoted by an algorithmic change. The rub was that other similar vertical search engines were not affected by the change. Other sites were apparently whitelisted out of the algorithms. Foundem has produced emails from Google's AdWords team indicating that there are whitelists on the search-ad side, but Google continues to deny that whitelists exist. Company spokesman Adam Kovacevich reiterated the denial on the phone with The Register late last year.
Foundem spent more than three years fighting the demotion – to no avail. But after the company took its story public in late 2009, the site returned to Google's "organic" search results. Foundem – which was granted an audience with Google's Search Quality Team – says its site was manually whitelisted.
In his video, Cutts did not mention whitelists. But he did say that if a site owner makes a "reconsideration request" – i.e., if it appeals Google's search ranking – the company will only examine the request if the site has been manually demoted. "If you do a reconsideration request, at least right now, we're checking against whether you have penalty in place manually," Cutts said. "But if something is only affected by our algorithms and not by any sort of manual action that flagged something that violated our guidelines, then normally you wouldn't be able to apply against the algorithm."
One of the apparent results of the EU investigation is that this sort of information is slowly trickling out of the company. But Google should have provided such guidelines years ago, and it's yet to come clean on such basic issues as whitelists. Whitelists, Foundem says, amount to a safety net, allowing Google to manually correct the inevitable mistakes that arise in its complex and ever-evolving penalty algorithms.
On one level, Foundem's EU antitrust complaint attacks Google's so-called Universal Search setup, which inserts links from other Google services – such as Google Maps and Google Product Search – into prominent positions on the company's main search result pages. Foundem says the setup is transforming Google's search engine into an "immensely powerful marketing channel" for its own services. Google Product Search (formerly known as Froogle) is a price-comparison service similar to Foundem.
But on another level, Foundem takes aim at the way Google handles manual interventions – and how little information we have about how they work. "The half of our Complaint detailing the anticompetitive power of Google's exclusionary penalties is as much about the systemic failures of Google's manual review process as it is about the legitimacy of the penalty algorithms themselves," Foundem CEO Shivaun Raff says.
Even though Foundem was returned to Google's search results, the company went ahead with its antitrust complaint. The change in ranking shouldn't have taken three years. It shouldn't have required pressure applied through the press. The process, Foundem says, is broken. And when you step back and examine the entire situation, the company contends, you see that Google has an unfair advantage when it expands into new markets, including price-comparison shopping or, well, travel search. Google's acquisition of flight-data outfit ITA Software is currently under review by the Department of Justice.
Raff sums up Foundem's EU complaint like this: "You have an overwhelmingly dominant search engine. If you add to that that search engine's ability to apply discriminatory penalties - they're discriminatory because some services are manually rendered immune through whitelists - and you add the ability of that search engine to preferentially insert its own services at or near the top of the search results, all of that adds up to an unparalleled and unassailable competitive advantage." ®
Sponsored: Becoming a Pragmatic Security Leader