Whatever happened to... The Wisdom of Crowds?

Future social historians looking back at the web cult - which met in San Francisco this week for a $3,000-a-head "summit" - may wonder what made them tick. Scholars could do worse than examine their superstitions. We'll bet that lurking on the bookshelf of almost every "delegate" was a copy of James Surowiecki's The Wisdom of Crowds. It's as ubiquitous as Erik Von Daniken books were in the 1970s.

In Silicon Valley this year, "collective intelligence" is the mandatory piece of psycho-babble necessary to open a Venture Capitalist's cheque book. Surowiecki's faith in prediction markets appears unshakeable. Writing in Slate three years ago, in an attempt to save Admiral Poindexter's "Terror Casino" - punters were invited to bet on the probability of state leaders being assassinated, for example - Mystic Jim begged for understanding:

"Even when traders are not necessarily experts, their collective judgment is often remarkably accurate because markets are efficient at uncovering and aggregating diverse pieces of information. And it doesn't seem to matter much what markets are being used to predict."

"Whether the outcome depends on irrational actors (box-office results), animal behavior (horse races), a blend of irrational and rational motives (elections), or a seemingly random interaction between weather and soil (orange-juice crops), market predictions often outperform those of even the best-informed expert. Given that, it's reasonable to think a prediction market might add something to our understanding of the future of the Middle East."

A heart-warming fable, then, for a population robbed of their pensions, and beset by uncertainty after the dot.com bubble. Suroweicki failed to mention however that experts are regularly outperformed by chimps, or dartboards - but no one talks about "The Wisdom of Chimps".

This week however the people spoke - and the markets failed.

In the wake of the Democratic Party's capture of the Senate this week, we took a look at how well they performed at predicting the result.

Here's the result from TradeSports.com:

You'll want to study that last week in more detail, so here it is:

And here's the one from NewsFutures:

When the people really did decide to make a collective decision, the predictive "collective intelligence" of the crowd failed to foresee it. Awkward humans - especially those elusive humans who don't go near the web - keep getting in the way.

What next?

It's obvious. We need to cull the population so the result meets the market-markers prediction. That way, the facts will eventually fit with the Maoists' theory. ®