Feeds

User-generated reviews - blessing or bull?

Crowd wisdom v mob stupidity

Build a business case: developing custom apps

Analysis Last week, connectivity-hardware maker Belkin admitted that one of its employees had been using Amazon's Mechanical Turk hiring service to pay for positive - and false - reviews of Belkin products.

While we congratulate Belkin for quickly admitting its employee's unethical behavior, and while we can only assume that said employee - identified by Engadget as bizdev rep Michael Bayard - has either joined the growing ranks of the unemployed or is now on a very short leash, the entire affair raises a larger question: How reliable are user-generated reviews, anyway?

More specifically, how can an online reviews service ensure that the product evaluations its users contribute aren't tainted by self-serving bias?

After speaking with spokesfolks from a selection of online reviews sites, we've concluded that there's no way user-generated reviews can be guaranteed 100 per cent accurate.

But we've also concluded that this is OK. Sorta.

Wisdom of crowds or stupidity of mobs?

User-generated content (UGC) is the bedrock of Web 2.0. Videos on YouTube, photos on Flickr, recipes on Recipezaar ("Choose from 341,000 member-contributed recipes") - the intertubes are stuffed with the creativity, considered commentary, and crackpot craziness of literally millions of UGCers.

Such participation is all well and good. But when you want an authoritative product recommendation before making a purchase, which would you prefer as your guide - expert analysis from an experienced reviewer or a consensus opinion derived from "The Wisdom of Crowds"?

Or both?

Some product evaluation websites, such as Yelp and Epinions, rely exclusively on UGC reviews. Others, most prominently Amazon, mix UCG with "Editorial Reviews". Still others - Digital Photography Review, Macworld, Consumer Reports, and many, many more - select and vet experts before publishing their often comprehensive analyses.

One clear advantage to the site choosing to host UGC reviews is cost. The Us Ging the C offer their opinions for free, while expert reviewers are paid for their time and trouble.

But there are three other less mercenary advantages to UGC reviews: breadth of coverage, the averaging of bias, and community-based expertise.

Take Yelp, for example, which publishes UGC reviews of over 20 categories of businesses and services in over 160 US cities. A single city can have thousands of reviews - San Francisco, for example, currently has 26,409 UGC reviews of everything from fondue restaurants to dog walkers. No expert review site could come close to that reach.

Such an enormous collection of reviews also provides a large UGC site with some protection against spurious reviews by their sheer volume. On Amazon, for example, JK Rowling's Harry Potter And The Goblet Of Fire has 5,180 UGC reviews. Even if you agree with the 75 reviewers who gave that book one star, it's still instructive to know that 4,402 others gave it five stars.

Then there's the intimate, streetwise smarts that come from living in a community whose offerings are being reviewed, such as those 26K-plus San Francisco reviewers. According to Stephanie Ichinose of Yelp, "Our posters take a certain amount of pride in the reviews they post, and in their community - New York, Chicago, San Francisco, and the rest."

But there's an even more important aspect of community: the desire for a UGC community to police itself to keep bad apples out of its beloved barrel. As Ichinose says, "The community self-polices themselves. If someone spots a review they think is inappropriate, they submit a note to our customer-service team."

Patty Smith of Amazon agrees, saying "Over time, the marketplace will out false reviews as 'unhelpful'," referring to how Amazon allows UGC reviewers to review not only products, but also each other.

Boost IT visibility and business value

More from The Register

next story
KDE releases ice-cream coloured Plasma 5 just in time for summer
Melty but refreshing - popular rival to Mint's Cinnamon's still a work in progress
Leaked Windows Phone 8.1 Update specs tease details of Nokia's next mobes
New screen sizes, dual SIMs, voice over LTE, and more
Mozilla keeps its Beard, hopes anti-gay marriage troubles are now over
Plenty on new CEO's todo list – starting with Firefox's slipping grasp
Apple: We'll unleash OS X Yosemite beta on the MASSES on 24 July
Starting today, regular fanbois will be guinea pigs, it tells Reg
Another day, another Firefox: Version 31 is upon us ALREADY
Web devs, Mozilla really wants you to like this one
Secure microkernel that uses maths to be 'bug free' goes open source
Hacker-repelling, drone-protecting code will soon be yours to tweak as you see fit
Cloudy CoreOS Linux distro declares itself production-ready
Lightweight, container-happy Linux gets first Stable release
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.
Maximize storage efficiency across the enterprise
The HP StoreOnce backup solution offers highly flexible, centrally managed, and highly efficient data protection for any enterprise.