Feeds

Cokeheads slip AI onto Yahoo! front page

Optimizing Britney

Combat fraud and increase customer satisfaction

NPUC 08 Yahoo! is now using real-time automated algorithms to select news stories on its famous front page, claiming a 25 to 30 per cent increase in click through rate - and millions of dollars in additional yearly revenue.

In other words, Yahoo! is making more money than ever from shameless celebrity gossip.

Senior research scientist Deepak Agarwal unveiled the new system this morning at NPUC 08, an IBM-sponsored conference dedicated to all things AI. A small step on the long road to true artificial intelligence, the system sprung from a Yahoo! Research project dubbed the Content Optimization Knowledge Engine. Agarwal calls it Coke for short, and his fellow researchers are affectionately known as Cokeheads.

"This is a project we started about a year ago," Agarwal told his audience of AI obsessives at IBM’s Almaden Research Center. "The goal was to use data mining and machine learning to optimize the content shown to users on web portals, and a version of the system is now in production on the Yahoo! front page."

Agarwal wouldn’t tell us when the system actually went live. Clearly, he’s been told not to speak with the press – or at least The Register. But during his prepared speech, he said the company ran preliminary tests for "several months."

As they have for years, real live editors still choose the pool of stories for the Yahoo! home page. But on the page’s main news tab (the "Featured" tab), automated algorithms decide which of these stories get placed where – and how long they stay there. And each decision is made with an eye towards click through rate.

Yahoo! Front Page

Yahoo! does AI (kinda)

Based on the Kalman algorithm – a filtering method first developed in the early 60s – the system determines story placement by analyzing millions upon millions of user clicks as they happen. Agarwal and his team considered a method that juggled stories based solely on their characteristics – is it a Britney Spears story, say, or a Paris Hilton? – but they soon realized they could deliver a bigger click boost if they tracked the clicks themselves. "We track user responses," Agarwal said, "and then we respond ourselves - in real-time."

Yes, Agarwal says this has improved click through rates by nearly a third. And yes, with extra clicks delivering extra ads, this translates to several millions in extra revenue. Naturally, Agarwal hopes to bring the system to other parts of the Yahoo! portal.

But even on that tiny Featured tab – which holds only four stories – the technology has some growing up to do. Agarwal and crew hope to boost click through rates even further by personalizing article placement for each user. But at the moment, this doesn’t quite fly. Personalization still loses out to raw click analysis.

Agarwal suspects this has something to do with the fact that real live editors are still selecting the initial pool of stories. They’re intent on fingering stories that appeal to everyone, the theory goes, so personalization isn't as personal as it could be.

As Agarwal described Yahoo!'s new click optimizer, he was closely watched by John McCarthy, one of the AI pioneers who organized the seminal Dartmouth Summer Research Conference on Artificial Intelligence back in the summer of 1956. And when the speech was over, McCarthy spoke up from the auditorium's front row. "I'm a little puzzled," he said. "What does this have to do with AI?"

OK, the Yahoo! news page still relies on editors. And even the automated bit ends up making some rather simple choices. But in making those choices, it's already beating human thinkers. And there's more work on the way. "Over the past 50 years, we've lowered the bar for what we consider AI," Stephan Nusser, senior manager of the User-Focused Systems research group at Almaden and the conference's MC, told us. "We have to start with AI that's practical." ®

3 Big data security analytics techniques

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Inside the Hekaton: SQL Server 2014's database engine deconstructed
Nadella's database sqares the circle of cheap memory vs speed
Oh no, Joe: WinPhone users already griping over 8.1 mega-update
Hang on. Which bit of Developer Preview don't you understand?
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
IRS boss on XP migration: 'Classic fix the airplane while you're flying it attempt'
Plus: Condoleezza Rice at Dropbox 'maybe she can find ... weapons of mass destruction'
OpenSSL Heartbleed: Bloody nose for open-source bleeding hearts
Bloke behind the cockup says not enough people are helping crucial crypto project
Ditch the sync, paddle in the Streem: Upstart offers syncless sharing
Upload, delete and carry on sharing afterwards?
prev story

Whitepapers

Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.