Feeds

Yahoo! exposes very own stuffed elephant code

Distributed data-crunching distro

3 Big data security analytics techniques

Hadoop Summit Yahoo! has released its own Hadoop distro, an internet-scale distributed data-crunching platform based on the Apache open-source project that underpins several of the web’s highest profile sites, including Yahoo!, Facebook, and - amusingly - Microsoft’s Bing.

Inspired by Google-published research papers describing Mountain View’s proprietary software infrastructure, Hadoop is the brainchild of open-source guru Doug Cutting, the Nutch crawler founder who’s now on the Yahoo! payroll.

Yahoo! has used Hadoop code on its production infrastructure for more than a year now, and after calls from the ever-growing Hadoop community, the company is opening up its internal implementation of the project.

"We’ve put a lot of investment on our testing and deployment," Yahooligan Eric Baldeschwieler said Wednesday at the Yahoo!-sponsored Hadoop Summit in Santa Clara, California. "We’re going to take that work that we put into it and put it out on the web."

The new release - known as the Yahoo! Distribution of Hadoop - is not a commercially supported distro. "We’re not getting into a new business," Baldeschwieler explained. Yahoo! is leaving that business to Cloudera, the Silicon Valley startup that unveiled a commercial Hadoop distro this spring.

According to Baldeschwieler, Yahoo! will release code identical to that tested and deployed on the company’s internal machines. "The source code release will be exactly like we use on Yahoo! clusters," he said. And he expects Yahoo!-tweaked code will be released three to six months after general release of the Apache project code it's based on.

Yahoo! will not restrict access to the code, which will be available here from the Yahoo! developer network. It will merely require an agreement before downloading. The first release will be Hadoop version 0.20, which is now under alpha test inside the company.

Yahoo! contributes about 72 per cent of all Apache Hadoop patches. And it now uses Hadoop code to crunch data for myriad Yahoo! services, including its search index and the automated system that chooses news stories for its homepage.

Cutting cooked up Hadoop in 2004, naming the project after his son’s yellow stuffed elephant. Along with Yahoo!, one of its early users was Powerset, the semantic search engine recently acquired by Microsoft. Powerset now drives at least a small portion of Redmond’s latest Google challenge, which it insists on calling Bing. ®

SANS - Survey on application security programs

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Oh no, Joe: WinPhone users already griping over 8.1 mega-update
Hang on. Which bit of Developer Preview don't you understand?
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
Pre-Update versions of new Windows version will no longer support patches
Microsoft TIER SMEAR changes app prices whether devs ask or not
Some go up, some go down, Redmond goes silent
Ditch the sync, paddle in the Streem: Upstart offers syncless sharing
Upload, delete and carry on sharing afterwards?
prev story

Whitepapers

Designing a defence for mobile apps
In this whitepaper learn the various considerations for defending mobile applications; from the mobile application architecture itself to the myriad testing technologies needed to properly assess mobile applications risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.