Feeds

Yahoo! kills its stuffed elephant

For the love of Apache

  • alert
  • submit to reddit

Secure remote control for conventional and virtual desktops

Yahoo! has killed off its Hadoop distro, choosing to put its weight behind the core Apache Hadoop project instead.

Yahoo! was instrumental in bootstrapping Apache Hadoop, an open source distributed number-crunching platform based on Google's proprietary infrastructure, and in the summer of 2009, the web giant open sourced its own Hadoop distro, based on the code used in its "production" infrastructure. The idea was to let the rest of the world benefit from the work Yahoo! had done to build a platform that actually drives an internet-scale operation.

"We’ve put a lot of investment on our testing and deployment," Yahooligan Eric Baldeschwieler said in announcing the distro. "We’re going to take that work that we put into it and put it out on the web."

But on Tuesday, with a blog post, Baldeschwieler put an end to the Yahoo! distro, saying the company plans to remove all references to the distro on its website (developer.yahoo.com/hadoop), close its github repo (yahoo.github.com/hadoop-common), and concentrate on working with the Apache Hadoop community.

The Yahoo! distro, he said, was causing a bit of a community split. "As the community grew, we experimented with using the 'Yahoo! Distribution of Hadoop' as the vehicle to share our work," the post reads. "Unfortunately, Apache is no longer the obvious place to go for Hadoop releases. The Yahoo! team wants to return to a world where anyone can download and directly use releases of Hadoop from Apache.

"We want to contribute to the stabilization and testing of those releases. We also want to share our regular program of sustaining engineering that backports minor feature enhancements into new dot releases on a regular basis, so that the world sees regular improvements coming from Apache every few months, not years."

But Yahoo! still wants its work open sourced, so now it has to find a way of getting years of code into the Apache project.

Currently, Yahoo! offers two code branches: a stable release and a "future" release. The latest stable code – which Yahoo! currently runs on 40,000 nodes – is based on Hadoop 0.20, and the company is now working to move this to Apache (in the branch: hadoop/common/branches/branch-0.20-security). The idea is to call this the 20.100 release, and if the community approves, it will become an official Apache release.

Yahoo!'s future branch includes, yes, new features, and this will be rolled into Apache as well. "We are working on a set of new features for Hadoop to improve its availability, scalability and interoperability to make Hadoop more usable in mission critical deployments. You're going to see another burst of email activity from us as we work to get hadoop-future patches socialized, reviewed and checked in. These bulk checkins are exceptional. They are the result of us striving to be more transparent."

Named for the yellow stuffed elephant cherished by son of project founder Doug Cutting, Hadoop also underpins online services operated by everyone from Facebook and Twitter and StumbleUpon to, apparently, Apple. The original open source project mimicked GFS, Google's distributed file system, and MapReduce, Mountain View's distributed number-crunching platform. And it now includes various other infrastructure pieces, including HBase, based on Google's BigTable distributed database.

In 2004, Google published a pair of research papers on GFS and MapReduce, and Cutting used these to build a platform that would back Nutch, his open source web crawler. Yahoo! hired Cutting in 2006, but he has since left Yahoo! for Cloudera. ®

The essential guide to IT transformation

More from The Register

next story
The Return of BSOD: Does ANYONE trust Microsoft patches?
Sysadmins, you're either fighting fires or seen as incompetents now
Munich considers dumping Linux for ... GULP ... Windows!
Give a penguinista a hug, the Outlook's not good for open source's poster child
Intel's Raspberry Pi rival Galileo can now run Windows
Behold the Internet of Things. Wintel Things
Linux Foundation says many Linux admins and engineers are certifiable
Floats exam program to help IT employers lock up talent
Microsoft cries UNINSTALL in the wake of Blue Screens of Death™
Cache crash causes contained choloric calamity
Eat up Martha! Microsoft slings handwriting recog into OneNote on Android
Freehand input on non-Windows kit for the first time
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 10 endpoint backup mistakes
Avoid the ten endpoint backup mistakes to ensure that your critical corporate data is protected and end user productivity is improved.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.