Yahoo! kills its stuffed elephant

Yahoo! has killed off its Hadoop distro, choosing to put its weight behind the core Apache Hadoop project instead.

Yahoo! was instrumental in bootstrapping Apache Hadoop, an open source distributed number-crunching platform based on Google's proprietary infrastructure, and in the summer of 2009, the web giant open sourced its own Hadoop distro, based on the code used in its "production" infrastructure. The idea was to let the rest of the world benefit from the work Yahoo! had done to build a platform that actually drives an internet-scale operation.

"We’ve put a lot of investment on our testing and deployment," Yahooligan Eric Baldeschwieler said in announcing the distro. "We’re going to take that work that we put into it and put it out on the web."

But on Tuesday, with a blog post, Baldeschwieler put an end to the Yahoo! distro, saying the company plans to remove all references to the distro on its website (developer.yahoo.com/hadoop), close its github repo (yahoo.github.com/hadoop-common), and concentrate on working with the Apache Hadoop community.

The Yahoo! distro, he said, was causing a bit of a community split. "As the community grew, we experimented with using the 'Yahoo! Distribution of Hadoop' as the vehicle to share our work," the post reads. "Unfortunately, Apache is no longer the obvious place to go for Hadoop releases. The Yahoo! team wants to return to a world where anyone can download and directly use releases of Hadoop from Apache.

"We want to contribute to the stabilization and testing of those releases. We also want to share our regular program of sustaining engineering that backports minor feature enhancements into new dot releases on a regular basis, so that the world sees regular improvements coming from Apache every few months, not years."

But Yahoo! still wants its work open sourced, so now it has to find a way of getting years of code into the Apache project.

Currently, Yahoo! offers two code branches: a stable release and a "future" release. The latest stable code – which Yahoo! currently runs on 40,000 nodes – is based on Hadoop 0.20, and the company is now working to move this to Apache (in the branch: hadoop/common/branches/branch-0.20-security). The idea is to call this the 20.100 release, and if the community approves, it will become an official Apache release.

Yahoo!'s future branch includes, yes, new features, and this will be rolled into Apache as well. "We are working on a set of new features for Hadoop to improve its availability, scalability and interoperability to make Hadoop more usable in mission critical deployments. You're going to see another burst of email activity from us as we work to get hadoop-future patches socialized, reviewed and checked in. These bulk checkins are exceptional. They are the result of us striving to be more transparent."

Named for the yellow stuffed elephant cherished by son of project founder Doug Cutting, Hadoop also underpins online services operated by everyone from Facebook and Twitter and StumbleUpon to, apparently, Apple. The original open source project mimicked GFS, Google's distributed file system, and MapReduce, Mountain View's distributed number-crunching platform. And it now includes various other infrastructure pieces, including HBase, based on Google's BigTable distributed database.

In 2004, Google published a pair of research papers on GFS and MapReduce, and Cutting used these to build a platform that would back Nutch, his open source web crawler. Yahoo! hired Cutting in 2006, but he has since left Yahoo! for Cloudera. ®