Feeds

Yahoo! kills its stuffed elephant

For the love of Apache

  • alert
  • submit to reddit

Choosing a cloud hosting partner with confidence

Yahoo! has killed off its Hadoop distro, choosing to put its weight behind the core Apache Hadoop project instead.

Yahoo! was instrumental in bootstrapping Apache Hadoop, an open source distributed number-crunching platform based on Google's proprietary infrastructure, and in the summer of 2009, the web giant open sourced its own Hadoop distro, based on the code used in its "production" infrastructure. The idea was to let the rest of the world benefit from the work Yahoo! had done to build a platform that actually drives an internet-scale operation.

"We’ve put a lot of investment on our testing and deployment," Yahooligan Eric Baldeschwieler said in announcing the distro. "We’re going to take that work that we put into it and put it out on the web."

But on Tuesday, with a blog post, Baldeschwieler put an end to the Yahoo! distro, saying the company plans to remove all references to the distro on its website (developer.yahoo.com/hadoop), close its github repo (yahoo.github.com/hadoop-common), and concentrate on working with the Apache Hadoop community.

The Yahoo! distro, he said, was causing a bit of a community split. "As the community grew, we experimented with using the 'Yahoo! Distribution of Hadoop' as the vehicle to share our work," the post reads. "Unfortunately, Apache is no longer the obvious place to go for Hadoop releases. The Yahoo! team wants to return to a world where anyone can download and directly use releases of Hadoop from Apache.

"We want to contribute to the stabilization and testing of those releases. We also want to share our regular program of sustaining engineering that backports minor feature enhancements into new dot releases on a regular basis, so that the world sees regular improvements coming from Apache every few months, not years."

But Yahoo! still wants its work open sourced, so now it has to find a way of getting years of code into the Apache project.

Currently, Yahoo! offers two code branches: a stable release and a "future" release. The latest stable code – which Yahoo! currently runs on 40,000 nodes – is based on Hadoop 0.20, and the company is now working to move this to Apache (in the branch: hadoop/common/branches/branch-0.20-security). The idea is to call this the 20.100 release, and if the community approves, it will become an official Apache release.

Yahoo!'s future branch includes, yes, new features, and this will be rolled into Apache as well. "We are working on a set of new features for Hadoop to improve its availability, scalability and interoperability to make Hadoop more usable in mission critical deployments. You're going to see another burst of email activity from us as we work to get hadoop-future patches socialized, reviewed and checked in. These bulk checkins are exceptional. They are the result of us striving to be more transparent."

Named for the yellow stuffed elephant cherished by son of project founder Doug Cutting, Hadoop also underpins online services operated by everyone from Facebook and Twitter and StumbleUpon to, apparently, Apple. The original open source project mimicked GFS, Google's distributed file system, and MapReduce, Mountain View's distributed number-crunching platform. And it now includes various other infrastructure pieces, including HBase, based on Google's BigTable distributed database.

In 2004, Google published a pair of research papers on GFS and MapReduce, and Cutting used these to build a platform that would back Nutch, his open source web crawler. Yahoo! hired Cutting in 2006, but he has since left Yahoo! for Cloudera. ®

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
Download alert: Nearly ALL top 100 Android, iOS paid apps hacked
Attack of the Clones? Yeah, but much, much scarier – report
You stupid BRICK! PCs running Avast AV can't handle Windows fixes
Fix issued, fingers pointed, forums in flames
Microsoft: Your Linux Docker containers are now OURS to command
New tool lets admins wrangle Linux apps from Windows
Facebook, working on Facebook at Work, works on Facebook. At Work
You don't want your cat or drunk pics at the office
Soz, web devs: Google snatches its Wallet off the table
Killing off web service in 3 months... but app-happy bonkers are fine
First in line to order a Nexus 6? AT&T has a BRICK for you
Black Screen of Death plagues early Google-mobe batch
prev story

Whitepapers

Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
10 ways wire data helps conquer IT complexity
IT teams can automatically detect problems across the IT environment, spot data theft, select unique pieces of transaction payloads to send to a data source, and more.
5 critical considerations for enterprise cloud backup
Key considerations when evaluating cloud backup solutions to ensure adequate protection security and availability of enterprise data.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?