Feeds

Yahoo! kills its stuffed elephant

For the love of Apache

  • alert
  • submit to reddit

Secure remote control for conventional and virtual desktops

Yahoo! has killed off its Hadoop distro, choosing to put its weight behind the core Apache Hadoop project instead.

Yahoo! was instrumental in bootstrapping Apache Hadoop, an open source distributed number-crunching platform based on Google's proprietary infrastructure, and in the summer of 2009, the web giant open sourced its own Hadoop distro, based on the code used in its "production" infrastructure. The idea was to let the rest of the world benefit from the work Yahoo! had done to build a platform that actually drives an internet-scale operation.

"We’ve put a lot of investment on our testing and deployment," Yahooligan Eric Baldeschwieler said in announcing the distro. "We’re going to take that work that we put into it and put it out on the web."

But on Tuesday, with a blog post, Baldeschwieler put an end to the Yahoo! distro, saying the company plans to remove all references to the distro on its website (developer.yahoo.com/hadoop), close its github repo (yahoo.github.com/hadoop-common), and concentrate on working with the Apache Hadoop community.

The Yahoo! distro, he said, was causing a bit of a community split. "As the community grew, we experimented with using the 'Yahoo! Distribution of Hadoop' as the vehicle to share our work," the post reads. "Unfortunately, Apache is no longer the obvious place to go for Hadoop releases. The Yahoo! team wants to return to a world where anyone can download and directly use releases of Hadoop from Apache.

"We want to contribute to the stabilization and testing of those releases. We also want to share our regular program of sustaining engineering that backports minor feature enhancements into new dot releases on a regular basis, so that the world sees regular improvements coming from Apache every few months, not years."

But Yahoo! still wants its work open sourced, so now it has to find a way of getting years of code into the Apache project.

Currently, Yahoo! offers two code branches: a stable release and a "future" release. The latest stable code – which Yahoo! currently runs on 40,000 nodes – is based on Hadoop 0.20, and the company is now working to move this to Apache (in the branch: hadoop/common/branches/branch-0.20-security). The idea is to call this the 20.100 release, and if the community approves, it will become an official Apache release.

Yahoo!'s future branch includes, yes, new features, and this will be rolled into Apache as well. "We are working on a set of new features for Hadoop to improve its availability, scalability and interoperability to make Hadoop more usable in mission critical deployments. You're going to see another burst of email activity from us as we work to get hadoop-future patches socialized, reviewed and checked in. These bulk checkins are exceptional. They are the result of us striving to be more transparent."

Named for the yellow stuffed elephant cherished by son of project founder Doug Cutting, Hadoop also underpins online services operated by everyone from Facebook and Twitter and StumbleUpon to, apparently, Apple. The original open source project mimicked GFS, Google's distributed file system, and MapReduce, Mountain View's distributed number-crunching platform. And it now includes various other infrastructure pieces, including HBase, based on Google's BigTable distributed database.

In 2004, Google published a pair of research papers on GFS and MapReduce, and Cutting used these to build a platform that would back Nutch, his open source web crawler. Yahoo! hired Cutting in 2006, but he has since left Yahoo! for Cloudera. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
Microsoft to bake Skype into IE, without plugins
Redmond thinks the Object Real-Time Communications API for WebRTC is ready to roll
Microsoft promises Windows 10 will mean two-factor auth for all
Sneak peek at security features Redmond's baking into new OS
Mozilla: Spidermonkey ATE Apple's JavaScriptCore, THRASHED Google V8
Moz man claims the win on rivals' own benchmarks
FTDI yanks chip-bricking driver from Windows Update, vows to fight on
Next driver to battle fake chips with 'non-invasive' methods
PEAK APPLE: iOS 8 is least popular Cupertino mobile OS in all of HUMAN HISTORY
'Nerd release' finally staggers past 50 per cent adoption
DEATH by PowerPoint: Microsoft warns of 0-day attack hidden in slides
Might put out patch in update, might chuck it out sooner
Ubuntu 14.10 tries pulling a Steve Ballmer on cloudy offerings
Oi, Windows, centOS and openSUSE – behave, we're all friends here
Was ist das? Eine neue Suse Linux Enterprise? Ausgezeichnet!
Version 12 first major-number Suse release since 2009
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Top 5 reasons to deploy VMware with Tegile
Data demand and the rise of virtualization is challenging IT teams to deliver storage performance, scalability and capacity that can keep up, while maximizing efficiency.
Mitigating web security risk with SSL certificates
Web-based systems are essential tools for running business processes and delivering services to customers.