Feeds

Yahoo! girds Google's bastard grid child

Open sources Hadoop with Security

Secure remote control for conventional and virtual desktops

Hadoop Summit Yahoo! has released a beta version of Hadoop with built-in security, while open sourcing the latest version of its in-house workflow engine for the Google-mimicking distributed number-crunching platform.

Speaking this morning at the Yahoo!-sponsored Hadoop Summit in Santa Clara, California, company Hadoop guru Eric Baldeschwieler said that both the Hadoop with Security beta and the Oozie workflow engine have been open sourced at Apache. The security beta has been deployed on the Hadoop clusters that Yahoo! maintains for research organizations, and Baldeschwieler tells The Reg that it is now part of Yahoo!'s Hadoop distro.

Based on Google’s (proprietary) software infrastructure, Hadoop is a means of crunching epic amounts of data across a network of distributed machines. Named for the stuffed elephant belonging to the son of project founder Doug Cutting, the open source platform now underpins online services operated by everyone from Yahoo! and Facebook and Twitter to — gasp! — Microsoft.

Hadoop duplicates GFS, Google's distributed file system, and MapReduce, Google's distributed number-crunching platform. In 2004, Google published a pair of research papers on these technologies, and Cutting used the papers to build a platform that would back Nutch, his open source web crawler. Hadoop was open sourced at Apache, and Yahoo! is still its largest contributor.

Hadoop with Security, Baldeschwieler says, integrates the platform with Kerberos, the open source authentication standard, while adding enhanced logging features. According to Baldeschwieler, the release is designed to allow the sharing of sensitive data with appropriate permissions and to "ease regulatory compliance". It's designed to prevent unauthorized access to data stored on Hadoop clusters, more effectively co-locate business sensitive data, and reduce costs by consolidating clusters.

Oozie — named with the Burmese term for, yes, elephant keeper — is designed for particularly complex workflows and data pipelines. At Yahoo! it's the de facto standard for ETL (extract, transform, and load) processing. This open source workflow engine is also part of the new Hadoop distro from startup Cloudera, released today.

Baldeschwieler tells The Reg that Yahoo! has been using Oozie for about six months. The version open sourced today is essentially Oozie 2.0. A 1.0 version was previously open sourced. ®

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
New 'Cosmos' browser surfs the net by TXT alone
No data plan? No WiFi? No worries ... except sluggish download speed
iOS 8 release: WebGL now runs everywhere. Hurrah for 3D graphics!
HTML 5's pretty neat ... when your browser supports it
Mathematica hits the Web
Wolfram embraces the cloud, promies private cloud cut of its number-cruncher
Mozilla shutters Labs, tells nobody it's been dead for five months
Staffer's blog reveals all as projects languish on GitHub
'People have forgotten just how late the first iPhone arrived ...'
Plus: 'Google's IDEALISM is an injudicious justification for inappropriate biz practices'
SUSE Linux owner Attachmate gobbled by Micro Focus for $2.3bn
Merger will lead to mainframe and COBOL powerhouse
iOS 8 Healthkit gets a bug SO Apple KILLS it. That's real healthcare!
Not fit for purpose on day of launch, says Cupertino
prev story

Whitepapers

Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.