Feeds

Yahoo! girds Google's bastard grid child

Open sources Hadoop with Security

Securing Web Applications Made Simple and Scalable

Hadoop Summit Yahoo! has released a beta version of Hadoop with built-in security, while open sourcing the latest version of its in-house workflow engine for the Google-mimicking distributed number-crunching platform.

Speaking this morning at the Yahoo!-sponsored Hadoop Summit in Santa Clara, California, company Hadoop guru Eric Baldeschwieler said that both the Hadoop with Security beta and the Oozie workflow engine have been open sourced at Apache. The security beta has been deployed on the Hadoop clusters that Yahoo! maintains for research organizations, and Baldeschwieler tells The Reg that it is now part of Yahoo!'s Hadoop distro.

Based on Google’s (proprietary) software infrastructure, Hadoop is a means of crunching epic amounts of data across a network of distributed machines. Named for the stuffed elephant belonging to the son of project founder Doug Cutting, the open source platform now underpins online services operated by everyone from Yahoo! and Facebook and Twitter to — gasp! — Microsoft.

Hadoop duplicates GFS, Google's distributed file system, and MapReduce, Google's distributed number-crunching platform. In 2004, Google published a pair of research papers on these technologies, and Cutting used the papers to build a platform that would back Nutch, his open source web crawler. Hadoop was open sourced at Apache, and Yahoo! is still its largest contributor.

Hadoop with Security, Baldeschwieler says, integrates the platform with Kerberos, the open source authentication standard, while adding enhanced logging features. According to Baldeschwieler, the release is designed to allow the sharing of sensitive data with appropriate permissions and to "ease regulatory compliance". It's designed to prevent unauthorized access to data stored on Hadoop clusters, more effectively co-locate business sensitive data, and reduce costs by consolidating clusters.

Oozie — named with the Burmese term for, yes, elephant keeper — is designed for particularly complex workflows and data pipelines. At Yahoo! it's the de facto standard for ETL (extract, transform, and load) processing. This open source workflow engine is also part of the new Hadoop distro from startup Cloudera, released today.

Baldeschwieler tells The Reg that Yahoo! has been using Oozie for about six months. The version open sourced today is essentially Oozie 2.0. A 1.0 version was previously open sourced. ®

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
NO MORE ALL CAPS and other pleasures of Visual Studio 14
Unpicking a packed preview that breaks down ASP.NET
DARPA-derived secure microkernel goes open source tomorrow
Hacker-repelling, drone-protecting code will soon be yours to tweak as you see fit
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
Put down that Oracle database patch: It could cost $23,000 per CPU
On-by-default INMEMORY tech a boon for developers ... as long as they can afford it
Google shows off new Chrome OS look
Athena springs full-grown from Chromium project's head
Apple: We'll unleash OS X Yosemite beta on the MASSES on 24 July
Starting today, regular fanbois will be guinea pigs, it tells Reg
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.