Feeds

Yahoo! girds Google's bastard grid child

Open sources Hadoop with Security

Beginner's guide to SSL certificates

Hadoop Summit Yahoo! has released a beta version of Hadoop with built-in security, while open sourcing the latest version of its in-house workflow engine for the Google-mimicking distributed number-crunching platform.

Speaking this morning at the Yahoo!-sponsored Hadoop Summit in Santa Clara, California, company Hadoop guru Eric Baldeschwieler said that both the Hadoop with Security beta and the Oozie workflow engine have been open sourced at Apache. The security beta has been deployed on the Hadoop clusters that Yahoo! maintains for research organizations, and Baldeschwieler tells The Reg that it is now part of Yahoo!'s Hadoop distro.

Based on Google’s (proprietary) software infrastructure, Hadoop is a means of crunching epic amounts of data across a network of distributed machines. Named for the stuffed elephant belonging to the son of project founder Doug Cutting, the open source platform now underpins online services operated by everyone from Yahoo! and Facebook and Twitter to — gasp! — Microsoft.

Hadoop duplicates GFS, Google's distributed file system, and MapReduce, Google's distributed number-crunching platform. In 2004, Google published a pair of research papers on these technologies, and Cutting used the papers to build a platform that would back Nutch, his open source web crawler. Hadoop was open sourced at Apache, and Yahoo! is still its largest contributor.

Hadoop with Security, Baldeschwieler says, integrates the platform with Kerberos, the open source authentication standard, while adding enhanced logging features. According to Baldeschwieler, the release is designed to allow the sharing of sensitive data with appropriate permissions and to "ease regulatory compliance". It's designed to prevent unauthorized access to data stored on Hadoop clusters, more effectively co-locate business sensitive data, and reduce costs by consolidating clusters.

Oozie — named with the Burmese term for, yes, elephant keeper — is designed for particularly complex workflows and data pipelines. At Yahoo! it's the de facto standard for ETL (extract, transform, and load) processing. This open source workflow engine is also part of the new Hadoop distro from startup Cloudera, released today.

Baldeschwieler tells The Reg that Yahoo! has been using Oozie for about six months. The version open sourced today is essentially Oozie 2.0. A 1.0 version was previously open sourced. ®

Internet Security Threat Report 2014

More from The Register

next story
Download alert: Nearly ALL top 100 Android, iOS paid apps hacked
Attack of the Clones? Yeah, but much, much scarier – report
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Whistling Google: PLEASE! Brussels can only hurt Europe, not us
And Commish is VERY pro-Google. Why should we worry?
Microsoft: Your Linux Docker containers are now OURS to command
New tool lets admins wrangle Linux apps from Windows
Soz, web devs: Google snatches its Wallet off the table
Killing off web service in 3 months... but app-happy bonkers are fine
First in line to order a Nexus 6? AT&T has a BRICK for you
Black Screen of Death plagues early Google-mobe batch
prev story

Whitepapers

10 ways wire data helps conquer IT complexity
IT teams can automatically detect problems across the IT environment, spot data theft, select unique pieces of transaction payloads to send to a data source, and more.
Why CIOs should rethink endpoint data protection in the age of mobility
Assessing trends in data protection, specifically with respect to mobile devices, BYOD, and remote employees.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Business security measures using SSL
Examines the major types of threats to information security that businesses face today and the techniques for mitigating those threats.