Feeds

Yahoo! girds Google's bastard grid child

Open sources Hadoop with Security

3 Big data security analytics techniques

Hadoop Summit Yahoo! has released a beta version of Hadoop with built-in security, while open sourcing the latest version of its in-house workflow engine for the Google-mimicking distributed number-crunching platform.

Speaking this morning at the Yahoo!-sponsored Hadoop Summit in Santa Clara, California, company Hadoop guru Eric Baldeschwieler said that both the Hadoop with Security beta and the Oozie workflow engine have been open sourced at Apache. The security beta has been deployed on the Hadoop clusters that Yahoo! maintains for research organizations, and Baldeschwieler tells The Reg that it is now part of Yahoo!'s Hadoop distro.

Based on Google’s (proprietary) software infrastructure, Hadoop is a means of crunching epic amounts of data across a network of distributed machines. Named for the stuffed elephant belonging to the son of project founder Doug Cutting, the open source platform now underpins online services operated by everyone from Yahoo! and Facebook and Twitter to — gasp! — Microsoft.

Hadoop duplicates GFS, Google's distributed file system, and MapReduce, Google's distributed number-crunching platform. In 2004, Google published a pair of research papers on these technologies, and Cutting used the papers to build a platform that would back Nutch, his open source web crawler. Hadoop was open sourced at Apache, and Yahoo! is still its largest contributor.

Hadoop with Security, Baldeschwieler says, integrates the platform with Kerberos, the open source authentication standard, while adding enhanced logging features. According to Baldeschwieler, the release is designed to allow the sharing of sensitive data with appropriate permissions and to "ease regulatory compliance". It's designed to prevent unauthorized access to data stored on Hadoop clusters, more effectively co-locate business sensitive data, and reduce costs by consolidating clusters.

Oozie — named with the Burmese term for, yes, elephant keeper — is designed for particularly complex workflows and data pipelines. At Yahoo! it's the de facto standard for ETL (extract, transform, and load) processing. This open source workflow engine is also part of the new Hadoop distro from startup Cloudera, released today.

Baldeschwieler tells The Reg that Yahoo! has been using Oozie for about six months. The version open sourced today is essentially Oozie 2.0. A 1.0 version was previously open sourced. ®

Top three mobile application threats

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Inside the Hekaton: SQL Server 2014's database engine deconstructed
Nadella's database sqares the circle of cheap memory vs speed
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
Oh no, Joe: WinPhone users already griping over 8.1 mega-update
Hang on. Which bit of Developer Preview don't you understand?
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
IRS boss on XP migration: 'Classic fix the airplane while you're flying it attempt'
Plus: Condoleezza Rice at Dropbox 'maybe she can find ... weapons of mass destruction'
Ditch the sync, paddle in the Streem: Upstart offers syncless sharing
Upload, delete and carry on sharing afterwards?
prev story

Whitepapers

Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.