Feeds

Got no idea what Hadoop is, but think you need it? You're not alone

It won't happen without Appening

Mobile application security vulnerability report

Open ... and Shut Hadoop is quickly becoming essential infrastructure for enterprises hoping to glean insights from the massive quantities of data they collect. The problem is that relatively few enterprises have the necessary competence to make effective use of the still-complex open-source project. While Hadoop vendors like Cloudera, Hortonworks, EMC, and MapR are doing their parts to simplify Hadoop, the real breakthrough for Hadoop may come from the applications that run on it, and not improvements to the infrastructure, according to Cloudera CEO Mike Olson.

Olson joined Hadoop heavyweights from Metamarkets, Oracle, Facebook, and MapR to discuss "The Elephant in the Enterprise: What Role Will Hadoop Play?" at a Churchill Club event earlier this week. As the panel debated the pros and cons of Hadoop, it became clear that while Hadoop is the heartbeat of the Big Data movement today, it still has a long way to go to become mainstream.

From left: Metamarkets CEO Michael Driscoll; Andrew Mendelsohn, SVP, Oracle Server Technologies; Cloudera CEO Mike Olson; Jay Parikh, VP Infrastructure Engineering, Facebook; and MapR CEO John Schroeder at the panel discussion.

For example, Metamarkets CEO Michael Driscoll criticised Hadoop's dated batch processing in a world that increasingly demands real-time. "Hadoop is not like having a conversation with your data," he insisted. "Instead it's like having a pen pal that you write from time to time." To this Olson quipped, "To be fair to Hadoop, it's like having thousands of pen pals that write to you."

Apparently this is enough for now, with Hadoop skills topping the list of skills needed across the IT landscape, according to research firm Ovum and jobs data compiled by Indeed.com:

But how much of this is real versus media-inspired mania for the next big thing?

While Driscoll feels the need for Hadoop is real, he was also quick to caution that "we're in a Hadoop hangover" that has executives saying "I have no idea what Hadoop is but I think we need it." The problem, however, is that the reason so many executives want Hadoop (data analytics) is not what an enterprise gets out of the box. As Driscoll pointed out, "Hadoop is a technology, not a solution."

Which is why Olson is probably right when he declared, "There will be enormous Hadoop adoption, but you'll get it by virtue of the applications you run." Cloudera board member and Workday co-CEO Aneel Bhusri echoed this Olson sentiment:

In other words, Hadoop, like Linux, MySQL, PHP, MongoDB, and other open-source technology, is very important, but mostly because it will enable a rich ecosystem of applications running on top of it. Enterprises won't need to become Hadoop experts. The experts will be the SaaS providers who build Big Data applications that take advantage of Hadoop, but don't shove it in the user's face.

Driscoll's Metamarkets is set up on the premise that customers can tap into analytics as a service, with Hadoop running behind the scenes. Many enterprises, however, simply aren't yet comfortable pushing their data to the cloud, leading Oracle's SVP of Server Technologies, Andy Mendelsohn, to conclude that Hadoop appliances are a necessary intermediate step for cautious enterprises on the road to true Hadoop-based analytics-as-a-service.

In sum, Hadoop is great technology, but not yet a tool that mere mortals can manage. While Hadoop training will undoubtedly mint many more data scientists, the real value this Big Data technology promises to deliver is as a behind-the-scenes infrastructure for cloud services, including everything from Facebook to Workday to Metamarkets. While the panel participants didn't estimate the timing for an onslaught of new Big Data applications, there's enough money at stake that the answer is almost certainly: soon. Very soon. ®

Matt Asay is senior vice president of business development at Nodeable, offering systems management for managing and analysing cloud-based data. He was formerly SVP of biz dev at HTML5 start-up Strobe and chief operating officer of Ubuntu commercial operation Canonical. With more than a decade spent in open source, Asay served as Alfresco's general manager for the Americas and vice president of business development, and he helped put Novell on its open source track. Asay is an emeritus board member of the Open Source Initiative (OSI). His column, Open...and Shut, appears three times a week on The Register.

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
Seagate chances ARM with NAS boxes for the SOHO crowd
There's an Atom-powered offering, too
prev story

Whitepapers

Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.