Feeds

Got no idea what Hadoop is, but think you need it? You're not alone

It won't happen without Appening

Intelligent flash storage arrays

Open ... and Shut Hadoop is quickly becoming essential infrastructure for enterprises hoping to glean insights from the massive quantities of data they collect. The problem is that relatively few enterprises have the necessary competence to make effective use of the still-complex open-source project. While Hadoop vendors like Cloudera, Hortonworks, EMC, and MapR are doing their parts to simplify Hadoop, the real breakthrough for Hadoop may come from the applications that run on it, and not improvements to the infrastructure, according to Cloudera CEO Mike Olson.

Olson joined Hadoop heavyweights from Metamarkets, Oracle, Facebook, and MapR to discuss "The Elephant in the Enterprise: What Role Will Hadoop Play?" at a Churchill Club event earlier this week. As the panel debated the pros and cons of Hadoop, it became clear that while Hadoop is the heartbeat of the Big Data movement today, it still has a long way to go to become mainstream.

From left: Metamarkets CEO Michael Driscoll; Andrew Mendelsohn, SVP, Oracle Server Technologies; Cloudera CEO Mike Olson; Jay Parikh, VP Infrastructure Engineering, Facebook; and MapR CEO John Schroeder at the panel discussion.

For example, Metamarkets CEO Michael Driscoll criticised Hadoop's dated batch processing in a world that increasingly demands real-time. "Hadoop is not like having a conversation with your data," he insisted. "Instead it's like having a pen pal that you write from time to time." To this Olson quipped, "To be fair to Hadoop, it's like having thousands of pen pals that write to you."

Apparently this is enough for now, with Hadoop skills topping the list of skills needed across the IT landscape, according to research firm Ovum and jobs data compiled by Indeed.com:

But how much of this is real versus media-inspired mania for the next big thing?

While Driscoll feels the need for Hadoop is real, he was also quick to caution that "we're in a Hadoop hangover" that has executives saying "I have no idea what Hadoop is but I think we need it." The problem, however, is that the reason so many executives want Hadoop (data analytics) is not what an enterprise gets out of the box. As Driscoll pointed out, "Hadoop is a technology, not a solution."

Which is why Olson is probably right when he declared, "There will be enormous Hadoop adoption, but you'll get it by virtue of the applications you run." Cloudera board member and Workday co-CEO Aneel Bhusri echoed this Olson sentiment:

In other words, Hadoop, like Linux, MySQL, PHP, MongoDB, and other open-source technology, is very important, but mostly because it will enable a rich ecosystem of applications running on top of it. Enterprises won't need to become Hadoop experts. The experts will be the SaaS providers who build Big Data applications that take advantage of Hadoop, but don't shove it in the user's face.

Driscoll's Metamarkets is set up on the premise that customers can tap into analytics as a service, with Hadoop running behind the scenes. Many enterprises, however, simply aren't yet comfortable pushing their data to the cloud, leading Oracle's SVP of Server Technologies, Andy Mendelsohn, to conclude that Hadoop appliances are a necessary intermediate step for cautious enterprises on the road to true Hadoop-based analytics-as-a-service.

In sum, Hadoop is great technology, but not yet a tool that mere mortals can manage. While Hadoop training will undoubtedly mint many more data scientists, the real value this Big Data technology promises to deliver is as a behind-the-scenes infrastructure for cloud services, including everything from Facebook to Workday to Metamarkets. While the panel participants didn't estimate the timing for an onslaught of new Big Data applications, there's enough money at stake that the answer is almost certainly: soon. Very soon. ®

Matt Asay is senior vice president of business development at Nodeable, offering systems management for managing and analysing cloud-based data. He was formerly SVP of biz dev at HTML5 start-up Strobe and chief operating officer of Ubuntu commercial operation Canonical. With more than a decade spent in open source, Asay served as Alfresco's general manager for the Americas and vice president of business development, and he helped put Novell on its open source track. Asay is an emeritus board member of the Open Source Initiative (OSI). His column, Open...and Shut, appears three times a week on The Register.

Beginner's guide to SSL certificates

More from The Register

next story
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
Turnbull should spare us all airline-magazine-grade cloud hype
Box-hugger is not a dirty word, Minister. Box-huggers make the cloud WORK
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
Microsoft adds video offering to Office 365. Oh NOES, you'll need Adobe Flash
Lovely presentations... but not on your Flash-hating mobe
prev story

Whitepapers

Seattle children’s accelerates Citrix login times by 500% with cross-tier insight
Seattle Children’s is a leading research hospital with a large and growing Citrix XenDesktop deployment. See how they used ExtraHop to accelerate launch times.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
How to determine if cloud backup is right for your servers
Two key factors, technical feasibility and TCO economics, that backup and IT operations managers should consider when assessing cloud backup.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Business security measures using SSL
Examines the major types of threats to information security that businesses face today and the techniques for mitigating those threats.