Feeds

Got no idea what Hadoop is, but think you need it? You're not alone

It won't happen without Appening

Internet Security Threat Report 2014

Open ... and Shut Hadoop is quickly becoming essential infrastructure for enterprises hoping to glean insights from the massive quantities of data they collect. The problem is that relatively few enterprises have the necessary competence to make effective use of the still-complex open-source project. While Hadoop vendors like Cloudera, Hortonworks, EMC, and MapR are doing their parts to simplify Hadoop, the real breakthrough for Hadoop may come from the applications that run on it, and not improvements to the infrastructure, according to Cloudera CEO Mike Olson.

Olson joined Hadoop heavyweights from Metamarkets, Oracle, Facebook, and MapR to discuss "The Elephant in the Enterprise: What Role Will Hadoop Play?" at a Churchill Club event earlier this week. As the panel debated the pros and cons of Hadoop, it became clear that while Hadoop is the heartbeat of the Big Data movement today, it still has a long way to go to become mainstream.

From left: Metamarkets CEO Michael Driscoll; Andrew Mendelsohn, SVP, Oracle Server Technologies; Cloudera CEO Mike Olson; Jay Parikh, VP Infrastructure Engineering, Facebook; and MapR CEO John Schroeder at the panel discussion.

For example, Metamarkets CEO Michael Driscoll criticised Hadoop's dated batch processing in a world that increasingly demands real-time. "Hadoop is not like having a conversation with your data," he insisted. "Instead it's like having a pen pal that you write from time to time." To this Olson quipped, "To be fair to Hadoop, it's like having thousands of pen pals that write to you."

Apparently this is enough for now, with Hadoop skills topping the list of skills needed across the IT landscape, according to research firm Ovum and jobs data compiled by Indeed.com:

But how much of this is real versus media-inspired mania for the next big thing?

While Driscoll feels the need for Hadoop is real, he was also quick to caution that "we're in a Hadoop hangover" that has executives saying "I have no idea what Hadoop is but I think we need it." The problem, however, is that the reason so many executives want Hadoop (data analytics) is not what an enterprise gets out of the box. As Driscoll pointed out, "Hadoop is a technology, not a solution."

Which is why Olson is probably right when he declared, "There will be enormous Hadoop adoption, but you'll get it by virtue of the applications you run." Cloudera board member and Workday co-CEO Aneel Bhusri echoed this Olson sentiment:

In other words, Hadoop, like Linux, MySQL, PHP, MongoDB, and other open-source technology, is very important, but mostly because it will enable a rich ecosystem of applications running on top of it. Enterprises won't need to become Hadoop experts. The experts will be the SaaS providers who build Big Data applications that take advantage of Hadoop, but don't shove it in the user's face.

Driscoll's Metamarkets is set up on the premise that customers can tap into analytics as a service, with Hadoop running behind the scenes. Many enterprises, however, simply aren't yet comfortable pushing their data to the cloud, leading Oracle's SVP of Server Technologies, Andy Mendelsohn, to conclude that Hadoop appliances are a necessary intermediate step for cautious enterprises on the road to true Hadoop-based analytics-as-a-service.

In sum, Hadoop is great technology, but not yet a tool that mere mortals can manage. While Hadoop training will undoubtedly mint many more data scientists, the real value this Big Data technology promises to deliver is as a behind-the-scenes infrastructure for cloud services, including everything from Facebook to Workday to Metamarkets. While the panel participants didn't estimate the timing for an onslaught of new Big Data applications, there's enough money at stake that the answer is almost certainly: soon. Very soon. ®

Matt Asay is senior vice president of business development at Nodeable, offering systems management for managing and analysing cloud-based data. He was formerly SVP of biz dev at HTML5 start-up Strobe and chief operating officer of Ubuntu commercial operation Canonical. With more than a decade spent in open source, Asay served as Alfresco's general manager for the Americas and vice president of business development, and he helped put Novell on its open source track. Asay is an emeritus board member of the Open Source Initiative (OSI). His column, Open...and Shut, appears three times a week on The Register.

Internet Security Threat Report 2014

More from The Register

next story
Docker's app containers are coming to Windows Server, says Microsoft
MS chases app deployment speeds already enjoyed by Linux devs
IBM storage revenues sink: 'We are disappointed,' says CEO
Time to put the storage biz up for sale?
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
SDI wars: WTF is software defined infrastructure?
This time we play for ALL the marbles
Windows 10: Forget Cloudobile, put Security and Privacy First
But - dammit - It would be insane to say 'don't collect, because NSA'
Oracle hires former SAP exec for cloudy push
'We know Larry said cloud was gibberish, and insane, and idiotic, but...'
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.