Original URL: http://www.theregister.co.uk/2011/06/29/hadoop_cdh_substantial_update/

Cloudera promises 'Google-like' Big Data dream in minutes

Hadoop shop automates so you don't have to

By Gavin Clarke

Posted in Business, 29th June 2011 12:35 GMT

Updated Cloudera has delivered a "substantial" update to its open source Hadoop distribution.

On Wednesday, Cloudera rolled out Cloudera Enterprise 3.5, two months after shipping a major upgrade to its Hadoop distribution called Cloudera Distribution of Apache Hadoop (CDH) 3.0.

Whereas CDH 3.0 expanded Cloudera's Hadoop stack from three components to 10, the idea behind Cloudera Enterprise 3.5 is to make that distro easier to manage and deploy for IT shops outside the ranks of Hadoop's super-users like Facebook, Yahoo!, and LinkedIn. Cloudera provides services and support for such mainstream users.

The changes will let you install and configure a full Google-like infrastructure "in a couple of minutes", product vice president Charles Zedlewski told The Register. "We have done a substantial update." Hadoop is based on Google's GFS and MapReduce platforms.

Zedlewski said Cloudera Enterprise 3.5 automates configuration changes, service restarts, and the addition and removal of hardware. There's also an Activity Monitor that consolidates user activity across components to provide both a real-time and historical view of user activities and jobs.

"[We have] expanded the capability of the management suite from monitoring and discovery of issues, to diagnostic problems, to automating changes, and setting long-term changes," Zedlewski said.

Cloudera has also enhanced the Hadoop Resource and Authorization Manager, facilitating rollbacks and improving security with LDAP systems.

Hadoop is an architecture for crunching huge amounts of data using a network of distributed servers. Nutch web crawler creator Doug Cutting based the platform on research papers describing Google GFS and MapReduce., and it is now an Apache Software Foundation (ASF) project.

Today, Cloudera also released a free "Express" edition of its Service and Configuration Manager module used in Cloudera Enterprise 3.5 that will automate the installation and configuration of Hadoop on a cluster of up to 50 nodes. Meanwhile, the company has also donated code for its packaging and testing suite to ASF, under a project called Bigtop. The idea is to help improve packaging and interoperability testing for Hadoop and related modules.

Among Bigtop's initial committers is Canonical, chief commercial steward of Ubuntu. Cloudera has supported packaging for Ubuntu Linux for the last two-and-a-half years.

Zedlewski said Cloudera will add a further three or four modules to the current stack, among them a compression algorithm that leverages Google's Snappy to speed up data import and export.

This will be added in an update to CDH in the next month or so, Zedlewski said. Other components are due in "the CDH 4 timeframe", he said, while Cloudera is also looking at enhancements around high-availability features in the core Hadoop module. Zedlewski would not provide a date for CDH 4, but he said "work is well underway". ®

This article has been updated to clarify Cloudera has delivered Cloudera Enterprise 3.5.