Feeds

Revolution speeds stats on Hadoop clusters

R language teaches 'meaningful' math to elephants

Mobile application security vulnerability report

Revolution Analytics, the company that is extending R, the open source statistical programming language, with proprietary extensions, is making available a free set of extensions that allow its R engine to run atop Hadoop clusters.

Now statisticians that are familiar with R can do analysis on unstructured data stored in the Hadoop Distributed File System, the data store used for the MapReduce method of chewing on unstructured data pioneered by Google for its search engine and mimicked and open sourced by rival Yahoo! as the Apache Hadoop project.

R can now also run against the HBase non-relational, column-oriented distributed data store, which mimics Google's BigTable and which is essentially a database for Hadoop for holding structured data. Like Hadoop, HBase in an open source project distributed by the Apache Software Foundation.

With MapReduce, unstructured data is broken up and spread across server nodes, where the data is mapped across the nodes (with replication at multiple points for performance as well as fault tolerance) and chewed on in parallel (that's the reduce part) rather than in series like you would have to do on a single machine.

Revolution R Hadoop logo combo

With the marriage of R and Hadoop, explains David Champagne, chief technology officer at Revolution Analytics, the R engine is installed atop each Hadoop node in the cluster. Instead of programming a reduction algorithm in Java, as you do in Hadoop, you set up an R algorithm from an R workstation, and it is parsed out to the Hadoop nodes by Hadoop's mapping function. Statistical analysis is thus done in parallel on the data is stored in HDFS.

You don't do MapReduce and then extract data that comes back to the workstation for the analysis, but you chew on data right where it is in the cluster, and then aggregate it. In essence, R is using Hadoop as a grid controller, managing where specific algorithms run and the data they run against.

"We allow not just sums, means, and averages, which can be done easily in Java or Python, but statistically meaningful analysis," says Champagne. And you don't have to know jack about Java or MapReduce to run an R algorithm against a Hadoop cluster with either HDFS or HBase as its data store. "We want to hide some of the complexity of the MapReduce approach from R programmers and statisticians."

The tool that makes this integration possible is called RevoConnectR for Apache Hadoop, and Champagne tells El Reg that it was tested against Cloudera's CDH3 commercial distribution of Hadoop combined with the Revolution R Enterprise 4.3 stats engine. But you can take the Revolution R Community Edition and plunk it down on an open source Hadoop cluster (from Apache or from Cloudera) and the Hadoop connector for R will also work.

You can download the R connector for Hadoop from GitHub. Both Revolution Analytics and Cloudera want to encourage customers to use their commercial releases – and pay for tech support, of course. But this R-Hadoop connector is supplied for free even if they do. ®

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
Seagate chances ARM with NAS boxes for the SOHO crowd
There's an Atom-powered offering, too
Gartner: To the right, to the right – biz sync firms who've won in a box to the right...
Magic quadrant: Top marks for, er, completeness of vision, EMC
prev story

Whitepapers

Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.