Feeds

Revolution speeds stats on Hadoop clusters

R language teaches 'meaningful' math to elephants

Beginner's guide to SSL certificates

Revolution Analytics, the company that is extending R, the open source statistical programming language, with proprietary extensions, is making available a free set of extensions that allow its R engine to run atop Hadoop clusters.

Now statisticians that are familiar with R can do analysis on unstructured data stored in the Hadoop Distributed File System, the data store used for the MapReduce method of chewing on unstructured data pioneered by Google for its search engine and mimicked and open sourced by rival Yahoo! as the Apache Hadoop project.

R can now also run against the HBase non-relational, column-oriented distributed data store, which mimics Google's BigTable and which is essentially a database for Hadoop for holding structured data. Like Hadoop, HBase in an open source project distributed by the Apache Software Foundation.

With MapReduce, unstructured data is broken up and spread across server nodes, where the data is mapped across the nodes (with replication at multiple points for performance as well as fault tolerance) and chewed on in parallel (that's the reduce part) rather than in series like you would have to do on a single machine.

Revolution R Hadoop logo combo

With the marriage of R and Hadoop, explains David Champagne, chief technology officer at Revolution Analytics, the R engine is installed atop each Hadoop node in the cluster. Instead of programming a reduction algorithm in Java, as you do in Hadoop, you set up an R algorithm from an R workstation, and it is parsed out to the Hadoop nodes by Hadoop's mapping function. Statistical analysis is thus done in parallel on the data is stored in HDFS.

You don't do MapReduce and then extract data that comes back to the workstation for the analysis, but you chew on data right where it is in the cluster, and then aggregate it. In essence, R is using Hadoop as a grid controller, managing where specific algorithms run and the data they run against.

"We allow not just sums, means, and averages, which can be done easily in Java or Python, but statistically meaningful analysis," says Champagne. And you don't have to know jack about Java or MapReduce to run an R algorithm against a Hadoop cluster with either HDFS or HBase as its data store. "We want to hide some of the complexity of the MapReduce approach from R programmers and statisticians."

The tool that makes this integration possible is called RevoConnectR for Apache Hadoop, and Champagne tells El Reg that it was tested against Cloudera's CDH3 commercial distribution of Hadoop combined with the Revolution R Enterprise 4.3 stats engine. But you can take the Revolution R Community Edition and plunk it down on an open source Hadoop cluster (from Apache or from Cloudera) and the Hadoop connector for R will also work.

You can download the R connector for Hadoop from GitHub. Both Revolution Analytics and Cloudera want to encourage customers to use their commercial releases – and pay for tech support, of course. But this R-Hadoop connector is supplied for free even if they do. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
Just don't blame Bono! Apple iTunes music sales PLUMMET
Cupertino revenue hit by cheapo downloads, says report
The DRUGSTORES DON'T WORK, CVS makes IT WORSE ... for Apple Pay
Goog Wallet apparently also spurned in NFC lockdown
Cray-cray Met Office spaffs £97m on VERY AVERAGE HPC box
Only 250th most powerful in the world? Bring back Michael Fish
Microsoft brings the CLOUD that GOES ON FOREVER
Sky's the limit with unrestricted space in the cloud
'ANYTHING BUT STABLE' Netflix suffers BIG Europe-wide outage
Friday night LIVE? Nope. The only thing streaming are tears down my face
Google roolz! Nest buys Revolv, KILLS new sales of home hub
Take my temperature, I'm feeling a little bit dizzy
Cisco and friends chase WiFi's searing speeds with new cable standard
Cat 5e and Cat 6 are bottlenecks for WLAN access points
CAGE MATCH: Microsoft, Dell open co-located bit barns in Oz
Whole new species of XaaS spawning in the antipodes
prev story

Whitepapers

Choosing cloud Backup services
Demystify how you can address your data protection needs in your small- to medium-sized business and select the best online backup service to meet your needs.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
New hybrid storage solutions
Tackling data challenges through emerging hybrid storage solutions that enable optimum database performance whilst managing costs and increasingly large data stores.
Business security measures using SSL
Examines the major types of threats to information security that businesses face today and the techniques for mitigating those threats.