Feeds

Revolution speeds stats on Hadoop clusters

R language teaches 'meaningful' math to elephants

Internet Security Threat Report 2014

Revolution Analytics, the company that is extending R, the open source statistical programming language, with proprietary extensions, is making available a free set of extensions that allow its R engine to run atop Hadoop clusters.

Now statisticians that are familiar with R can do analysis on unstructured data stored in the Hadoop Distributed File System, the data store used for the MapReduce method of chewing on unstructured data pioneered by Google for its search engine and mimicked and open sourced by rival Yahoo! as the Apache Hadoop project.

R can now also run against the HBase non-relational, column-oriented distributed data store, which mimics Google's BigTable and which is essentially a database for Hadoop for holding structured data. Like Hadoop, HBase in an open source project distributed by the Apache Software Foundation.

With MapReduce, unstructured data is broken up and spread across server nodes, where the data is mapped across the nodes (with replication at multiple points for performance as well as fault tolerance) and chewed on in parallel (that's the reduce part) rather than in series like you would have to do on a single machine.

Revolution R Hadoop logo combo

With the marriage of R and Hadoop, explains David Champagne, chief technology officer at Revolution Analytics, the R engine is installed atop each Hadoop node in the cluster. Instead of programming a reduction algorithm in Java, as you do in Hadoop, you set up an R algorithm from an R workstation, and it is parsed out to the Hadoop nodes by Hadoop's mapping function. Statistical analysis is thus done in parallel on the data is stored in HDFS.

You don't do MapReduce and then extract data that comes back to the workstation for the analysis, but you chew on data right where it is in the cluster, and then aggregate it. In essence, R is using Hadoop as a grid controller, managing where specific algorithms run and the data they run against.

"We allow not just sums, means, and averages, which can be done easily in Java or Python, but statistically meaningful analysis," says Champagne. And you don't have to know jack about Java or MapReduce to run an R algorithm against a Hadoop cluster with either HDFS or HBase as its data store. "We want to hide some of the complexity of the MapReduce approach from R programmers and statisticians."

The tool that makes this integration possible is called RevoConnectR for Apache Hadoop, and Champagne tells El Reg that it was tested against Cloudera's CDH3 commercial distribution of Hadoop combined with the Revolution R Enterprise 4.3 stats engine. But you can take the Revolution R Community Edition and plunk it down on an open source Hadoop cluster (from Apache or from Cloudera) and the Hadoop connector for R will also work.

You can download the R connector for Hadoop from GitHub. Both Revolution Analytics and Cloudera want to encourage customers to use their commercial releases – and pay for tech support, of course. But this R-Hadoop connector is supplied for free even if they do. ®

Internet Security Threat Report 2014

More from The Register

next story
Docker's app containers are coming to Windows Server, says Microsoft
MS chases app deployment speeds already enjoyed by Linux devs
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
'Urika': Cray unveils new 1,500-core big data crunching monster
6TB of DRAM, 38TB of SSD flash and 120TB of disk storage
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
SDI wars: WTF is software defined infrastructure?
This time we play for ALL the marbles
Windows 10: Forget Cloudobile, put Security and Privacy First
But - dammit - It would be insane to say 'don't collect, because NSA'
Oracle hires former SAP exec for cloudy push
'We know Larry said cloud was gibberish, and insane, and idiotic, but...'
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.