Feeds

Big data elephant mates with RainStor

RainStor Hadoops its storage

Choosing a cloud hosting partner with confidence

RainStor, the deduping database supplier, is bringing its analytics engine and enterprise database to Hadoop, rather than bringing Hadoop data to its engine.

Hadoop is becoming a standard for storing big data but most business intelligence analytics software – such as that pushed out by GreenPlum, Netezza and Teradata – does not natively support the Hadoop file system, HDFS, so data has to be extracted and moved to the analytics engine. This takes time and needs disk space for the copied data.

John Bantleman, RainStor's CEO, briefed us on Hadoop support by RainStor, RainStor Big Data Analytics on Hadoop, and said existing business intelligence (BI) analytics routines run against extracted Hadoop data can take hours – whereas RainStor's Hadoop-supporting analytics engine can run analytics much, much faster, 10 to 100 times faster it's claimed. Before we get to that, let's just acquaint ourselves with RainStor's history.

The story starts with a UK company called Clearpace back in 2008. It's NParchive product archived less-frequently accessed data from an Oracle database or other RDBMS uniquely in deduplicated form, with a 20:1 or better dedupe ratio, on cheap SATA drives. SQL routines could be run against the NParchive and there was no need for data rehydration to do this.

Bantleman moved Clearpace across to Silicon Valley and renamed it and the product RainStor, although there was no RAIN – redundant array of internet nodes – aspect to the name though. The second phase of its development saw a move into telecommunications and using its database to cope with storing records of the tens of billions of network events a day.

A four-hour Hadoop MapReduce run to find a single stock's NYSE average price for a day, ran 1,800 times faster using a SQL query against Hadoop data stored natively in RainStor.

One RainStor customer is Softbank in Japan. It stores 2PB of raw data, compressed and deduped to 135TB held on HP scale-out NAS drive array storage. It gets answers to questions about what individual subscribers did in a day in two to five seconds. A traditional database/data warehouse scheme would involve many petabytes of data at an average cost of $20,000/TB, meaning a 3PB setup would cost upwards of $60m. The RainStor/HP hardware system cost around $5m.

Big data elephant

This is big data under any definition and big data means Hadoop and RainStor's third development phase. It has spent over a year integrating Hadoop support into its product, enabling RainStor to run natively on Hadoop, and execute both MapReduce queries and SQL queries against compressed and deduped Hadoop data. The company claims it can dedupe and compress such data with an up to 40:1 ratio; 97.5 per cent compression. Telco records are, for example, highly repetitive in their content and rewardingly susceptible to compression and deduplication.

RainStor says: "The compressed multi-structured data set running on HDFS delivers maximum efficiency and reduces the cluster size by 50 per cent to 80 per cent, which significantly lowers operating cost."

What about EMC Isilon's Hadoop integration and integration with Greenplum?

Bantleman says: "Greenplum doesn't allow you to run MapReduce; it's actually a Postgres database inside and about parallel relational SQL queries.. We are the only enterprise database that can run on HDFS ... [and] we've added the ability to support MapReduce.

"Greenplum, Teradata, Netezza and Vertica have built connectors to allow you to bring data out of Hadoop into their own databases. They can't run natively on Hadoop clusters; we can. .. RainStor allows you to run ad hoc analytics directly on the Hadoop environment."

Bantleman added that he thinks data transfers at big data scale are silly.

Internet Security Threat Report 2014

More from The Register

next story
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
Turnbull should spare us all airline-magazine-grade cloud hype
Box-hugger is not a dirty word, Minister. Box-huggers make the cloud WORK
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
Microsoft adds video offering to Office 365. Oh NOES, you'll need Adobe Flash
Lovely presentations... but not on your Flash-hating mobe
prev story

Whitepapers

Designing and building an open ITOA architecture
Learn about a new IT data taxonomy defined by the four data sources of IT visibility: wire, machine, agent, and synthetic data sets.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
5 critical considerations for enterprise cloud backup
Key considerations when evaluating cloud backup solutions to ensure adequate protection security and availability of enterprise data.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Managing SSL certificates with ease
The lack of operational efficiencies and compliance pitfalls associated with poor SSL certificate management, and how the right SSL certificate management tool can help.