Feeds

Cassandra can FINALLY predict the future

Via Spark-based real-time analytics, that is

5 things you didn’t know about cloud backup

The company behind the Cassandra database has partnered with a big-brained computer science company to add real-time analytics to its technology.

Datastax announced on Thursday that it had partnered with Apache Spark's steward Databricks to bring the service to Cassandra, giving users of the system a way to rapidly generate insights over ingested data. This could potentially let them spot the warning signs for bank fraud, along with other hard-to-spot occurrences.

Spark is an open source data-processing system that was developed at UC Berkeley's computer science hothouse AMPLab in 2009, then published as open source in 2010.

The technology "provides high-level APIs in Scala, Java, and Python that make parallel jobs easy to write, and an optimized engine that supports general computation graphs. It also supports a rich set of higher-level tools including Shark (Hive on Spark), MLlib for machine learning, GraphX for graph processing, and Spark Streaming," according to an FAQ.

When it launched, it was designed to be run on top of the Hadoop File System (HDFS) and was therefore chiefly used as an extremely fast batch processing system.

Now, Databricks has worked with Datastax to layer Spark on top of Cassandra, bringing a capable data processing engine right on top of a database management system, giving companies involved in ecommerce, fraud detection, and others a handy tool.

"Our customer base and community concentrates on real-time apps that needs a real-time database," explained Datastacks' executive vice president of engineering, Martin van Ryswyk, in a chat with El Reg. "There are a lot of use cases where you're processing data very quickly that is immediately fed back to users in a web application. A database like Cassandra under the covers is the most appropriate choice."

To get Spark to work with Cassandra, the two company's worked to make sure that Spark's underlying storage interface, the Resilient Distributed Dataset (RDD), could make sense of Cassandra-stored data.

"You have to worry about connections like connecting to the database and being efficient and doing threading. We're using [Cassandra Query Language] – making sure it's up to date, [and there's been] a lot of work in making sure that things like datatypes map between the two, and how to do that," van Ryswyk said.

Datastax plans to publish the technology as open source soon, and may develop a paid option as well.

"We have a candidate we are working on that is well into development. It will be released into open source and you do not have to pay anybody for it," he said.

The tie-up between Spark and Cassandra mirrors a similar partnership that was announced last week between MongoDB and Cloudera.

That deal saw the companies agree to work together to more tightly integrate Hadoop and associated analytics technologies with MongoDB, and to build on existing work like MongoDB's "Hadoop Connector".

By partnering with Databricks, Cassandra-company Datastax is hoping to skip the use of HDFS altogether, and load analytics directly on top of its DBMS. It's an interesting move and one that highlights how the worlds of data analysis and data storage are, as they have many times in the past, dancing closer together. ®

Build a business case: developing custom apps

More from The Register

next story
The Return of BSOD: Does ANYONE trust Microsoft patches?
Sysadmins, you're either fighting fires or seen as incompetents now
Linux turns 23 and Linus Torvalds celebrates as only he can
No, not with swearing, but by controlling the release cycle
China hopes home-grown OS will oust Microsoft
Doesn't much like Apple or Google, either
Sin COS to tan Windows? Chinese operating system to debut in autumn – report
Development alliance working on desktop, mobe software
Apple promises to lift Curse of the Drained iPhone 5 Battery
Have you tried turning it off and...? Never mind, here's a replacement
Eat up Martha! Microsoft slings handwriting recog into OneNote on Android
Freehand input on non-Windows kit for the first time
Linux kernel devs made to finger their dongles before contributing code
Two-factor auth enabled for Kernel.org repositories
This is how I set about making a fortune with my own startup
Would you leave your well-paid job to chase your dream?
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Scale data protection with your virtual environment
To scale at the rate of virtualization growth, data protection solutions need to adopt new capabilities and simplify current features.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?