Feeds

Machine learning climbs atop Hadoop

Pattern hoists machine-learning models onto HDFS

Security for virtualized datacentres

Hadoop whisperer Concurrent has released a free tool for porting machine-learning models over to Hadoop.

The Pattern tool lets you run machine-learning models on top of the Hadoop compute and storage framework via either exported Predictive Model Markup Language (PMML) files or a Pattern Java API.

Designing machine-learning models requires a precise set of skills, and though the technology can bring great efficiencies by creating automated programs that can, say, automatically score query results by relevance, it is rare that machine-learning experts – who are a subcategory of the data scientist breed of tech bod – are also familiar with the vagaries of MapReduce jobs.

Rather, many data scientists work within the confines of mathematical or machine-learning programs such as R or MicroStrategies – and it can be a tall order for these people to learn HDFS and MapReduce sufficiently to re-implement their algorithms on large HDFS-stored datasets.

With Patterns, Concurrent has created a free technology that can take machine-learning models exported into PMML files and run them atop Hadoop. "You should be able to export from your favorite tools your PMML docs and get into production at least at scale," Concurrent founder Chris Wensel says. "The goal with Pattern is to be able to apply a [machine-learning] scoring model and run it at scale."

Pattern is the third prong in Concurrent's pitchfork for getting useful data in and out of Hadoop without having to learn the vagaries of the application. It sits alongside the company's Java API for Hadoop and its Lingual add-on for making SQL queries on Hadoop easy.

The tool is designed for data scientists who are unfamiliar with Hadoop but want to use the technology to run machine-learning models against large pools of data. It works with any program capable of exporting a model as a PMML file – R, MicroStrategies, SAS, and so on.

"We've used the Cascading APIs and implemented the scoring aspect of these models against the cascading APIs," Wensel says. "It'll generalize itself thanks to the facilities Hadoop provides. If you export the model from R into PMML and run [it] across Hadoop, it'll parallelize itself appropriately."

Pattern is part of Concurrent's overall strategy of shifting Cascading into an all-purpose translation layer for people who want to access the inherent scalability of Hadoop without having to invest time in learning its peculiarities.

Its closest contemporary would be the open source Apache Mahout project. However, Mahout is more a selection of HDFS-compatible machine learning algorithms than anything else, so it lacks the flexibility and tooling that software like R may have.

"Mahout is a set of standalone and independent applications that have to be orchestrated with other applications to do their job, each using different file formats," Wensel says. "This is fundamentally very brittle and adds lots of latency to the applications."

The company expects existing Cascade users such as Airbnb will start experimenting with the Patterns tool imminently. It is already in use by AgileOne.

Over time, Concurrent hopes to build an ecosystem of complementary tools for Hadoop around the Cascading data analysis software. This announcement comes after the company took $4m from VCs to give it time to follow through on Wensel's ambition to "build a sustainable business around Cascading." ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
Microsoft to bake Skype into IE, without plugins
Redmond thinks the Object Real-Time Communications API for WebRTC is ready to roll
Microsoft promises Windows 10 will mean two-factor auth for all
Sneak peek at security features Redmond's baking into new OS
Mozilla: Spidermonkey ATE Apple's JavaScriptCore, THRASHED Google V8
Moz man claims the win on rivals' own benchmarks
FTDI yanks chip-bricking driver from Windows Update, vows to fight on
Next driver to battle fake chips with 'non-invasive' methods
PEAK APPLE: iOS 8 is least popular Cupertino mobile OS in all of HUMAN HISTORY
'Nerd release' finally staggers past 50 per cent adoption
DEATH by PowerPoint: Microsoft warns of 0-day attack hidden in slides
Might put out patch in update, might chuck it out sooner
Ubuntu 14.10 tries pulling a Steve Ballmer on cloudy offerings
Oi, Windows, centOS and openSUSE – behave, we're all friends here
Was ist das? Eine neue Suse Linux Enterprise? Ausgezeichnet!
Version 12 first major-number Suse release since 2009
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
The hidden costs of self-signed SSL certificates
Exploring the true TCO for self-signed SSL certificates, including a side-by-side comparison of a self-signed architecture versus working with a third-party SSL vendor.