Feeds

Machine learning climbs atop Hadoop

Pattern hoists machine-learning models onto HDFS

High performance access to file storage

Hadoop whisperer Concurrent has released a free tool for porting machine-learning models over to Hadoop.

The Pattern tool lets you run machine-learning models on top of the Hadoop compute and storage framework via either exported Predictive Model Markup Language (PMML) files or a Pattern Java API.

Designing machine-learning models requires a precise set of skills, and though the technology can bring great efficiencies by creating automated programs that can, say, automatically score query results by relevance, it is rare that machine-learning experts – who are a subcategory of the data scientist breed of tech bod – are also familiar with the vagaries of MapReduce jobs.

Rather, many data scientists work within the confines of mathematical or machine-learning programs such as R or MicroStrategies – and it can be a tall order for these people to learn HDFS and MapReduce sufficiently to re-implement their algorithms on large HDFS-stored datasets.

With Patterns, Concurrent has created a free technology that can take machine-learning models exported into PMML files and run them atop Hadoop. "You should be able to export from your favorite tools your PMML docs and get into production at least at scale," Concurrent founder Chris Wensel says. "The goal with Pattern is to be able to apply a [machine-learning] scoring model and run it at scale."

Pattern is the third prong in Concurrent's pitchfork for getting useful data in and out of Hadoop without having to learn the vagaries of the application. It sits alongside the company's Java API for Hadoop and its Lingual add-on for making SQL queries on Hadoop easy.

The tool is designed for data scientists who are unfamiliar with Hadoop but want to use the technology to run machine-learning models against large pools of data. It works with any program capable of exporting a model as a PMML file – R, MicroStrategies, SAS, and so on.

"We've used the Cascading APIs and implemented the scoring aspect of these models against the cascading APIs," Wensel says. "It'll generalize itself thanks to the facilities Hadoop provides. If you export the model from R into PMML and run [it] across Hadoop, it'll parallelize itself appropriately."

Pattern is part of Concurrent's overall strategy of shifting Cascading into an all-purpose translation layer for people who want to access the inherent scalability of Hadoop without having to invest time in learning its peculiarities.

Its closest contemporary would be the open source Apache Mahout project. However, Mahout is more a selection of HDFS-compatible machine learning algorithms than anything else, so it lacks the flexibility and tooling that software like R may have.

"Mahout is a set of standalone and independent applications that have to be orchestrated with other applications to do their job, each using different file formats," Wensel says. "This is fundamentally very brittle and adds lots of latency to the applications."

The company expects existing Cascade users such as Airbnb will start experimenting with the Patterns tool imminently. It is already in use by AgileOne.

Over time, Concurrent hopes to build an ecosystem of complementary tools for Hadoop around the Cascading data analysis software. This announcement comes after the company took $4m from VCs to give it time to follow through on Wensel's ambition to "build a sustainable business around Cascading." ®

Combat fraud and increase customer satisfaction

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
Inside the Hekaton: SQL Server 2014's database engine deconstructed
Nadella's database sqares the circle of cheap memory vs speed
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
Oh no, Joe: WinPhone users already griping over 8.1 mega-update
Hang on. Which bit of Developer Preview don't you understand?
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
Pre-Update versions of new Windows version will no longer support patches
IRS boss on XP migration: 'Classic fix the airplane while you're flying it attempt'
Plus: Condoleezza Rice at Dropbox 'maybe she can find ... weapons of mass destruction'
prev story

Whitepapers

Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.