Feeds

Doug Cutting: Hadoop dodged a Microsoft-Oracle stomping

Elephant daddy on breaking into mainstream IT

High performance access to file storage

Name change, game changer

The last big change went to the very heart of Hadoop’s identity: the MapReduce engine was rewritten so there is a global resource manager and per-application master that manages different applications. The rewrite was called MapReduce 2.0 (MRv2) or YARN, and has been folded into Apache Hadoop 2.0 and Cloudera’s CDH 4.

MRv2 is designed to bring the power of Hadoop to applications outside of large datasets, such as graph-processing algorithms. "Graph" is the name given to software maps constructed by social networks such as Facebook and LinkedIn to connect people, to find out who knows whom and their relationships to each other.

'Right now we are getting the low-hanging fruit of the companies that are sophisticated users of technology and that have the most glaring big data problems,' – Doug Cutting

“Graph will be moved to shared node,” Cutting said. “So you can have a node for a time doing graph process and then a minute later running a task from a MapReduce job – there’d be time-sharing on the cluster at a finer level. It’s going tog give people better resource utilisation.”

Cutting wrote Hadoop while working on Nutch, a web search project. Nutch had used a lot of manual steps and lacked a framework that would automate large-scale data crunching. MapReduce provided the automation framework and two years later Cutting was hired by Yahoo!.

Despite improvements, Hadoop still remains relatively difficult for newcomers. More work is needed on vertical-specific apps, nice UIs and tools to integrate Hadoop with existing platforms, Cutting says.

Cutting, who today spends just a third of his time working on Hadoop, is focused on the need for broader language support. He spends the rest of his time working on Apache's Avro project, designed to make Hadoop “less of a Java shop”.

“[Java] gets you 90 per cent of the performance,” Cutting told us. “Over time, the last 10 per cent of performance might become more critical and as things stabilise, we might see more things move into C/C++. In Hadoop, several critical elements have been moved into C/C++. It would be nice if wasn’t just a Java world and we worked better with other languages, but it [Java] has been a good platform for the technology.”

Cutting is bullish on Hadoop's potential to become established in the mainstream in the next five to 10 years. MRv2 should help broaden Hadoop’s adoption among different sizes of user with different types of challenges, he says. "Making Hadoop easier" should accompany such changes, he added.

The old order liveth

“Right now we are getting the low-hanging fruit of the companies that are sophisticated users of technology and that have the most glaring big data problems. Over time we can take new classes of problems and support people more easily and do it more efficiently,” Cutting says.

Cutting said he is relieved Microsoft and Oracle are on board. In fact, he says, the will actually help Hadoop grow by adding more contributors. “We want to build a large community to develop the common software... the larger the community the more refined the tools will get. Users turn into contributors over time; the more users, the more contributors over time.”

As long as Cloudera and Hortonworks don’t become Cold War proxies to the giants, maybe this IS manifest destiny. Maybe more makers of proprietary software in the field of data and analytics should be worried.

By allying with Hadoop, though, Microsoft and Oracle have done more than reassure Cutting. They have ensured the survival of their RDBMS. ®

High performance access to file storage

More from The Register

next story
Android engineer: We DIDN'T copy Apple OR follow Samsung's orders
Veep testifies for Samsung during Apple patent trial
Microsoft: Windows version you probably haven't upgraded to yet is ALREADY OBSOLETE
Pre-Update versions of Windows 8.1 will no longer support patches
OpenSSL Heartbleed: Bloody nose for open-source bleeding hearts
Bloke behind the cockup says not enough people are helping crucial crypto project
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Windows XP still has 27 per cent market share on its deathbed
Windows 7 making some gains on XP Death Day
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
US taxman blows Win XP deadline, must now spend millions on custom support
Gov't IT likened to 'a Model T with a lot of things on top of it'
Microsoft TIER SMEAR changes app prices whether devs ask or not
Some go up, some go down, Redmond goes silent
prev story

Whitepapers

Mainstay ROI - Does application security pay?
In this whitepaper learn how you and your enterprise might benefit from better software security.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.