Feeds

Doug Cutting: Hadoop dodged a Microsoft-Oracle stomping

Elephant daddy on breaking into mainstream IT

Secure remote control for conventional and virtual desktops

Name change, game changer

The last big change went to the very heart of Hadoop’s identity: the MapReduce engine was rewritten so there is a global resource manager and per-application master that manages different applications. The rewrite was called MapReduce 2.0 (MRv2) or YARN, and has been folded into Apache Hadoop 2.0 and Cloudera’s CDH 4.

MRv2 is designed to bring the power of Hadoop to applications outside of large datasets, such as graph-processing algorithms. "Graph" is the name given to software maps constructed by social networks such as Facebook and LinkedIn to connect people, to find out who knows whom and their relationships to each other.

'Right now we are getting the low-hanging fruit of the companies that are sophisticated users of technology and that have the most glaring big data problems,' – Doug Cutting

“Graph will be moved to shared node,” Cutting said. “So you can have a node for a time doing graph process and then a minute later running a task from a MapReduce job – there’d be time-sharing on the cluster at a finer level. It’s going tog give people better resource utilisation.”

Cutting wrote Hadoop while working on Nutch, a web search project. Nutch had used a lot of manual steps and lacked a framework that would automate large-scale data crunching. MapReduce provided the automation framework and two years later Cutting was hired by Yahoo!.

Despite improvements, Hadoop still remains relatively difficult for newcomers. More work is needed on vertical-specific apps, nice UIs and tools to integrate Hadoop with existing platforms, Cutting says.

Cutting, who today spends just a third of his time working on Hadoop, is focused on the need for broader language support. He spends the rest of his time working on Apache's Avro project, designed to make Hadoop “less of a Java shop”.

“[Java] gets you 90 per cent of the performance,” Cutting told us. “Over time, the last 10 per cent of performance might become more critical and as things stabilise, we might see more things move into C/C++. In Hadoop, several critical elements have been moved into C/C++. It would be nice if wasn’t just a Java world and we worked better with other languages, but it [Java] has been a good platform for the technology.”

Cutting is bullish on Hadoop's potential to become established in the mainstream in the next five to 10 years. MRv2 should help broaden Hadoop’s adoption among different sizes of user with different types of challenges, he says. "Making Hadoop easier" should accompany such changes, he added.

The old order liveth

“Right now we are getting the low-hanging fruit of the companies that are sophisticated users of technology and that have the most glaring big data problems. Over time we can take new classes of problems and support people more easily and do it more efficiently,” Cutting says.

Cutting said he is relieved Microsoft and Oracle are on board. In fact, he says, the will actually help Hadoop grow by adding more contributors. “We want to build a large community to develop the common software... the larger the community the more refined the tools will get. Users turn into contributors over time; the more users, the more contributors over time.”

As long as Cloudera and Hortonworks don’t become Cold War proxies to the giants, maybe this IS manifest destiny. Maybe more makers of proprietary software in the field of data and analytics should be worried.

By allying with Hadoop, though, Microsoft and Oracle have done more than reassure Cutting. They have ensured the survival of their RDBMS. ®

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
Google+ goes TITSUP. But WHO knew? How long? Anyone ... Hello ...
Wobbly Gmail, Contacts, Calendar on the other hand ...
Preview redux: Microsoft ships new Windows 10 build with 7,000 changes
Latest bleeding-edge bits borrow Action Center from Windows Phone
UNIX greybeards threaten Debian fork over systemd plan
'Veteran Unix Admins' fear desktop emphasis is betraying open source
Microsoft promises Windows 10 will mean two-factor auth for all
Sneak peek at security features Redmond's baking into new OS
Google opens Inbox – email for people too stupid to use email
Print this article out and give it to someone techy if you get stuck
DEATH by PowerPoint: Microsoft warns of 0-day attack hidden in slides
Might put out patch in update, might chuck it out sooner
Redmond top man Satya Nadella: 'Microsoft LOVES Linux'
Open-source 'love' fairly runneth over at cloud event
prev story

Whitepapers

Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.