Feeds

Microsoft answers Google MapReduce with 'Dryad' beta

It's like Hadoop. Without the open source bit

Remote control for virtualized desktops

Microsoft has released a beta version of its Dryad platform, a Windows-happy answer to Google's MapReduce distributed number crunching platform.

Yes, there's already a burgeoning open source version of MapReduce: the Apache Hadoop project. And at least one Microsoft unit is both using Hadoop and actually contributing code to it. But a certain amount of Microsoft schizophrenia isn't surprising – especially where open source is concerned.

As Microsoft announced with a blog post, the company has released "community technology previews" of the Dryad platform and the DryadLINQ programming language, a means of building applications atop the platform. Like Google's proprietary MapReduce platform, Dryad is designed to crunch epic amounts of data across a network of distributed machines. But unlike MapReduce, it's meant for clusters running Windows HPC Server. And, yes, DryadLINQ uses .NET's LINQ query language model.

In 2004, Google published a pair of research papers on MapReduce and its equally-proprietary distributed file system GFS, and these served as the basis for Hadoop. Originally developed by Doug Cutting – the man behind the open source Nutch web crawler – Hadoop was open sourced at Apache and bootstrapped by Yahoo! It now underpins much of Yahoo!'s online infrastructure, and it's been adopted by several other big-name tech outfits, including Facebook, Twitter, Apple, and, yes, Microsoft. In 2008, Redmond acquired semantic search outfit Powerset, whose Hadoop-happy technology has since been rolled into Bing.

Microsoft even lets former Powersetters commit code to Hadoop. Powerset founded the Hadoop sister project HBase, which mimics Google's BigTable distributed database.

But Dryad and DryadLINQ predate Microsoft's purchase of Powerset. They've long been under development inside Microsoft's research arm. The company first published a research paper (PDF) on the technology in 2007. Microsoft released non-commercial Dryad and DryadLINQ builds to academics in the summer of 2009. But this year, it moved Dryad from research to its Technical Computing Group.

The Dryad community technology preview (CTP) requires a cluster based on Windows HPC Pack 2008 R2 Enterprise, with Service Pack 1 installed. If you've already registered with an HPC Pack 2008 R2 program, you already have access to the CTP. If not, you can sign up at Microsoft's Connect Beta website.

Microsoft acknowledges that the beta is still, well, a beta. The current version can't handle more than 2028 individual partitions. It's been tested on only 128 individual nodes. And the DryadLINQ LINQ provider can't yet handle all LINQ queries.

Like Dryad, Hadoop offers its own query language. In fact, it offers two. Yahoo! built a query language called Pig, and then Facebook cooked up a higher-level language dubbed Hive. ®

Remote control for virtualized desktops

More from The Register

next story
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
NASA launches new climate model at SC14
75 days of supercomputing later ...
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
You think the CLOUD's insecure? It's BETTER than UK.GOV's DATA CENTRES
We don't even know where some of them ARE – Maude
BOFH: WHERE did this 'fax-enabled' printer UPGRADE come from?
Don't worry about that cable, it's part of the config
Want to STUFF Facebook with blatant ADVERTISING? Fine! But you must PAY
Pony up or push off, Zuck tells social marketeers
Oi, Europe! Tell US feds to GTFO of our servers, say Microsoft and pals
By writing a really angry letter about how it's harming our cloud business, ta
prev story

Whitepapers

Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
How to determine if cloud backup is right for your servers
Two key factors, technical feasibility and TCO economics, that backup and IT operations managers should consider when assessing cloud backup.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
The hidden costs of self-signed SSL certificates
Exploring the true TCO for self-signed SSL certificates, including a side-by-side comparison of a self-signed architecture versus working with a third-party SSL vendor.