Feeds

Microsoft answers Google MapReduce with 'Dryad' beta

It's like Hadoop. Without the open source bit

Choosing a cloud hosting partner with confidence

Microsoft has released a beta version of its Dryad platform, a Windows-happy answer to Google's MapReduce distributed number crunching platform.

Yes, there's already a burgeoning open source version of MapReduce: the Apache Hadoop project. And at least one Microsoft unit is both using Hadoop and actually contributing code to it. But a certain amount of Microsoft schizophrenia isn't surprising – especially where open source is concerned.

As Microsoft announced with a blog post, the company has released "community technology previews" of the Dryad platform and the DryadLINQ programming language, a means of building applications atop the platform. Like Google's proprietary MapReduce platform, Dryad is designed to crunch epic amounts of data across a network of distributed machines. But unlike MapReduce, it's meant for clusters running Windows HPC Server. And, yes, DryadLINQ uses .NET's LINQ query language model.

In 2004, Google published a pair of research papers on MapReduce and its equally-proprietary distributed file system GFS, and these served as the basis for Hadoop. Originally developed by Doug Cutting – the man behind the open source Nutch web crawler – Hadoop was open sourced at Apache and bootstrapped by Yahoo! It now underpins much of Yahoo!'s online infrastructure, and it's been adopted by several other big-name tech outfits, including Facebook, Twitter, Apple, and, yes, Microsoft. In 2008, Redmond acquired semantic search outfit Powerset, whose Hadoop-happy technology has since been rolled into Bing.

Microsoft even lets former Powersetters commit code to Hadoop. Powerset founded the Hadoop sister project HBase, which mimics Google's BigTable distributed database.

But Dryad and DryadLINQ predate Microsoft's purchase of Powerset. They've long been under development inside Microsoft's research arm. The company first published a research paper (PDF) on the technology in 2007. Microsoft released non-commercial Dryad and DryadLINQ builds to academics in the summer of 2009. But this year, it moved Dryad from research to its Technical Computing Group.

The Dryad community technology preview (CTP) requires a cluster based on Windows HPC Pack 2008 R2 Enterprise, with Service Pack 1 installed. If you've already registered with an HPC Pack 2008 R2 program, you already have access to the CTP. If not, you can sign up at Microsoft's Connect Beta website.

Microsoft acknowledges that the beta is still, well, a beta. The current version can't handle more than 2028 individual partitions. It's been tested on only 128 individual nodes. And the DryadLINQ LINQ provider can't yet handle all LINQ queries.

Like Dryad, Hadoop offers its own query language. In fact, it offers two. Yahoo! built a query language called Pig, and then Facebook cooked up a higher-level language dubbed Hive. ®

Security for virtualized datacentres

More from The Register

next story
It's Big, it's Blue... it's simply FABLESS! IBM's chip-free future
Or why the reversal of globalisation ain't gonna 'appen
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
Bitcasa bins $10-a-month Infinite storage offer
Firm cites 'low demand' plus 'abusers'
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
CAGE MATCH: Microsoft, Dell open co-located bit barns in Oz
Whole new species of XaaS spawning in the antipodes
Microsoft and Dell’s cloud in a box: Instant Azure for the data centre
A less painful way to run Microsoft’s private cloud
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
New hybrid storage solutions
Tackling data challenges through emerging hybrid storage solutions that enable optimum database performance whilst managing costs and increasingly large data stores.