Microsoft answers Google MapReduce with 'Dryad' beta
It's like Hadoop. Without the open source bit
Microsoft has released a beta version of its Dryad platform, a Windows-happy answer to Google's MapReduce distributed number crunching platform.
Yes, there's already a burgeoning open source version of MapReduce: the Apache Hadoop project. And at least one Microsoft unit is both using Hadoop and actually contributing code to it. But a certain amount of Microsoft schizophrenia isn't surprising – especially where open source is concerned.
As Microsoft announced with a blog post, the company has released "community technology previews" of the Dryad platform and the DryadLINQ programming language, a means of building applications atop the platform. Like Google's proprietary MapReduce platform, Dryad is designed to crunch epic amounts of data across a network of distributed machines. But unlike MapReduce, it's meant for clusters running Windows HPC Server. And, yes, DryadLINQ uses .NET's LINQ query language model.
In 2004, Google published a pair of research papers on MapReduce and its equally-proprietary distributed file system GFS, and these served as the basis for Hadoop. Originally developed by Doug Cutting – the man behind the open source Nutch web crawler – Hadoop was open sourced at Apache and bootstrapped by Yahoo! It now underpins much of Yahoo!'s online infrastructure, and it's been adopted by several other big-name tech outfits, including Facebook, Twitter, Apple, and, yes, Microsoft. In 2008, Redmond acquired semantic search outfit Powerset, whose Hadoop-happy technology has since been rolled into Bing.
Microsoft even lets former Powersetters commit code to Hadoop. Powerset founded the Hadoop sister project HBase, which mimics Google's BigTable distributed database.
But Dryad and DryadLINQ predate Microsoft's purchase of Powerset. They've long been under development inside Microsoft's research arm. The company first published a research paper (PDF) on the technology in 2007. Microsoft released non-commercial Dryad and DryadLINQ builds to academics in the summer of 2009. But this year, it moved Dryad from research to its Technical Computing Group.
The Dryad community technology preview (CTP) requires a cluster based on Windows HPC Pack 2008 R2 Enterprise, with Service Pack 1 installed. If you've already registered with an HPC Pack 2008 R2 program, you already have access to the CTP. If not, you can sign up at Microsoft's Connect Beta website.
Microsoft acknowledges that the beta is still, well, a beta. The current version can't handle more than 2028 individual partitions. It's been tested on only 128 individual nodes. And the DryadLINQ LINQ provider can't yet handle all LINQ queries.
Like Dryad, Hadoop offers its own query language. In fact, it offers two. Yahoo! built a query language called Pig, and then Facebook cooked up a higher-level language dubbed Hive. ®