Feeds

Microsoft answers Google MapReduce with 'Dryad' beta

It's like Hadoop. Without the open source bit

Internet Security Threat Report 2014

Microsoft has released a beta version of its Dryad platform, a Windows-happy answer to Google's MapReduce distributed number crunching platform.

Yes, there's already a burgeoning open source version of MapReduce: the Apache Hadoop project. And at least one Microsoft unit is both using Hadoop and actually contributing code to it. But a certain amount of Microsoft schizophrenia isn't surprising – especially where open source is concerned.

As Microsoft announced with a blog post, the company has released "community technology previews" of the Dryad platform and the DryadLINQ programming language, a means of building applications atop the platform. Like Google's proprietary MapReduce platform, Dryad is designed to crunch epic amounts of data across a network of distributed machines. But unlike MapReduce, it's meant for clusters running Windows HPC Server. And, yes, DryadLINQ uses .NET's LINQ query language model.

In 2004, Google published a pair of research papers on MapReduce and its equally-proprietary distributed file system GFS, and these served as the basis for Hadoop. Originally developed by Doug Cutting – the man behind the open source Nutch web crawler – Hadoop was open sourced at Apache and bootstrapped by Yahoo! It now underpins much of Yahoo!'s online infrastructure, and it's been adopted by several other big-name tech outfits, including Facebook, Twitter, Apple, and, yes, Microsoft. In 2008, Redmond acquired semantic search outfit Powerset, whose Hadoop-happy technology has since been rolled into Bing.

Microsoft even lets former Powersetters commit code to Hadoop. Powerset founded the Hadoop sister project HBase, which mimics Google's BigTable distributed database.

But Dryad and DryadLINQ predate Microsoft's purchase of Powerset. They've long been under development inside Microsoft's research arm. The company first published a research paper (PDF) on the technology in 2007. Microsoft released non-commercial Dryad and DryadLINQ builds to academics in the summer of 2009. But this year, it moved Dryad from research to its Technical Computing Group.

The Dryad community technology preview (CTP) requires a cluster based on Windows HPC Pack 2008 R2 Enterprise, with Service Pack 1 installed. If you've already registered with an HPC Pack 2008 R2 program, you already have access to the CTP. If not, you can sign up at Microsoft's Connect Beta website.

Microsoft acknowledges that the beta is still, well, a beta. The current version can't handle more than 2028 individual partitions. It's been tested on only 128 individual nodes. And the DryadLINQ LINQ provider can't yet handle all LINQ queries.

Like Dryad, Hadoop offers its own query language. In fact, it offers two. Yahoo! built a query language called Pig, and then Facebook cooked up a higher-level language dubbed Hive. ®

Internet Security Threat Report 2014

More from The Register

next story
Docker's app containers are coming to Windows Server, says Microsoft
MS chases app deployment speeds already enjoyed by Linux devs
IBM storage revenues sink: 'We are disappointed,' says CEO
Time to put the storage biz up for sale?
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
SDI wars: WTF is software defined infrastructure?
This time we play for ALL the marbles
Windows 10: Forget Cloudobile, put Security and Privacy First
But - dammit - It would be insane to say 'don't collect, because NSA'
Symantec backs out of Backup Exec: Plans to can appliance in Jan
Will still provide support to existing customers
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.