The Channel logo


By | Cade Metz 20th December 2010 23:21

Microsoft answers Google MapReduce with 'Dryad' beta

It's like Hadoop. Without the open source bit

Microsoft has released a beta version of its Dryad platform, a Windows-happy answer to Google's MapReduce distributed number crunching platform.

Yes, there's already a burgeoning open source version of MapReduce: the Apache Hadoop project. And at least one Microsoft unit is both using Hadoop and actually contributing code to it. But a certain amount of Microsoft schizophrenia isn't surprising – especially where open source is concerned.

As Microsoft announced with a blog post, the company has released "community technology previews" of the Dryad platform and the DryadLINQ programming language, a means of building applications atop the platform. Like Google's proprietary MapReduce platform, Dryad is designed to crunch epic amounts of data across a network of distributed machines. But unlike MapReduce, it's meant for clusters running Windows HPC Server. And, yes, DryadLINQ uses .NET's LINQ query language model.

In 2004, Google published a pair of research papers on MapReduce and its equally-proprietary distributed file system GFS, and these served as the basis for Hadoop. Originally developed by Doug Cutting – the man behind the open source Nutch web crawler – Hadoop was open sourced at Apache and bootstrapped by Yahoo! It now underpins much of Yahoo!'s online infrastructure, and it's been adopted by several other big-name tech outfits, including Facebook, Twitter, Apple, and, yes, Microsoft. In 2008, Redmond acquired semantic search outfit Powerset, whose Hadoop-happy technology has since been rolled into Bing.

Microsoft even lets former Powersetters commit code to Hadoop. Powerset founded the Hadoop sister project HBase, which mimics Google's BigTable distributed database.

But Dryad and DryadLINQ predate Microsoft's purchase of Powerset. They've long been under development inside Microsoft's research arm. The company first published a research paper (PDF) on the technology in 2007. Microsoft released non-commercial Dryad and DryadLINQ builds to academics in the summer of 2009. But this year, it moved Dryad from research to its Technical Computing Group.

The Dryad community technology preview (CTP) requires a cluster based on Windows HPC Pack 2008 R2 Enterprise, with Service Pack 1 installed. If you've already registered with an HPC Pack 2008 R2 program, you already have access to the CTP. If not, you can sign up at Microsoft's Connect Beta website.

Microsoft acknowledges that the beta is still, well, a beta. The current version can't handle more than 2028 individual partitions. It's been tested on only 128 individual nodes. And the DryadLINQ LINQ provider can't yet handle all LINQ queries.

Like Dryad, Hadoop offers its own query language. In fact, it offers two. Yahoo! built a query language called Pig, and then Facebook cooked up a higher-level language dubbed Hive. ®

comment icon Read 15 comments on this article alert Send corrections


Frank Jennings

What do you do? Use manual typwriters or live in a Scottish croft? Our man advises
A rusty petrol pump at an abandoned gas station. Pic by Silvia B. Jakiello via shutterstock

Trevor Pott

Among other things, Active Directory needs an overhaul
Baby looks taken aback/shocked/affronted. Photo by Shutterstock

Kat Hall

Plans for 2 million FTTP connections in next four years 'not enough'
Microsoft CEO Satya Nadella


League of gentlemen poster - Tubbs and Edward at the local shop. Copyright BBC
One reselling man tells his tale of woe