Feeds

Intel and Yahoo! spawn open-source 'Tashi' cluster

Big Data goes back to school

Choosing a cloud hosting partner with confidence

Backed by Yahoo!, HP, and Intel, the computer science mavens at Carnegie Mellon University have added a new compute cluster to the worldwide Open Cirrus test bed, a collection of clusters designed to explorer the frontiers of interwebs-scale distributed computing.

The test bed now spans ten clusters at locations across the globe.

Open Cirrus was launched in 2006 by Yahoo!, HP, and Intel to advance so-called cloud computing through open collaboration between academia, government, and, yes, the corporate world. Carnegie Mellon was already using an Open Cirrus cluster set up on its campus by Intel Labs Pittsburgh, and it was one of the first universities to make use of Yahoo!'s 4,000-processor M45 cluster, meant to promote research on Hadoop, the open source distributed number-crunching platform.

Juiced with CPU and cash donation from Intel and a power management and cooling systems contribution from APC, the new Carnegie Mellon cluster spans 159 servers and 1,165 processing cores, 2.4 terabytes of memory, and almost 900 terabytes of storage. Greg Ganger, a professor of electrical and computer engineering and director of Carnegie Mellon’s Parallel Data Lab, tells The Reg that the new cluster will run Tashi, an open source "infrastructure cloud" platform similar to Amazon's EC2.

Unlike Eucalyptus - another open source mirror of EC2 - Tashi is meant specifically for Big Data - i.e. the sort of number crunching handles by Hadoop, which the university will run atop its Tashi installation. "We do things Amazon wouldn't do," Ganger said, "things tailored specifically to something like Hadoop."

Hadoop is based on the Google File System (GFS) and MapReduce technologies developed at the Mountain View Chocolate Factory. In 2004, the Google published a pair of papers on these distributed computing technologies, and the platforms were reconstructed in open source by Doug Cutting, the man known previously for developing Lucene, the open-source retrieval library. He called his project Hadoop, after his son's stuffed elephant.

The Apache-based Tashi was co-developed by Carnegie Mellon and Intel, and it's currently running on that Intel Open Cirrus cluster on the university's campus. This cluster spans 25 nodes with 1000 cores and 450 disks providing more than 400 terabytes of storage.

With the new cluster, Ganger says his Parallel Data Labs will explorees all sorts of data intensive apps, including machine translation between languages like English and Chinese and deep analysis of the way netizens interact on the world's social networking sites. ®

Security for virtualized datacentres

More from The Register

next story
It's Big, it's Blue... it's simply FABLESS! IBM's chip-free future
Or why the reversal of globalisation ain't gonna 'appen
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
Bitcasa bins $10-a-month Infinite storage offer
Firm cites 'low demand' plus 'abusers'
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
Pssst. Want to buy a timeshare in the clouds?
The Google dilemma — controller or spreader of knowledge?
CAGE MATCH: Microsoft, Dell open co-located bit barns in Oz
Whole new species of XaaS spawning in the antipodes
Microsoft and Dell’s cloud in a box: Instant Azure for the data centre
A less painful way to run Microsoft’s private cloud
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
New hybrid storage solutions
Tackling data challenges through emerging hybrid storage solutions that enable optimum database performance whilst managing costs and increasingly large data stores.