Feeds

HPC 2.0: The monster mash-up

It's storage size and speed that counts...

Boost IT visibility and business value

Blog When Big Data gets big, data centers should get nervous Part 3

In the first two parts of this trilogy, we set the stage with a discussion of how HPC and the Big Data trend, combined with the increasing use of business analytics, are presenting existing data centers with some pretty big challenges. The first article is here, the second is here, and you’re reading the third (and final) story.

The genesis of these articles was an IBM HPC analyst event in New York last month and, more specifically, the discussion of these topics by IBM’s VP of Deep Computing, Dave Turek.

In the second installment, we covered the way that analytics processes (and HPC too) often consist of several different workloads, each of which uses a massive amount of data and relies on output from some other application.

Traditionally, this would mean moving data to the systems that will be doing the processing. But today’s disk and network technology, although fast, is still to slow to feed the analytics beast at a high enough rate to meet depth-of-analysis and time-to-solution requirements.

So if the usual data center workflow arrangements won’t work, what will? Turek spoke about a ‘Workflow Optimized System’ which, to me, looks and sounds a lot like either a big system or a cluster with virtualization in a MapReduce wrapper. In a workflow optimized infrastructure, mass moves of data from the network and disk storage is minimized to the greatest extent possible.

Data transfers take place over the system interconnect, which could be 40GB/sec to 100GB/sec, or even 400GB/sec at the high end. Even at 40GB/sec, you’d need 320 spinning hard drives running at 128 MB/ sec to equal the transfer speed of the system interconnect – which also assumes you have a network that can provide the bandwidth.

Combining different workloads on the same system or cluster presents some challenges. What each workload needs in terms of hardware resources varies.

Traditional HPC typically needs fast interconnect and high core counts – disk access speed and network bandwidth aren’t all that important. But velocity analytics depends on high network flows, a fast interconnect, and storage – not so much on core count. With volume analytics, storage size and speed and core counts are most important.

In order to maximise processing efficiency, a workflow optimized analytics infrastructure will be able to adapt on the fly to bring the right set of processing resources to the workload. While you can get this kind of resource granularity and workload management in a large SMP system – like a mainframe or commercial Unix box – these capabilities aren’t quite there yet for clusters.

There’s a lot of work happening along these lines, and some products out there right now that get us part of the way there (like ScaleMP’s memory aggregation), so we are seeing some progress. New system architectures with extensible bus mechanisms should take us farther down the road. ®

The Power of One Brief: Top reasons to choose HP BladeSystem

More from The Register

next story
BBC goes offline in MASSIVE COCKUP: Stephen Fry partly muzzled
Auntie tight-lipped as major outage rolls on
iPad? More like iFAD: We reveal why Apple fell into IBM's arms
But never fear fanbois, you're still lapping up iPhones, Macs
Stick a 4K in them: Super high-res TVs are DONE
4,000 pixels is niche now... Don't say we didn't warn you
Philip K Dick 'Nazi alternate reality' story to be made into TV series
Amazon Studios, Ridley Scott firm to produce The Man in the High Castle
Sonos AXES support for Apple's iOS4 and 5
Want to use your iThing? You can't - it's too old
There's NOTHING on TV in Europe – American video DOMINATES
Even France's mega subsidies don't stop US content onslaught
You! Pirate! Stop pirating, or we shall admonish you politely. Repeatedly, if necessary
And we shall go about telling people you smell. No, not really
Too many IT conferences to cover? MICROSOFT to the RESCUE!
Yet more word of cuts emerges from Redmond
Joe Average isn't worth $10 a year to Mark Zuckerberg
The Social Network deflates the PC resurgence with mobile-only usage prediction
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Securing Web Applications Made Simple and Scalable
Learn how automated security testing can provide a simple and scalable way to protect your web applications.