Football U goes to 3Leaf for HPC

Hut one, hut two, MPI

HP ProLiant Gen8: Integrated lifecycle automation

Florida State University, like most state schools in the US with a storied (American) football program, also has a respectable comp sci department that is not afraid to spend a little cash on a new technology when it comes along. And that's how the Seminoles have ended up being one of the first customers of upstart server maker 3Leaf Systems.

Back in November 2009, after nearly six years of development, 3Leaf brought to market a shared memory clustered system called the Dynamic Data Center Server line. The DDC Server machines take Opteron-based SMP servers and a homegrown chipset called Voyager to lash together and virtualize the cores, memory, and I/O slots in the machines in the cluster.

When equipped with 3Leaf's own virtualization layer, the DDC machine can be made to look and feel like a giant SMP server or a regular HPC cluster based on the Message Passing Interface (MPI) cluster. Which is one of the reasons why the DDC machines may be the vanguard of future systems that will allow users to toggle from loose coupling with memory unshared to tighter coupling with shared memory being implemented over super-fast networks. This way, as workloads change, the hardware doesn't have to. It just reconfigures itself on the fly.

FSU's Department of Scientific Computing has a bunch of systems that support processing related to molecular biophysics, evolutionary biology, network modeling, and Monte Carlo algorithm development. The key HPC system at the university has 256 of Dell's PowerEdge SC1435 blade servers, which are two-socket Opteron-based blade servers from a few generations ago, plus eight Dell PowerEdge 6950 servers that act as head nodes for the cluster. The nodes in the cluster are glued together with a 288-port 20 Gb/sec InfiniBand switch from Cisco Systems, and two Cisco 6500 switches link to a 156 TB storage array from Panasas. This machine is rated t a relatively modest 8 teraflops, and it was clearly due for some kind of boost.

According to Bob Quinn, the founder, chairman, and chief technology officer at 3Leaf, FSU is starting out relatively small, with a twelve-node DDC Server that is based on two-socket systems crafted by motherboard and whitebox server maker Super Micro (which has created a special motherboard with room for the Voyager ASIC and which is also 3Leaf's hardware manufacturing partner). The system uses the 2.6 GHz Opteron 8400 processors from Advanced Micro Devices (these are the six-core chips known by the code-name "Istanbul"), and it has 144 cores in total. The dozen nodes are linked by a 40 Gb/sec InfiniBand switch, and the cluster has 576 GB of shared coherent memory.

The nodes have a modest 6 TB of local storage. This box, says Quinn, sells for $206,000. Assuming the Istanbuls do a peak of four floating point operations per clock cycle, then this machine should be rated at around 1.5 teraflops. By modern standards, this is not a lot of number-crunching power. But Florida State is not the University of Illinois, either, which is getting the 1 petaflops "Blue Waters" super made by IBM, the fastest machine currently installed at a university in the United States. (The Seminoles might be able to pull even with the Fighting Illini on the gridiron, even though they have been bested on the compute grid).

The DDC Server lineup can scale up to 16 server nodes in a shared memory configuration, so FSU has a little room to grow. And Quinn says that it will be able to support the impending twelve-core "Magny-Cours" Opteron 6100s in the DDC Servers by the third quarter of 2010.

Scalability was not really top-of mind for FSU, according to Quinn. What mattered to them, and what matters to the companies that 3Leaf is going to chase with its current and future (and much more scalable) machines, is the flexibility of the DDC Server systems software, which allows for nodes to be partitioned and repartitioned in a matter of seconds and configured with new software stacks in 30 seconds or so.

The MPI protocol can run atop the shared memory architecture without any changes to the code, too, so shifting between shared memory and distributed applications is no big deal. Up until now, customers had to decide what way they wanted to go and buy a machine to do one or the other.

While the initial DDC Servers are interesting, it looks like the systems will be far more scalable with the "Sandy Bridge" Xeon servers due from Intel in 2011. Quinn says that since the middle of 2008, 3Leaf has been working with Intel on modifying the Voyager ASIC so it can make use of the second generation of Intel's QuickPath Interconnect. The DDC Servers using the Sandy Bridge Xeons will be able to support up to 64 TB of shared memory, and "a large number" of server nodes and cores, according to Quinn. He would not specify how many nodes. But the word on the street is that it will scale to 32 nodes.

The Sandy Bridge Xeons aimed at two-socket servers are expected to have at least eight cores, and possibly more in machines where thermals are not as big an issue. Assume for Sandy Bridge-EP class machines Intel can push it up to a dozen cores, which they have to do to compete with Magny-Cours. So a single 3Leaf machine in the Sandy Bridge era should be able to house 768 cores using two-socket boxes.

How is 64 TB is going to be crammed into 32 nodes, you ask? Well, brace yourself, because a terabyte per node is going to be normal with the server chips that Intel and AMD are cooking up for this year. So doubling that to 2 TB per node in a year is so, well, that's just Moore's Law. ®

Reducing security risks from open source software

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
SHOCK and AWS: The fall of Amazon's deflationary cloud
Just as Jeff Bezos did to books and CDs, Amazon's rivals are now doing to it
Amazon Reveals One Weird Trick: A Loss On Almost $20bn In Sales
Investors really hate it: Share price plunge as growth SLOWS in key AWS division
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
BlackBerry: Toss the server, mate... BES is in the CLOUD now
BlackBerry Enterprise Services takes aim at SMEs - but there's a catch
The triumph of VVOL: Everyone's jumping into bed with VMware
'Bandwagon'? Yes, we're on it and so what, say big dogs
Carbon tax repeal won't see data centre operators cut prices
Rackspace says electricity isn't a major cost, Equinix promises 'no levy'
prev story


Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.