Work begins on radical Gordon super flash-computer

One of Intel's secret Xeon E5 testbeds, perhaps?

Beginner's guide to SSL certificates

Supercomputer maker Appro International has finally begun building the "Gordon" flash-heavy supercomputer at the San Diego Supercomputer Center, which was funded by a $20m grant from the National Science Foundation nearly two years ago. The machine is a testbed to analyze what happens when you get the I/O and floating point operations in a parallel supercomputer in balance.

Gordon has been waiting for Intel's "Sandy Bridge-EP" Xeon E5 processors and its "Lyndonville" Series 710 solid state drives. The latter was announced at Intel Developer Forum in San Francisco last week, while the former was not. Sort of.

Intel has said that the Xeon E5 processors, which sport on-chip PCI-Express peripheral controllers and Advanced Vector Extensions (AVX) for processing up to 8 floating point operations per clock, are actually shipping to selected cloud and HPC customers but will not be formally announced until next year.

It looks like SDSC might be getting early access to the Xeon E5 chips, since Allan Snavely, associate director at SDSC and co-principal investigator for the Gordon system, said that the SDSC hopes to have it up and running by January 2012.

Bang for the IOPS

The bulk of the flash storage in the Gordon system is not going into the server processing nodes, says Snavely, but rather in dedicated I/O nodes that use current two-socket "Westmere-EP" Xeon 5600 motherboards and el cheapo LSI disk controllers to play traffic cop for the 710 SSDs.

SDSC has picked the 300GB version of the Series 710 SSDs, which come in a 2.5-inch form factor and which link to the system board in the I/O node through a 3Gb/sec SATA interface. The 710 SSDs are based on 25 nanometer MLC flash and delivers 2,700 random write I/O operations per second (IOPS) and 38,500 random read IOPS using 4KB data chunks; they cost $1,929 each when bought in 1,000-unit quantities

The Gordon machine will have a total of 64 I/O nodes, which is based on a 3U rack server from Appro. Each node has over 4,200GB of flash capacity (14 flash drives, for 539,000 aggregate IOPS) and a number of controllers to link the flash through the motherboard to ConnectX-3 host adapters from Mellanox Technologies, exposing them to the 3D torus interconnect of the supercomputer.

The flash drives are aggregated on I/O nodes so you can create a larger file stored on the I/O node that you would be able to do on a single server, and the latency from the server out over the InfiniBand network to external storage is not all that large thanks to the peppiness of flash storage.

This is thanks to using Quad Data Rate (QDR) InfiniBand, which runs at 40Gb/sec, as well as having the InfiniBand cards plug into the on-chip PCI-Express controllers in the Xeon E5 processors, which deliver up to 80GB/sec of bandwidth into each processor socket, according to specs El Reg published earlier this year on the not-yet-announced processors.

Interestingly, Snavely says that SDSC started out thinking it would have to buy high-end SAS or SATA controllers with their own processing to handle the I/O bandwidth ion all of that flash. But it soon discovered that even if these are not the peppiest SSDs on the market, the Intel 710s can quickly choke even a high-end SATA controller and that the best thing to do was put a dumb controller into the server and let the Xeon 5600s handle the I/Os coming into and out of the I/O nodes. Snavely says that the 1.1 petabytes of write endurance of the 710 SSDs is more than enough to outlast the expected three to four year life of the Gordon super.

As El Reg has pointed out in its coverage of the 710 SSD launch last week, these flash drives are not speed demons by comparison with some of the other enterprise-grade SSDs out there on the market in PCI-Express and drive bay form factors. It's tough to beat Intel on dollars per GB or dollars per IOPS, explained Snavely, who just graduated a PhD student who did a thesis including a thorough price/performance analysis of flash technologies.

"We did an extremely comprehensive flash analysis," says Snavely. "There are some good ones out there, but looking at price per IOPS, Intel SSDs were among the best. And we also have a partnership with Intel, which gives us a good price."

Bang for the flops

The 1,024 server nodes in the Gordon supercomputer will also have flash and will be based on a future GreenBlade blade server from Appro packaged up its Xtreme-X chassis. Each node gets an Intel X25-M2 SSD, which is used to store the operating system image for the node. Those servers are configured with a fairly modest 64GB of main memory.

All told, the Gordon machine will have well over the contracted 200 teraflops of floating point performance as well as 35 million aggregate IOPS of random flash read performance across its I/O nodes. The cluster will run SDSC's own derivative of Red Hat Enterprise Linux and its open source Rocks cluster management software.

Rather than use a fat-tree network, Gordon uses a 3D torus interconnect that will, in theory, allow it to be extended more easily but there is no indication that this machine will be upgraded at this time. The InfiniBand network has two rails for redundancy and multipathing between the server and I/O nodes.

SDSC is going to use ScaleMP's vSMP virtual symmetric multiprocessing to create fat memory nodes with 2TB of main memory on the fly when workloads call for it. The machine will link to an external 4PB Lustre disk file system that can pump data into Gordon at a sustained 100GB/sec data rate.

So a lot of new ideas are being tested in the Gordon machine all at the same time. That is what NSF grants to academia are all about. ®

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
IT crisis looming: 'What if AWS goes pop, runs out of cash?'
Public IaaS... something's gotta give - and it may be AWS
Linux? Bah! Red Hat has its eye on the CLOUD – and it wants to own it
CEO says it will be 'undisputed leader' in enterprise cloud tech
Oracle SHELLSHOCKER - data titan lists unpatchables
Database kingpin lists 32 products that can't be patched (yet) as GNU fixes second vuln
Ello? ello? ello?: Facebook challenger in DDoS KNOCKOUT
Gets back up again after half an hour though
Hey, what's a STORAGE company doing working on Internet-of-Cars?
Boo - it's not a terabyte car, it's just predictive maintenance and that
Troll hunter Rackspace turns Rotatable's bizarro patent to stone
News of the Weird: Screen-rotating technology declared unpatentable
prev story


Providing a secure and efficient Helpdesk
A single remote control platform for user support is be key to providing an efficient helpdesk. Retain full control over the way in which screen and keystroke data is transmitted.
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.