Fujitsu fires up first petaflopper PrimeHPC FX10
A chip off the new K
IT conglomerate Fujitsu has fired up the first installation of its PrimeHPC FX10 massively parallel Sparc-based supercomputer, a machine called Oakleaf-FX that weighs in at 1.13 petaflops of peak raw performance.
The University of Tokyo's Super Computing Division/Information Technology Center is the home of the Oakleaf-FX super, which has a list price of around $32m. Oakleaf-FX is being installed in the Second Research Complex on UT's Kashiwa Campus.
As El Reg detailed exhaustively last November in the wake of the SC11 supercomputer conference, the PrimeHPC FX10 is a follow-on machine to the K supercomputer built by Fujitsu for the Japanese government, sporting the eight-core Sparc64-VIIIfx processor running in a whopping 22,032 nodes (by which Fujitsu means CPU sockets not server blades since the blades do not implement SMP or NUMA clustering across the blade) and delivering a peak theoretical performance of 11.28 petaflops and a sustained performance on the Linpack Fortran benchmark of 10.51 petaflops.
Japan's Oakleaf-FX petaflopper at the University of Tokyo
For the PrimeHPC FX10 machines, Fujitsu has etched a new 16-core Sparc64-IXfx processor that runs at 1.85GHz and delivers 236 gigaflops of double-precision floating point oomph.
The Sparc64IXfx has two memory controllers on the die as well as an interface into the "Tofu" 6D mesh/torus interconnect that is at the heart of the K and PrimeHPC FX10 machines. That Tofu interface reaches out to its own Tofu interconnect chip (just like Cray talks to its XE interconnect through the HyperTransport ports on AMD's Opteron processors), which provides remote direct memory access (RDMA) across the processors in the cluster and which also provides PCI-Express 2.0 links for external storage and other peripherals. There are four Sparc64-IXfx processors and four Tofu interconnect chips per server blade.
The PrimeHPC FX machine is designed to scale up to 1,024 racks with 98,304 compute nodes, 6PB of main memory across those nodes, and a little more than 23 petaflops of peak number-crunching power, at a cost of $655.4m and with 22.4 megawatts of power consumed.
That's about half the power draw of the K super, and half the estimated budget of the original "Next Generation Supercomputer" project that resulted in the Fujitsu-only K super that is now at the top of the Top 500 supercomputer charts – NEC and Hitachi were supposed to help add vector math units and other stuff to the machine, but bagged on the Japanese government.
The University of Tokyo is not about to test the upper limits of the PrimeHPC FX10, but is nonetheless committing to a very powerful machine. The Oakleaf-FX system has 50 server racks with a total of 4,800 nodes with a total of 76,800 cores to deliver its 1.13 petaflops of raw performance. Those nodes have 150TB of main memory in total, and no, this is not a shared memory system. (Wouldn't that be a hell of a Crysis machine? It's a pity that the box is running a Sparc variant of Linux.)
The Sparc nodes in the Oakleaf-FX system are front-ended by 16 Primergy RX200S6 and 58 Primergy RX300S6 servers, which are used as log-in management nodes for the machine. The processing capacity in the log-in nodes alone would have qualified as HPC a decade ago.
In addition to running Fujitsu's homegrown Linux for Sparc, the company has Fortran, C, and C++ compilers, and a number of math libraries for common routines in what it calls the Technical Computing Suite.
The machine's storage, which consists of 150 Eternus DX80S2 storage arrays with RAID 5 across 1.1PB of capacity, and 80 Eternus DX410S2 arrays with RAID 6 across 2.1PB of capacity, runs Fujitsu's own homegrown Fujitsu Exabyte File System, or FEFS, a variant of the open source Lustre file system popular with the HPC crowd.
Fujitsu says that Oakleaf-FX was installed specifically because it is compatible with the K super. It will be used to run a slew of applications, including simulations for earth science, seismology, weather modeling, biology, hydrodynamics, astrophysics, materials science, energy, and solid-state physics. The machine will be a foundation of the application development efforts as Japan pursues an exascale super.
The University of Tokyo's current machines include two Hitachi parallel supers: an HA8000 and an SR16000. The HA8000 has been running since June 2008, and is an Opteron-based box rated at 140.1 teraflops. The SR16000 is none other than a rebadged Power 775 behemoth from IBM, and this particular machine is a 56-node box rated at 54.9 teraflops.
The Power 755 was supposed to be at the heart of the petaflops-busting "Blue Waters" super at the University of Illinois, but IBM pulled the plug on that deal last August. Last November Cray swooped in and landed a $188m contract with the university's National Center for Supercomputer Applications to build a 10 petaflops hybrid AMD Opetron-Nvidia GPU machine.
Back in Japan, the University of Tokyo has started an effort called the Large Scale HPC Challenge, a competition to seek proposals for how to use the machine. If you win the challenge, you get to run your apps on all of the nodes on the Oakleaf-FX for 24 hours straight. ®