IDC takes a second pass at supercomputer rankings

The Balanced Rating

ComputerWire: IT Industry Intelligence

Last November, the supercomputer analysts at International Data Corp developed a new way to rank the performance for HPC machines that, it reckoned, was superior to the Linpack Fortran benchmarks that have been used to rank supercomputers for the past several decades,

Timothy Prickett Morgan writes


The new rankings were in developed in response to the analysts' frustratation at the gaping holes in the supercomputer performance rankings developed by corporations, vendors, research institutions, and the governments that sponsor their high-performance computing projects.

This composite benchmark, called the IDC Balanced Rating, was generated for many of the biggest installed parallel cluster supercomputers in the world as well as for many midrange HPC machines, which are essentially goosed-up enterprise Unix servers or specialized vector machines in most cases.

This IDC Balanced Rating is arguably a better rating than the Linpack test used to rank the Top 500 supercomputers in the world, but it probably has had HPC vendors upset for the past six months. The latest rankings from IDC will doubtless also make many vendors unhappy.

IDC likes to think in categories, so it comes as no surprise that its supercomputer performance ratings for installed HPC machines are available in four different flavours: capability computers, enterprise computers, divisional computers, and departmental computers. For each category, IDC has found and ranked the 50 most powerful machines installed in the world and provided the number of processors, number of nodes, and the IDC Balanced Rating.

In the original rankings announced last year, IDC ranked the performance of single processors used in each HPC server or cluster using Linpack and SPEC benchmarks. Memory bandwidth for single nodes and the complete cluster were based on the STREAM benchmark. A scalability factor was then calculated based on the total processor count and the memory bandwidth of the full HPC cluster.

The Balanced Rating was the average of these three numbers after they had been normalized to the same number of digits. This initial IDC Balanced Rating gave Compaq Computer Corp's ASCI Q 3,024-processor AlphaServer cluster at the Pittsburgh Computing Center a performance rating that was 56% higher than IBM Corp's 8,192-processor ASCI White machine installed at the Lawrence Livermore National Lab. Using these ratings, a number of vector machines from NEC Ltd and Cray Inc rose much higher in the rankings than their Linpack ratings in the Top 500 list, too.

This time around, based on feedback from the HPC community and, presumably intense criticism from the vendors which make and sell supercomputers based in part on the publicity that the rankings and prestige that the Top 500 rankings provide, IDC has come up with a Balanced Rating for machines that does not show how it came up with these ratings.

Under this new ranking system, the Earth Simulator massively parallel vector supercomputer created for the Yokohama Institute for Earth Sciences in Japan using 5,120 vector processors spread out across 640 nodes was given a rating of 40,478. The number two machine was not Hewlett Packard's ASCI Q (now that Compaq is part of HP), but rather IBM's ASCI White, which with its 8,192 processors in 512 nodes was rated at 4867 on the IDC rankings. The 3,016-processor, 7564-node ASCI Q machine fell to the number four spot in the list with a ranking of 2914, behind a 4,096-processor, 1,025-node parallel AlphaServer cluster installed by the Los Alamos National Laboratory.

Another AlphaServer cluster came in at spot number six and SGI's ASCI Blue Mountain, without any changes to its hardware, rose in the IDC rankings from position 28 to position 6 because of revisions that IDC has made to its performance ranking algorithms.

In the capability HPC category, which means the monster clusters that are created to crack the world's toughest number crunching jobs, NEC has six machines with an aggregate IDC rating of 45,769 IBRs, with 88% of that aggregate power coming from the Earth Simulator. IBM had 28,679 IBRs of aggregate capacity installed in 16 machines, with the ASCI White machine representing 17% of its installed HPC capacity.

A number of pSeries 690 Regatta clusters with 12, 24 or 32 nodes are in the IBM installed base. The Regatta machines, which have only been shipping since December 2001, comprise 48% of the processing capacity at its biggest sites, based on these IBR rankings. Hewlett Packard, by virtue of its acquisition of Compaq, has 12,278 IBRs of processing capacity in the Top 50 IDC rankings for high-end HP machines with six big clusters installed using AlphaServers. SGI has seven machines with a total of 6,572 IBRs, Cray has two machines with 1,411 IBRs, and Fujitsu Ltd has two machines with 2,149 IBRs.

In the enterprise HPC category, which IDC says includes traditional commercial clusters that cost $1m or more but which fall far short of the high-end HPC clusters installed at the world's largest supercomputer centers, Sun Microsystems Inc and SGI are holding their own against IBM in terms of aggregate IBRs installed and the number of machines they have among the top 50 machines ranked in this category by IDC.

IBM leads this category with 1,495 IBRs of power in eight clusters, giving it 27.8% of the installed 5,385 IBRs of capacity in this subset of the HPC market. Sun, with 15 machines largely based on HPC versions of its "Starfire" Enterprise 10000 servers, had an aggregate of 1,309 IBRs installed, giving it 24.3% of the capacity in this enterprise HPC sector of the market. SGI had nine machines with 1,167 IBRs of capacity, giving it 21.7% of capacity. HP had 789 IBRs installed in eight machines, 14.7% of this enterprise HPC base; Cray had 377 IBRs installed in six machines, 7% of this base; and NEC had 247 IBRs in four machines, 4.6% of the base.

© ComputerWire. All rights reserved.

Sponsored: 5 critical considerations for enterprise cloud backup