Have a reality check, you Itanium lover

Letter We don't typically spend a lot of time debating benchmarks here at Vulture Central, primarily because of all the work vendors pour into tuning their systems to perform well. You often find servers with unusual features such as excessive memory or you see tweaks to processors that would not likely occur in the real world.

One Reg reader took the time to have a close look at some of the most popular benchmarks and show how this vendor-tweaking plays out - at least in the case of Intel's Itanium processor.

In your recent article "Analyst sees St. Fister in Itanium wafer," you included the statements "in 2003, Intel managed to capture most of the leading performance benchmarks with EPIC/Itanium architecture" and "Clabby rightly points out that Intel has destroyed the competition on most benchmarks with IBM's Power processor the only rival even close to keeping pace."

This primarily proves that if one sprays around enough Kool-Aid it will touch even the lips of those who don't actually care for it: Intel has had considerable success in doing this for Itanic, and HP perhaps even more so.

Let's look at the record.

SPECint (the processor metric of most interest to most server users): By virtue of its monstrous on-chip L3 cache (rather than anything intrinsic to its EPIC architecture), Itanic briefly grabbed partial SPECint leadership (in base but not peak scores) over its RISC competition when the McKinley product was nominally launched in July, 2002. But its base scores still fell significantly behind those of its P4/Xeon sibling and AMD's Athlon, which without question also qualify as server competition, so no brass ring there at all. When Madison was launched in June, 2003 it did somewhat better: doubling the size of the already-monstrous L3 cache gave it at least a marginal SPECint win over *all* competition for a while.

But again there are qualifications to note: 1) this SPECint leadership was only obtainable using HP's zx1 chipset and its HP-UX compiler (performance in other configurations only matched that of IBM's new POWER4+, and fell short of the fastest P4/Xeons and new AMD64 products), and 2) AMD64 and P4 have since forged ahead of even those leading HP platform results. So Itanic not only failed to 'destroy' the competition in SPECint, it isn't currently even a leader at all, and from published roadmaps seems unlikely to retake the lead any time soon.

SPECfp (the processor metric of most interest to many HPTC users): This is the *one* area where Itanic can legitimately boast of a clear lead, and can actually credit that lead to its EPIC architecture rather than to other chip or system features. However, there's a good reason why HPTC users haven't jumped on the Itanic bandwagon to the exclusion of all other platforms: they're also sensitive to things like performance per Watt and performance per dollar, and Itanic does not lead in either of these.

TPC-C (a server metric of significant interest to database users, at least in its unclustered form): Itanic's performance here (again) in no way 'destroys' - or even clearly leads - its competition.

1) HP's zx1-based 4-processor platform has a clear TPC-C lead at that node size, but not due to any EPIC features: 4-way Xeon systems support only 1/3 the on-chip cache, 1/3 the maximum RAM, and 1/3 the aggregate FSB bandwidth to work with, IBM hasn't submitted any top-of-the-line systems at this system size, and the cost of setting up a full-bore TPC-C test seems to have prevented any serious AMD64 submissions since the initial one nearly a year ago (its submission date was changed last summer due to insignificant revisions).

2) HP's 64-processor Superdome indeed posts the top TPC-C score, but IBM's POWER4+ p690 attains 76% of that score using only half as many processors (and only 1/4 as many processor chips, since each is dual-core - you mention future Itanic multi-core chips later in your article, so this might be of interest): on a per-processor basis POWER4+ delivers 33% higher TPC-C performance, and on a per-processor-chip basis it delivers 167% higher TPC-C performance (these numbers based on comparison with Itanic's best 32-processor TPC-C result to avoid apples-to-oranges issues: relative per-processor performance in the 64-processor Superdome system listed is even worse).

SAP SD 2-tier (another server metric of significant interest to database users): IBM simply hasn't bothered to submit top-of-the-line configurations here, but their previous-generation 1.3 GHz POWER4 32-processor p690 system beats the best current-generation top-of-the-line 1.5 GHz Itanic 32-processor score (so, for that matter, does the 32-processor 1.15 GHz Alpha GS1280 system - again using previous-generation chip technology), and the score obtained by IBM's mid-range 1.45 GHz p650 8-way server suggests that its high-end model would approximately equal the top-of-the-line Itanic 2 at the 8-processor node size - so no 'destruction' (or even clear leadership at all) by Itanic here, either.

Oracle Applications Standard Benchmark (yet another significant database benchmark): No direct comparison with POWER4+ is available, and AMD64 systems haven't been submitted here. But the 16-processor POWER4+ score is over 3.4 times the 4-processor Itanic2 score, so given the significantly less than linear scaling up from 4 processors that HP's systems have demonstrated in other database-related areas POWER4+ seems likely to hold at the very least per-processor equality here.

SPECweb99_SSL (an important web-serving benchmark): HP initially seemed to have chosen to submit a SPECweb99_SSL Itanic test instead of one using its more popular SPECweb99 sibling because there was no serious competition there, but times have changed: in the hotly-contested 4-processor node size AMD64 now holds a modest lead, with Itanic2 and POWER4+ in a tie for second.

SPECjbb2000 (an important enterprise server benchmark): Itanic holds a slim (under 3%) lead over AMD64 at the 4-processor node size and another slim (under 4%) lead over POWER4+ at the 32-processor node size - hardly 'destroying' the competition, once again. And while Superdome's support for up to 64 processors lets Itanic exceed the p690's absolute headroom, the top score in the benchmark is held by a 112-processor Fujitsu SPARC64 (gasp!) system.

So there you have Itanic's benchmark 'dominance', using about the same set of benchmarks that HP uses to try to prove it (but without the apples-to-oranges spin that they add): one clear win (SPECfp), three approximate ties (OASB - maybe, SPECweb99_SSL, and SPECjbb2000), and three (TPC-C, SPECint, and SAP SD 2-tier) where Itanic is clearly behind at some system sizes and/or apparently mostly benefiting from the lack of competing
submissions elsewhere).

POWER4+ is not some struggling also-ran as your quotes above suggest: on average, it's measurably superior to Itanic 2 (and should increase that lead significantly when POWER5 appears this year), and where AMD64 has shown up to compete it's on average at least Itanic 2's equal (and with comparable benchmarketing effort would likely be shown to be its superior). Alpha would be putting Itanic to shame if Compaq hadn't decided to start de-emphasizing Alpha development over 4 years ago, and PA-RISC might as well had HP not made a similar decision even longer ago - but Itanic still has plenty of competition left (as noted above, even SPARC64 can't be ignored), and another serious threat may develop if Intel decides that it has to counter AMD64 with an x86 64-bit extension of its own.

Bill Todd ®