TPC slaps Oracle on benchmark claims

Watchdogs to Larry: 'You lie!'

Internet Security Threat Report 2014

Oracle let its marketing mouth get ahead of its brain with the Exadata 2 cluster system. Today, the Transaction Processing Council, which administers the TPC family of transaction and data warehouse processing benchmarks, slapped Oracle with a fine and a muzzle order relating to claims it has been making in advertisements about how its iron stacks up against alternatives such as IBM's Power 595 behemoth.

The TPC is quite picky about the rules for citing benchmark results, and Oracle has stepped over the line, according to Michael Majdalany, the administrator of the benchmarks. You're not allowed to estimate results for one machine from results for another or tell people about benchmarks that have not been through the formal TPC vetting process, among other restrictions.

Oracle may have done both. The advertisements ran in the Wall Street Journal and The Economist a month ago, in which Oracle proclaimed "Sun + Oracle is Faster" and told everyone to expect some sort of product announcement on October 14 that would demonstrate that some sort of hybrid Oracle-Sun setup would offer two-digit performance on the TPC-C online transaction processing test compared to IBM's 6 million transaction per minute result on its Power 595 running AIX and DB2. As El Reg previously told you, the original ad showed Oracle and Sun working together on a Sparc cluster of some sort.

Two weeks later, Oracle ran another set of adverts, saying that should it prevail in its $7.4bn (net around $5bn, perhaps) acquisition of Sun, it would increase spending on Sparc platforms, Solaris Unix, and sales efforts for the Sparc/Solaris platform, and would tightly integrate Oracle database and middleware software with the platform to boost its performance. And then, ironically, only five days later, Oracle ditched x64 HP iron, adopted Sun x64 iron and flash storage, and cranked out the Exadata 2 cluster.

Presumably we will see a Sparc/Solaris variant of the Exadata on October 14, most likely using the quad-socket T5440 Sparc T2+ servers instead of x64 servers as database nodes. Why Oracle decided to do the Exadata 2 launch ahead of Oracle's OpenWorld trade show rather than waiting is a mystery. When you're Larry Ellison, chief executive officer at Oracle, you can do whatever you want. Most of the time.

Ever since Oracle and Sun Microsystems speed-launched the Exadata 2 cluster for data warehousing and online transaction processing two weeks ago, I have been scouring the usual suspects of public benchmarks, trying to get proof of the claims that Oracle has been making in its ads and at the Exadata launch. Ellison said that the Exadata 2 setup was five times as fast as data warehousing boxes from Teradata and Netezza (leading everyone to believe there was a TPC-H benchmark to prove this) and said further that for online transaction processing (by which everyone knew Ellison was insinuating a TPC-C test), two racks of the Exadata 2 setup would have the same performance as an IBM Power 595 and cost one quarter the price.

No surprises here, but it was TPC consortium member IBM that made a formal complaint that Oracle was bending the rules. Teradata and Netezza are also TPC members and should have been equally annoyed, but Oracle didn't make any claims in the ads relating to their products. That said, what Oracle said in the launch was as much a violation of the TPC principles, so it's a wonder that Oracle wasn't cited for this as well.

The TPC has fined Oracle $10,000 for violating its fair use rules, as you can see from the letter that Majdalany sent to Michael Brey, Oracle's representative at the TPC consortium. It has also required that Oracle immediately stop running the ads in any publications and to remove a Web page at www.oracle.com/sunoraclefaster where it was making performance claims. That page has indeed been removed.

"Oracle's claim that it is faster than IBM using a TPC-C benchmark result it claimed would be announced on October 14 was not supported because Oracle did not have a TPC result at the time of publication," Majdalany said in a statement. "The TPC requires that claims based on TPC benchmarks must be demonstrable using publicly available data from official TPC benchmark results." Majdalany added that as of today, Oracle has not submitted any paperwork to uphold its performance claims.

That doesn't mean there won't be a double-digit TPC-C OLTP benchmark on October 14 for Sparc-based machines of some sort, or that Oracle and Sun are not cooking up OLTP and data warehousing benchmarks for x64 and Sparc variants of the Exadata boxes.

While IBM will count this as a victory, all IT vendors game the TPC-C test, which was useful when it was announced almost two decades ago but now has so much tweaking and tuning that you have to be careful about using the test results.

Big Blue, for one, seems to have mastered the black art of de-randomizing the data coming out of the TPC-C transactions, steering data to precise bits of system cache on Power Systems boxes running AIX and DB2 - or so other server makers have claimed to me. This tuning may account for nearly half of the performance of the box. I base this assertion on IBM's own Commercial Processing Workload (CPW) benchmarks for the i/OS-Power platform, which are loosely based on the TPC-C test, and its similar Relative Performance (rPerf) test for its AIX-Power platform, which is also derived from the TPC-C test. Several years ago, the OLTP performance of Power servers started to diverge, with AIX machines suddenly able to perform around 50 per cent more OLTP work than their i/OS counterparts. I have attempted to get clarification on this divergence from Big Blue's benchmarkers, to no avail, for many years.

I don't expect it today. But by bringing this up, maybe Oracle and Sun (and HP, who has complained about this to me for years) can encourage the TPC to take a hard look at what IBM is doing, too. Or, at the very least, get Big Blue to explain how one machine seems to do a lot more work than the other. I personally don't believe that the DB2 on the i platform is that much worse than the DB2 for Unix boxes. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Fat fingered geo-block kept Aussies in the dark
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
Turnbull should spare us all airline-magazine-grade cloud hype
Box-hugger is not a dirty word, Minister. Box-huggers make the cloud WORK
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
Do you spend ages wasting time because of a bulging rack?
No more cloud-latency tea breaks for you, users! Get a load of THIS
prev story


Free virtual appliance for wire data analytics
The ExtraHop Discovery Edition is a free virtual appliance will help you to discover the performance of your applications across the network, web, VDI, database, and storage tiers.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
The total economic impact of Druva inSync
Examining the ROI enterprises may realize by implementing inSync, as they look to improve backup and recovery of endpoint data in a cost-effective manner.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Website security in corporate America
Find out how you rank among other IT managers testing your website's vulnerabilities.