It was Microsoft wot done it
What did happen to that Bloor Report?
Regcast training : Hyper-V 3.0, VM high availability and disaster recovery
Database myths and legends (Part 9) In this series we're looking at the myths and legends of the database world - some are true, some false. The myth under the spotlight today is: Bloor Research once produced a report that slammed SQL Server so much that Microsoft had it suppressed.
OK, let's start at the top. Bloor Research is a highly respectable analyst company. Has it ever looked at scalability in database engines? A quick Google of the words "Bloor Scalability Database" produces over 40,000 hits and simply following a couple of links will convince you that, of course, the company has done so. This kind of report is meat and drink to Bloor.
Now try and find a reference to the highly critical one. I'll give you two clues.
- It did exist.
- It was published in March 1997.
It's hard to find any references to it. In fact, only by knowing, and searching for, the actual title of the report, The Realities of Scalability, was I able to turn up two references to it: here and here.
Both are contemporary news pieces, published shortly after it appeared. It seems very likely that these originally appeared in print and have subsequently migrated to the web as archive material.
So the report clearly did exist, indeed rumour has it that certain copies are still extant...
The 'supressed' Bloor report
It is still, today, a very impressive body of work. Over 130 detailed pages of complex tests that really put three database engines: DB2 on AIX, DB2 on WindowsNT, and Microsoft SQL Server 6.5 on WindowsNT, through the wringer.
Bloor tested the databases' performance under read-only and update conditions, varied the number of users and the number of processors. Finally, and unusually, Bloor actually performed statistical analysis on the results to verify whether the differences were significant.
It is also undeniable that the report is highly critical of SQL Server 6.5.
The summary says, for example, of the user scalability: "Put simply, Microsoft SQL Server for Windows NT at high numbers of users performs dramatically worse than either of the other two databases."
And of the stability, it says: "This database had a number of failure states that could repeatably be generated. With large numbers of users, it was found that the database would grind to a halt."
DB2 for NT generally acquitted itself well: "Despite the occasional unexplained server or database crash, it proved impossible to consistently generate a fatal error in DB2 for Windows NT. It seemed to be capable of taking most things thrown at it."
And DB2 for AIX was a positive paragon of virtue: "DB2 for AIX proved to be by far the most stable of the platforms tested. No repeatable errors were found, and the system never fell over. The performance never fell appreciably below that which was expected in any of our tests."
COMMENTS
Freedom
Now what I'd really like to know is how is it possible that the EULA can suppress constitutional rights to freedom of speech. Is it? Probably yes, but isn't it sort of absurd?
If you think something published was a libel, slander, whatever, than sue. Same if you think the benchmark was "bad". Isn't that how it's supposed to work?
Mr Crank
"An average 1 query a second, results in timeout errors?"
This was an estimation by another commenter (not a respected database benchmark tester) based on a performance of a single website, not inspection of the database itself and certainly not taking into account how badly the database was configured.
And what's more even 1 query a year can still result in a time out error if a badly written query meets a badly designed and un-optimised database.
You seem to have completely missed the point that you cannot slate a product by one person highlighting one bad implementation.
yes the products should be benchmarked, but at the same time, database installations and configurations are not always straight forward and it would be fair to invite all vendors to send representatives along to assist with the testing.
And as for JD Power... a completely different method of assessing product function/performance through user surveys. Not really benchmarking now is it
Can't handle 1 query a second reliably
"er no, any badly designed and configured database is going to result in errors, especially time out errors."
An average 1 query a second, results in timeout errors?
Again, I'd want to see free and independent benchmarks before accepting that no other database can deliver and average load of 1 query a second. Am I asking too much?
"Next you'll be blaming every broken down car on the manufacturer regardless of how conscientious or not the owner is regarding routine servicing and checking."
J.D.Power annual survey exists for car, why not databases? Why should they have the right to block bad benchmarks?
Fraser:
"So, (whoever the anonymous post was) you think that if someone/some company has done nothing wrong they have nothing to hide?"
Clearly I think they're hiding their benchmarks because they show something is wrong. Nested selects performance for example, I get very poor performance from SQL Server for queries that use nested selects. I'd like to be able to read benchmarks from sources I trust to tell me how each database performs with these.
But as it stands, bad benchmarks would prevent the report being distributed.
This is not acceptable, why should Boors be prevented from telling the truth about a bad product?
IT infrastructure monitoring strategies
Requirements Checklist for Choosing a Cloud Backup and Recovery Service Provider
Data control in the cloud
Cloud based data management
Enabling efficient data center monitoring
