Original URL: http://www.theregister.co.uk/2007/05/30/myths_legends_nine/

It was Microsoft wot done it

What did happen to that Bloor Report?

By Mark Whitehorn

Posted in Developer, 30th May 2007 12:18 GMT

Database myths and legends (Part 9) In this series we're looking at the myths and legends of the database world - some are true, some false. The myth under the spotlight today is: Bloor Research once produced a report that slammed SQL Server so much that Microsoft had it suppressed.

OK, let's start at the top. Bloor Research is a highly respectable analyst company. Has it ever looked at scalability in database engines? A quick Google of the words "Bloor Scalability Database" produces over 40,000 hits and simply following a couple of links will convince you that, of course, the company has done so. This kind of report is meat and drink to Bloor.

Now try and find a reference to the highly critical one. I'll give you two clues.

  1. It did exist.
  2. It was published in March 1997.

It's hard to find any references to it. In fact, only by knowing, and searching for, the actual title of the report, The Realities of Scalability, was I able to turn up two references to it: here and here.

Both are contemporary news pieces, published shortly after it appeared. It seems very likely that these originally appeared in print and have subsequently migrated to the web as archive material.

So the report clearly did exist, indeed rumour has it that certain copies are still extant...

Bloor Research's supressed report

The 'supressed' Bloor report

It is still, today, a very impressive body of work. Over 130 detailed pages of complex tests that really put three database engines: DB2 on AIX, DB2 on WindowsNT, and Microsoft SQL Server 6.5 on WindowsNT, through the wringer.

Bloor tested the databases' performance under read-only and update conditions, varied the number of users and the number of processors. Finally, and unusually, Bloor actually performed statistical analysis on the results to verify whether the differences were significant.

It is also undeniable that the report is highly critical of SQL Server 6.5.

The summary says, for example, of the user scalability: "Put simply, Microsoft SQL Server for Windows NT at high numbers of users performs dramatically worse than either of the other two databases."

And of the stability, it says: "This database had a number of failure states that could repeatably be generated. With large numbers of users, it was found that the database would grind to a halt."

DB2 for NT generally acquitted itself well: "Despite the occasional unexplained server or database crash, it proved impossible to consistently generate a fatal error in DB2 for Windows NT. It seemed to be capable of taking most things thrown at it."

And DB2 for AIX was a positive paragon of virtue: "DB2 for AIX proved to be by far the most stable of the platforms tested. No repeatable errors were found, and the system never fell over. The performance never fell appreciably below that which was expected in any of our tests."

Was the report suppressed? Yes. The acid test is looking on Bloor's own site. There you'll find an archive that does, indeed, go back to 1997, but there is no record of any database scalability report.

Screen shot of Bloor website with no mention of the report.

Who was responsible for the suppression? When the report was initially published I was very impressed with the rigour of the testing, so when it suddenly disappeared I approached Bloor to ask where it had gone – whereupon the sorry tale emerged. Microsoft confirmed the basic facts although, as this contemporary column that I wrote in Personal Computer World shows, the stories from the two companies differed in detail.

November 1997

History ain't what it used to be (allegedly)

Those of you who have been worrying about the implications of the Bloor report on Scalability in Databases can stop worrying: the report essentially no longer exists.

The facts described in the report? Oh, they don't appear to have changed. Bloor did find that Microsoft's SQL Server did not scale well, did find the other points of concern, and the company still stands by those findings. What has changed is that the report itself is no longer an available part of the sum total of human knowledge. If you don't have a copy of it, your chances of obtaining all the information therein are slim indeed.

Here are some more facts, so far as I have been able to ascertain them. The report looked at RDBMSs from two companies, IBM and Microsoft. Bloor claims that both companies (Microsoft included) provided software and support people to set up the RDBMSs for testing; Microsoft disputes this and says it did not supply people to help with the test.

The licence for SQL Server says (in broad legalese) that you must have Microsoft's written permission before publishing the results of testing. Bloor did not obtain this written permission, and says that it took Microsoft's assistance in the project as tacit acceptance of the fact that the results would be published.

A couple of months after publication of the report, Bloor received a "cease and desist" letter from Microsoft's lawyers. This letter says Bloor is in breach of contract because it failed to obtain the required written permission. Since Bloor had indeed failed to do so, it felt that it had no option but to cease distribution of the report.

A reasonable person might ask why, if Microsoft felt that the report was inaccurate, it didn't try to stop the report being published on those grounds rather than on what appears to be a technicality.

The greatest loss from all of this may not be the actual information contained in the report, but the methods it used for testing databases. Testing databases for speed is a very complex area, too often treated in a trivial way. This Bloor report was, in my opinion, a genuine contribution to our attempts to derive a sensible methodology of testing databases.

As another piece of opinion, I feel that if Microsoft knew the report was inaccurate, it should have attacked on that basis. To attack it on a technicality rather than technical grounds seems to me the worst of all worlds, making the company look high-handed and arrogant.

Of course, I don't want to get any "cease and desist" letters myself, so I want to stress that I am not making any suggestion that Microsoft has behaved in an unreasonable way in this matter. I have every confidence that you can continue to think of Microsoft as one of the family; like, say, a brother – a big brother.

Have I proved the cause and effect – that Microsoft invoked the clause because the report was damning? No, at this point we run out of evidence so it is up to you to draw your own conclusions. I clearly drew mine 10 years ago.

Perspective

In the hopes of forestalling a "so this proves that Microsoft is a tyrant and that SQL Server is rubbish" response, bear in mind that this is history – it certainly tells us that 10 years ago Microsoft could be heavy handed and that SQL Server 6.5 was a poor product. But companies and products do change.

My belief is that SQL Server is now an excellent product. I would also stick my neck out and guess that Microsoft would probably not act in the same way today. This is not because, as a corporation, it cares particularly about truth and honesty; corporations are not the same as people and have different drivers and objectives. But I do think that Microsoft has learnt that a heavy hand brings bad PR...with, of course, the odd minor hiccough. ®