Lies, damn lies, benchmarks and awards

Ill-considered awards are damaging the IT industry's health

  • alert
  • submit to reddit

Top 5 reasons to deploy VMware with Tegile

Opinion Few would disagree that Microsoft deserves every marketing award ever invented - providing of course its business practices do not cause it to be disqualified. It is therefore surprising that Microsoft does not seem to receive the recognition of marketers, while other award givers fall over themselves to make awards to Microsoft for its products. Some of these are almost self-awarded. An example is the TRUSTe seal that Microsoft was awarded to "guarantee" that all Microsoft's web sites met TRUSTe's privacy standards. This is all very well, but how many web users realise that Microsoft and seven other portal companies set up TRUSTe in the first place? Then there's James Gray of Microsoft, who received the 1998 Turing Award from the ACM. This is a real honour, but although Gray acknowledged that it was for work done over 30 years with various colleagues, Microsoft was not so keen to admit that he had only been at Microsoft Research for the last 10 per cent or so of his career, after his corpus of research was largely completed. He's a Microsoft employee who gets away with using Adobe Acrobat .PDF document format. Some awards received must also be totally unexpected and unsought: an example in this category is the African American Institute Award in recognition of Microsoft's "commitment to the economic empowerment and development of Africa". Mutually beneficial Most awards are of course given by trade magazine publishers for mutual benefit: the award giver gets mentioned in advertisements, and sometimes in other editorial: "PC Universal Monthly awarded Microsoft its prestigious product of the year award for Microsoft Bob". Microsoft is also more likely to advertise in the magazine. The magazine publisher would claim that the awards are to serve the readers. For its part, Microsoft frequently uses them for marketing purposes, so "winning" is important. A particularly good example of this was seen in the Microsoft trial, where the defence made a great point of all the IE4 reviews that Microsoft 'won' against Netscape. There have also been cases where if Microsoft did not win one of the reviews it regarded as being important, it demanded a rerun - and there are documented accounts of pressure being exerted on the publisher behind the scenes to produce a revised, more favourable review. We even have the absurd situation of beta software being reviewed, and the bugs being ignored. The strange case of COM Other awards are more puzzling. Recently, Microsoft's COM/DCOM was awarded a 1999 Well-Connected award by CMP's Network Computing, in competition with Corba and EJB. We thought, as did many middleware professionals, that this was a mistake or a joke, but not so apparently. We wondered if the editors were following orders, but when we saw that RealPlayer was used to show the awards, and also noted that Caldera's OpenLinux 1.3 won in the network operating system category, we dismissed the idea. We saw that a letter in a recent issue of CMP's VAR Business said that "Microsoft has little or no chance against Linux. When academia starts discarding NT for Linux, the effect will propagate to students" so there seemed to be no sign of pressure being brought to bear from above - even though Microsoft does advertise in Network Computing. Nor does CMP's pending acquisition by United News seem to be connected. Our attention therefore turned to how the Microsoft products had been fingered. Editor-in-chief Fritz Nelson wrote that "the products chosen were tested over a networked enterprise firsthand by editors themselves". Furthermore, the winners had been chosen for "their mature capabilities for creating distributed applications built from components." We searched in vain for the methodology used, and the precise equipment specification (and of course what non-Wintel kit, if any, was included), but to no avail. Even the accompanying background article about middleware did not get round to discussing any reasons for the award. We did discover a couple of strange coincidences. The editor of Network Computing, Art Wittmann, was previously associate director of IT for the University of Wisconsin's College of Engineering. And which lab conducted DCOM tests that made DCOM "positively shine" a couple of years back: well, the University of Wisconsin. Nothing wrong with that, of course. We also found that last October, Wittmann wrote that "Ten of us from Network Computing recently spent a couple of days in Redmond getting the lowdown from Microsoft on everything from NT to COM". Microsoft elaborated on "the shortcomings of CORBA" but to give Wittmann credit, he was perceptive enough to realise that Microsoft's claim to be serious about porting COM to various versions of Unix was spurious, and said that after a few years Microsoft would cite poor sales and drop support. He was so right - except the support only lasted another four months until February, when James Utzschneider of Microsoft's application developer unit announced that Microsoft was dropping COM+ porting to Unix platforms, because "we'd need to see a market... we built the ports... promoted them heavily and we didn't see much interest." Microsoft decided to use XML as an intermediate data format to link applications across platforms. Wittmann also observed that in a previous Network Computing jolly in Redmond, a Microsoft product manager had described a heterogeneous environment as "one that contained NT4, NT3.5, Win95 and Win3.1". A clue? Coincidences aside, the explanation seems to be in a recent article by Richard Hoffman in Network Computing on distributed object models (the title is "Sneaking up on COBRA [sic]..."). It put the spike in CORBA largely because it was regarded as "highly complex". Hoffman says that CORBA is "too low-level for anyone but highly-specialised engineers to use", which is a value judgement that will be questioned by many practitioners. It is a real-world requirement of the enterprise that it be possible for data to be transferred between different platforms, and expecting this to be achievable by point-and-click is unrealistic. It is true that dealing with distributed objects is a job for professionals, and not for amateurs who never progressed beyond BASIC. Hoffman validly criticises CORBA cross-vendor compatibility issues that can cause concern, but such problems should be ironed out with a toughening of the testing arrangements that the OMG has set up with the Open Group. It's worth noting what the serious professionals with no vendor axe to grind have been saying about COM/DCOM and CORBA. The US Defense Information Systems Agency (no need to guess how critical their work might be) noted in a report prepared by the Mitre Corporation that DCOM "does not appear to be used extensively" and that although Software AG (encouraged by Microsoft) was porting DCOM to UNIX for the OS/390, the cost was "mind-boggling" at a cool $200,000. [This is reminiscent of when IBM over-priced CP/M-86 at $240 because it wanted MS-DOS to be successful, and to avoid legal problems arising form the fact that DOS was a rip off from CP/M.] The report concluded that DCOM should be considered to be "platform-specific middleware for Windows", and also notes that "DCOM interfaces tend to be larger, less intuitive, more manually intensive, and more error prone than the equivalent CORBA interfaces which in contrast are visually simpler and much more Java-like". As a consequence, Microsoft has had to develop an array of tools and components to compensate for this weakness, which means life's tougher for programmers. Another comparison study from distributed component specialist Quoin of Cambridge, Mass. concluded that "the COM specification is substantially incomplete with respect to the foundations of the architecture". Thomas Mowbray, author of the best-selling CORBA text, noted in an article in Object News the difficulties caused by Microsoft trying to eliminate its interface definition language in COM+, and that anyway the language has changed significantly with every release. As a result, the interfaces are demoted to language-specific source code that cannot support a multi-vendor environment, which is exactly what Microsoft intended of course. The CORBA Interface Definition Language IDL is, in Mowbray's view, "one of the most significant contributions in the history of computer science" and aids interoperation across a wide range of platforms. He also observed that DCOM cannot be used on the Internet - something that even Microsoft has had to admit. Mowbray came up with a stunning factoid: IDL and DCOM were both invented by the same person: Paul Leach. Hopeless patchwork Steve Cohen, an attendee at the last WinDev seminar nominally arranged by Boston University, but in practice stuffed with Microsoft evangelists, said afterwards that COM is "a hopeless patchwork quilt of an attempt to unify all manner of poorly designed pieces of the Microsoft monopoly under one technical banner. The COM evangelists freely made cracks about Visual Basic being the root of all evil, but they are acutely aware that VB is the dog that is pulling this sled. Everything must work with VB, which distorts all technologies that have to work with it." [Perhaps this is because Bill Gates never really progressed beyond programming in Basic.] Cohen considers that Microsoft is trying to retrofit technologies under the COM banner, and suggests this cannot be done satisfactorily. So the award given by Network Computing to Microsoft for its middleware does not stand up to serious scrutiny. If any award were justified, it should have been made in a different category - for connectivity in a proprietary world (e.g. all Microsoft). We do not really asperse the CMG staff involved, other than to observe that they are far from being alone in feeling the hot breath of Microsoft at every turn. It is easy to abandon objective analysis in the face of such pressure. Microsoft will use such "wins" (in the same way as the results of "surveys" that it has commissioned - or predestined -- as we now know from the trial evidence) to further its momentum marketing efforts for products that would have been better not seeing the light of day. Time for a challenge? The consequences of ill-considered COM/DCOM adoption are very serious for large organisations that have a mixed environment: it means the abandonment of open-connectivity possibilities in the future. With the prevalence of corporate mergers, it can never be known when it will subsequently prove important to be able to link to the non-Microsoft world. There were once some superior alternatives that have now withered away -- SOM, DSOM, and OpenDoc come to mind - but they were largely ignored by the then pro-Microsoft media, sidelined by IBM (probably because of its desire to get a deal with Microsoft for NT), and consequently little appreciated by users. The CORBA community, through the OMG, should be demanding its own re-run of a true mixed-environment test of COM, CORBA (and EJB, which should merge with CORBA), to be conducted independently, but with vendor tuning allowed. The methodology must be disclosed, and be seen to be fair, although it is unlikely that Microsoft would be a willing participant in a truly open contest. After all, Microsoft says it will not support a mixed-platform environment in the future. This award should not have been made in the face of such strong evidence that COM & Co are vendorware rather than middleware. ®

Choosing a cloud hosting partner with confidence


Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
The Heartbleed Bug: how to protect your business with Symantec
What happens when the next Heartbleed (or worse) comes along, and what can you do to weather another chapter in an all-too-familiar string of debilitating attacks?
Top 5 reasons to deploy VMware with Tegile
Data demand and the rise of virtualization is challenging IT teams to deliver storage performance, scalability and capacity that can keep up, while maximizing efficiency.