Test lab ‘proves’ both cases in Novell vs. MS directory wars

Microsoft has made another attempt at beating Novell in the directory game, but has tripped itself up again in the process. This time Microsoft chose a Utah testing house called KeyLabs that Novell had used in the past, and asked it to carry out some tests so that Microsoft could have third party verification. Novell corporate strategist Brad Anderson told The Register that he thought it probable that Microsoft had found some non-real-world cases in its labs where NDS eDirectory (as NDS 8 is now called) could be beaten by Active Directory, and then asked KeyLabs to verify the tests independently. Novell was incensed, and also commissioned KeyLabs to run some real-world tests that, perhaps not surprisingly, show NDS performance in a very different light. While Microsoft referred to KeyLabs as a "Novell Authorised Testing Site" in its release, Novell countered that KeyLabs was a "registered Microsoft Certified Solutions Partner". You get the picture. Credit is due to KeyLabs for mentioning in its report commissioned by Microsoft that SSL security was switched off for its tests, and that IIS was not installed (increasing performance of course) - something that Microsoft forgot to mention in its release. Microsoft made a big issue of 96 scenarios being tried, but they were hardly realistic ones. The Novell tests, with more realistic configurations, included security and some searches not attempted by Microsoft for reasons that will become clear in a moment. An important issue not mentioned in the tests is that Active Directory only works with Windows 2000 (and not even NT), whereas NDS is happy with Solaris, NetWare, NT, Windows 2000 (and Tru64 this summer). There is something of a problem with Linux and NDS, since client connectivity with SAMBA does not yet support NDS authentication, but we digress. With multi-faceted, simultaneous searches, KeyLabs found NDS outperformed Active Directory by 250 percent on average. But worse was to come. Microsoft did not do any LDAP "contains" searches (completing a search across a sub-tree after a few characters are typed) when it defined the tests that KeyLabs was to run. Microsoft evidently knew that if more than one client workstation were used, Active Directory would not return any results at all: so much for Windows 2000's reliability. Where Active Directory was able to produce results with one workstation, NDS was 1250 times faster (yes, that is "times", and not "per cent"). Another example of Active Directory's inability to perform was seen in a mixed search test, where NDS returned results in 100 percent of cases, but AD failed to return results in 715 out of 6,000 cases - so that one in eight searches with Active Directory failed. Another scalability test showed that in a scenario with 10,000 searches - 100 clients performing 100 searches - NDS produced 250 results/second, but Active Directory failed to produce a single result after 60 seconds. Two different and important issues arise from these tests. The first is concerned with the ethical problems that are posed when a test organisation appears to bend both ways in endorsing test results from clients with horns locked. However scrupulously they perform their work - and there is no suggestion that KeyLabs did otherwise - such organisations are effectively muzzled by their clients from making any useful comments. Instead, we find the absurd situation of Matt Mace, director of quality at KeyLabs, being quoted by Microsoft as saying: "In each of the 96 LDAP-based tests we performed, Active Directory running on Windows 2000 Server exceeded the performance on Novell's NDS Version 8 running NetWare 5.1". But just three days later, he was quoted by Novell as saying that "In each of the LDAP-based tests we performed, using simultaneous different search types, Novell's NDS exceeded the performance of Active Directory running on the Windows 2000 Server." Mace had not returned our call inviting comment at press time. It seems that benchmark testing, which has been used in the industry since the days when big iron was compared, has not resolved the problems that were manifest nearly forty years ago. The other issue concerns the truthfulness of marketing messages. Yes, it is marketing, and of course there will be hype, but facts are supposed to be facts. Novell has been a little too quiet about its multiprocessing capability, promised for the next iteration of NetWare, and how this might affect present performance. Microsoft has been making excessive claims about the performance, scalability and reliability of Windows 2000. Yesterday, CIO Jim Yost of the Ford Motor Co told the Windows 2000 meeting in San Francisco that "Windows 2000 was chosen [by Ford] for both its scalability and reliability". He's not the only one who needs to dig deeper than the hype. ®