Linux camp slashes out at ‘NT beats Linux’ survey

A Microsoft-sponsored test has 'proved' that the combination of Windows NT and Microsoft Internet Information Server outperform Linux and Apache as a Web server, and Linux and Samba as an SMB file server. Unsurprisingly the survey has come under withering (and, it appears, justified) fire from the Linux camp. The test was carried out by Mindcraft, Inc., whose previous greatest hits include providing evidence that supported Microsoft's claims that it was all Apple's fault that Quicktime didn't work with Internet Explorer. The outfit concludes that NT 4.0 is 2.5 times faster than Linux as a file server and 3.7 times faster as a Web server. Presumably this means that an awful lot of customers out there (including The Register, which scores 100 on the Dumbo scale by running Linux and Apache) are just plain stupid. NT isn't exactly the world's Web server of choice, whereas Apache is. Mindcraft's results somewhat cheekily suggest that NT is more scalable than Linux, and show NT's NetBench throughput starting to climb above Linux at 32 users. We could note here that even on Microsoft's figures this indicates that Linux is a more appropriate fileserver for most small businesses than NT, but the Linux people are already pointing out that Mindcraft's Linux installation was maimed. We'd be the last to suggest this was deliberate of course. If it wasn't the case that the installation was effectively crippled, then the results are still difficult to credit. NT is repeatedly criticised for scalability limitations, and Microsoft itself has repeatedly promised scalability improvements for the product. Steve Ballmer himself did so again, just last week, at WinHEC. Here's another little bit of cheek from Mindcraft: "The Linux 2.2.x kernel is not well supported and is still changing rapidly… We started the tests using Red Hat Linux 5.2 but had to upgrade it to the Linux 2.2.2 kernel because its Linux 2.0.36 kernel does not support hardware RAID controllers and SMP at the same time. In addition, there are comments in the Red Hat Linux 5.2 source code noting that the SMP code is effectively Beta-level code and should not be used at the same time as the RAID driver. For this reason, we upgraded to the Linux 2.2.2 kernel, which has full support for both hardware RAID controllers and SMP to be used simultaneously. As of the date this report was written, Red Hat did not ship or support a product based on the Linux 2.2.x kernel." There's certainly some justification for this, but look who's talking. For inexperienced users it's probably reasonable to say that the installation of a large-scale Linux server system supporting both RAID and SMP is non-trivial, and this isn't helped by the fact that Linux is very much a moving target. But an inexperienced user who tries to do this, rather than hiring experienced Linux support that knows what it's doing, is a maniac. A user without serious NT experience trying to do so is the same. In support of its case, Mindcraft lists six Linux kernel updates so far this year. We know from the Halloween papers that Microsoft really sees this development momentum as a massive advantage for Linux, so flipping it on its head in this way is cynicism of heroic proportions. There is of course absolutely no danger that NT users will be threatened by anything like this rate of change - mind you, those NT service packs just seem to keep coming. The main challenge to the Mindcraft report however comes at the technical level. According to the Linux Weekly News critique: "no adjustments were made to the 'bdflush' and 'file system cache size' parameters supported by the 2.2 kernel," possibly accounting for a factor of two performance improvement in Samba, which was used for the fileserver test, when these are set correctly. The survey also used Windows clients rather than NT - Windows clients will tend to drive the results in favour of NT, while NT clients go better with Samba. LWN continues: "Their Samba configuration sets the widelinks parameter to 'no'. This setting increases the system call overhead for file name lookups considerably. The penalty is especially severe on SMP systems. "Their Apache configuration disables KeepAlive, an important real-world optimisation. (It has been pointed out that the tests do not use KeepAlive in any case). The Apache configuration is also not suited to large loads. It initially starts 10 servers, and MinSpareServers is set to 1. In particular, quick response to sudden, heavy loads will be reduced by this configuration. It is not an 'enterprise' configuration." Eric Green at Slashdot meanwhile points out that the system was probably running out of file handles, and that increasing them would improve performance, while there's also a possibility that running both systems on the same disk (which was the case with the test) would give whichever system was on the outer part of the disk a 1.5-2 times transfer rate advantage. Mindcraft's claims that it searched for support in newsgroups have also been questioned. According to Samba project member Jeremy Allison: "I just did searches on DejaNews in comp.protocols.smb, the samba mailing lists and also in the samba-bugs Jitterbug database (isn't it nice that we keep records of all email we get :-) looking for a request for help from Mindcraft with respect to their benchmark tests. Or even *any* email from Mindcraft. The only Mindcraft references I found were discussions of their previous benchmark trashing Syntax running on Solaris when benchmarked against Windows NT." Register reader Luca Lizzeri however suggests Mindcraft may have sought support under the pseudonym will@whistlingfish.net, but now tells us his eariler suggestion that peter@flyingsaucer.org is a possible ringer was a joke. Great domain name anyway, Peter... ®