Linux loses in NT tests – Mindcraft numbers still wrong?

PC Week Labs has unveiled the results of the latest round of the Linux versus NT benchmarks and, as generally expected, Linux lost. But the results of the five day rematch also tend to indicate that the initial Microsoft-funded Mindcraft benchmarks weren't quite all they should be. After the publication of the initial Mindcraft test results earlier this year various Linux luminaries criticised the way they'd been set up and the class of hardware they'd been run on. The Mindcraft tests, it was argued, unfairly hobbled Linux and ought to be re-run on a level playing field. That was the point of the PC Week Lab tests, which had Microsoft, Red Hat, Penguin Computing and Mindcraft representatives present to ensure fair play. Mindcraft's tests had claimed NT outperformed Linux as a file server by 2.5 times and as a Web server by 3.7 times. PC Week, bless 'em, doesn't seem to draw attention to the point, but its Labs results come up with NT being 2.3 times faster as a Web server (IIS 4.0 versus Apache) and around about the same as a file server. They also found that Linux performance didn't collapse when client load exceeded 16 computers, as Mindcraft's Apache benchmark had found, and the performance difference was a lot narrower on a single CPU box, although NT still won. The performance difference here was 41 per cent for Web server and 52 per cent for a file server. The PC Week Labs results also arguably understate their divergence from the Mindcraft ones in some areas. Earlier PC Week Labs tests (still however conducted after the Mindcraft test) had found Linux-Samba could outperform NT when NT Workstation clients were used. MS performance engineers turned loose on NT after that found performance improved if four NTFS partitions, each with their own transaction log, were used rather than one. So NT's performance in the latest tests benefited from this discovery. PC Week also reports that the lack of a multithreaded IP stack in the Linux networking subsystem causes a performance plateau in the OS, and the charts in the report are particularly interesting. They show fairly conclusively that Linux was barely able to take advantage of multiprocessor boxes at all, so this is a hill the OS needs to climb soon. PC Week Labs Report. ®