Penguin hits out at ‘artificial’ Linux benchmarks

Linux hardware outfit Penguin Computing has responded to the results of last week's Linux versus NT rematch at PC Week Labs. Penguin director of technical staff Mark Willey attended the tests, which Linux lost again, and has commented that he doesn't think the Linux community should have bothered with them. Penguin president Sam Ockman however yesterday described the tests as artificial. "Imagine," he says, "that there was a test that proved that a Ford could corner better than a Chevy at 120 m.p.h. The result of such a study, while technically accurate, would not be relevant to many customers. That is just the sort of study that we have seen." The controversy over Linux versus NT started with tests run by Mindcraft earlier this year, and tests by Ziff-Davis and then last week's 'official' rematch have followed approximately the same approach, and come up with results of at least a similar order (Mindcraft's earliest figures were extreme, and haven't been repeated). After some debate the Linux community seems to be moving towards a consensus that Linux should be assessed in more 'real life' type tests, of the sort that's just been published by c't magazine (Now Linux beats NT). Says Ockman: "Linux is far superior to Windows NT in four very important categories that were not considered in the tests: reliability, stability, security and expandability. These are some of the most important factors for any IT manager in making a purchasing decision." And then of course, Penguin adds, there's price/performance. Sour grapes? Up to a point, perhaps. The tests have given Microsoft valuable ammunition for use in the areas of the business Penguin makes its money out of, so a response of some kind was probably necessary. But it would still probably make sense for Penguin to come up with a longer term reply with some data attached to it. ®