After the controversial Mindcraft benchmark tests this spring, which showed Windows NT 4.0 outperforming OS Linux by a factor of 2 to 3, rival factions have endlessly debated the tests, the results, and the relevance of such benchmarks. More recently, other tests have shown NT beating Linux as well, though not by nearly the extreme scores that the Mindcraft test portrayed. But the biggest sticking point, really, was that the Linux community wasn't taking part in these tests, fine-tuning the Linux machines as needed so as to best compete with NT.
Until now, that is.
PC Week Labs released the results of its latest Windows NT vs. Linux testing, which was audited by representatives from Microsoft Corporation and Linux vendor Red Hat Software. And like previous tests, Windows NT 4.0 beat Linux handily, despite heaving tuning and tweaking of the Apache Web server and Samba file-sharing feature used by Linux. And lest the Linux camp cry foul again (fear not, they will), the PC Week test took into account virtually every Linux tweaking tip it received. For example, Linux advocates point out that Apache isn't designed for speed and that an alternate Web server such as Zeus would be more appropriate. So PC Week tested Zeus on Linux as well (and, incidentally, found it almost exactly as anemic as Apache, performance-wise).
"In all the areas in which the Linux community cried foul, its assumptions were wrong," the report reads. "We tested Zeus on Linux and found its performance peaked almost exactly where Apache's did."
It turns out that the performance problems in Linux are caused by a poorly written IP stack in the Linux networking subsystem, and are not caused by the Web server at all. Red Hat says the problem will be fixed in the next major revision to Linux, version 2.3.
So let's look at the numbers. For Web serving, Windows NT's Internet Information Server (IIS) 4.0 handily beat Apache or Zeus on Linux, responding to an incredible 4166 requests per second, compared with 1842 for Linux, an advantage of 226% percent. Linux advocates complained that earlier tests used high-end multiprocessing systems that weren't typical Linux configurations, so the test also included single processor systems, where Windows NT 4.0 once again beat Linux, 1863 requests to 1314. So even on cheaper low-end systems, Windows won by 41%, contrary to the expectations of many Linux users.
What's particularly impressive for NT is that a single processor Windows NT Web server performs equally to a four-processor Linux server. This reverses the theory that a low-end Linux box can outperform a more expensive, high-end NT box. For Web serving, the exact opposite is, in fact, true.
File serving was another big win for NT: Linux and Samba lost to Windows NT in every test that was run. Linux's best number, 155.9 Mbps, fell well short of NT's 338.3 Mbps. And an earlier test that showed Linux and Samba beating NT with NT Workstation clients was turned around, as Microsoft representatives found the problem with the earlier testing: In this comparison, NT won yet again.
When it comes down to it, it's not surprising that NT performed so well: It's been in development for over a decade, was designed with networking as a primary component, and has been tweaked and put through the paces by the largest user base there is. But this isn't a black mark for Linux: For the open source OS to have performed as well as it did is indeed a testament to its prowess. While the gap may seem wide today, it's clear that the Linux community will be working on the very components that are responsible for this gap. In the future, Linux will likely come closer to the performance of Windows NT. By that time, of course, the heavily modified Windows 2000 will have shipped, and the game begins anew.
For more information about the tests, including some interesting performance charts, please visit the PC Week Web site