MS paper touts Unix in Hotmail's Win2k switch
But concludes 'we should eat our own dog food'
An older MS internal whitepaper from August 2000 on switching Hotmail, which MS acquired in 1997, from front-end servers running FreeBSD and back-end database servers running Solaris to a whole farm running Win2K, reads like a veritable sales brochure for UNIX, but concludes that the company ought to set the right example by ensuring that each division "should eat its own dogfood."
The whitepaper, by MS Windows 2000 Server Product Group member David Brooks, has been posted on the Web by Security Office, which says it discovered the item and numerous other confidential MS documents on a poorly protected server. There are a number of other fascinating documents posted, in which the careful reader will find a veritable treasure map for hacking the citadel, but the one I enjoyed best was the comparison between Win2K and UNIX.
Among the observations is a very basic one about security: "A fact about UNIX is that it is easy for an administrator to ensure that there are no irrelevant services running. As well as giving the potential for maximizing performance, it is useful to be sure that there are no random TCP/IP or UDP ports open that could be used as a basis for an attack," the paper notes.
Next there's kernel stability: "Both the UNIX kernel, and the design techniques it encourages, are renowned for stability. A system of several thousand servers must run reliably and without intervention to restart failed systems," the author notes, and adds that, "Apache is also designed for stability and correctness, rather than breadth of features or high performance demands."
Then of course there's the cost of ownership, which MS insists, against overwhelming contradictory evidence, gives Windows an advantage: "FreeBSD is free. Although there are collateral costs (it's not particularly easy to set up) the freedom from license costs is a major consideration, especially for a startup."
And it's easy to minimize a UNIX system: "It is particularly easy to cut down the load on the system so that only the minimum number of services is running. This reduced complexity [and] aids stability and transparency."
Whereas: "A Windows server out of the box is an elaborate system. Although it performs specific tasks well (such as being a web server) there are many services that have a complex set of dependencies, and it is never clear which ones are necessary and which can be removed to improve the system's efficiency."
Another good thing about UNIX is that everything is out in the open, for admins, anyway: "It's easy to look at a UNIX system and know what is running and why. Although its configuration files may have arcane (and sometimes too-simple) syntax, they are easy to find and change."
Whereas in Win2K: "Some parameters that control the system's operation are hidden and difficult to fully assess. The metabase is an obvious example. The problem here is that is makes the administrator nervous; in a single-function system he wants to be able to understand all of the configuration-related choices that the system is making on his behalf."
Another strike against Windows is the GUI: "GUI operations are essentially impossible to script. With large numbers of servers, it is impractical to use the GUI to carry out installation tasks or regular maintenance tasks."
Then we have the ease of UNIX administration: "Most configuration setups, log files, and so on, are plain text files with reasonably short line lengths. Although this may be marginally detrimental to performance (usually in circumstances where it doesn't matter) it is a powerful approach because a small, familiar set of tools, adapted to working with short text lines, can be used by the administrators for most of their daily tasks. In particular, favorite tools can be used to analyze all the system's log files and error reports," the author explains, and notes further that:
"Over the years, UNIX versions have evolved a good set of single-function commands and shell scripting languages that work well for ad-hoc and automated administration. The shell scripting languages fall just short of being a programming language (they have less power than VBScript or JScript). This may seem to be a disadvantage, but we must remember that operators are not programmers; having to learn a block-structured programming language is a resistance point." Furthermore, "PERL ... is more of a programming than scripting language. It is popular for repeated, automated tasks that can be developed and optimized by senior administrative staff who do have the higher level of programming expertise required."
We find also that the Windows image size can be a real inconvenience on a big farm: "The team was unable to reduce the size of the image below 900MB; Windows contains many complex relationships between pieces, and the team was not able to determine with safety how much could be left out of the image. Although disk space on each server was not an issue, the time taken to image thousands of servers across the internal network was significant. By comparison, the equivalent FreeBSD image size is a few tens of MB."
And finally, we're reminded that Windows often needs a re-boot when a UNIX admin can simply edit a configuration file, stop the process in question, and immediately run it again with the new configuration.
This is also a great advantage when things go wrong: "A service may be hung, and rather than take the time to find and fix the problem, it is often more convenient to reboot [a Windows machine]. By contrast, UNIX administrators are conditioned to quickly identify the failing service and simply restart it; they are helped in this by the greater transparency of UNIX and the small number of interdependencies."
Another item worth mentioning, though not directly related to a UNIX comparison, is the cost of load-balancing technology and its supporting software. Using Windows load balancing service requires Advanced Server, whereas using Cisco's Local Director needs only Server. The costs, we discover, are dramatically different:
"Although Hotmail uses Microsoft software without license fees, we must consider this project as a model for real customers. Use of WLBS requires Advanced Server, but Server provides all the other features used by Hotmail. Using list prices, the cost comparison for a farm of 3500 servers is: Using WLBS (hence Advanced Server): $15M+ / Using LD and Server: $6M+"
Also very entertaining is the dramatic difference between the internal whitepaper and its public version on MS TechNet in terms of facts.
For example, TechNet assures us that, "administrators generally find benefit from porting 'cron' jobs to Windows Task Scheduler events. Both Microsoft Interix 2.2 and SFU allow administrators to port 'cron' files to Windows 2000 without any changes in most cases, allowing administrators to gradually transition scheduled events and scripts without impacting operations i.e. at migration scheduled events can still run as 'cron' jobs. After the migration, the 'cron' jobs can be migrated to Windows Task scheduler events. The Windows task scheduler has better integration with event logs."
But the whitepaper had found that, "using FreeBSD, such tasks are scheduled by the cron service. Jobs are scheduled by being listed in a file, one line per job. Changing the file is easy to accomplish using the command line (or rdist), and replacing the entire file is a good way to ensure that each server has exactly the schedule of jobs that the administrator intended. Jobs can be scheduled to execute once, or at intervals down to one minute.
"Although the Windows Task Scheduler service is fundamentally able to look after such jobs, the interfaces provided in Windows does not measure up to the task. The usual interface is the GUI, which is appropriate for setting up jobs on a machine at a time, is labor-intensive and error-prone.
"The command at is deprecated, is not able to schedule repeated jobs at a frequency of less than one day.
"The command jt was offered by the Task Scheduler team, but it is unsupported and awkward to use (it was intended for testing).
"None of the three interfaces offers an easy way to replace the current task schedule entirely. The team met the need by running the cron service provided in Services for UNIX. As described earlier, relying on Services for UNIX (or any other package subject to extra license costs) provides a bad model for other customer deployments."
So once again we see that TechNet is more a source of rhetoric than information, just in case their painfully-cheerful security bulletins had left anyone in doubt.
It is terrifying to contemplate the efficiency bonus MS would have enjoyed if it had only been willing to base its entire corporate operations on UNIX instead of eating its own dog food. The software monopolist might today be in the bizarre position of being the world's only consumer of unices. ®