Upgrade soon to improve Exchange quality and stability
All products go through a life cycle that varies in intensity. At the beginning, the pace of change is hectic as engineers strive to add features to build a product that takes market share and wins deals. Over time, the need to add features lessens. Engineers fix bugs, but the product stabilizes into a shape that holds until engineers are ready to release a new generation of the product, at which time the cycle restarts.
Microsoft Exchange 2000 Server is the second generation of Exchange and incorporates a lot of changes from the previous generation (Exchange Server 4.0 through Exchange Server 5.5). The Store is partitioned, a new SMTP routing engine replaces the X.400-based Message Transfer Agent (MTA), and Active Directory (AD) provides directory services. Exchange 2000 Service Pack 1 (SP1) and SP2 refine the product by fixing bugs that appeared during production use and introducing new features such as the Mailbox Manager in SP1 and a rewritten directory access (DSAccess) component in SP2.
In late July 2002, Microsoft released Exchange 2000 SP3 (which you can download from http://www.microsoft.com/exchange/downloads/2000/sp3/default.asp), but don't expect to find many new features. Instead, Microsoft expects SP3 to improve quality and stability by delivering a comprehensive set of fixes accumulated since SP2 went out the door. At the same time, SP3 provides the platform for future upgrades to the next functional release of Exchange (code-named Titanium), expected in mid-2003. You won't be able to upgrade from SP2 or SP1 to Titanium.
Additionally, although Microsoft doesn't support running Exchange 2000 SP3 on a Windows .NET Server (Win.NET Server) 2003 system, SP3 provides some initial support for the first deployments of Win.NET Server by being able to use a Win.NET Server domain controller (DC). However, support for Win.NET Server isn't a high priority for SP3 because Microsoft doesn't expect the new OS to be in widespread use before Titanium ships.
The only visible evidence that you've upgraded a server to SP3 is a new build number. Figure 1, page 40, shows the build number as seen through the Exchange System Manager (ESM) console. Note that the build number of the shipping release is 6249.4, rather than the 6249.1 you see in the figure, because Microsoft fixed a couple of last-minute bugs just after I wrote this article. Microsoft thinks of Exchange 2000 as version 6.0. Titanium is known as version 6.5, which indicates that it's an evolutionary upgrade to Exchange rather than the major change that Exchange 2000 marked.
Because SP3 doesn't include any major new features, you might conclude that you can overlook it. However, this view is a little short-sighted because SP3 includes some important security upgrades among its bug fixes. Also, you'll eventually have to deploy SP3 if you want to upgrade a server to Titanium. A review of SP3's contents might help you determine whether deploying the service pack early is a good idea.
Lower Network Demand
The SP2 DSAccess component provides a new set of suitability tests to improve the way that Exchange selects DCs and Global Catalogs (GCs) and streamlines the failover process that occurs when a DC goes offline. SP3 builds on the new platform by reducing the traffic between Exchange and DCs, largely through better caching of configuration data and information about AD objects such as mail-enabled accounts and contacts. SP3 also reduces the number of Lightweight Directory Access Protocol (LDAP) queries made when the routing service validates addresses in message headers.
The impact of these changes varies from network to network. If your servers operate across low-bandwidth links that connect Exchange servers to GCs, you can expect to see fewer outages or delays in message routing caused by loss of connectivity between Exchange and GCs. On the other hand, if you always place Exchange servers in close network proximity to GCs (i.e., the Exchange servers and GCs are on the same LAN segment whenever possible), the SP3 changes have less impact. I always like to see Exchange's network demand driven down because not every company has high-speed LAN-quality connectivity always available.
SP3 also updates DS2MB (the System Attendant component that synchronizes changes made to the Microsoft IIS metabase with the Exchange configuration data held in the AD configuration naming context) so that it makes fewer LDAP calls to the AD. The IIS metabase holds configuration data about many services that Exchange uses, including the protocol stacks (e.g., HTTP, POP3, IMAP4) and SMTP virtual servers. DS2MB ensures that any changes made to the metabase or to AD (through ESM) are synchronized. The need for synchronization might seem strange, but it's a side effect of the way that Exchange now consumes services from IIS rather than implementing its own support for Internet protocols, as Exchange 5.5 does.
Better Memory Management
Virtual memory management has proved to be a major challenge for high-end clustered Exchange servers that host thousands of concurrently connected clients. (Most of the problems observed to date have occurred with MAPI clients, but the same symptoms appear with thousands of clients that use other protocols.) Problems occur when not enough contiguous virtual memory is available to move storage groups and databases to another node. As the Store takes and releases virtual memory to handle client demands, the memory becomes fragmented (just like a heavily used disk over a period of time). If a cluster transition then occurs and the Store process on the node that's taking over responsibility for a storage group and its databases doesn't have enough free contiguous virtual memory to mount the databases, users won't be able to access their mailboxes. This problem restricts the number of concurrent clients that a cluster node can support in an active-active configuration.
As Microsoft has improved memory management, the number of supported client connections has increased from approximately 1000 in the original release of Exchange 2000 to 1900 in SP2. Microsoft hasn't yet completed its testing for SP3 to determine the new approximate client limit, but you can expect it to be higher because Microsoft has worked hard to improve the way that the Store manages memory internally. Stay tuned to Microsoft's Web site for the test results.
You need to take the client limit into account when designing and operating a cluster. Your mileage may vary depending on your mix of clients, the workload the clients generate, other software running on the server, and the length of time that the server has been up since the last reboot. Note that there are no restrictions on the number of clients in active-passive clusters, nor does Exchange 2000 place any formal restriction on the number of clients that can connect to a standalone server. However, memory-fragmentation problems sometimes occur on high-end standalone servers that handle the heavy workloads that thousands of concurrent clients generate. The Microsoft article "XGEN: The Exchange 2000 Information Store Reports an Event ID 327 Warning Message and Virtual Memory May Be Fragmented" (http://support.microsoft.com/default.aspx?scid=kb;en-us;q319682) describes how memory fragmentation can prevent an SMTP server from sending or receiving messages (functions that require substantial interaction with the Store). You can restart the SMTP service or reboot the server to address the problem, but SP3 should make this type of situation occur far less frequently. The Microsoft article "XADM: Monitoring for Exchange 2000 Memory Fragmentation" (http://support.microsoft.com/default.aspx?scid=kb;en-us;q296073) describes how to monitor Exchange to detect excessive virtual memory fragmentation.
Enhanced stability for software products comes from fewer bugs and more robust code, and those are Microsoft's major goals in SP3. The service pack incorporates more than 400 separate fixes, some of which exist as hotfixes for SP2 and earlier versions. The Microsoft article "XGEN: List of Bugs Fixed by Exchange 2000 Server Service Packs" (http://support.microsoft.com/default.aspx?scid=kb;en-us;q311456) provides a reasonably comprehensive description of the contents of all Exchange 2000 service packs.
The number of fixes in SP3 might surprise some people, but Exchange has so many components and is used in so many different configurations that you can expect people to report a wide variety of problems ranging from Help topic problems to bugs that cause the server to crash. Large flaws, such as security and memory management problems, are a small percentage of the overall bug total. Microsoft doesn't necessarily describe in detail all the fixes it makes to products, especially security-related fixes. However, Table 1, page 42, lists some of the major fixes that you can read about.
Assuming that your server already runs Windows 2000 SP2, applying SP3 won't take you long—expect to be able to upgrade a midsized server (single CPU, 1GHz or less) in less than 30 minutes, and you should be able to upgrade even large clusters in a few hours. SP3 is a cumulative release, meaning that you can apply it to servers running SP2, SP1, or the original version of Exchange 2000. You don't need to apply previous service packs before you apply SP3. You don't have to upgrade the AD schema before deploying SP3—the next AD schema upgrade for Exchange is expected in the Titanium release.
Microsoft recommends that you apply SP3 to servers that run the Active Directory Connector (ADC) first, largely because these servers can then take advantage of SP3's improvements in communication between Exchange and AD DCs. Front-end servers (protocol proxies) running SP2 can't access back-end SP3 servers in the same administrative group, so upgrade front-end servers to SP3 before upgrading back-end servers.
As with Exchange 2000, you can install SP3 only from an account that holds Exchange Administrator rights and is a member of the local administrators group. You must have Exchange Full Administrator rights to upgrade a cluster or a server that has the Exchange Key Management Service (KMS) installed.
Back Up Before and After
The actual upgrade is only part of the picture—you must perform a full backup before and after the upgrade. The backups are necessary because SP3 upgrades Extensible Storage Engine (ESE) components, which means that you can't roll forward transactions from SP3 transaction logs into a backup restored from SP2 because the database signatures don't match.
SP2 has the same problem, which the Microsoft article "XADM: Exchange 2000 SP2 Does Not Allow You to Restore Exchange 2000 or Exchange 2000 SP1" (http://support.microsoft.com/default.aspx?scid=kb;en-us;q316794) documents. In any case, backing up a system before upgrading important applications is always a good practice, and backing up afterward is certainly sensible, if only to prevent the need to redo the upgrade should future problems occur.
You can perform online or file-level offline backups. Users can be active during online backups, so these backups reduce system downtime, but many experienced administrators prefer a full-system, file-level backup of all disks before applying important upgrades. If you take a server offline, consider performing other maintenance tasks at the same time, such as moving files between disks or even rebuilding some of the Store databases (an action that can reduce backup times and save some disk space). But be sure to give users plenty of warning that the server will be unavailable.
Consider Add-On Products
Most Exchange servers run some add-on products, including backup software, messaging connectors, and antivirus scanners. After you install SP3, test that these products support SP3 and apply any required updates. For example, some products such as Sybari Software's Antigen (http://www.sybari.com) depend on a certain version of the Store or another Exchange component that SP3 updates. In these situations, the add-on products won't function properly until you apply an upgrade that supports SP3. Naturally, you should try out SP3 and all dependent product updates on a test server before going anywhere near a production server, just in case.
Permissions and Registry Settings
One annoying feature of an SP3 upgrade is that it resets to the default setting any permissions on an Exchange 2000 object (such as a server) that you've manually changed since you last upgraded your Exchange 2000 installation. Microsoft recommends keeping track of manual permission changes (something that isn't always easy to do) so that you can reapply them after installing SP3. SP3 also overwrites any changes you've made to the system registry.
I can see why SP3 sanitizes the registry—a changed value lurking there might interfere with the proper operation of the server—but why the service pack resets permissions is harder to understand. However, you can expect similar behavior in the future from full upgrades and service packs, so I underline once again the need to fully document any change you make to Windows and the applications that run on a server.
As I mentioned earlier, you can't run SP3 on a Win.NET Server system (at least, not while receiving support from Microsoft), but you can deploy SP3 on Win2K servers running in a forest that includes Win.NET Server systems. When you do so, code in SP3 can take advantage of some new AD features in Win.NET Server. However, SP3 won't be able to take advantage of at least one important Win.NET Server AD feature—the ability to rename domains and DCs.
Exchange 2000 depends on AD as a store for configuration data, including information about servers and connectors. When Exchange 2000 is present in a Win.NET Server forest, you can't rename domains and DCs. The reason is simple—renaming introduces a period of instability in the domain until replication is completed. Although the period of instability might be short in a small domain or when a high-speed network connects every DC, predicting the impact in a domain that's very distributed or that uses some low-speed connectors is tremendously difficult. If you change the shape of your organization by renaming domains or DCs, you run the risk that your messaging infrastructure won't be able to route messages. Microsoft hasn't yet determined how to let AD perform domain renames and preserve the integrity of the messaging system. Thus, for now, you can forget the renaming feature if Exchange is present in your Win.NET Server forest.
ESM on Workstations
SP3 includes an updated version of ESM with notably better message-queue viewing. You can install the updated ESM on other Win2K servers and workstations so that you can monitor and manage Exchange servers from those machines, but if ESM is already installed, upgrading to the SP3 version isn't mandatory. Unfortunately, Microsoft doesn't support installing ESM on Windows XP Professional workstations. This lack of support is a pity, and I hope that Microsoft soon provides the necessary components to run ESM on XP Pro.
An SP3 security modification removes broadly available read access to the IIS metabase. As a result, an existing application that works under SP2 and uses Collaboration Data Objects for Exchange Management (CDOEXM) or Collaboration Data Objects for Windows 2000 (CDOSYS) to send SMTP messages could fail. Although this change might cause a disruption for some customers, Microsoft says that the end result is a more secure system.
The most common types of applications that this change affects are those that don't authenticate themselves to Windows or use an Exchange mailbox. One example is a Web application that lets visitors to a site create and submit an Exchange email message to send feedback. If you have an application of this type, you'll need to change the code to provide the necessary credentials to use an Exchange mailbox when it submits the message. I haven't encountered many such applications, so I think Microsoft is right to make this low-impact modification to increase security. But you need to evaluate your applications in light of this change as you prepare your system for SP3.
How quickly should you deploy SP3? Applying the upgrade doesn't take much time or require a huge amount of preparation. SP3 is stable in production and increases the overall resilience of Exchange through better memory management, bug fixes, increased security, and reduced network overhead. My advice is to test SP3 and deploy it in production as soon as convenient.