Real-life lessons from Exchange 2000's first year

Microsoft Exchange 2000 Server hit the shelf about 1 year ago. During the past year, those of us who deployed the new product have learned many lessons—among them, that experience with Exchange Server 5.5 is no guarantee we'll automatically understand Exchange 2000. Something new occurs daily to remind me just how different Exchange 2000 is from Exchange 5.5. (These changes are a good argument for testing new Exchange 2000 deployments before you introduce them into your production environment, as the sidebar "Test First," page 84, explains.)

Nothing teaches like real-world deployment experience, and we can now begin to identify the most important elements in successful Exchange 2000 deployments. If you've been waiting to deploy Exchange 2000, you can benefit from this experience. To keep your Exchange 2000 organization running smoothly, consider the following factors: where you place Global Catalog (GC) servers, how you use the Active Directory Connector (ADC), how add-on products work with Exchange 2000, which Exchange clients you employ, which Windows 2000 features affect Exchange 2000, and whether you can consolidate Exchange servers.

Locate the GC
Each GC is a specially configured Win2K domain controller (DC) that holds full read/write copies of objects in its domain and partial read-only copies of objects in all the other domains in an Active Directory (AD) forest. Exchange 2000 fetches the Global Address List (GAL) from a GC and uses the GC to resolve message addresses during the routing process.

Unless the Exchange server can establish and maintain contact with a GC, Exchange 2000 can't start its services or route messages. Without a GC, Exchange users can't see the GAL, and the Exchange DSAccess component issues countless error messages.

DSAccess is the Exchange component that handles directory access. (In effect, DSAccess replaces the Directory Service—DS—that shipped with earlier versions of Exchange.) In concept, the DSAccess component's work is straightforward: When the Exchange services (e.g., the Routing Engine, the Message Transfer Agent—MTA) start, the component connects (i.e., binds) to a suitable GC and maintains connectivity so long as the Exchange server is running.

When deciding which GC to connect to, the DSAccess component weights site information heavily. Exchange generates a significant load on a GC, which must respond to queries against the directory for routing information, configuration data, and user details. (The exact load depends on the volume of message traffic, but many deployments use a rule of thumb that dictates one GC for every four Exchange servers.) The constant interaction between Exchange and the GC mandates that every Exchange 2000 server connect to the closest—in network terms—GC. To determine which GC an Exchange server connects to, open the Microsoft Management Console (MMC) Exchange System Manager (ESM) console. Select the Exchange server, open the server's Properties dialog box, and go to the General tab, which Figure 1 shows. The name of the GC appears in the Domain controller used by services on this server text box at the bottom of the tab.

Many organizations experience problems (e.g., slow message delivery, Microsoft Outlook clients freezing up) when an Exchange 2000 server connects to a GC in a remote Win2K site (sometimes because an administrator placed the Exchange server in the wrong Win2K site during installation). My personal experience confirms the importance of deploying Exchange 2000 servers close to GCs. In this context, the word close means that the network link between the Exchange and GC servers must be reliable, fast, and resistant to interruption. The safest option is to locate a GC on the same LAN as the Exchange server; stray from this approach only when you're certain that the link you intend to use is reliable.

Clearly, you need to perform some design work during your Win2K implementation to determine how many GCs you need and their optimum locations. If you're lucky enough to have a single domain, then the task is much easier because every DC can function as a GC. However, multidomain implementations are the most common model in the corporate world. Of course, you can still make every DC a GC, but in a multidomain environment, this approach incurs significant replication traffic. Most of us prefer to use valuable network resources for other work, so you must balance Exchange's need for GC proximity with available bandwidth. The choice is simple: Restrict replication and force Exchange to connect over extended links with unpredictable service levels, or bite the bullet and deploy more GCs to ensure that Exchange (and users) can always access the directory.

Synchronize the ADC Way
Most Exchange 2000 deployments integrate into existing Exchange 5.5 organizations, so the "old" DS must synchronize with AD through the ADC (preferably the version that comes with Exchange 2000 rather than the basic version that ships with Win2K). The ADC is a configurable utility that manages connection agreements (CAs), which define how part or all of the DS replicates to AD. CAs can be uni- or bidirectional.

In essence, ADC deployment has three goals: Initially populate AD with DS data (i.e., information about existing mailboxes, custom recipients, and distribution lists—DLs), replicate DS-object changes to AD objects and vice versa, and transfer details of user mailboxes from Exchange 5.5 to Exchange 2000. The ADC can cope with organizations that vary from simple one-site designs to complex multisite designs with multiple recipient containers.

Unless your Exchange organization is extremely straightforward, you'll probably end up tweaking your initial configuration of the CAs that the ADC manages. Become familiar with the subtleties in the tool's methods of dealing with DLs (which become AD groups), mapping recipient containers to AD organizational units (OUs), and ensuring that mailboxes that multiple Windows NT accounts share transfer to unique Win2K accounts with unique Exchange 2000 mailboxes. I suggest that you back up your live Exchange 5.5 server, restore the backup to a test system, and work through the steps necessary to set up replication to a Win2K DC. Confirm that uni- and bidirectional replication work correctly and that the ADC replicates changes in a timely manner, replicates all mailbox details, and correctly populates DLs. Repeat your tests until you're absolutely confident that your configuration will work well in your production environment. (For information about using the ADC, see Kieran McCorry, "Real-Life ADC Deployment, Part 1," May 2001, and "Real-Life ADC Deployment, Part 2," June 2001.)

Review Add-ons
Almost every production Exchange server uses one or more add-on products. The most popular categories are backup utilities (e.g., Legato Systems' Legato NetWorker, VERITAS Software's VERITAS Backup Exec), fax connectors (e.g., Fenestrae's Faxination) or other connectors, systems management products (e.g., NetIQ's AppManager, BMC Software's PATROL), and antivirus software (e.g., Trend Micro's ScanMail for Microsoft Exchange, Sybari Software's Antigen for Microsoft Exchange).

You don't always have a choice about the products you use for your Exchange organization. (For example, backup utilities are often part of an overall backup strategy that must cover different OSs and applications.) However, migrating to Exchange 2000 gives you the opportunity to review the add-on products you're using and—if you're in the position to make such choices—decide whether to change products. Add-on products designed for Exchange 2000 cope with Exchange-specific elements such as Store partitioning, the streaming file, the new cluster model, the new Transport Core, and front-end and back-end server configurations. If you use add-on products that fully support all the Exchange 2000 features you want to use, you end up with a more stable environment.

Reconsider Your Clients
While you're evaluating your add-on products, take a look at your Exchange clients and determine whether you're using the best clients for your environment. You've never had a wider array of Exchange clients to choose from (for a discussion of your options, see "Selecting an Exchange Server Client," Summer 2000), but one of the primary contenders might surprise you: Outlook Web Access (OWA) 2000.

OWA 2000 comes with Exchange 2000 and provides a glimpse at the possibilities of a highly functional browser-based client. Earlier OWA versions are handicapped by the Active Server Pages (ASP) architecture, which doesn't scale well in terms of supported users per server. The earlier versions can't deliver sparkling performance and don't provide a huge feature set, but in OWA 2000 performance is acceptable with even the largest mailboxes and even over slow dial-in connections. Tight integration between the Store and Win2K's Microsoft Internet Information Services (IIS) 5.0, as well as the movement away from ASP, contributes to better performance. Microsoft has also removed some code bottlenecks, so servers can now scale up to support more concurrent client connections.

When users employ Microsoft Internet Explorer (IE) 5.0 or later, OWA 2000's features make OWA a reasonable client for day-to-day use. As Figure 2, page 85, shows, users get the preview pane, rich text editor, and default views (e.g., view unread messages, view messages by sender) that you find in Outlook. (If users employ an earlier version of IE or a Netscape browser, OWA restricts them to what Microsoft refers to as a reach client—a nice way of saying "please upgrade.")

Certainly, many predominantly desktop-bound users who need access only to email and a calendar can use OWA 2000 in place of Outlook. IE 5.0 and later offer extended protocol support (i.e., support for Dynamic HTML—DHTML, WWW Distributed Authoring and Versioning—WebDAV, XML, and Extensible Style Language—XSL). This support is important and the key to OWA 2000's enhanced functionality.

Of course, gaps in functionality exist between OWA 2000 and Outlook. For example, OWA users can't send or read encrypted email, recover deleted items, or access task features. However, most users simply want to get to their email and calendars. Furthermore, Exchange 2000 Service Pack 1 (SP1) closed some of the gaps, and future service packs will surely close several more.

From an administrator's perspective, OWA 2000 is easy to deploy and manage. And its range of features makes it worth considering—even as a direct replacement for Exchange 2000 Messaging API (MAPI) Outlook clients.

Use Win2K Features
As hardware-software combinations become more complex, additional effort and attention to detail are necessary to ensure users a predictable level of service. Becoming familiar with certain Win2K features can help you achieve operational excellence in your Exchange 2000 organization.

Deployments—especially at the enterprise level—are proving that Store databases are larger than ever before. Microsoft designed Exchange 2000 servers to be bigger and to host more mailboxes, and many systems administrators are willing to increase mailbox quotas to facilitate users' pack-rat tendencies. When Store databases pass 100GB or so, determining the best method to back up these large databases becomes a concern. The traditional approach is to back up from disk to tape—a method that's appropriate for small to midsize servers. However, the general rule of thumb is to keep the time required for full backups to 4 hours or less, on the assumption that a restore will take twice as long—a full 8-hour workday. Even the fastest tape arrays process data at no faster than 35GB per hour, so processing a large server's mailbox stores and public store within the 4-hour target might be impossible.

Exchange 2000 can exploit Win2K's version of NT Backup to back up from disk to disk. Depending on the space available and the I/O capability of your disk subsystem, you can back up to disk, then copy the backup set to tape at your convenience. (However, the disk-to-tape approach isn't always viable. Tests on my server demonstrated that backups to disk proceeded at 6GB per hour when the server was loaded; a DLT tape drive could have processed data faster.)

You can also employ other Win2K features to your benefit. For example, you can build a customized MMC console from the snap-ins that deal with AD, the ADC, and the ESM. My favorite trick, however, is to use Win2K Server Terminal Services to manage Exchange 2000 servers remotely, as Figure 3 shows. (Win2K Server includes a basic Terminal Services license that permits two concurrent connections.) You can even use Terminal Services to manage servers over a dial-up connection. I've found that response over a 28.8Kbps link is slow but acceptable.

Explore Consolidation
Microsoft designed both Win2K and Exchange 2000 to scale up further than their predecessors can, so server consolidation is a pretty hot topic at the moment. A critical look at most corporate NT deployments will probably reveal that too many servers are in use. Servers tend to last between 2 and 3 years in a production environment. Because of advances in processor power, disk capacity and speed, and throughput, you can often consolidate several 3-year-old NT servers into one Win2K server. For example, if you operate 10 NT servers that each support 500 Exchange 5.5 mailboxes, you might be able to consolidate the servers into 2 or 3 large Win2K and Exchange 2000 servers or clusters. (Lack of network availability is a typical constraint on consolidation projects, but you should compare the expense of upgrading your network links to the lower operating costs of running fewer servers.) An Exchange 2000 migration project can be the perfect opportunity for consolidation, especially because Exchange 2000 includes two advances—the partitioning of the Store and better clustering—that can help you effectively consolidate servers.

Exchange 2000 Enterprise Server running on Win2K Advanced Server supports active/active clustering, which means that all the systems in a cluster can run services and support users at the same time. Figure 4, page 86, shows the set of active Exchange resources on a two-node cluster, including Exchange services (e.g., System Attendant, MTA, Routing Engine) and physical resources (e.g., disks, network names, IP addresses). Collectively, the resources form an Exchange Virtual Server (EVS). Typically, you run EVSs on a one-EVS-per-physical-system basis. You can configure a system to support multiple EVSs, but because of the demands that Exchange makes on system resources, I recommend against this type of configuration. (For information about clustering Exchange 2000, see Jerry Cochran, "Clustering Exchange 2000, Part 1," December 2000, and "Clustering Exchange 2000, Part 2," January 2001.)

Exchange 2000 is limited to two-node clusters, but Exchange 2000 SP1 increases support for as many as four nodes when running on Win2K Datacenter Server. (The SP1 download site, at http://www.microsoft.com/exchange/downloads/2000/sp1.asp#sp1_improvements, includes descriptions of some of the improvements in SP1. For information about using SP1 on Datacenter, see Jerry Cochran, "Exchange 2000 SP1 on Datacenter," July 2001.) However, Datacenter—and thus four-node clustering—is an expensive proposition, so the majority of current deployments use two-node clusters, which support as many as 1000 mailboxes on each node. In production, Exchange 2000 clusters are a mixed bag. Active/active clusters are nice because you can support active users on all the systems in the cluster, but state transitions can still take a long time, so clustering doesn't necessarily improve system availability.

Even administrators who have worked with Exchange 5.5 clusters can find Exchange 2000 clusters a challenge, so be prepared to take time to learn the difference between the two and to master the operating environment before you put clusters into action. Also, many administrators connect clusters to a Storage Area Network (SAN), which provides the necessary storage, flexibility, expansion, and stability to support many mailboxes. Clusters and SANs can deliver great advantages, but these configurations are complex to set up and manage, and you must master additional Exchange 2000 tasks (e.g., disaster-recovery procedures for a Store that's divided into multiple storage groups or databases). Make sure that you receive appropriate training to prepare to run the new servers before you begin a consolidation or clustering process.

Lesson Learned
Like all new software, Exchange 2000 has taken time to get used to. (And like all software, the initial release includes bugs that appeared only after the software was exposed to the stresses of a production environment. For a short list of Microsoft articles that address these problems, see "Related Microsoft Articles." Exchange 2000 SP1 also provides fixes for many problems; see the Microsoft article "XGEN: List of Bugs Fixed in Exchange 2000 Server Service Pack 1" at http://support.microsoft.com/support/kb/articles/q301/4/52.asp for details.) The biggest lesson from a year in production is not to rush. Take the time to prepare a solid migration plan and to carry out proper testing procedures, and your Exchange 2000 deployment will run smoothly.

RELATED MICROSOFT ARTICLES
"BUG: PdhExpandWildCardPath() ANSI Version May Fail on Windows 2000"
http://support.microsoft.com/support/kb/articles/q274/8/53.asp

"XADM: Additional Strings Created for 5.0.0 NDR"
http://support.microsoft.com/support/kb/articles/q287/6/02.asp

"XADM: Information Store Generates an Access Violation During Backup"
http://support.microsoft.com/support/kb/articles/q290/1/00.asp

"XADM: Users Receive Random 5.7.1 and 5.7.3 NDRs"
http://support.microsoft.com/support/kb/articles/q283/2/87.asp

"XCON: Large Number of 5.0.0 NDRs in a Routing Group"
http://support.microsoft.com/support/kb/articles/q282/5/06.asp

"XGEN: Rollup of Selected Exchange 2000 Server Post-Release Fixes"
http://support.microsoft.com/support/kb/articles/q291/2/22.asp