Check up on your Exchange organization's well-being
As email has evolved into a mission-critical application and Microsoft has pumped up Exchange Server's feature set to meet customer demands, Exchange—and the job of administering it—has become increasingly complicated. Exchange Server 2003 is the most complex Exchange version yet, but it also provides more features and a tighter integration with Windows and Active Directory (AD) than earlier versions do. Compared with Exchange Server 5.5, Exchange 2003 is far more scalable, secure, and robust—when deployed and administered correctly. One way to help ensure that you're doing just that is to implement best practices for your Exchange organization.
Best practices change over time, but Exchange configurations often remain the same. Out of a lack of knowledge or resources, many Exchange administrators fail to keep up with current recommendations for achieving maximum performance, security, and availability. To help rectify this problem (and, theoretically at least, to generate fewer support calls and greater customer satisfaction), Microsoft debuted the Exchange Server Best Practices Analyzer (ExBPA) tool in September of last year. ExBPA is an automated scanning and reporting tool that you can use to check your Exchange organization against current Microsoft best practices. Because its analyses are restricted to Exchange and the Windows configuration that Exchange depends on, ExBPA is better thought of as a proactive health checker for Exchange servers and organizations than as a full-featured reporting tool (such as Ecora Software's Ecora Reporter). And you'll need to take your organization's specifics into consideration when reviewing ExBPA's results.
Best practices for any software product evolve as customers and vendors increase their knowledge of and experience with the product. In the case of Exchange, an initial set of best practices for a given version is laid down by customers who participate in early adoption programs and by Microsoft developers. Over time, these recommendations are amended according to customers' practical experiences because Exchange never quite behaves as the engineering specifications say it will. For example, initial advice about Exchange 2000 Server claimed that you could safely deploy active-active clusters. However, it turned out that Exchange 2000 encounters problems with virtual-memory management, especially on clusters, so Microsoft evolved a more pragmatic stance advising you to keep a passive member in the cluster (to better handle server transitions if a problem occurs). Exchange's increasing interaction with Windows provides another example. In the early days of Exchange 2000, we all grappled with the product's new dependency on AD and followed the guideline that a domain provides an adequate security boundary. Now, we understand that the only true security boundary is at the forest level.
Microsoft spent months developing ExBPA. The company involved many customers in the tool's development, chiefly to run ExBPA against their Exchange organizations and provide feedback about the results. Similar analyzers have been around for a long time, and all analyzers depend heavily on the quality of the knowledge built into the software. In the case of ExBPA, this knowledge comes from Microsoft's internal deployment of Exchange and the company's many customer projects. As such, the tool provides a reasonable knowledge base against which to check your Exchange organization.
ExBPA supports Exchange 2003, Exchange 2000, and Exchange 5.5 (in a mixed-mode organization only, because the tool gathers much of its data from AD). Before you get started with the tool, be aware of the caveats. Although you can run ExBPA on the Microsoft Small Business Server (SBS) version of Exchange or on a single-box configuration, the tool is designed for multiserver environments. Exchange 5.5 doesn't support some of the interfaces, such as Windows Management Instrumentation (WMI), that ExBPA uses to extract data, so analysis of your Exchange 5.5 servers will be limited. Exchange 2003 supports a more complete implementation of WMI across all server components, so you'll get the best results when you run ExBPA against Exchange 2003 servers. (Non-WMI data—such as performance counters, DNS, file locations, the Microsoft IIS metabase, and the registry—are largely common across all Exchange versions.)
Get Set ...
Installing ExBPA is easy. First, go to http://www.exbpa.com for information about the tool and instructions for downloading it. Downloading the kit gives you a Windows Installer file that you can install onto the system from which you want to run ExBPA. You can install the tool on an Exchange server, but to prevent the tool from interfering with other applications, I suggest you run ExBPA on a Windows XP Professional or Windows 2000 workstation. The workstation must be able to connect to all the Exchange servers in your organization—or at least to all the Exchange servers that you want to analyze—and to a WMI provider. The workstation needs to be reasonably powerful, but any fairly new laptop or desktop with more than 256MB of RAM will be sufficient for small to midsized organizations. The larger your organization, the more memory ExBPA requires; a good rule of thumb is to allocate 10MB of RAM for each server that you want to scan. You need to install the Windows Administration Tools pack and Exchange System Management Tools, then the Microsoft .NET Framework 1.1, on the XP or Win2K workstation. If you want to collect data from the IIS metabase, you must install IIS Common Files on each Exchange server from which you want to collect metabase data. ExBPA installs itself in the workstation's \program files\exbpa directory. At the time of this writing, ExBPA supports English and Japanese, with further language support scheduled as soon as Microsoft can translate the ExBPA components. Therefore, the ExBPA directory contains subdirectories that hold language-specific files.
To ensure that ExBPA's assessments keep up with the ever-evolving list of recognized best practices, Microsoft plans frequent updates to the rules that drive ExBPA. To get these updates, you can either use ExBPA's Update Exchange Best Practice Analyzer option to manually update the tool, or you can rely on the automatic download that ExBPA attempts every time you start the program. (Whichever option you choose, I suggest you check the ExBPA Web site for updates and retest your organization every month or so.) Both methods involve ExBPA connecting to microsoft.com to check for an updated version of exbpa.config.xml, which contains the tool's rules and baseline measurements. If the file at the Microsoft site contains a version identifier greater than the local copy's, ExBPA prompts you to download the newer file. ExBPA also downloads an updated version of exbpa.chm, the compiled Help file, if available. ExBPA uses the updated files immediately after downloading them.
After you download and install ExBPA, you can run the program interactively by selecting it from the Microsoft Exchange program menu, or you can run exbpacmd.exe in batch mode or from the command line. This executable gives you a way to call ExBPA from the command line so that you can script scans or incorporate ExBPA into your scheduled system-maintenance routine. To see the available command switches, which let you specify parameters such as which Exchange server to review, run
When you launch ExBPA, its first step is to connect to an AD server. Exchange 2003 and Exchange 2000 store organization configurations in AD, and ExBPA needs to read this data to build a picture of the organization before beginning its scan. You can supply the name of a domain controller (DC) or Global Catalog (GC) server, or you can let ExBPA find one on its own.
The majority of ExBPA's processing is passive; the scanned server merely responds to queries from the ExBPA workstation. (The exception is ExBPA's test of SMTP communications, during with the workstation attempts to submit a message to the SMTP virtual server.) Because ExBPA needs to read information from various locations on the target servers, the tool must run under a suitably privileged account that has permission to access the registry and other locations. Microsoft recommends that you use an account that is a member of the Domain Admins group (so that ExBPA can read configuration data from AD and system data from WMI, the registry, and the IIS metabase) and that possesses Exchange View Only permission for the Exchange organization. If the account under which you're running ExBPA doesn't have all the permissions required to fetch all necessary data from all the servers you want to scan—perhaps because you've divided responsibility for Windows and Exchange between different administrators—you can input explicit account username and password information to connect to AD, as Figure 1 shows. Otherwise, ExBPA uses the credentials of the account that you use to run the program.
After ExBPA reads AD to discover information about the Exchange organization, the tool displays the page that Figure 2 shows. On this page, you select the servers that you want to scan. You can select individual servers, a specific administrative group, or the complete organization. ExBPA uses multiple threads to scan servers. If you run the tool on a single-processor workstation, ExBPA uses as many as 25 threads; on multiprocessor systems, the tool uses as many as 50 threads.
You can tell ExBPA the speed of the network connection between the workstation and the servers. The tool uses this setting to determine the times that it reports as it progresses and to set the underlying timeout that it uses when it attempts to contact a server to collect data. If you scan servers on the same WAN segments, you can safely use the Fast WAN setting, but you must specify a setting such as WAN (64 kbps) when you want ExBPA to collect data from servers across low-speed links. Expect as much as 3MB of network data to pass between the scanning workstation and each target server.
The original version of ExBPA checked against approximately 830 conditions, so it should come as no surprise that scanning even one server can take some time. Still, given reasonable network connectivity, scans proceed fairly quickly and I expect Microsoft to continue to improve the tool's performance as the company discovers how to optimize the queries that the tool makes to the servers. Microsoft cites itself as an example: ExBPA takes 2 to 3 hours to scan its approximately 100 servers. However, the company's Exchange deployment, which is highly centralized around a few core datacenters, is atypical. A more distributed organization, especially one in a hub-and-spoke network, can expect longer scan times. For example, HP's Exchange organization spans 297 servers in a highly distributed environment; ExBPA takes up to 1 day to scan all those servers. I don't mean this as a criticism of ExBPA, which gathers a huge amount of data from each Exchange server and from AD and performs a more accurate and detailed analysis than a human could perform in the same amount of time—but you need to be aware of the time that will be involved. Furthermore, ExBPA caches a scan's results in memory (for the sake of performance) and doesn't checkpoint data, so if a scan is interrupted, that data is lost; you'll have to start the scan again.
ExBPA keeps you informed of its progress as it reaches out to contact Exchange servers and collect data. In the instance that Figure 3 shows, ExBPA has successfully completed processing one server but has encountered problems with other servers. The tool uses one of three icons to display a scan's status. A yellow triangle with an exclamation mark (!) indicates that ExBPA was able to contact the server but that errors occurred during data collection—perhaps because some essential Exchange service (e.g., the Information Store service) wasn't running on the server. A red circle with a white "×" indicates that ExBPA couldn't connect to the server. ExBPA tests connectivity by attempting to read a well-known registry key on the server; if the attempt fails, ExBPA considers the server unavailable and doesn't attempt further processing. A green icon with a check mark indicates that ExBPA successfully collected data from the server. This doesn't mean that the Exchange server is in good health, it just means that ExBPA was able to collect the required data for analysis.
The Finish Line
After ExBPA processes all the specified servers, it analyzes the collected data against the rules and best-practice indicators in exbpa.config.xml and generates several predefined reports, which you can then view, export, or print. The most useful report is the Critical Issues List, which Figure 4 shows. This report lists all the pain points that Microsoft believes you need to address to run a well-managed Exchange organization. If you select a reported issue's Tell me more about this issue and how to resolve it link, ExBPA will attempt to connect to microsoft.com to retrieve more information. If no connection is available, ExBPA will direct you to its Help file.
Some reported issues are easy to resolve or might even turn out not to be critical. For example, ExBPA considers an offline server to be a critical issue. That designation is certainly accurate if someone has installed an Exchange server, then decommissioned it without removing it from the organization, but a server could simply be offline for maintenance or for some other good reason. The Global outgoing message size issue, which you can see in the report that Figure 4 shows, can be critical if your organization has low capacity connections; in such a situation, a 50MB message could quickly slow message traffic to a trickle. On the other hand, if your servers are all connected with high-capacity links or users absolutely need to send large attachments, you might have a good business reason to allow such large outgoing messages. ExBPA will flag the NIC is DHCP enabled issue if you use DHCP to allocate IP addresses to Exchange servers. Microsoft supports this practice but recommends against it because it can cause routing problems unless you carefully manage your IP leases. Still, DHCP is an appropriate mechanism to manage IP addresses for servers in some circumstances. These examples underline the need for you to interpret ExBPA's reports before you take action regarding a reported issue.
Another useful report is the Detailed View - Full Issues List report, which Figure 5 shows. As you can see, this report lists information about the organization's key attributes, including the AD environment, the number of servers and mailboxes, the present connectors, and the mailbox stores. You can drill down into various areas to see even more detail and compare that against your knowledge of the environment. This process is often illuminating, and you'll probably discover things about Exchange or your organization that you'd be hard pressed to find through Exchange System Manager (ESM). ExBPA provides an even more detailed report that can be of use to teams that need to investigate problems further.
ExBPA stores its output data as XML files in the \exbpa subdirectory under the \documents and settings root directory on the ExBPA workstation. The tool also stores log files that report the progress of the analysis, along with any errors that ExBPA encounters (e.g., not being able to contact an Exchange server). You can view the XML data to get an idea of the kind of queries that ExBPA makes to build its view of the organization. (For example, the data that Figure 6 shows reveals that the ExBPA workstation, HPQNET-DC9, is a dual-processor system with 512MB of RAM.) These XML files can be quite large. For instance, a scan of the complete HP production organization generates a file larger than 500MB.
ExBPA offers an import and export facility for its XML reports. For example, in a distributed organization, you can allocate responsibility for best-practice validation to the administrators of each Exchange administrative group and require them to send report copies to a central location, from which all the data can be reviewed. You can even provide copies of your organization's reports to an external consultant, who can then load the reports and perform a review or audit of your environment.
Practice Makes Perfect
As with any automated service or tool, you need to consider ExBPA's results' and recommendations' applicability to your installation to get full value from the tool. Rushing into action at every recommendation without considering the bigger picture might break something in your Exchange organization. For example, if you remove a permission that ExBPA considers to be too elevated, you might stop an add-on product from working. Use ExBPA as a pointer for situations you should check into, rather than as a definitive source of wisdom.
Anything that helps hard-pressed administrators to take better care of systems is worthwhile, and ExBPA definitely fits this category. I can honestly say that ExBPA is one of the best Exchange add-on utilities that Microsoft has ever released; used with care, it will be an essential part of your toolkit. Expect more Microsoft server products to provide similar tools in the future.