New tool gives your Exchange servers the once over
One of the best things about working with software is that it can be designed and built so that it's flexible and adaptable. Real-world physical objects are limited by their design and construction: The minivan my wife uses to take our kids to school can't morph into a sports car or a school bus. Software systems such as Exchange Server, on the other hand, can be deployed into a wide range of environments, but this flexibility raises the question of whether the software is being deployed properly. Not every administrator is aware of every nuance of Exchange setup and configuration. Microsoft has worked hard to build a product that "just works" in most situations, but it requires a fair amount of specialized knowledge to examine an Exchange server and see whether it's optimally configured.
As a secondary problem, consider what happens in midsized to large organizations. Humans aren't very good at repetitive tasks. Sure, we can do them, but tasks that require both attention to detail and repetition are especially difficult because we're easily bored. If someone walked into your office and asked you to examine every aspect of the Exchange configuration on one server, that wouldn't be too bad. On 10 servers, it would be somewhat tedious; on 400 servers, you'd probably go mad.
Microsoft has addressed both Exchange's complexity and the needs of organizations that have multiple servers with the release of the Exchange Server Best Practices Analyzer (ExBPA), an automated tool that runs on a Windows workstation and scans your Exchange servers for proper configuration. The tool checks more than 1200 distinct settings against a database of more than 1000 rules and produces a detailed report telling you what it found. Actually, there are two types of reports. First, the tool reports on problems it finds when checking your configuration against the rules database. Most problems link to pages on the Microsoft Web site that contain more information about the change needed. Some items are self-explanatory, but the ExBPA team is madly adding articles to flesh out the Web documentation. The report that ExBPA generates also contains some of the information found on the associated Web pages. The other type of report is a configuration snapshot of your Exchange environment, which gives you a way to view configuration changes over time. The snapshot report is an XML file that you can parse with your own tools or load into ExBPA on another machine. As ExBPA becomes more widely deployed, I expect to see Microsoft Product Support Services (PSS), Microsoft Consulting Services (MCS), and third-party consultants routinely asking for ExBPA reports as part of their initial troubleshooting procedures. Of course, ExBPA is no substitute for a detailed reporting package such as Ecora Reporter or Quest Software's Quest MessageStats.
How ExBPA Works
ExBPA obtains information in three primary ways:
- by making Windows Management Instrumentation (WMI) calls to the Exchange servers. WMI exposes a wealth of configuration and performance information, especially on Exchange Server 2003. For example, WMI provides an easy programmatic way to find out how many storage groups (SGs) and databases exist on a server or what the mailbox or public folder size limits are for a particular database.
- by looking up configuration parameters in Active Directory (AD). Recall that Exchange keeps its organizational configuration data (including the list of known Exchange servers, administrative groups, and routing groups) in the configuration naming context of the AD forest.
- by inspecting registry and metabase values on the Exchange servers. These values indicate which version of Exchange and related hotfixes are installed and contain some per-server configuration data that's not otherwise exposed. To scan a server's metabase remotely, your scanning machine must have the IIS Common Files component installed. It will if you've followed Microsoft's recommendation of installing the Exchange management tools on the workstation before you install ExBPA.
When you start an ExBPA scan, the tool begins by querying the domain controller (DC) that you specify; from this DC, it gets information about your Exchange topology. The tool then begins scanning individual servers by attempting to connect to each server and retrieve a registry key. If the connection attempt succeeds, the server will be queued for scanning. ExBPA will adjust the number of threads it uses to scan according to the number of Exchange servers that exist. A uniprocessor workstation can scan as many as 25 servers concurrently; a multiprocessor machine can scan as many as 50 servers at once.
This scanning takes a while, of course. In Microsoft's network, which features approximately 100 servers and excellent interserver bandwidth, a complete scan takes 2 to 3 hours. The exact time required to scan a server varies according to the load on that server, the load on the scanning machine, and the bandwidth available between them. Larger and more distributed environments can take up to 24 hours to scan.
Preparing to Use ExBPA
Microsoft has made ExBPA freely available at http://www.microsoft.com/exchange/downloads/2003/exbpa. After you download it, you can easily install it on any Windows Server 2003, Windows XP, or Windows 2000 machine, although Microsoft recommends running it on an XP or Win2K workstation. The machine you use must have network connectivity to the Exchange servers you want to scan, and you must have access to an account that has administrative privileges on each Exchange server you're scanning and at least View Only permissions on the Exchange organization.
ExBPA will scan Exchange 2003 and Exchange 2000 Server systems, and it will scan Exchange Server 5.5 servers in a mixed-mode organization—but if you have only Exchange 5.5, you're out of luck. ExBPA attempts to perform all scans and queries on all servers it can see, so it's normal to see some errors as ExBPA tries to test servers that don't support the queries it's making or for which you don't have administrative permissions. (Note that you can specify the account credentials to use when connecting to Exchange servers, so you don't have to log on using a privileged account.)
There seems to be some confusion about when you can, or should, run ExBPA. The ExBPA documentation says that you can use the tool for troubleshooting but points out that if it can't connect to a server, it won't be able to give you any useful information. Microsoft's recommendation is that you plan on using ExBPA for regular proactive health checks, using the data it gathers to pinpoint and fix any problems before they get out of control and cause data loss or other problems.
One of the coolest things about ExBPA is that its rules database is easy to update. When you launch ExBPA, it tries to download a file named ExBPA.Config.xml from Microsoft's Web site. The file's ConfigVersion attribute (a four-digit attribute in the form W.X.Y.Z, where W and X are the major and minor version numbers) is checked against your local copy's Config-Version attribute; if the Web version of the file is newer, ExBPA prompts you to download the new version. If you agree, the new ExBPA.Config file (and the associated Help file) is downloaded and used immediately; any existing versions of the configuration file are renamed. When you get a new version of ExBPA.Config, it can be used to reinterpret existing scan results as long as the major and minor version numbers (the W and X that I mentioned above) are the same; if either or both of them have changed, the new ExBPA can still display the old scan data but can't analyze it. Updates to the rule set are also posted on Microsoft's ExBPA page, so that you can download them manually if your Web proxy requires authentication or if you can't connect for some other reason.
The first update to the rule set came out less than a month after the tool's initial release, and Microsoft has said that it will update the rule set frequently. In addition, (and in rather unusual behavior for Microsoft), the ExBPA team has already been talking about the next release. Sometime in 2005, as part of the Exchange Web Release (WR) process, we'll get a new version of ExBPA that features better integration with the Microsoft Operations Manager (MOM) tool set and more localized language versions. Even better, Microsoft will be releasing best practices analyzers for its other server products. It's too early to tell whether these products will coalesce into one mega-BPA, but it's an interesting thought.
When you're ready to run ExBPA to scan your servers, you can do so in one of two modes. The regular ExBPA scanner (exbpa.exe) presents a graphical interface that leads you through the process of conducting a scan. Or, you can use the exbpacmd.exe tool to start scans from the command line so that you can schedule scans and archive the reports for future analysis. The two tools are equivalent in function; if you want a complete list of the command-line options, just use the /? command-line switch to display a list of switches and arguments. I'll walk through the process of scanning by using the GUI tool.
The first step in running ExBPA is the update check, which I described earlier. After that, the next step is to specify which DC you want to use as the basis for the initial scan. ExBPA will cache information about the Global Catalog (GC) servers and DCs it finds, but it has to start somewhere. This step also gives you the opportunity to pick an alternative set of credentials to use when binding to the DCs. After ExBPA has completed its bind, the next screen lets you select the scan scope and type and the LAN speed to be used in the scan.
The scope lets you control which servers are scanned. ExBPA shows your Exchange organization as a tree; by default, the entire organization will be checked, but you can select individual administrative groups or servers within the groups for scanning, as Web Figure 1 (http://www.windowsitpro.com/microsoftexchangeoutlook, InstantDoc ID 44793) shows. If you have two Exchange forests connected by a trust, you'll have to scan the two separately because ExBPA can't follow a trust from one forest to the other.
The scan type determines what's actually checked. There are three types, each of which is appropriate for a particular set of circumstances:
- The connectivity check tries to connect to the servers in the scope, testing that you can successfully establish a network connection and that you have appropriate permissions to connect. This test runs quickly; you use it to verify that you have connectivity to the servers before you run a full health check.
- The health check is a complete scan of the selected servers. This scan type requires the most time because it does the most work, but it gives you the most complete picture of how well your servers' configuration matches ExBPA's ideal state.
- The baseline check is for comparing one server with a set of others. This scan type is a fast way to compare the settings of a known-good server with those of other servers whose configuration you might not be sure of. For this scan type, you pick the server that you want to be the baseline, then select the servers you want to check against that baseline.
The network speed setting doesn't control how long the scan takes, as you might expect. Instead, Microsoft recommends that you specify the speed of the slowest link in your topology, and ExBPA uses that speed to calculate the amount of time it will take to complete a scan (a full scan returns approximately 1MB of data per server) and the timeout values it should use when connecting to servers. If you have slow links between routing groups or remote sites, you can always run ExBPA in those sites via the command line (or via Terminal Services).
As the scan proceeds, you'll see the ExBPA progress bar. Under most circumstances, it will be pretty accurate unless you picked a network speed that doesn't accurately reflect your connection speed or if network or server loads slow scanning down. When the scan is finished, you'll see a link that you can click to show a report of your scan results.
Examining the Scan Report
Before I talk about the scan report, I want to point out that you shouldn't rush into making changes according to ExBPA's results. You can easily break things by blindly tweaking settings, so resist the temptation to immediately fix everything that ExBPA reports. Wait until you've researched the suggested settings and understand the impact of any changes before you touch anything. In addition, remember that you might have made specific configuration changes that are right for your environment but that differ from ExBPA's recommendations—it's important to think before you accept all of ExBPA's suggestions.
That said, the first thing you'll see is a summary page that shows which servers ExBPA located, as Web Figure 2 shows. Each server will have an icon next to it that indicates whether the server was completely scanned, partially scanned, or not scanned at all. This summary view has a link that you can click to see a detailed report. When you click the link, the first report that you see by default is the Critical Issues list. As you'd expect, this list shows the items that you should probably focus on fixing first. If you don't find any items in this list, good for you! If you do see items in this list (e.g., SMTP relaying problems, page table entries, out-of-date Exchange binary files), you should plan to fix them after you understand what the fix entails.
A pulldown menu at the top of the page lets you select another report type: the Full Issues list, which is probably where most of us are likely to find at least one configuration problem flagged by ExBPA. For example, if you're not using WINS, you'll probably have a blank WINS primary address; if you have a blank address, ExBPA will flag it. If you haven't turned on crash upload logging, ExBPA will flag that too. Other issues that fall into this category include being behind on installing service packs, an SMTP queue configuration that differs from ExBPA's recommendation, and running Exchange on a DC. A detailed view of the Full Issues list shows you the same data with more information (such as the number of DCs and public folder stores found in the organization) and specifics. The detailed view lists all the settings that ExBPA checks, flagging those on your servers that don't match the recommendations in its rule set. Looking at this view is a good way to get a feel for exactly what ExBPA checks, although you can get the same information by inspecting the ExBPA.Config file.
Should you fix every item you see reported in the Full Issues category? Probably not. Many of the items ExBPA reports will actually be legitimate configuration choices you've made, such as running Exchange on a DC. In these cases, you can tell ExBPA to exclude the rules that you know your site doesn't meet, and it will ignore those rule violations in subsequent runs.
ExBPA archives its reports in the \Application Data\Microsoft\ExBPA subfolder of the logged-in user's Documents and Settings folder. The reports are XML, so you can parse them with your own XML tools or you can apply an Extensible Style Language Transformations (XSLT) style sheet to give them the appearance you want. If all you want is a simple HTML rendering, you can obtain that by selecting the Print to file option in the Print dialog box. You can also export and import ExBPA reports, as I mentioned earlier. Don't be surprised if you're asked to submit an ExBPA report the next time you call Microsoft PSS with an Exchange server problem.
If you've been around Exchange a while, you've probably used the Exchange 5.5 Performance Wizard, which asks questions about the role your server plays and performs some simple performance analysis to try to determine the best configuration. Subsequent versions of Exchange are more self-tuning but don't give you a way to influence the tuning process or to find out whether the settings that you configure are correct. ExBPA reintroduces the concept of an automated analyzer that applies Microsoft's knowledge of best configuration practices to your servers. I expect continued updates to the ExBPA rule set, perhaps with a future extension that lets you have the tool automatically fix problems it finds.
ExBPA has some bugs (see "Microsoft Exchange Server Best Practices Analyzer Tool Known Issues" at http://www.microsoft.com/technet/prodtechnol/exchange/2003/exbpakil.mspx), and no doubt Microsoft will consider fixing these in ExBPA 2.0. In the meantime, though, you should definitely download and run the tool to see what it finds in your environment. ExBPA is an excellent addition to your arsenal of diagnostic tools, and over time I expect its use to become a best practice all its own.