Message tracking logs and the Message Tracking Center help you hunt down missing messages
Microsoft Exchange Server has always included the Message Tracking Center facility, which tracks the progress of messages as they make their way between servers in an organization. The Message Tracking Center is part of the Exchange Server 5.5 administration program and resides under Tools in Exchange 2000 Server's Exchange System Manager (ESM) console. In both versions of Exchange, the feature depends on the availability of message tracking logs for interrogation. Let's look at how Exchange 2000 logs a message's progress and how you can use the Message Tracking Center to track a message.
Message Tracking Logs
By default, the properties of an Exchange 2000 server enable message tracking. The server creates a new message tracking log each day, which it names according to the date (e.g., 20030725.txt). Exchange 2000 stores the logs in the server_name.log network share, which usually resides on the same drive as the Exchange 2000 binaries. Figure 1, page 36, shows a typical set of message tracking logs.
When a client submits a message to the server, Exchange 2000 generates a unique identifier for the message and records the identifier in the message tracking logs, thus making it possible to trace a message as it makes its way to its final destination. A sample message identifier that a Messaging API (MAPI) client generated is BE8B1DCC92D77E4C9CC70E141E3B583B02226F@EXCSERVER.acme.org. (Note that this identifier isn't the same as the MAPI message identifier that you see when you view the message properties.) The only part of this identifier that makes any sense to humans is the server name (EXCSERVER.acme.org) at the end of the string. The rest of the identifier gives the Message Tracking Center information to help it track messages.
POP3 and IMAP4 clients generate their own message identifiers in the form 000001c19dbf$78c6bf90$7705d110@acme.org. Aside from formatting, the major difference between MAPI identifiers and the identifiers that other clients (such as Microsoft Outlook Express) generate is that the other clients omit the originating server name. You find client-specific message identifiers in the message tracking log's Linked-MSGID field. You also see the client-specific message identifier when you track a message and view its history.
You use three server properties to enable message tracking. To set or check these properties, open ESM, right-click the server on which you want to start message tracking, select Properties, and click the General tab. Figure 2, page 36, shows the properties, which aren't enabled by default. Alternatively, you can use a system policy, as Figure 3, page 36, shows, to set the properties across a set of servers that might span multiple administrative groups. If the check boxes are shaded, the server is under the control of a system policy and you might not have the permissions to change the property values. The properties are as follows:
Obviously, you can track messages only if all the servers that process those messages record details in the tracking logs. Thus, you should make an enterprisewide decision about whether to log messages. You should either enable logging on all servers or not at all. I recommend enabling logging because it provides a useful facility at very little cost, and the easiest way to apply settings organizationwide is with a suitable system policy.
Tracking Messages
You use the Message Tracking Center to follow a message's path in the logs. The Message Tracking Center is available as an Exchange 2000 snap-in that you can load into a customized console, but you typically access it from the ESM Tools node, as Figure 4 shows. When you open the Message Tracking Center, you can enter the criteria necessary to begin a search.
Obviously, the more information (e.g., sender, time sent, subject) you have about a message that you want to track, the better. Starting a search based on a fuzzy description of a message that a user thinks he or she sent 4 days ago to a group of people is much harder than looking for a message sent yesterday afternoon. Figure 4 shows the result of a search that has located one message that I sent to the specified user. Note that on this server, the Enable subject logging and display property isn't set, so message subjects aren't included in the message tracking logs and the Message Tracking Center can't display this information.
The Sender and Recipients buttons let you search Active Directory (AD) for users, contacts, or groups that sent or received the message. The Message Tracking Center won't begin a search until you provide valid sender and recipient data. By default, the Message Tracking Center assumes that you want to begin your search on the server that you select by clicking the Server button, but you can connect to another server to start searching if you need to (you'll need to if the sender's mailbox is on another server).
Even on large servers that handle heavy volumes of email, searching typically proceeds rapidly, yielding a result within a couple of minutes. When the Message Tracking Center must contact other servers to retrieve data (and especially when a message is addressed to a large distribution list—DL), it can appear unresponsive, but it will respond if you let it complete its processing.
In addition to introducing a better Message Tracking Center UI (which resembles Outlook's Find Message functionality), Exchange 2000 Service Pack 2 (SP2) makes an important architectural change to search processing. In versions earlier than SP2, the server performing the search contacts each server in a message's path, retrieves all the log data from the remote servers, then processes the search of the logs. In SP2 and later, the local server sends a search request to the Exchange Management Service running on each server involved in a message's progress. The service processes the search of the logs on that server (upgraded to SP2 or later) and returns only the relevant data to the requesting server.
The better the search criteria, the fewer the number of messages retrieved. The search in Figure 4 found only one message, so it's likely the one that I'm looking for. However, if the criteria are inexact, the Message Tracking Center might find 10 or more messages. In that case, you must decide which message is the one you want. To view comprehensive details about the progress of a message through the Routing Engine, select a message and double-click it.
Figure 5 shows a sample message history. In this instance, the history is reasonably simple because the message was sent to two SMTP recipients. The message subject isn't displayed, again because of server property settings. The history shows that after the message was sent (i.e., submitted from the Store), Advanced Queueing and the Categorizer (both components of the Routing Engine) processed it and determined that it should be delivered to a remote SMTP recipient. The Routing Engine then transferred the message to a server that hosts an SMTP connector so that the message could be sent to its final destination.
If you look at a message sent to users in other routing groups, you might see that the Routing Engine transfers the message to the Message Transfer Agent (MTA) for delivery to an X.400 recipient or to an Exchange 5.5 server. Or a message might use SMTP to go to a bridgehead server in the same routing group for onward transfer to other routing groups or, if your server hosts the routing group connector, directly to servers in other routing groups.
In situations in which messages are transferred to other routing groups, the Message Tracking Center lists destination bridgehead servers in a tree in the Location (left-hand) pane. As tracking proceeds, the bridgehead server tree expands to show all the servers within the routing group that receive a copy of the message.
Among deliveries to local mailboxes, the Message Tracking Center might also list deliveries to mail-enabled Public Folders. Organizations often use Public Folders as archives for messages sent to distribution groups and, in terms of delivery, Exchange 2000 processes these messages the same way it does those sent to a local mailbox. The only way you can tell that a recipient is a Public Folder is if you recognize the recipient name as that of a Public Folder rather than a person.
The Message Tracking Center follows the progress of a message until the message reaches its destination or meets a server that doesn't have tracking enabled. As I mentioned earlier, the Message Tracking Center depends on message tracking being enabled on each server, preferably in a consistent manner across all servers in the organization. Obviously, if you don't enable message tracking on some Exchange servers in your organization, those servers won't create tracking logs and the Message Tracking Center might not give you a complete message history.
Tracking Log Format
The message tracking log format uses tab-delimited fields, including the ones listed here. You can see some of these fields in the Microsoft Excel spreadsheet that Figure 6 shows, which is an extract of a log edited for clarity.
Understanding Message Tracking Log Data
The easiest way to begin to understand the data held in the tracking log is to take a copy of a log from a test server, load it into Excel, then examine the different events that occur as the Routing Engine processes a message from submission to final dispatch.
Of course, you can review log data from a production system, but typically these logs are large (more than 50MB), so picking a particular message and then tracking its progress is more difficult. Large logs slow tracking down, especially if the message was sent to a long DL or if you must track a message across a number of different servers within an organization.
Servers that host active Public Folders can generate heavy replication traffic, which is noted in the message tracking logs. You can identify replication messages by sender name, which is either the Exchange 2000 internal X.500 address for the Public Folder Replication Agent (e.g., EX:/O=Acme/OU=Central/CN=Configuration/CN=Servers/CN=Server1/CN=Microsoft Public MDB) or the SMTP address for the Public Folder Replication Agent (e.g., Server1-IS@acme.org). The message tracking logs store the name of the destination server (the replication partner) in the Recipient-Address field. The message tracking logs also store data for incoming messages, including nondelivery notifications and delivery and read receipts.
The Excel spreadsheet that Figure 6 shows contains the raw message tracking log data that the Message History dialog box in Figure 5 shows. The message originates from a MAPI client and goes to two external SMTP recipients. Therefore, the logical processing that the Routing Engine performs is straightforward. The engine accepts the message, checks the SMTP addresses, and routes the message onward to either an SMTP connector on the same server (if present) or an SMTP connector on the server that has the lowest possible routing cost. You can track the events in the figures as follows, noting that entries exist for each recipient address on the message:
In comparison, the sequence of events for a message delivered to a mailbox on the same server is as follows. The same events occur for deliveries to a mailbox in another storage group (SG).
Messages that Outlook Web Access (OWA) clients submit log the same sequence of entries. POP3 and IMAP4 clients use SMTP to submit messages directly to the Routing Engine. The messages don't go through the Store Driver, so the sequence begins at event ID 1019.
Other common entries include event ID 1023 (successful delivery to a local mailbox) and event ID 1029 (transfer to the MTA for onward delivery). Unlike Exchange 5.5 logs, Exchange 2000 logs have no special entry to mark the expansion of a distribution group. Instead, you'll see an event ID 1020 entry written into the log for every member of the group after expansion.
Complex messages addressed to a mixture of single recipients and distribution groups that resolve to local and remote addresses pose a particular challenge for email servers, especially if some of the recipients are members of multiple distribution groups. The fact that the number of messages sent to a distribution group is determined by the properties of the group further complicates processing. For example, the group properties that Figure 7 shows reveal that any delivery reports for messages sent to this group go to the sender. In other words, if the group contains an invalid address (e.g., a contact whose email address is no longer valid, a mailbox with an exceeded quota), the sender will receive a nondelivery notification. You can set the group properties to suppress delivery reports to decrease email traffic, but then users can never be quite sure that their message arrived. The Routing Engine performs the same per-message processing for out-of-office notifications, which you can also suppress or enable through distribution group properties. Obviously, the different values you can select for these properties result in message-generation permutations.
Because Exchange 2000 processes delivery reports and out-of-office notifications on a per-message basis, if you send a message addressed to two groups— one that permits delivery reports and one that suppresses reports—the Routing Engine must generate two sets of messages. Thus, the recipients in the first group receive a copy of the message with its properties set to allow delivery reports and the recipients in the other group receive a copy with its properties set to suppress delivery reports.
You don't want users to receive multiple copies of a message, and this situation can easily occur if the Routing Engine code doesn't handle all possible circumstances when the Routing Engine builds the recipient list for a message. To optimize the recipient list, the Routing Engine must expand all distribution groups, check the delivery reports and out-of-office-notification properties of the distribution groups, then generate the appropriate number of messages to meet the requirements as determined by each group's properties. The result of the complex nature of recipient lists is that you might see multiple event ID 1020 and event ID 1028 entries for each address on a message when you look through tracking log data.
By comparison, Exchange 5.5 MTA uses a process called fan-out to expand DLs (or groups). MTA builds the address list, then determines the best possible route for sending a single copy of the message to every addressee. Exchange 5.5 doesn't support the same set of properties to control out-of-office messages and delivery reports, so building a recipient list is easier.
In Exchange 2000 or Exchange 5.5, if duplicate messages get through, the Store on the receiving server can use the message identifier to detect the duplicates and suppress them. This backstop mechanism ensures that Exchange doesn't deliver duplicate messages.
Further Analysis
Loading a message tracking log into a program such as Microsoft Access, in which you can analyze the data to your heart's content, is easy. However, if you want to use message tracking data as the basis for analyzing traffic volume, identifying the generators of the most messages, or analyzing traffic on individual servers, you'll probably want to write some code to reduce the data to its essential numbers (such as the number of messages sent during a period, the number of addressees, and the size of the messages). Expect to process a large amount of data because one message sent to a large distribution group generates hundreds of entries in the tracking log. For example, a message sent to two common distribution groups that total 1350 users (with some duplicates) at my company generates more than 4000 entries. Judging by comments made at conferences and in Internet mailing lists, I'd say that Perl is a particular favorite of those who write homegrown code to analyze message tracking log data.
Alternatively, you can buy a commercial product such as PROMODAG Reports for Microsoft Exchange Server to do the job. Quest Software's MessageStats provides more sophisticated analyses of message log contents to report on many different statistics, such as the most popular DL and who sends the most messages. Utilities don't handle some situations well—for example, some utilities can't process messages sent to nested DLs (lists composed of lists rather than individual recipients)—but in most cases, an off-the-shelf product is an easier and more functional choice than developng your own message tracking tool.
You might have overlooked the Message Tracking Center if you aren't often asked to track a message. However, if you are asked to track a message, you can be sure that the message is important—and you need to know how to go about the tracking, including analyzing the raw data if necessary.
