Tweak MRS settings to avoid interruptions
The words "corrupted mailbox data" tend to make Microsoft Exchange administrators break out in a cold sweat. After all, the first thing that those words bring to mind is the possible necessity of a database restore. And we all know how much fun those are.
The TechNet Exchange 2010 Server forum contains multiple discussions about encountering bad or corrupted items during mailbox moves and the resulting fuss when administrators see messages such as Error: This mailbox exceeded the maximum number of corrupted items that were specified for this move request. This error appears when administrators run the Get-MoveRequestStatistics cmdlet to check the status of a mailbox move that's in progress. The error can also be seen logged as event 1100 from the source MSExchange Mailbox Replication in the Application event log.
Just what causes such problematic items? How worried should you be if you encounter this error? And what can you do about the issue? Read on to find out.
Mailbox Replication Service and Bad Items
Conceptually, the error is easy to understand. The Exchange 2010 Mailbox Replication Service (MRS) runs on Client Access servers and is responsible for moving mailbox contents asynchronously. In other words, MRS moves mailboxes behind the scenes so that users can continue to work until their new mailboxes are available and MRS switches the AD points for the user account to point to the new mailbox location. MRS can deal with Exchange Server 2003, Exchange Server 2007, Exchange 2010, and Exchange Online (part of Microsoft), so it's a pretty capable component. As we'll discuss later, MRS can also play a role in keeping your Exchange databases in good health.
MRS processes mailboxes according to move requests. A move request describes the target database and other parameters that influence how MRS should deal with the mailbox. For example, one of the parameters (SuspendWhenReadyToComplete) for a move request can be used to suspend the move after the mailbox contents are nearly fully copied. This action allows an administrator to review the move report, decide that everything is OK, and then resume the move request to allow MRS to flip the Active Directory (AD) settings, and redirect the user to his or her new mailbox before resuming the move. Another parameter indicates the permitted bad item limit -- in other words, the number of corrupted items that MRS can tolerate before completing a mailbox move request. In this context, "corrupted" means that MRS determines a property or properties of the item to be incorrect, malformed, or in some other form that is unacceptable to Exchange.
The default value for the bad item limit is 0, meaning that MRS will tolerate no data loss in a mailbox move. This is sometimes appropriate when data absolutely must be retained. However, you might find that you can't move mailboxes at all because MRS continues to encounter corrupted data and fail with the error message mentioned earlier. For this reason, best practice is to set a reasonable limit for bad items. You should also check mailbox-move reports after requests complete, to determine whether the limit is in fact reasonable or if valuable data is being lost. I think anything in the 10-to-20 range is reasonable, but that depends on your circumstances and organizational requirements.
If you decide that a mailbox really must be moved no matter how much data is lost en route, you can set a very high bad item limit, which forces MRS to move the mailbox at all costs. The maximum setting is 2147483647, but 100 is probably a better starting point. Note that after you specify a bad item limit equal to or higher than 51, you also need to pass the AcceptLargeDataLoss switch parameter to tell MRS that it's acceptable to drop all corrupted items.
Positive Intervention Is Required
If a move request is halted because MRS encounters too many bad items, an administrator can intervene to amend the move-request parameters and get things moving again. Use the Set-MoveRequest cmdlet to amend the BadItemLimit parameter and increase the tolerance for bad items until MRS can move the mailbox, and then instruct MRS to resume the move.
For example, suppose that the move for TRedmond's mailbox has stalled because MRS has encountered too many bad items. The first order of business is to view the move report. Use the Exchange Management Console (EMC) to view the properties of the move request, find the items that are causing trouble, and perhaps figure out whether you need to intervene in the mailbox before continuing. See the Microsoft article "View Move Request Properties" for details about how to access a mailbox-move report.
Suppose that the original bad item limit for the move request was 10, and MRS encountered more than 10 corruptions as it copied data from the source to the new target mailbox. We can scan the mailbox move report to discover how far MRS managed to get before reaching this limit. MRS creates checkpoints on a frequent basis so that it can resume a move request without redoing work; MRS reports this checkpoint status in terms of percentage complete. This percentage reports the overall progress of the move and takes into account other processing, such as connecting to source and target servers, setting up the target mailbox, and preparing to copy. The phase between 25 and 90 percent of the overall move is when copying of mailbox data occurs. If MRS encounters a bad item limit, it will happen at some point in this range.
In our example, suppose that the mailbox-move report indicates that MRS reached 80 percent before stopping the move. The last 10 percent or so of a mailbox move is used to perform a final incremental synchronization of mailbox contents and to switch the mailbox pointers in AD. At 80 percent, there isn't much more mailbox data to copy, so there's a fair chance that MRS can finish copying data from the source mailbox if you increase the bad item count to 20. To do so, you can run the following commands in the Exchange Management Shell (EMS):
Get-MoveRequest -Identity "TRedmond" | Set-MoveRequest -BadItemLimit 20 | Resume-MoveRequest
The first command retrieves information about the move request. This information is piped to the second command, which increases the bad item limit to 20 and pipes the data to the third command. That final command tells MRS to resume working on the move request. If things happen as we expect, MRS will complete the move request and all will be well in the world.
Fools Rush In Where MRS Discards Data
Before you increase bad item limits, you must understand that MRS discards every bad item that it meets; it doesn't copy this data from the source to the target mailbox. Thus, a limit of 20 bad items means that MRS can discard as many as 20 discrete pieces of data from a user's mailbox while it is en route to a new home. This data might be terribly unimportant, as in the case of an old calendar meeting request. Then again, it might be a message that contains an Excel attachment describing the current year's budget. Fortunately, in 99.9999 percent of cases, bad items are extremely corrupted and highly unlikely to be accessible by a client.
Letting MRS dump a lot of bad data as it moves mailboxes might be deemed a very bad thing. However, bad items are corrupted, and it's good to give the mailbox the equivalent of a colonic irrigation from time to time, to remove accumulated debris. Indeed, Microsoft moves mailboxes frequently between databases in its Exchange Online cloud service, both to balance workload across available servers and to ensure that mailboxes stay squeaky clean and don't introduce corruption that could compromise the databases' service.
Corrupted items arise through many sources. Bugs are the most obvious and can occur on both the server and its clients. Exchange is now a very mature product, and its developers have had years to eliminate potential problems that might introduce corruption at the database level. You're likely to see more corrupted items when you move mailboxes from an Exchange 2003 server than when moving mailboxes from Exchange 2007 or Exchange 2010.
The root cause of item corruption is more likely to be a client problem than a server issue. A profusion of clients can connect to Exchange, using a multitude of connection methods, including add-on software products that integrate with clients such as Microsoft Outlook. Clashes and corruption can occur when multiple clients attempt to access the same item concurrently. Given the number of mobile devices that connect to Exchange, it should be no surprise that corruptions exist, especially in calendar items.
Based on purely anecdotal evidence in online forums, users of Research In Motion (RIM) BlackBerry devices seem to experience more corrupted calendar items than do users who access Exchange only through Outlook or Microsoft ActiveSync devices. A look through calendar items in a mailbox (use the Outlook list view) often identifies problems such as apparently blank items or items with multiple versions (i.e., conflicts). Items such as these can cause MRS problems and are best removed before mailbox moves are attempted.
The general rule is that the older the item, the more likely it is to cause a problem. If you, like me, have items in your mailbox that go back to 1996, it should be no surprise that some item properties that Exchange expects a well-behaved MAPI client to populate might not have been written correctly into the database.
Other Bad Item Limits
Mailbox moves aren't the only work that MRS does. Any request that MRS processes can have a defined bad item limit. Think of MRS as a component that shuttles data around for Exchange, and you can understand why it implements bad item limits in everything it does. Clearly, there's no point in moving corrupted data from one place to another. It's best to detect and drop the bad stuff.
For example, if you use the New-MailboxImportRequest cmdlet to import data from a PST into an Exchange 2010 or archive mailbox, or you use the New-MailboxExportRequest cmdlet to export data from a mailbox to a PST, you'll find that you can specify bad item limits. The same zero default value is used and you'll need to specify the AcceptLargeDataLoss parameter if you decide to use a bad item limit higher than 51. The New-MailboxRestoreRequest cmdlet, which Microsoft introduced in Exchange 2010 SP1 as the preferred method to restore a soft-deleted or disconnected mailbox, also supports bad item limits. You might need to use this cmdlet to restore content to a mailbox from a copy in a backup-recovery database.
Importing data from a PST can be tricky. The PST file format was never designed to cater to the many ways that it's used in practice. Older ANSI-format PSTs are particularly problematic when it comes to item corruptions, and I often need to set high bad item limits (i.e., greater than 50) to successfully import data. The problem appears to be less severe with the newer UNICODE-format PSTs created since Outlook 2003 was introduced. Still, I rarely see imports of large PSTs (i.e., greater than 1GB) without encountering some corruptions.
Rules Can Hiccup
Many users like to create mailbox rules to help automatically process incoming messages. Users like rule processing so much that they might create more than 32KB of rules in a mailbox. MRS won't process a mailbox that has more than 32KB of rules. The only solution is to specify the IgnoreRuleLimitErrors parameter when creating the move request or amending an existing move request with the Set-MoveRequest cmdlet. Unfortunately, this parameter allows MRS to move the mailbox without moving any of the user's rules. The user must then recreate rules when he or she starts to use the new mailbox. Hopefully, this issue will be addressed in a future version of Exchange.
The Good News
The good news is that a mailbox that has been successfully moved to Exchange 2010 is clean and free of corrupted items -- at least, according to MRS. How long the mailbox remains clean is entirely dependent on the software that accesses and updates the mailbox. Software is getting better too, so fewer corrupted items are being created. The combination of better software and MRS sanitization should minimize bad mailbox items in the future. At least, that's the plan!