A Tricky Migration to Exchange Server

Late last year, my company decided to migrate a few hundred customers from a foreign mail system to Exchange Server. For the past 2 years, the company had supported the customers (i.e., users) through a foreign mail connector to Exchange. All the users had custom recipient objects. The custom recipient objects representing these users were members of a specific custom recipients container on a routing site server. The custom recipient objects were also members of various Exchange distribution lists (DLs) that included regular Exchange mailboxes.

The standard migration procedure involved creating a new Exchange mailbox and deleting the old custom mailbox for each user. The plan was to record all the details of each custom recipient mailbox object, including DL membership, before deleting it.

However, the migration team inadvertently deleted the custom recipient mailbox objects for one group of users before we had recorded the DLs the users were members of. We didn't discover the error for several days, so the regular directory replication cycle removed the objects and their membership from all the other sites.

We faced the problem of finding out which DLs the users were members of so that we could add the users' new Exchange mailboxes as members of those DLs. A procedure to manually capture this kind of information wasn't in place so we couldn't use a directory export, a Comma Separated Values (CSV) file, or any kind of list. Therefore, we decided to try to perform a full server restore of the Exchange Directory Store (dir.edb) to an offline machine.

This plan presented another host of problems because the Exchange dir.edb is tied explicitly to the Windows NT server on which you originally install the Directory. In our case, the server was a BDC to a production domain and was still in use. Exchange uses SID information contained in the SAM to validate access to directory objects, so we needed access to the original domain SAM. When you restore the Exchange dir.edb, the Directory looks at the security properties of all the objects (e.g., Site Container, Server) and compares the properties with the domain SAM. The Directory expects the security properties of all objects to match the SAM properties. If the restored server uses the local SAM created during the Exchange installation and not the original SAM from the domain, you can't access the Exchange data.

To restore the NT Registry from tape so that the new server maintains its unique SID and SAM, you must restore the Registry to the same physical machine. NT Registries are machine-specific; you can't restore a Registry to a different NT computer.

The Exchange Directory also maintains information about domain NT user IDs. Security for the service account, user mailboxes, and other accounts are all tied to the security contained in the domain SAM. To restore the Directory of the original Exchange server to an offline machine requires access to the SAM from the original domain that the Exchange service account belonged to. To restore the Directory to an offline server, we needed to name the offline server the same as the production server and have a copy of the production server's domain SAM. We had to restore the server to the offline machine while leaving the other machine in operation and still get a copy of the SAM.

Here are the steps we followed:

  1. We installed NT 4.0 on the offline server with the same NT service pack as the original Exchange server and enough disk space for the restore. We gave the server a new unique name (e.g., MAILRESTORE) and made it a BDC to the domain containing the production Exchange service account. By making the restore machine a BDC for this domain, we kept the domain SID and the domain SAM the same. We then unplugged the network cable to remove any chance of damaging the production network.
  2. We used Server Manager to promote the new BDC to become the PDC of the domain, and we rebooted the server. Promoting the BDC let us authenticate access to the original domain while remaining independent of the production network.
  3. When the server came up, we logged on to the domain, opened Server Manager, and removed the account for the production Exchange server. We then added the production server account name back into the domain. We removed and then re-added the production server account to remove its old domain SID from the domain SAM and generate a new, blank one.
  4. We renamed the offline server to the same name as the production Exchange server and rebooted. Be warned that if you rename the server and have already cataloged your restore tape, you need to recatalog the tape.

    To allocate a new domain SID and computer account, we used the name of the production server to reregister the new server with the SAM. With this approach, we're still able to use the production NT computer name.

  5. When the server booted with its new name and SID, we logged on to the server with the Exchange service account from the original domain and installed Exchange. We then used the same organization and site name, making sure to take the option to create a new site and not join the existing site. We also installed the same Exchange service packs and hotfixes we had on the original Exchange server.
  6. The installation created an empty dir.edb. To replace the data and let us locate the missing information, we ran the backup software and restored the dir.edb from the production system to the offline server.
  7. We started the server and retrieved all the membership information we were missing.

Lessons Learned
As a result of this experience, we now maintain an offline PDC of the email domain. We made a Compaq Deskpro class machine a BDC of the email domain, unplugged it from the live network, and promoted it to a PDC. Using a small 8-port hub, we connected the PDC with our MAILRESTORE machine. This action gave us an isolated copy of our email domain for authentication purposes that doesn't affect the production network.

As an additional safety measure, we regularly perform an automatic export of the DLs in the directory. Using a batch job and the Windows AT Scheduler service to run the job, we export all the DLs in the Global Address List (GAL) and save the output as a CSV file for emergencies. The batch file uses the Admin /e switch, then references a CSV file and an Option file to specify the information we want and its format.

You can learn from our experience by remembering these essential points:

  1. Keep track of user information during migrations. One easy way to track user information is to export the contents of each recipient container to a CSV file each week.
  2. Practice restores to an offline server. You never know when you'll need to restore information during a disaster-recovery exercise, and you might need to perform a restore to recover data for other reasons.
  3. Understand the relationship between domain SIDs, machine accounts, machine names, and the information the Exchange Directory Store holds.