Get up and running whether Exchange is on a physical or virtual platform
The worst fear of any Microsoft Exchange Server administrator is a complete crash of the Exchange server. What every administrator needs are step-by-step instructions for recovering that Exchange system—from beginning to end. So, let’s apply that how-to treatment to the most difficult scenario of an Exchange recovery, which is the inability to replace the crashed Exchange system on the same hardware. You have to rebuild your server on new hardware!
The rebuild scenario significantly complicates the restoration process: You can’t restore the OS, system state, and Exchange program files because of hardware differences on the new server. Fortunately, most of the configuration details about your dead Exchange server are stored in Active Directory (AD), and for the purposes of this article, I'll assume that your AD infrastructure is still intact. (If that's not the case, you must recover your AD infrastructure before rebuilding your Exchange server.)
If your Exchange server is running as a virtual server guest and you have a backup of the virtual server guest disk images, the recovery process will be greatly simplified, as you'll see later in the article. But even if you’re fortunate enough to have Exchange running as a virtual server guest, it’s still a good idea to know the recovery procedures for a failed Exchange server—just in case. This article reviews the recovery steps for an Exchange 2007 Server with the following roles: Mailbox, Client Access, Hub Transport, and Management Tools.
A Basic Exchange-Recovery Scenario
Writing one foolproof Exchange-recovery scenario is impossible, because each Exchange environment is unique. However, you can use the following steps as a guideline for your recovery procedures. Understand, though, that you’ll probably need to modify them to fit the needs of your Exchange environment.
Order the new server replacement—When you’re ordering your new server, make sure to order an adequate number of drives to duplicate the exact same drive letters, with storage space on each drive letter equal to or greater than that of the original server. If you don’t know where Exchange was installed on the old server, you can use ADSIEdit.msc (installed with the Windows Support Tools or here) on any server in AD. Expand Configuration, CN=Configuration, DC=<domain>, DC=<domain_ext>, CN=Services, CN=Microsoft Exchange, CN=<company_name>, CN=Administrative Groups, CN=Exchange Administrative Group (admin_group_id), CN=Servers,CN=<Exchange_Server>. Right-click the server name, and select Properties. Select the Show only attributes that have values check box, as Figure 1 shows, and click on the Value column to sort by value. Scroll down until you see the full path of the Exchange server attributes, and make a note of each drive letter and path. You can also view the path of each storage group and database on the server by expanding CN=<server_name>, CN=Information Store, CN=<storage_group> and examining the properties of each storage group and database stored on the server. As you know, Exchange 2007 needs a lot of memory. For a relatively small Exchange installation, I suggest ordering a server that has at least 8GB of memory. With the x64 platform and Exchange 2007, more memory equals better performance.
Install the same OS as the failed server on the new hardware—Exchange 2007 will run on Windows Server 2008 x64 or Windows Server 2003 x64. Install the latest service pack with all the patches. Avoid the temptation to upgrade the OS during the disaster-recovery process. The last thing you want to troubleshoot are new OS problems that might arise because you decided to upgrade the OS during the recovery process. Installing the same OS reduces the chance of having problems with backup, antivirus, or other related programs during the recovery process.
Name the new server the same as the failed server with the same IP address configuration—This step is important because when you perform an Exchange reinstallation, the system will use the server name to gather the failed server’s configuration from AD. If you can’t remember the failed server’s name or IP address, you can look that up on any DC that’s running AD-Integrated DNS.
Reset the computer account in AD—Before you join the domain, start the Microsoft Management Console (MMC) AD Users and Computers snap-in, locate the Exchange system, right-click it, and select Reset. Doing so will let you use the old server’s name to join the new server to the domain.
Join the domain—Join the domain, and restart the server.
Install server antivirus—If you had antivirus protection on the server, install the same version on the new server.
Stop inbound mail flow into the server—Before you start the reinstallation of Exchange, I suggest stopping inbound mail flow to the server. You can typically accomplish this at the firewall or spam-filtering appliance. Stopping mail flow will prevent you from losing any Internet inbound messages that could be overwritten during the recovery process.
Install the necessary prerequisites and roles on the server for Exchange—If you’re running Exchange 2007 on Windows 2003, these include MMC 3.0; .NET 2.0 with SP1; Windows PowerShell; and IIS, World Wide Web, and Common Files. If you're running the Unified Messaging Role, you'll also need:
If you’re running Exchange 2007 on Windows 2008 with the Mailbox, Client Access, Hub Transport, and Management Tools, you can script these requirements through the following commands:
Install Windows PowerShell:
ServerManagerCmd -i PowerShell
Run the following commands, in this order:
ServerManagerCmd -i Web-Server
ServerManagerCmd -i Web-ISAPI-Ext
ServerManagerCmd -i Web-Metabase
ServerManagerCmd -i Web-Lgcy-Mgmt-Console
ServerManagerCmd -i Web-Basic-Auth
ServerManagerCmd -i Web-Digest-Auth
ServerManagerCmd -i Web-Windows-Auth
ServerManagerCmd -i Web-Dyn-Compression
If the server will support Outlook Anywhere clients, install the RPC over HTTP proxy by running the following command:
ServerManagerCmd -i RPC-over-HTTP-proxy
Run Setup /m:RecoverServer—Install Exchange 2007 with this switch is similar to performing a disaster-recovery installation on Exchange 2003. Most of the Exchange settings are stored in AD. The /m:RecoverServer switch tells the Exchange Setup program to query AD for the Exchange settings based on server name. For more information about running Setup /m:RecoverServer, refer to the Microsoft article "How to Recover a Lost Exchange Server." Setup will run through the prerequisite check, and if everything is in place, Exchange 2007 should be installed on your recovered server. After a successful installation, reboot the server. At this point, you should have your Exchange implementation restored without any current data. Open the Exchange Management Console (EMC), review everything, and verify that the server was properly installed. By default, the empty Exchange databases dismounted after the Exchange installation. For any databases that you plan to restore, select the properties of the database and ensure that the This database can be overwritten by a restore check box is selected.
Install Exchange antivirus—If the failed server was running an antivirus package for Exchange, reinstall the package with the latest patches and virus patterns. Obtain the latest virus patterns before restoring the Exchange databases.
Install the backup agent—If the failed server had a backup agent installed, reinstall it with the same configuration you had on the original server.
Reinstall the SSL certificate—If you had a commercial SSL certificate on the failed server, you must reinstall it. If you didn't export the certificate, have your SSL vendor reissue the certificate or obtain a new SSL certificate. If you have to order a new certificate, make sure that you obtain an SSL certificate that's capable of multiple identities. Typically, you'll need the following identities for the SSL certificate: internal Fully Qualified Domain Name (FQDN) of the Exchange server, external FQDN of the Exchange server, and Autodiscover..com.
Install VSS Snapshot Patch KB940349—If you don't install this patch, the Exchange database-restore job might fail with the error Final error: 0xe00fed5—A Failure occurred initializing for restore.
Restore the databases to the Exchange server—Using your backup software, restore the Exchange databases to your server. Ensure that the existing databases are dismounted; otherwise, the restore will fail. Figure 2 shows a restoration that uses Backup Exec 12.5. After the restore is complete, verify that the database is mounted and that the mailboxes and public folders were properly restored. (Using the EMC, verify that the Do not mount this database at startup check box is cleared.
Reestablish mail flow from the Internet.
Test the restored mail server to verify that it's properly functioning—This process might include (but is not limited to) verifying Autodiscover functionality, ensuring that all mailbox and public folder information was properly restored, checking mail flow to and from internal Exchange users (as well as to and from the Internet), verifying ActiveSync functionality, checking Outlook Web Access (OWA) functionality, ensuring Unified Messaging functionality (if applicable), and verifying relay functionality for servers that use the Exchange server like SharePoint servers and other servers that use this server to send mail.
Exchange Recovery Through Virtualization
Virtualization can greatly simplify the recovery process and reduce the recovery time for Exchange. With virtualization, you don’t have to worry about new hardware, because the virtual server guest will always see the same hardware no matter what physical hardware is running on the virtual server host.
If your virtualization platform (e.g., VMware ESX Server, Microsoft Hyper-V Server) can perform an image backup while the virtual server guest is still running, you can back up the virtual server guest files on the virtual server host without taking down the virtual server. If your Exchange server is virtualized, I suggest obtaining an image backup at least once per week, while performing “traditional” backups of the Exchange server during the week. Assuming that your Exchange Server is virtualized on a VMware ESX host, the recovery steps are as follows:
Order the new server replacement—When you're ordering the new server, make sure that the replacement server is on the VMware Hardware Compatibility Guide. Ensure that the replacement hardware has the same (or greater) capabilities as the failed server in terms of CPU, memory, disk, and network cards.
Install ESX Server on the new server—Ideally, give the server the same name and IP address. Install the latest patches on the server.
Install the ESX host backup agent—If necessary, install the ESX backup agent software.
Recreate the virtual server guests—Recreate all virtual server guests that were on this host with the same configuration, but don't create any hard drives for the virtual server guests.
Restore the *.vmdk files to the host—Because the storage group will be different on the newly built ESX host, you must redirect each restored *.vmdk file to the proper folder. When you restore the *.vmdk files, don't restore the directory tree; otherwise, the *.vmdk files won't be restored to the proper location. You'll probably have to create a separate restore job for each virtual server that you restore.
Associate *.vmdk files with virtual server guests—Add the *.vmdk files to the virtual server guest. Because the virtualization platform creates a hardware-agnostic platform for the virtual server guests, the virtual server guests will always “see” the same hardware platform regardless of the physical hardware running on the host.
Start up the DC guests first—If your DCs were virtualized on the failed host, be sure to start them first and verify that they're properly functioning before you start any Exchange servers.
Start up the virtual server Exchange guest—Start the virtual server guest and verify that it's properly running.
Restore the latest Exchange database backup by following steps 13 through 15 in the previous section.
If you have a spare virtual server host available you can reduce the recovery time even more by pre-staging the virtual server guest files on the host. If the server crashes, you can start the virtual servers on the backup server, restore the latest Exchange database backup, and you’re up and running.
Plan for the Recovery Now!
Now is the time to test your recovery procedures and document your Exchange environment. When all your Exchange users are down, you don’t want the added stress of figuring out the recovery process. Virtualization is a good way to test your recovery procedures with a minimum investment in hardware and minimal risk to your production environment. You should at least document the following:
Server hardware—Make, model, CPU, disk configuration, network configuration, and OS and service-pack level of your Exchange server.
Exchange server name—Server name, IP address, and version of Exchange installed on each Exchange server, as well as the installed roles of each server.
DCs—DC names and locations.
Exchange installation location—Drive and folder where Exchange is installed.
SSL certificate—Export the SSL certificate, and store it in a safe location.
Storage groups and databases, and database locations for each Exchange server.
Backup software burned to CD with installation keys.
Current OS and Exchange software burned to CD/DVD.
Recovering a crashed Exchange server is something you hope you never have to do, but it’s a skill that every Exchange administrator should have. I hope I've provided some insight into the necessary steps to gracefully recover your Exchange environment.
