In Windows NT 4.0, the domain controller (DC) promotion and demotion process requires a complete rebuild of the server. You can turn a server into a DC only during Windows setup. This situation presents a problem to administrators who want to change the role of a server while retaining other shared services, and rebuilding a system from scratch isn't exactly an attractive prospect. Windows 2000 Server improves on the NT 4.0 DC promotion and demotion process by permitting an administrator to use the Dcpromo utility to promote a standalone or member server to a DC. A demotion is also a simple Dcpromo task, assuming you have connectivity to a healthy, functional DC within the domain.
However, even in the improved Win2K Dcpromo process (pre—Service Pack 4—SP4), a problem lingers: To perform clean demotions, Win2K Dcpromo requires the DC to have physical network connectivity to another functioning DC. Other stumbling blocks, such as corrupt objects in the directory, can also lead to demotion problems. If a clean demotion is impossible, you're looking at a complete rebuild of the server, as well as a metadata cleanup, to remove the DC from the directory. Another promotion limitation that's been around since the early days of NT is the requirement of replicating all data over the network—you've never been able to perform a promotion from a compressed file or storage medium shipped to a remote location. With pre—Active Directory (AD) domains, such replication wasn't a huge problem, given the size limitations of the SAM databases in those versions. With AD, one Directory Information Tree (DIT) file can be 10GB or larger. In large environments that use Win2K AD, promoting a Global Catalog (GC) server in the far reaches of the network can take as long as 4 to 5 days.
Vastly improving on Win2K Server's promotion and demotion process, Windows Server 2003's Dcpromo lets you force a demotion regardless of connectivity to the domain and regardless of the status of objects relating to the DC. (The Dcpromo version in Win2K SP4 also permits forced demotions.) But arguably the most popular feature of Windows 2003 Dcpromo is its ability to promote a DC from media. The medium necessary to promote a DC is essentially a directed restore of a system state backup from another Windows 2003 DC. This backup medium lets you promote DCs in remote locations in far less time than Win2K requires. By using Windows 2003 Dcpromo to promote a DC from media, you can promote the same GC that took 4 to 5 days with Win2K in a matter of minutes, depending on the age of the system state backup you use. These added features will not only save you hours of wasted time rebuilding servers from scratch but also open up the door to new methods of disaster recovery and deployment.
Dcpromo from Media
Windows 2003 Dcpromo's new ability to promote from media can greatly improve your ability to recover from DC failures, particularly in remote sites where repromoting a DC over the wire can take hours or even days, depending on WAN bandwidth and the size of your directory. The feature can also dramatically speed your ability to roll out AD services to remote sites as well as decrease the amount of WAN traffic necessary to promote DCs. You can use this process to create only replica DC or GC servers, and both systems should have the same level of OS patches. Promoting a DC from media requires that you first ensure the following:
- The system state backup must be no more than 60 days old, which is the default tombstone lifetime. (Tombstone lifetime is the length of time that deleted objects remain before the DC permanently removes them from the directory. Promoting from a backup that's older than the tombstone lifetime results in the reintroduction of tombstoned objects.)
- The system state backup must be taken from a Windows 2003 DC in the same domain. If you want the DC to also be a GC server, you need to create the system state backup from an existing DC that's also a GC in the same domain.
- The source DC must be completely healthy at the time of the backup. Use netdiag.exe and dcdiag.exe to ensure that the DC has clean event logs and correct DNS registrations. Use the repadmin.exe /showrepl /all command to ensure that clean replication is occurring. Test with dummy files to make sure that you have clean File Replication Service (FRS) replication. Ensure that the DC is responsive to Lightweight Directory Access Protocol (LDAP) queries and remote procedure calls (RPCs). Make sure the DC has Netlogon and Sysvol shares. Finally, ensure that the DIT contains little fragmentation.
- The server you want to promote from media must be on the network and must be able to communicate with other healthy DCs.
The Basic Steps
You first need to use NTBackup to perform a system state backup of a healthy DC (and optionally GC), as Figure 1 shows. To do so, start NTBackup and click Advanced Mode on the first screen. On the next screen, go to the Backup tab and select the System State check box. As a best practice, consider a naming convention for the backup file that conveys the name of the source server, the date of the backup, whether the server is a GC, and the OS build number. After you type the pathname, click Start Backup. On the next screen, you can select advanced options. Clear the Automatically back up system protected files with the system state check box because those files aren't necessary for using Dcpromo from media. When the backup has completed, be sure to view the report to ensure that the backup didn't fail on any crucial files. If a failure occurred, remove any file locks and rerun the backup. (If the NT File Replication System—NTFRS—service is running, you can expect a DO_NOT_REMOVE_NtFrs_PreInstall_Directory error. You can safely ignore this error. NTFRS uses this directory, which isn't intended to be backed up.)
After you obtain the backup file, you might want to use your favorite file-compression program to compress it. By doing so, you minimize the amount of data you need to copy to the destination server you want to promote. However, some compression utilities won't compress files larger than 4GB. If you plan to use compression frequently, investigate whether your compression utility meets your needs. Copy the compressed file to the new server and expand the backup file.
You now need to use NTBackup to perform a directed restore of the files within the backup file's backup set. Open NTBackup and click Advanced Mode on the first screen. Go to the Restore and Manage Media tab. Select Catalog a backup file from the Tools menu and browse for the backup file. Expand the catalogued file and select the system state check box. Also, as Figure 2 shows, configure NTBackup to restore files to an alternative location. I chose a location on the same logical drive on which the DIT file resides. Figure 3 shows the items that will be restored in that folder.
To greatly speed the promotion process, I recommend that you restore to the same drive that you plan to use for the directory database. The benefit of this strategy is that Dcpromo simply moves the files to the correct locations during the promotion process rather than copying files across drives. This functionality also minimizes the overall disk space necessary to promote a DC.
Next, start Dcpromo in advanced mode by using the command dcpromo /adv. On the Domain Controller Type selection screen, choose the Additional domain controller for an existing domain radio button. On the next screen, which Figure 4 shows, under Copy domain information, select From these restored backup files, and browse to the location (e.g., F:\system-state-backup\restore) of your directed restore.
The next screen lets you turn the DC into a GC server. You can select this option as long as your backup comes from a GC in the same domain. Even if your backup comes from a GC, you don't need to make the new DC a GC. The software will prompt you for a domain account that has the rights to promote a DC into the domain. Next, the software will prompt you for the paths you want to use for the database transaction logs, directory database, and Sysvol share. Finally, you'll need to provide a directory-restore password in case you need to boot into Directory Services Restore Mode to perform directory maintenance or restores in the future.
Near the end of the promotion process, Dcpromo replicates delta changes (i.e., changes since the system state backup occurred) from a source DC or GC over the network. The length of time this process takes depends on the size of your directory, the number of changes that have occurred since the backup you used, and correspondingly, the length of time since the system state backup occurred.
If you're considering a disaster-recovery plan, Windows 2003 Dcpromo's ability to promote from media gives you even more options. Most administrators don't perform backups on all DCs in their environment because the data is basically redundant across all DCs in a given domain. Before Windows 2003, recovering a failed DC typically required rebuilding the server from scratch, then promoting it back to a DC (and possibly a GC) over the network. You could lose hours and sometimes days waiting for promotions to finish—particularly in the case of raising a DC's status to that of a GC, which requires additional replication of read-only naming contexts that can be quite large, depending on the environment. In a remote site, this long process could mean days of suboptimal LDAP query performance over the WAN and possibly the lack of a dynamic DNS (DDNS) server in the local site. Consider also the amount of data pushed over the WAN links to complete the promotion. Using Dcpromo's promote-from-media functionality, you can easily overcome corrupt DIT files and failed hardware in a mere fraction of the time that Win2K requires. You simply need to build a base-level server and repromote the server to its former DC or GC status from a system state backup.
As I noted earlier, Windows 2003's (and Win2K SP4's) Dcpromo lets you force a demotion even when communications with the domain aren't possible. Suppose you discover that a DC hasn't replicated inbound changes for more than 60 days (the default tombstone lifetime). You wouldn't want to bring that DC online because it would replicate previously deleted objects (objects deleted on DCs that are still online) back into the environment, thereby introducing lingering objects. Lingering objects are also referred to as ghosts (in read-only partitions) or zombies (in writable partitions) because they've essentially come back from the dead. When lingering objects become trapped within a read-only naming context, they can be difficult to find and remove. Their existence can halt replication, so you need to be careful about keeping them out of your environment. The Dcpromo /forceremove command lets you bring the server back to member server status despite the inability to communicate with other DCs. You can then use Dcpromo's promote-from-media functionality to quickly repromote the server to a DC. Understand that if a forced demotion or other "dirty" removal of a DC occurs, you'll need to perform a metadata cleanup of the directory and possibly a DNS record cleanup. For detailed information about this process, see the Microsoft article "HOW TO: Remove Data in Active Directory After an Unsuccessful Domain Controller Demotion" (http://support.microsoft.com/?kbid=216498).
You must consider the location of any Flexible Single-Master Operation (FSMO) roles that the DC you're recovering might hold. If a forced demotion is necessary for some reason, you might need to seize a FSMO role to keep the forest or domain operating properly.
Scheduled System State Backups
To facilitate quick recovery of DCs, you should perform daily system state backups of at least one GC in each AD site for each domain that has a DC present in that site. You can use simple shell scripting to centrally manage and deploy a large number of scheduled DC backups. The script relies on the schtasks.exe command-line utility, which Windows 2003 and Windows XP include in the %windir%\system32 directory. Schtasks.exe is similar to NT's at.exe command, in that it lets you remotely manage schedule jobs. However, schtasks.exe is much more robust than at.exe. If you're trying to push out scheduled jobs to systems in trusted domains and you're using trusted domain accounts for the scheduled job's user context, schtasks.exe will fail to apply the account information to the scheduled job. To avoid this problem, deploy scheduled jobs from a system within the same domain as the target systems, and use service accounts from within the same domain. Another consideration is that you can't use schtasks.exe to create a local scheduled job under the context of a different account from the locally logged-on account. Therefore, to programmatically manage your system state backups, you should run SSBU-Control.bat on a server that's not in use (e.g., a domain member server). Web Listing 1 (http://www.winnetmag.com, InstantDoc ID 39767) contains this script.
SSBU-Control.bat is the engine that controls the entire process of deploying and verifying the backups and schedule jobs. Specifically, it verifies the existence of a "system-state-backup" scheduled job on any number of DCs. If the scheduled job doesn't exist, SSBU-Control.bat creates a new one automatically and logs the event to a file. The script also creates the necessary folder structure on the target DC and copies to the DC a couple of files needed to configure the backup. Further, the script verifies that a system state backup file exists on each DC and logs the details—including the file-modification date—to a central file for viewing or emailing. For example, the target directory on the DC is J:\system-state-backup\job. When the scheduled backup job runs, the backup will automatically create the backup file in J:\system-state-backup.
You must run SSBU-Control.bat under the context of the same service account you use for the system state backup jobs you're deploying. Doing so prevents conflicting credentials on the target server at script runtime. If you run the script manually within a command session, make sure that the command session is running under the context of the service account.
The steps for using shell scripting to set up scheduled system state backups are as follows:
- Copy and customize SSBU-Control.bat. Be sure to edit the variables at the top of the file to customize it to your environment. I recommend that you store the backups on a drive other than the drive that contains your directory. In the event of a DIT drive failure, you'll still have the backup file available for recovery.
- Create a batch file called ssbu.bat, which Web Listing 2 shows (http://www.winnetmag.com, InstantDoc ID 39767), that contains the command contents. SSBU-Control.bat will automatically copy this batch file to the DC and run the batch file locally on the DC to perform the backup. Make sure that the BKDrive variable at the top matches your target drive letter. Also, note that the backup time in this file is set for 2 a.m. If this time isn't suitable, insert a more convenient time. For further details about scheduling backups, see "Scheduling Command-Line Win2K Backups," September 2002, http://www.winnetmag.com, Instant Doc ID 25961.
- Set up a service account in the local domain with privileges to back up the system state data and access the share point where the backups will reside. Ssbu.bat requires access to the J$ share on the DC. You can use the variables at the top of ssbu.bat to modify the share point.
- On your source server (i.e., the server you want to use for managing the system state backups), create the C:\scripts\ssbu folder. This folder is the source directory.
- In the source directory, create a file called dclist.txt that contains a list of DCs you want to back up. Each line in the file should contain the name of only one server. You can use either Fully Qualified Domain Names (FQDNs) or NetBIOS names, assuming name resolution in either scenario is working correctly in your environment.
- Create a backup file named system-state-backup.bks. To do so, start NTBackup and click Advanced Mode on the first screen. On the next screen, go to the Backup tab and select the System State check box. Under the Job menu, select Save Selection As and save the file with the name system-state-backup.bks to the source directory. This file tells NTBackup what to back up.
- Make sure that schtasks.exe is in either the source directory or the path. I used schtasks.exe version 5.1.2600.0 in SSBU-Control.bat. This version runs on Windows 2003, XP, and Win2K.
- Create a scheduled job on your management server that runs SSBU-Control.bat once per day. This automated script will help you sleep better because you'll know that your scheduled system state backup jobs are in place and running properly. If a backup job is missing from a DC, the script fixes the problem and logs the information to a file. The SSBU_JOB_STATUS.txt file shows the backup job status on all DCs, and the SSBU-Log.txt file lists each DC and the system state backup files, so you can verify that the backup files are current and the appropriate size. Simply review these files each day.
DC promotions have come a long way since the early days of NT. Forced demotion and promotion from media give you new options for managing the deployment and disaster recovery of DCs and GCs. These new features, coupled with the ability to use schtasks.exe and NTBackup to deploy and maintain your system state backup jobs from a central location, will greatly simplify your job.