Patch Testing

Auditability
As part of your testing, you should also confirm that you can remotely verify that your patches have been installed successfully. Querying the remote system's registry to see whether a patch has been applied sometime in the past (e.g., by using srvinfo.exe from the Win2K resource kit) isn't enough; your audit tool must be able to check the file versions on the hard disk.

If Microsoft's downloadable XML database of patch information happens to include the patches you're testing, this check will be easy because many popular tools (e.g., Shavlik Technologies' HFNetChk, Microsoft Baseline Security Analyzer—MBSA) use this database. But if the patches you're applying aren't in the XML database, you'll need to use SMS or custom scripts to check the versions of the updated files on the disk. For this task, you could script the use of the filever.exe program (from the Support Tools for Windows 2003, XP, or Win2K), which can display verbose file-version information. If you don't want to make this effort for noncritical patches, at least audit the important ones.

Some companies give away special-purpose audit tools for the worst of the network-threatening vulnerabilities—for example, Microsoft's scanners for the Blaster and Slammer worms, described in the Microsoft articles "How to Use the KB 824146 Scanning Tool to Identify Host Computers That Do Not Have the 823980 (MS03-026) and the 824146 (MS03-039) Security Patches Installed" (http://support.microsoft.com/?kbid=827363) and "SQL Server 2000 Security Tools" (http://support.microsoft.com/?kbid=813944) and the eEye Digital Security scanners for Nimda and CodeRed (http://www.eeye.com/html/research/tools). But don't count on such tools always being available or always being released quickly. "Security Groups as PC Status Containers," April 2004, InstantDoc ID 41834, describes a method of scanning computers for a particular hotfix or security patch.

Patch Rollback
Just in case a patch is faulty and must be uninstalled, test your rollback method as well. You don't want to use the Control Panel Add/Remove Programs applet to uninstall a patch one machine at a time; thus, you should choose a patch-management solution that automates patch rollback. If your solution doesn't support patch rollback, you can at least remotely trigger the uninstallation.

When installed, most Microsoft patches create a folder named %system root%\$ntuninstallpatchnumber$\sp-uninst, where patchnumber is the number of the Microsoft article that documents the patch. In that folder is spuninst.exe, which can take the -u command-line switch to uninstall the patch, the -q switch to close all applications, and the -f switch to reboot the system. You can execute this program with all the necessary switches through a variety of methods; for example, you can use a custom script pushed out through Group Policy, use schtasks.exe to remotely schedule a job to do it, use a WMI script (such as exec.vbs from the resource kit) to execute commands on remote systems, or perhaps use Sysinternals' PsExec. Sometime this year, Microsoft will start releasing .msi patches that use Windows Installer 3.0, and these will support rollback.

After you remove a patch, you should employ the scripts and techniques I've described above to verify remote manageability. Installing a patch can cause problems, but trying to uninstall one can kill a box, and you'll definitely want to discover this truth in the lab rather than in the real world.

Clinical Trials
After all the lab work is done, the time has come to do clinical trials on human subjects—that is, to try out new patches on a subset of your production systems before rolling the patches out to everyone. Software developers and other IT-savvy people make great guinea pigs because they're more sensitive to subtle problems that occur and they might be working with new code or applications that will soon be deployed. You'll need at least a dozen machines because you'll need to have a representative sampling of the OSs, service pack levels, applications, and services on your network. Push out patches to these machines first, monitor for problems and complaints, then start rolling out the patches to the rest of your enterprise in stages if all goes well. Some will argue that you don't have time to test crucial patches in this way, but you can shorten the "live animal testing" to just a few hours if pressed for time because show-stopping problems typically surface immediately after you apply faulty patches.

You should also conduct clinical trials for your production servers, but do a full backup of the guinea pig servers first. Select one of each of your types of servers (e.g., Web, mail, domain controller—DC, print server) and apply the patches to just those machines one at a time. For 24 × 7 service availability during frequent patching, failover clustering and the Windows NT Load Balancing Service (WLBS) are a big help.

Don't forget that others are facing the same patch-management problems as you are. Browse newsgroups dedicated to the products you manage (one place to find such groups is at http://groups.google.com), and subscribe to patch-related or security mailing lists such as NTBugtraq (http://www.ntbugtraq.com) or the patch-management list at http://www.patchmanagement.org. You can learn a ton from your peers in the IT community, and when your testing discovers faulty patches, you can share your findings and advice with them too.