Scanning Flexibility
One significant weakness we found in most of the patch managers is that configuring the system to scan a complex network is difficult. Our test network had a variety of common but sometimes complicated configurations. Most of the products scan a network by using standard Windows protocols and remote registry access to query each system (i.e., they use an agentless system) or by installing on each system an agent that reports to a central station (i.e., they use an agent-based system). Table 1 compares the pros and cons of these two scanning methods.
The agentless products we tested were Ecora Patch Manager, Gravity Storm Software's Service Pack Manager 2000, and Shavlik Technologies' HFNetChkPro. The agent-based products were BigFix Patch Manager, PatchLink Update, and SecurityProfiling's SysUpdate. Only one product, St. Bernard Software's UpdateEXPERT, uses both scanning methods, although many products have future plans for using both methods.
Every product we tested had its quirks, and defining target systems in each product was a tedious and frustrating process. Most of the agentless products offer several methods for adding systems, such as by IP address range, by domain name, by AD OU, or by importing a list of host names in a text file. Ecora Patch Manager and HFNetChkPro provided the most flexibility for adding systems. All agentless products allowed for custom logon credentials for each system or group of systems.
Adding systems to our test network was difficult using any of the agentless products. Identifying all our test systems from different domains and workgroups, often with varying system credentials, was awkward and sometimes tricky. UpdateEXPERT and Ecora Patch Manager had problems seeing all the systems in each domain, HFNetChkPro had problems with conflicting credentials, and Service Pack Manager 2000 wouldn't let me add a particular system because the password was too long to fit into the password field on the credentials screen. All the agentless products had problems adding offline domain members. Perhaps some of these problems will be solved by the time you read this article and many of these problems won't show up on simpler networks, but after struggling with each of the agentless products, I believe that agent-based products might be easier to work with.
The agent-based products sidestepped some of the obstacles of the agentless products, but installing the agents on each system required a significant amount of work. Most of the agent-based products let you push agents to remote clients, but that functionality has the same limitations as agentless product installation: You need remote administrative access to each system. Most of the agent installations prompted for information that would make the agents difficult to mass-deploy using automated methods. For example, UpdateEXPERT requires that you manually enter a serial number when installing the client agent. However, BigFix Patch Manager provides easy installation by building custom client configurations that include everything necessary to connect to the server. One problem that the agent-based products had that agentless products didn't have is that after installation, communication between agents and the central console sometimes broke down.
Patch Detection
After running each patch manager against the network, I was surprised with the inconsistent results. Although we expected some false positives and some false negatives, not one product achieved 100 percent accurate results in every test. HFNetChkPro was the only product that achieved 100 percent accuracy on some tests. But most surprising was that no two products produced the same report and no one product produced the same report twice. Each product had different inaccuracies. To be fair, most of the problems occurred because of the confusing nature of some Microsoft fixes.
Patching Windows is more complicated than most people realize. It's not sufficient to simply replace older files with newer versions; a patch-management system also must take into account what other software is in use and which versions of applications such as Microsoft Internet Explorer (IE), Microsoft Data Access Components (MDAC), and Microsoft XML Core Services (MSXML) are installed to know exactly which file versions to use. To further complicate matters, sometimes a file has a more recent file date than the one installed but an earlier file version. And if you installed Windows with a slipstreamed service pack, you have even more hurdles to surmount.
Service Pack Manager 2000 usually returned the longest list of missing hotfixes, but many of the items were patches that had been superseded or weren't relevant to the current configuration. HFNetChkPro, Ecora Patch Manager, and PatchLink Update consistently produced the most accurate results. The rest of the products had varying levels of accuracy, with an average of 5 to 10 mistakes (i.e., false positives and false negatives) for each system. Since my testing, St. Bernard has added patch-validation support to UpdateExpert 6.1, which should improve the product's accuracy, although I haven't tested it yet.
Register
