3 Steps to Troubleshooting Device Drivers

I’m waiting for an application to start…again. As I stare at the hourglass on my Windows XP desktop, I wonder what is slowing things down this time. I press Ctrl+Shift+Esc to bring up Windows Task Manager and click the Processes tab. I’m ready to kill the offending process, but I find nothing but the usual System Idle Process at the top of the list.

Over time, Windows system performance tends to degrade. Many performance problems can be resolved by eliminating unnecessary programs from startup, defragmenting the hard disk, and cleaning the registry. In my case, however, none of these actions worked. I don’t usually get into desktop support issues but since it was my desktop, I was determined to find and resolve the problem.

When I searched the Internet, I found many people complaining about the same symptoms in blogs and forums, but they didn’t provide any solutions. I got a few clues when some people mentioned that the problem went away after they upgraded a driver or replaced a disk controller. At this point, I concluded that the performance problem was probably related to a driver, but I didn’t know for sure. So, I dug deeper. After wading through some device-driver development articles, I found a gem. The Microsoft TechNet article “Advanced DPCs” (www.microsoft .com/technet/sysinternals/information/ advanceddpcs.mspx) discusses how to diagnose driver performance problems by examining deferred procedure call (DPC) queue activity.

DPCs wreaking havoc on Windows system performance is a common problem in not only workstations but also servers. Knowing how to recognize, diagnose, and solve a DPC problem can save you hours of troubleshooting and possibly save you from having to reinstall or reconfigure the problematic workstation’s or server’s OS.

1. How to Recognize the Problem
To recognize the problem, you need to know about DPCs and how to perform an initial check in Windows Task Manager when you’re experiencing a performance problem. DPCs are a part of the Windows interrupt handling architecture. Interrupt handling consists of two components, both of which are part of a device driver. The first component is the Interrupt Service Routine (ISR), which quickly allows the hardware to get the device to stop interrupting. The driver actually handles the processing around the interrupt later by queuing a worker thread from the created DPC queue object. In short, this means that a driver does its initial work with a hardware device in a DPC. Because both the ISR and DPCs are vendor-provided within the driver for a device, a poorly written kernel-mode driver can have a significant impact on overall system performance.

When you’re experiencing a performance problem, you can use Windows Task Manager to see whether it might be due to DPCs. When a performance problem might be related to DPCs, you’ll see that the CPU usage is high on the Performance tab, but no processes are taking the blame on the Processes tab. You’ll also see that System Idle Process is showing the highest CPU usage on the Processes tab, which is typical for an idle system.

The System Idle Process typically doesn’t significantly affect system performance. However, when DPC problems exist, the kernel will be using a large percentage of the CPU, which can impact system performance. You can find out the kernel’s CPU usage by selecting the Performance tab and choosing the Show Kernel Times option on the View menu. The bottom red portion of the CPU Usage graph shows the CPU time being used by the Windows kernel. The sample CPU Usage graph in Figure 1 reveals that something loaded by the kernel is monopolizing the CPU.

2. How to Diagnose the Problem
A kernel can monopolize the CPU for several reasons, so the next step is to diagnose the problem to determine whether DPCs are in fact causing the high CPU usage and if so, which device is using those DPCs. You can use three tools to drill down and get the details needed for diagnosis:
• Performance Monitor (perfmon.exe). This is a built-in Windows tool.
• Process Explorer (procexp.exe). You can download this free utility from Microsoft at www.microsoft.com/technet/ sysinternals/ProcessesAndThreads/ ProcessExplorer.mspx.
• Kernel Profiling Tool (kernrate.exe). This tool is part of the Microsoft Windows Server 2003 Resource Kit, which you can download at www.microsoft.com/downloads/details.spx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en.

To determine whether DPCs are causing high CPU usage, you can use Performance Monitor or Process Explorer. I find Process Explorer easier to use than Performance Monitor for this purpose, but I’ll cover both in case you prefer to use Performance Monitor.

In Process Explorer, you can quickly and easily see whether DPCs are causing high CPU usage. In the main window, the System Idle Process is broken into three groups: Interrupts, DPCs, and System. Just doubleclick DPCs to bring up the DPCs properties page and select the Performance Graph tab. The top two graphs show the DPCs’ CPU current usage and CPU usage history, as Figure 2, page 53, shows.

Under the Processor ob-ject, Performance Monitor includes a counter named % DPC Time, which tells you the percentage of time a processor is spending on receiving and servicing DPCs. Comparing this percentage to the % Processor Time value reveals what portion of total CPU usage is consumed by a thread in the DPC queue, as Figure 3, page 53, shows. For information about how to use Performance Monitor, see “Performance Management in Windows” (March 2003, InstantDoc ID 37933).

After Process Explorer or Performance Monitor has confirmed that the CPU is being disproportionately consumed by activity in the DPC queue, you can use the Kernel Profiling Tool to determine which driver is causing the problem. This command-line tool lists kernel modules, including kernel-mode device drivers and the percentage of kernel time that they're consuming.

Continue on Page 2