Troubleshooting Kernel Memory Corruption

As an IT administrator, you've no doubt been the victim of the infamous drive-by blue screen crash. It typically happens at the most inconvenient time, often resulting in service interruptions and work stoppage causing monetary loss. Working on the Microsoft Windows escalation team, I'm tasked with debugging these crashes daily and reporting a timely action plan to prevent such problems from happening again. The work is a lot like the popular crime show, CSI, where crime scene investigators use low-level techniques to flesh out the guilty party. In some of my debugging investigations, the DNA left behind is as microscopic as a single bit flipped to a 1 when the code was expecting a 0. In a high percentage of the cases, the offender is long gone except for the memory corruption that remains, making it extremely difficult to find the misbehaving driver. How do we weed out culprits that cause memory corruption?

In this two-part article series, I'll discuss the tools used by the Microsoft Support Team to troubleshoot kernel memory corruption typically caused by a buggy driver that doesn't kindly leave behind its calling card. Because of the complexity of the memory manager, it may not be immediately clear to you why the support team prescribesf these tools. This article and the next one should clear up any confusion and will be especially helpful if you're responsible for reporting an action plan back to your management team. I should also mention that although there are many other reasons for bug-check crashes, this article will focus primarily on some of the tools used by Microsoft Support to diagnose crashes caused by kernel memory corruption.

Pool Memory Architecture
Before I delve into the tools, I'll provide a short primer on high-level pool memory architecture, which may help you better understand when I explain the tools in more detail. Most of the memory that drivers use is allocated from the system-provided pools, called paged pool and nonpaged pool. There are exceptions to this rule, but a discussion of them is beyond the scope of this article. As the names imply, the memory used by the nonpaged pool is always guaranteed to be resident in physical memory, while portions of the paged pool can be swapped out at any given point in time.

To better understand the output from our diagnostic tools, it's important to note that a pool is divided into smaller divisional units called pages, either small or large pages. A small page is 4KB. A large page is either 2MB on the x64 platform or 4MB on x86 systems. However, if the Physical Address Extension (PAE) option is enabled on x86, the large page size is reduced to 2MB. Both pages types have their advantages, but for the rest of this article we'll assume I'm talking about the smaller 4KB pages.

A page is further subdivided into the actual allocations made by the various drivers asking for memory. Figure 1 shows a random page of non-paged memory that I dumped, which shows an example of a 4KB memory page.

I used the !pool command to display the memory in the Microsoft debugger (windbg.exe) that's available with the Microsoft Debugging Tools from the Microsoft download site. (For more information about using the Microsoft debugger, see "Administrators' Intro to Debugging.") Windbg is the primary debugging tool used by the Microsoft Global Escalation Services team. (In future articles I plan to discuss the different debugging techniques used with Windbg for diagnosing these types of crashes.)

This output represents a single 4KB page of nonpaged memory. Each row is a block of memory either allocated to or freed from the page. Most of the blocks in this example are allocated, and you can sort of determine the owner of the block by the tag listed to the right of each allocation. When a driver makes a call to allocate pool memory, it passes the size of the requested block along with a four-character identifier called a tag. The tag and the size of the allocation requested are maintained in a bookkeeping structure called the pool header, which is parked at the top of each allocation. This structure also maintains the current size and previous size block, which is used by the memory manager to easily traverse the contents of the page for maintenance tasks such as coalescing adjacent freed blocks into one large freed block. The area following the pool header is arguably the most important area from the perspective of the drivers allocating the memory, as this is the actual storage area for the memory page's data.

The Crime Scene: Dusting for Driver Bug Fingerprints
Building on the context of how the memory manager organizes pool memory, let's discuss what happens when software bugs creep into the environment causing those unpleasant blue screen crashes. One of the most common driver bugs is to write beyond its allocation and spill into the next allocation overwriting data it doesn't own. As mentioned earlier, the pool header precedes the actual data area in each allocation, so when the driver writes into the next allocation, the pool header for the next allocation gets corrupted. If the memory manager later attempted to read from the corrupt pool header, the system would possibly crash with a Bug Check 0x19: BAD_POOL_HEADER or Bug Check 0xC2: BAD_POOL_CALLER. The parameters of these bug checks are documented fairly well on MSDN. A look at the first two parameters usually indicates the state of the corruption; however, it doesn't point you to the buggy driver.

Let's extend the scenario before investigating the tools we use to weed out the misbehaving drivers. Using the output in Figure 1, assume the driver using the VadS pool at virtual address fffffadcda813c50 wrote beyond the pool header of the Irp allocation at fffffadcda813c90 and continued writing into the data section. Now we have a scenario where not only the pool header is corrupted, but the driver data, too. If the driver owning the Irp pool used the corrupted data, there is a high likelihood of system instability. Even worse, the owner of the Irp pool would appear to be the guilty party because that driver might be on the stack if the system crashed.

The bug check code would vary depending on how the owner of the Irp pool used the corrupt data. It may manifest as a STOP 0x0000001e if the driver of the Irp pool attempted to de-reference the corrupt value as a pointer and the value was inaccessible. And what if the address was accessible and the driver wrote data to this random address? Now the corruption runs into another pool or perhaps a critical kernel structure. I point all this out to illustrate how extremely difficult it can be to trace back to a driver when there is no distinct path back to the guilty party.

While investigating this scenario, we could easily point the finger at the VadS pool, making this an open-and-shut case. But consider the case where the VadS pool has been freed and another driver allocated the block in its location, only after VadS had corrupted the Irp pool. Now the VadS owner is long gone, making this a cold case file. Enter Special Pool.

The Chase Continues
Introduced in Windows NT SP4, Special Pool was created to catch drivers corrupting memory in real time by allocating guard pages around the allocation. The idea is to catch a driver writing beyond its allocation by forcing it to write into a guard page, causing the system to crash immediately with the culprit on top of the stack. It's the smoking gun approach. In my next article, we'll take a deep look at the Special Pool mechanism and discuss how it's implemented.


Ron Stock (ronsto@microsoft.com) is an escalation engineer for Microsoft's Global Escalation Services team. He specializes in advanced Windows debugging and performance-related issues. For information about Windows debugging, visit his team's blog.