Executive Summary:

Two malware protection defenses in Vista and Server 2008 can help you protect against buffer-overrun–based attacks. They are Data Execution Protection (DEP) and Address Space Layout Randomization (ASLR). DEP and ASLR each use a slightly different proactive defense approach as a buffer-overrun defense. Where ASLR makes it more difficult for malware to find the right code, DEP makes it more difficult for malware to execute the code once the target code is found. You can leverage both techniques at the same time and they can also be leveraged in virtual computing environments such as Microsoft Virtual PC or VMware products.


Attacks based on buffer overruns (aka buffer overflows) have been a problem for a long time and are still considered one of the computer industry’s most important security problems. The first buffer-overrun–based attack distributed via the Internet, the Morris worm, did a lot of harm in 1988. The sad thing is that the creators of the Morris worm didn’t write the worm to cause harm but rather as an experiment for measuring the size of the Internet. The Morris worm exploited weak passwords and known vulnerabilities in UNIX programs such as sendmail and Finger. Two recent well-known attacks that involved exploiting buffer overruns, the Code Red and SQL Slammer worms, exposed many Internet-connected systems to attackers’ control. In 2001, the Code Red worm exploited a bufferoverrun vulnerability in Microsoft Internet Information Services (IIS) 5.0 (the IIS version that is bundled with Windows 2000), and in 2003, the SQL Slammer worm used a buffer-overrun vulnerability to compromise machines running Microsoft SQL Server 2000.

You can defend against buffer-overrun–based attacks by using defenses that Microsoft includes in Windows Vista and Windows Server 2008: Data Execution Prevention (DEP) and Address Space Layout Randomization (ASLR). (At the time of this writing, Microsoft was about to release Vista SP1 and had released Windows Server 2008 RC0.) I’ll explain why these defenses are important and how you can configure them and observe their behavior.

Understanding Buffer Overruns
Before going into more detail on the Vista and Server 2008 buffer-overrun defenses, it might be worthwhile to look at how a buffer overrun works and how it can harm your systems and data.

A buffer overrun occurs when a malicious or badly engineered program stores data beyond the boundaries of a fixed-length buffer in computer memory. The result is that the extra “overflowing” data overwrites adjacent memory locations. The data that’s overwritten can include other buffers, variables, and program logic and may cause a process to crash or produce incorrect results. An even bigger threat is that the injected data often includes executable code that the program under attack is then lured to execute. This executable code often contains the real payload of a buffer-overrun–based attack. It’s used to steal or delete data, create Denial of Service (DoS)–based service outages, trigger privilege elevations, or spread malware to other systems.

Figure 1 gives a simple example of a buffer overrun. A program has defined two variables that are stored in adjacent memory locations. The first variable is an eightbyte- long string called X; the second, a two-byte integer called Y. Initially, X contains nothing but zero bytes, and Y contains the number 30. Imagine that a user (whether unintentionally or maliciously) inputs a character string OVERFLOW to this program. The program then attempts to store this character string in X’s memory location followed by a 0 value to mark the end of the string. The program logic doesn’t check the length of the string and partially overwrites the value of Y. The result is that, although the programmer didn’t intend to change the value of Y when variable X receives input, variable Y’s original value 30 is now replaced by the number that’s part of the character string that was injected into the variable X memory location.

Developers can prevent buffer overruns by including sufficient boundary checks in their program code and by leveraging compilers or runtime services that perform boundary checks. Boundary checks ensure that input data are of the right length. Although boundary checking and enforcement have become best practices for developers, plenty of legacy code doesn’t include boundary checks. Also, coding best practices are worthless if some programmers don’t follow them.

These reasons explain why many hardware, application, and OS software vendors including Microsoft have developed proactive defenses that attempt to stop bufferoverrun attacks in badly engineered code. Let’s look at Microsoft’s implementations of DEP and ASLR.

Data Execution Protection
As I mentioned above, buffer-overrun–based attacks often write executable malicious code to another program’s memory buffers and then trick the program into executing the malicious payload. You can tackle the execution of maliciously injected code by using DEP. DEP lets Windows mark memory locations that should only contain data as non-executable (NX). When an application attempts to execute code from NX-marked memory locations, Windows’ DEP logic will block the application from doing so.

A negative side effect of the buffer-overrun protection offered by DEP is that the blocked application will typically halt. In other words, even though DEP stops malware from executing its malicious payload, this situation creates a new opportunity for malware to launch DoS attacks.

Microsoft includes DEP support not only in Vista and Server 2008, but also in Windows XP SP2, Windows Server 2003 SP1, Windows 2003 R2. Microsoft DEP implementation comes in two flavors: hardware-enforced DEP and software-enforced DEP.

Hardware-enforced DEP. Hardwareenforced DEP leverages a processor feature that AMD refers to as the no-execute pageprotection (NX) feature and that Intel refers to as the Execute Disable Bit (XD) feature. At the time of writing, AMD supported NX only on its 64-bit processors, and Intel supported XD only on the Itanium and EM64T 64-bit processors and a small number of 32-bit Prescott processors. Microsoft is not the only OS vendor that leverages the NX and XD processor features for stopping buffer overruns: NX- and XD-enabled software is also available in other OSs such as Linux and UNIX BSD (see en.wikipedia.org/wiki/Nx-bit for more information).

Software-enforced DEP. Softwareenforced DEP lets Microsoft provide DEP on 32-bit processor systems not equipped with an NX- or XD-compatible processor. In this software workaround, the processor-level NX- or XD-bit functionality is provided by a set of special pointers that the Windows OS automatically adds to data objects stored in the system memory.

Continue on Page 2

You can easily check whether your system supports hardwareor software-enforced DEP by checking the DEP configuration settings. You can access these settings using the Advanced Settings option in the System Control Panel applet and navigating to the Advanced and Performance Settings options.

At the bottom of the DEP configuration settings screen, there’s a reference to the type of DEP your system supports. Figure 2 shows the DEP configuration settings on a Vista system. (I’ll explain the other configuration options later in this section) The bottom line reads, “Your computer’s processor supports hardware-based DEP.”

If your system supports software-enforced DEP (meaning that your machine doesn’t have the NX- or XD-compatible processor), you’ll see “Your computer’s processor does not support hardware-based DEP. However, Windows can use DEP software to help prevent some types of attacks.”

An alternative way to check whether your system supports hardware- or software-enforced DEP is by using Windows Management Instrumentation (WMI) commands. The procedure is outlined in the Microsoft article at support.microsoft.com/kb/912923.

On XP SP2, Windows 2003 SP1, and later Microsoft OSs, DEP is enabled by default. However, DEP doesn’t always protect all programs running on your system. The exact list of programs that are protected by DEP is defined by DEP’s protection level. DEP supports two protection levels:

  • Level 1—The first level protects only the Windows system code and executables and doesn’t offer DEP protection for additional Microsoft or third-party applications that run on your system.
  • Level 2—The second level protects all executable code that runs on your system; it offers DEP protection for both Windows system code and the Microsoft or thirdparty applications that run on your system.

By default, XP SP2 and Vista run DEP at protection level 1; Windows 2003 SP1 and Server 2008 run DEP at protection level 2.

Administrators can configure the DEP protection levels from the DEP configuration screen, which you can see in Figure 2. In this example (which shows the default DEP configuration settings on a Vista system), DEP is enabled for essential Windows programs and services only—DEP protection level 1. You can use the other radio button Turn on DEP for all programs and services except those I select to switch to DEP protection level 2, which is the default setting on Windows 2003 SP1 and Server 2008.

Protection level 2 lets you exempt certain applications from DEP protection. This ability to exempt apps is important because some legacy applications don’t run properly when DEP is enabled—for example, at the time of writing, Microsoft Word was still automatically exempted from DEP. Before switching your DEP protection to level 2, you must run an application compatibility test to ensure that all applications run properly when DEP is enabled. To exempt one of your applications from DEP, you can add the application’s executable to the excluded list in the DEP configuration screen using the Add… button.

You can easily check whether a given application is protected by DEP by checking the DEP column of the application’s process in the Windows Task Manager, which Figure 3, page 66, shows. If you don’t see the DEP column on your system, you can add it using the Task Manager’s View\Select Columns… option.

Another way to exempt one of your applications from DEP is to create a software fix to distribute to your systems that automatically disables DEP for a given application on those systems. Microsoft refers to such a software fix as a DisableNX shim. To create this software fix, see the Microsoft Application Compatibility Toolkit (ACT), which also includes a tool called Compatibility Administrator that can help (technet.microsoft.com/en-us/windowsvista/aa905078.aspx). Application developers can also do the opposite—directly enable their applications for DEP support in their application binaries. To do so, they use the /NXCompat compilation switch.

One important final note is that when DEP is running in protection level 2, your system will run a bit slower because of all the extra DEP checks that are carried out on the processor and system memory level. That’s why for test systems that aren’t exposed to the Internet, for example, you can consider turning off DEP protection completely. The only way to turn off DEP completely on a given system is to specify the /NoExecute=AlwaysOff switch in the system’s boot.ini file.

Continue on Page 3

Note that you can also use the same boot.ini /NoExecute= switch with other values to turn DEP on and to set the DEP protection level. Table 1 shows all the / NoExecute values.

The boot.ini file is available only on XP and Windows 2003, and you can edit it using Notepad or going to the Startup and Recovery section in System properties.

On Vista and Server 2008, the boot.ini has been replaced by the Boot Configuration Data (BCD) file. To edit the BCD file, Microsoft provides a command-line utility called bcdedit.exe.

When you run bcdedit without switches, it shows your current boot configuration. Figure 4 shows the result of running bcdedit on a Vista system. Note the last line that holds the nx configuration OptIn. To change the nx configuration to alwaysoff, you would run the following bcdedit command:

 bcdedit /set nx alwaysoff

The values specified in Table 1 for the boot.ini /NoExecute= switch are also available for the BCD nx option.

For more information about Microsoft DEP and how to configure it, consult the Microsoft article at support.microsoft.com/kb/875352/en-us.

Address Space Layout Randomization
Another technique often used by bufferoverrun– based malware is to inject a system memory path that points to the location of an important system DLL into another program’s buffer. The malware then tricks the program into calling that particular system file to let the malware leverage the system DLL’s services without being detected.

This type of buffer-overrun attack is relatively easy to carry out if the OS always loads certain system DLLs on the exact same memory location. On XP, for example, the memory locations of system DLLs are always identical— they vary only slightly depending on the service pack status of the system. The new Vista and Server 2008 ASLR feature makes it harder for malware to leverage a system DLL’s services by randomizing DLL’s memory location. Unlike DEP, ASLR isn’t available on earlier Windows versions.

Each time a Vista and Server 2008 system reboots, ASLR randomly assigns system code (basically system DLLs and executables) to different memory locations. This means that the system code’s entry points (the addresses the malware would use to call on the service of a particular piece of system code) are in unpredictable locations. In Vista and Server 2008, a DLL or executable can be loaded into any of 256 locations. This means that an attacker has a 1/256 chance of getting the address right. As such, ASLR also makes it harder for hackers to write repeatable code such as worms that target identical system resources on many different systems.

You can observe the effect of ASLR by using the SysInternals Process Explorer tool, which you can download at www.microsoft.com/technet/sysinternals/utilities/processexplorer.mspx. To use the tool, start Process Explorer and ensure that you have selected the Show Lower Pane option in the View menu.

Then select the explorer.exe process in the upper pane and check the base address of the ntdll. dll in the base column in the lower pane. (If you don’t see the base column you can add it by using the View / Select Columns… menu option—the Base column can be added from the DLL tab.)

Write down the base address, then reboot your system. On an XP system, the base address for ntdll. dll remains identical after a system reboot (XP doesn’t support ASLR). On a Vista system, the base address is different after a system reboot (because Vista supports ASLR).

Figure 5 shows the Process Explorer interface and the base address for the ntdll.dll DLL. Table 2 shows the base addresses I found for the ntdll.dll and user32.dll DLLs when running Process Explorer on an XP SP2 system and on a Vista system.

Continue on Page 4

You can leverage ASLR not only for randomizing the memory locations of Windows system files but also for randomizing the memory locations of executables and DLLs of any application that runs on Vista or Server 2008. To do so, application developers must compile their code with the /dynamicbase linker option. Microsoft Visual Studio supports this option from Visual Studio 2005 SP 1 and later.

Like DEP, ASLR is not a Microsoft-only invention and implementation. ASLR was implemented long before Vista and Server 2008, on platforms such as Linux and UNIX. Also certain Host Intrusion Detection System (HIDS) solutions have been supporting ASLR on legacy Windows platforms long before the native Windows support.

A good analysis of the Microsoft ASLR implementation in Vista is offered in the Symantec research paper at www.symantec.com/avcenter/reference/Address_Space_Layout_Randomization.pdf. Unlike with DEP, Microsoft doesn’t offer ASLR-specific configuration settings for fine-tuning the use of ASLR.

Important Proactive Defenses
DEP and ASLR each use a slightly different proactive defense approach as a buffer-overrun defense. Where ASLR makes it more difficult for malware to find the right code, DEP makes it more difficult for malware to execute the code once the target code is found. You can leverage both techniques at the same time and they can also be leveraged in virtual computing environments such as Microsoft Virtual PC or VMware products.

From an application-support point of view, you should remember that you must test your applications for DEP compatibility prior to deploying them on a DEP-enabled Windows platform. DEP can cause certain applications to stop working properly or even halt.

Finally, it’s important to understand that DEP and ASLR aren’t a panacea for the buffer-overrun problem. Both techniques certainly make it much more difficult for malware to leverage buffer overruns. ASLR, for example, doesn’t make it impossible for malware to find system code, but it makes the process of finding system code much more challenging. In many cases, ASLR and DEP will also effectively stop bufferoverrun– based attacks.