Using IPsec with Exchange

The front-end and back-end topology that Exchange Server 2003 and Exchange 2000 Server offer lets you scale your Exchange infrastructure by separating the machines that clients communicate with from the (usually larger) machines that store the mail data. However, in a typical front-end/back-end setup, the front-end server is outside the corporate network boundary, often in a demilitarized zone (DMZ), which means that the front-end server and back-end server will likely be communicating across a trusted network boundary. Unfortunately, Exchange doesn't support the use of the Secure Sockets Layer (SSL) protocol to secure this traffic. You can work around this limitation by using a firewall that can do SSL bridging (such as Microsoft Internet Security and Acceleration—ISA—Server), but that solution isn't always practical. As an alternative to SSL bridging, you can use IP Security (IPSec) to secure your Exchange network for free.

Understanding IPSec
IPSec is a set of extensions to the basic IP technology that we use for Internet communications. IPSec operates at the transport layer, so applications don't need to be aware of whether IPSec security is in effect. That approach is a major advantage over SSL, which—as an application-level protocol—requires that the application on each end know about the protocol. Another IPSec advantage is that it's made up of two separate but complementary protocols:

• The Authentication Header (AH) protocol adds a cryptographic authentication header to each IP datagram on a secured connection. The AH protocol calculates and inserts a digital signature into the packet between the original IP datagram header and the packet's payload. This approach lets the packet be routed without losing the AH data; non­IPSec-capable devices think that the AH data is part of the payload. AH provides tamper-proofing, but no confidentiality—an attacker can still read AH-protected traffic in transit.
• The Encapsulating Security Payload (ESP) protocol provides confidentiality and integrity checking. ESP uses one of two modes to encrypt the datagram's contents: In tunnel mode, packets are protected to enable connections to two separate networks; transport mode provides end-to-end security between a client and a remote network. We'll use the transport mode with Exchange; you can use tunnel mode to establish IPSec-protected VPNs.

You can use the AH and ESP protocols in conjunction with each other or independently; each protocol also supports several cryptographic algorithms. Two IPSec-capable computers begin communication by using the Internet Key Exchange (IKE) protocol to exchange cryptographic keys. The computers then negotiate to find an algorithm and key length that they both support. This process establishes a secure channel—called a security association (SA)—which protects traffic between the two machines. . . .