Get ready to secure your IP network with IPSec

Early this year, hackers tore down NT systems in many sites with a denial of service attack called Teardrop2. (Teardrop2 sends an NT system deliberately constructed IP fragments that form invalid packets. The NT system receiving the packets allocates kernel memory to accommodate them. If the system receives a large number of the invalid packets, it will hang and stop working.) Although Microsoft immediately responded to these attacks with a hotfix that defends NT's IP stack against Teardrop2, NT remains a favorite target of hackers. Attacks such as these could happen again, unless network managers upgrade their NT networks' IP protocol to IP Security (IPSec).

The Internet Engineering Task Force (IETF) developed IPSec as a security protocol of the next generation IP--IPv6. IPSec is also an optional extension for the implementation of IPv4, the current-version IP. IPv4 is widespread on the Internet and in corporate networks, but its design does not include security provisions. IPSec provides confidentiality and integrity to information transferred over IP networks through network-layer encryption and authentication. IPSec protects your IP network from attacks, including denial of service, man-in-the-middle, and spoofing. (For more information about IPv6 and the development of IPSec, see my article, "The Next Generation IP in Action," June 1998.)

Microsoft is building IPSec into NT 5.0, which will let you implement a secure NT network without having to change your existing applications and network hardware. With NT 5.0 IPSec, you can define security policies for your entire organization, departments, groups, or individuals, and you can specify whom your NT computers can trust and talk to and what security methods those computers can use for communication. IPSec in NT 5.0 will be an important component of your network security. Study IPSec and perhaps test it to develop a good implementation plan for when NT 5.0 hits the street. In this article I'll help you understand IPSec, NT 5.0 IP security policy, and NT 5.0's implementation of IPSec.

IPSec Basics
The IETF defined the IPSec protocol in Request for Comments (RFC) 1825-1829 and several Internet Drafts. IPSec protects IP traffic with two protocols: the Authentication Header (AH) protocol and the Encapsulating Security Payload (ESP) protocol.

AH integrity ensures data integrity by authenticating a packet's IP header and payload (i.e., packet content). If a hacker alters an IP packet and replays it, AH lets the intended recipient know that the packet underwent modification during transmission. ESP confidentiality guarantees data confidentiality by encrypting IP packets so that hackers can't decode them. ESP confidentiality is mandatory in IPSec. The difference between AH integrity and ESP integrity is that ESP integrity doesn't authenticate IP headers. ESP integrity is an option in IPSec implementation, but Microsoft recommends using both ESP confidentiality and ESP integrity for high security. However, if you use a Network Address Translator (NAT) to translate your private IP addresses into Internet legitimate addresses, you can use only ESP integrity, because ESP integrity doesn't manipulate IP headers, as AH integrity does.

IPSec operates in two modes: transport mode and tunnel mode. In transport mode, AH or ESP resides in the original IP packet between the IP header and upper-layer extension header information (to learn about the content of IP headers, see the June 1998 sidebar "What's New in the IPv6 Header"). IPSec uses the transport mode to provide end-to-end security between two end systems: for instance, between an NT workstation and an NT server. In the tunnel mode, IPSec places an original IP packet in a new IP packet and inserts AH or ESP between the IP header of the new packet and the original IP packet. The new IP header points to the tunnel endpoint, and the original IP header specifies the ultimate destination of the packet. You can use the tunnel mode to set up an IPSec tunnel between two end systems, between an end system and a security gateway, or between two security gateways. A security gateway can be a tunnel server, router, firewall, or Virtual Private Network (VPN) device. One example of implementing tunnel mode is securing remote access to your corporate network through the Internet. When you have a tunnel server at the perimeter of your network, telecommuters must go through the tunnel server before reaching an internal system. The tunnel in this example is between the Internet and the tunnel server (i.e., between an end system and a security gateway).

IPSec uses authentication and encryption algorithms in AH and ESP to implement data integrity and confidentiality. There are several authentication algorithms, such as Hash Message Authentication Code (HMAC), Message Digest version 5 (MD5), and HMAC Secure Hash Algorithm (SHA). There are also several encryption algorithms: for example, Data Encryption Standard (DES), DES-Cipher Block Chaining (CBC), and 3DES. As a minimum requirement for IPSec compliance, IPSec vendors must implement HMAC MD5 and HMAC SHA for AH and ESP integrity, and DES for ESP confidentiality. In addition to the foregoing three security algorithms, Microsoft supports DES-CBC and 3DES for ESP confidentiality in NT 5.0 IPSec. The sidebar "Five Security Algorithms" explains HMAC MD5, HMAC SHA, DES, DES-CBC, and 3DES.

ISAKMP/Oakley
Before two systems can exchange data, they need to agree on a set of security parameters--such as a shared session key, identity authentication method, and data authentication and encryption algorithm--that is based on their IPSec policies. This set of security parameters is called Security Association (SA). After the IETF considered several security negotiation and key-management protocols, it chose Internet Security Association and Key Management Protocol (ISAKMP)/Oakley to negotiate and manage SAs in IPSec. Oakley is a protocol name taken from the developer's name, and it is used with ISAKMP for key exchange in IPSec. (The IETF also calls ISAKMP/Oakley the Internet Key Exchange--IKE.)

ISAKMP/Oakley consists of two phases or modes: phase I, or main mode, and phase II, or quick mode. The main mode establishes the ISAKMP/Oakley SA, and the quick mode establishes the IPSec SA. The ISAKMP/Oakley SA is bidirectional, and the IPSec SA is unidirectional. Because the IPSec SA can travel in only one direction, there are two IPSec SAs for each pair of communicating systems.

In the main mode, ISAKMP/Oakley decides which authentication method two communicating systems will use to authenticate each other. ISAKMP/Oakley supports multiple authentication methods. NT 5.0 IPSec uses the Kerberos, certificate, and preshared key authentication methods. After two communicating systems trust each other, ISAKMP/Oakley uses the Diffie-Helman protocol to generate and exchange a shared symmetric key, which creates a secure channel between the two systems.

After the ISAKMP/Oakley SA is in place, either communicating system can initiate the quick mode to set up the IPSec SA. In the quick mode, ISAKMP/Oakley negotiates which security algorithms the two systems will use to secure application data. Then, ISAKMP/Oakley generates a new shared symmetric key that the selected security algorithm uses to authenticate and encrypt data. This new key is different from the shared symmetric key the main mode used, but a hashing function can derive the new key from the main-mode key. To enact Perfect Forward Secrecy (PFS), which doesn't allow use of an exiting key to derive a new key, ISAKMP/Oakley can use Diffie-Helman again to generate the new key. NT 5.0 IPSec supports PFS. However, PFS requires additional processing time.

Both ISAKMP/Oakley modes have a rekeying mechanism that provides a highly secured session through the implementation of limited key lifetime. In limited key lifetime, when a key expires, ISAKMP/Oakley automatically generates a new key. By default, NT 5.0 IPSec sets key lifetime to 1 hour in main mode and 5 minutes in quick mode. You can adjust the length of NT 5.0 IPSec key lifetimes according to your security requirements.

NT 5.0 IP Security Policy
The core of NT 5.0 IPSec implementation is NT 5.0's security policy, which consists mainly of a rule that controls how IPSec works in NT 5.0. This rule is a collection that consists of a list of IP filters, negotiation policies, an authentication method, an IPSec tunnel setting, and a connection type. Let's look at each of these policy attributes. Then I'll set up a scenario in which I can demonstrate how to create a security policy.

IP filters. An IP filter defines whom a computer can talk to. IP filters specify allowable IP source and destination addresses, protocol, and source and destination port numbers. An allowable address can be any address, the address of a computer receiving the security policy, a specific DNS name, a specific address, or a specific subnet. Screen 1 shows an example of an IP filter specification in which any computers in the subnet 193.1.1.0 can talk to any computer that receives the security policy containing the same filter, and vice versa. You can use an IP filter to filter any protocol, such as TCP, UDP, Internet Control Message Protocol (ICMP), and Raw IP. For example, you can use TCP and the port 80 to filter HTTP traffic. A security policy can contain multiple IP filters. However, two communicating computers take further security action only when both contain a matching filter.

Negotiation policy. You've learned that IPSec uses ISAKMP/Oakley to negotiate the SA for communication between two computers. In addition, you must define security methods to specify how one computer talks to another. To define these methods, you choose algorithms depending on your security requirements. For example, you might apply 3DES for ESP confidentiality and HMAC SHA for ESP integrity to achieve a high level of security. You can define several security methods and put them in an ordered list according to your preference. When you do so, the negotiation protocol will move down the list and choose the first security method that appears on the negotiation policies of each of two communicating computers. Screen 2 shows a security method list in which 3DES and HMAC SHA has the highest priority.

Authentication method. NT 5.0 IPSec uses one of three authentication methods for machine-to-machine authentication. (Because IPSec is a network-layer protocol, it doesn't authenticate users.) The three IPSec authentication methods are Kerberos, certificate, and preshared key. Two communicating machines must use the same authentication method to validate each other. NT 5.0 uses Kerberos as its default authentication mechanism. In Kerberos authentication, a Kerberos server maintains the secret keys of all NT computers and users in its domain. When one machine needs to authenticate another machine in the domain, the first machine uses the Kerberos server for validation. This authentication method works well for a single administrative network, such as your NT intranet. However, if your NT computer must authenticate a computer in an external network, you can use the certificate method. In the certificate method, a certificate identifies a machine. Two machines in separate domains or networks will trust each other if their certificates are signed by a Certificate Authority (CA) that they both trust. Finally, the preshared key authentication method requires you to create a key or password string in your negotiation policy. If two machines share this key, they will trust each other. The preshared key method is not secure, however, because discovering a key shared by two systems or people is easier for a third party than discovering a private key that only one system or person holds. Shared keys must transfer from one party to another over a secure channel. If the channel is not secure, an intruder can steal a shared key.

IPSec tunnel setting. You might need to place an IPSec tunnel server between two machines to set up an IPSec tunnel. You would then use the tunnel server as a tunnel endpoint in the negotiation policy. For example, to let a remote computer access your corporate network through the public Internet, you can create a security policy specifying that the remote computer must traverse the corporate tunnel server from the remote computer's public Internet address to the corporate network address. The IPSec tunnel setting in a negotiation policy contains a tunnel server's IP address or DNS name.

Connection type. NT 5.0 IPSec supports two network connection types: LAN and dial-up. Through the connection type setting, you can specify whether a security policy is for LAN, dial-up, or both. If a machine has a LAN-based security policy, it can talk to another machine only through a LAN adapter--not through a dial-up modem. By adopting a LAN-based connection type, you can prevent users on remote computers from dialing up your network without using the appropriate dial-up policy.

Creating a Security Policy
NT 5.0 IPSec provides several preconfigured security policies: for example, the lockdown policy lets one machine use IPSec to talk to any other machine. However, you might need to create your own policies according to your company's security requirements. Let's walk through the process of creating a security policy for a company's human resources (HR) server. Suppose that the HR server contains confidential employee and company information and lets only members of the HR department and two managers from two other departments access HR information. The HR server and all HR workstations are in a dedicated subnet 193.1.1.0. The two departmental managers' workstation IP addresses are 193.1.2.25 and 193.1.3.25. The two subnet addresses for the departmental managers' workstations are 193.1.2.0 and 193.1.3.0, respectively. The HR department can access the server using any protocol, whereas the two managers can use only a Web browser to search the employee database in the HR server. Data integrity is the minimum requirement to secure the data transferred between the server and the workstations.

Figure 1 shows the HR server's security policy. To create this policy, you first generate the IP filter list. The list in Figure 1 contains three filters. Filter 1 specifies that the server will talk to any machines in the HR subnet 193.1.1.0 using any protocol. Filter 2 and Filter 3 define that the server will talk to the two manager workstations only by TCP at port 80 for HTTP access. The negotiation policy lists two security methods the HR server will use: The server prefers DES-CBC for ESP confidentiality, HMAC MD5 for ESP integrity, and HMAC MD5 for AH only. The server policy authentication method is Kerberos. There is no tunnel setting for intranet traffic. Finally, the HR server's policy allows only the LAN connection used for IPSec.

Using similar configurations, you can define policies for all HR departmental workstations, the two manager workstations, and the rest of the company. Defining security policies is an important step in implementing IPSec. Write down your security policies and save them with your network documentation.

Deploying Security Policies in NT 5.0
After you define IP security policies for your NT 5.0 network, you can set up the policies in either Active Directory (AD) or individual computers, through NT 5.0's IP Security Manager. AD is a better place to store policies, because you can manage them in a central repository within AD. Then, from AD you can assign a security policy to an entire organization, an organizational unit, or a computer. You can also manually configure a local computer's IPSec option in the TCP/IP property to use a specific security policy. (However, one computer can use only one security policy at a time.)

When the computer boots up, the IPSec policy agent in the computer will copy the assigned security policy from AD to the local Registry. Screen 3 shows the IPSec option, in which the custom policy is HR Server Security Policy. NT 5.0's IP Security Manager lets you dynamically change the security policy rule and settings. If you want to renew the security policy dynamically, the computer will check AD and automatically download the modified policy within the interval time parameter that you define. If you change the security policy often, you need to increase the frequency of the interval: for example, to once every hour.

When an application in a local computer needs to interact with data or applications in another computer, the IPSec driver built into the IP stack will first examine the IP filter list in the local computer's security policy to determine whether the local computer and application are allowed to communicate with the second computer. If a filter within the first computer's security policy matches a filter within the second computer's policy, the IPSec driver will call up ISAKMP/Oakley to negotiate the SA between the two computers, based on their polices, including machine authentication, security algorithm selection, and session key generation and exchange. ISAKMP/Oakley uses UDP at port 500 for SA negotiation. When ISAKMP/Oakley successfully establishes the SA, the two computers can exchange application data in a secure channel. This procedure is completely transparent to users and applications.

IPSec Is the Key
Now you've learned about IPSec in NT 5.0. With NT 5.0 IPSec, you can build a secure NT network and protect it from attackers without having to modify legacy applications and upgrade network equipment--AD will let you easily manage security polices in your enterprise from a central location. NT 5.0 IPSec is a big leap forward from the old IP protocol in NT 4.0. Together with Microsoft, about 40 networking and security vendors are hard at work implementing IPSec in their products so you can leverage IPSec in your heterogeneous network. In addition to heightening your intranet security, IPSec will let you safely conduct e-commerce over the Internet. IPSec is the key to your business success.