Protect Private Ports with IPSec

In "Close the Doors to Your Web Server," February 2002, InstantDoc ID 23573, I discussed the importance of disabling unneeded services to close down the related ports and prevent them from being potential targets for attack. But how do you protect ports related to the services that your Web server requires? I find it useful to divide the open ports on a Web server into two categories: ports that the general public accesses and ports that only computers under your control access. For example, the general public usually needs access to ports 80 (HTTP) and 443 (HTTP over Secure Sockets Layer—HTTPS). To protect against attacks that target public ports, you must secure both the internal application and the OS.

However, you probably have other ports open on your server—to support such activities as content maintenance, administration, and updates from databases in your network or a business partner's network. For example, if you administer an IIS Web server from a remote computer through that computer's Web browser, a much more limited set of computers should have access to the port assigned to IIS's Administration Web Site. Similarly, if you use Windows 2000 Server Terminal Services, a limited set of computers will have access to port 3389. If you have an FTP server enabled on your Web server to maintain content or support file transfers to and from a database, a similarly limited set of computers will have access to ports 20 and 21. You might also want to protect access to the Remote Registry (port 135) and Telnet (port 23) services.

Not only are these private ports vulnerable doorways into your Web server, but securing and keeping all the services that relate to these ports up-to-date with software updates and service packs is difficult. For these private ports, Win2K's IP Security (IPSec) functionality can come to the rescue. You can easily create an IPSec policy to require that an authenticated and encrypted IPSec connection be established before any computer can connect to these more private ports.

How an IPSec Policy Can Help
An IPSec policy lets you use strong authentication to limit which computers can connect to a port. An IPSec policy also ensures that IPSec encrypts all subsequent communications and performs integrity checking. These IPSec operations are transparent to the services and client applications that use the port; however, you're effectively wrapping a service such as Terminal Services in a new level of security. You've added protection in case you inadvertently configure Terminal Services insecurely or an intruder discovers a vulnerability in your version of Terminal Services. You're even protected against Denial of Service (DoS) attacks related to Terminal Services or any other private service you protect with IPSec.

You get the additional protection because IPSec prevents an untrusted computer from connecting to the relevant port in the first place. To connect to an IPSec-protected port, the computer must successfully authenticate as a trusted computer by using one of three authentication methods (Kerberos, preshared secret key, or certificate-based) you specify. Without IPSec, you can limit connections from untrusted computers only by filtering on a source IP address. (Source filtering is vulnerable to spoofing, whereas IPSec's filtering isn't. Spoofing involves an attacker sending packets in which the source address has been substituted with a fake address to fool the target computer into thinking it's communicating with a trusted computer. ) In addition to authenticating each packet, IPSec also performs integrity checking through which it detects modified packets and rejects them, and session hijacking.

The only requirement for using IPSec to protect your private ports is that the computers that access these ports must support IPSec. You can enable IPSec support more easily between computers that run Windows XP and Win2K. For purposes of this article, I assume that the computers accessing private ports on your Web server run Win2K. You need to configure complementary IPSec policies on these computers (i.e., if the server requires a certain level of security and authentication, the client must be configured to acquiesce) to let them communicate with your Web server. In this article, I show you how to plan your IPSec implementation; in a subsequent article, I'll show you how to configure IPSec for a sample Web server.

Assigning an IPSec Policy
For each Win2K computer, you can assign (i.e., activate) one IPSec policy. A policy is a set of rules, as the sample policy in Figure 1 shows. A rule catches packets that match specified filter criteria, then applies one of three actions to those packets: block, permit, or negotiate security. The three main parts of a rule are its IP filter list, its designated filter actions, and its authentication methods.

First, the filter list contains filters that specify any combination of protocols (e.g., TCP, UDP), source and destination IP addresses, and port numbers in much the same way that firewalls filter for specified elements. After the filter list catches a packet, Win2K moves to the second part of the rule and applies the filter action. You can specify whether Win2K permits the packet, blocks the packet, or negotiates a secure IPSec connection. Negotiation lets you require that the connection be subject to IPSec's Authentication Header (AH) mode or Encapsulating Security Payload (ESP) mode. In AH mode connections, IPSec authenticates and performs integrity checking for each packet. In ESP mode connections, IPSec also encrypts everything in the packet except the destination address.