Security tools that find holes before intruders attack

To secure an Internet-connected network, firewalls are a necessary component in your arsenal of tools. However, firewalls alone are no longer sufficient protection, because they're static devices that enforce a particular ruleset. This setup means that intruders can use valid, legal packets to formulate an attack on your network and compromise your security. You must use additional tools for complete protection, especially for Windows NT and the TCP/IP protocol. To spot potential problems before intruders exploit them from inside or outside your network, you can implement several security tools, such as port scanners, vulnerability tests, log analyzers, and intrusion detectors.

Port Scanners
TCP and UDP use port numbers to identify higher-layer services. Systems administrators use port scanners to determine what TCP/UDP services are available on a server. A cardinal rule of server security is to disable any service that the system isn't using because any open TCP/UDP service offers intruders a possible entry into your system. Thus, you can use a port scanner to ensure that only the desired TCP/UDP services are running.

Port numbers 0 through 1023 are well-known ports that systems administrators usually use for only system processes or for programs that privileged users are running. If attackers exploit a well-known port, they can potentially gain control of a server.

To compromise your network, attackers use several generic schemes to scan ports. The following examples focus on TCP scanning:

  • TCP connect() scanning is the most basic form of TCP scanning. An attacker's host issues a connect() system call to every interesting port on the target machine. If the port is listening, connect() will succeed; otherwise, the port is unreachable and the service is unavailable. This attack scheme is fast and doesn't require any special privileges; however, port scanners can easily detect and block this attack at the target system.
  • TCP SYN scanning attempts to set up a TCP virtual connection. Establishing a TCP virtual connection requires a three-way handshake, in which one host sends a TCP segment with the synchronize (SYN) flag set, the other host responds with a segment that has the acknowledge valid (ACK) and SYN flags set, and the first host responds with a segment that has only the ACK flag set. In SYN scanning, a querying host sends a SYN segment to every port. If the server responds with a SYN-ACK segment, the service is available; if the server responds with a reset (RST) segment, the service is unavailable. Port scanners can usually log these attempts at the target site.
  • TCP FIN scanning entails the attacker's querying host skipping the three-way handshake and sending a finish (FIN) segment to all interested TCP ports. In ordinary circumstances, sending a FIN segment closes an open TCP connection. However, if the port is open (i.e., listening or active), the system is supposed to ignore the FIN because there is no existing connection; if the port is closed (i.e., not listening or inactive), the system generates an RST segment. Thus, lack of a response identifies an active port to an attacker. This attack is a clever way around the problems of SYN scanning, and it's very hard to trace. Ironically, this method doesn't work well on most Windows systems because Microsoft's TCP implementation always sends a RST in response to a FIN.
  • FTP bounce attacks use the FTP PORT command and the upload capability at an FTP server behind the firewall. This attack lets a client connect with other systems that otherwise wouldn't be accessible. For example, an attacker creates a text file that contains the SMTP dialog of a spoofed email message and uploads the file to an FTP server behind a firewall. The attacker then issues an FTP PORT command to the server and provides the intended target system's IP address and the SMTP port number. Next, the attacker sends the file to the victim machine, so the file appears to come from the FTP server. Intruders can use this method to insert Trojan horse programs or viruses behind a firewall and might gain access to an FTP server and scan services behind a firewall. Microsoft's FTP server implementation is reportedly immune from this type of attack because it requires the IP address contained in the PORT command to match the original IP address provided on the FTP control channel (TCP port 21). In addition, by default, you can't use the PORT command to specify any privileged ports (i.e., port numbers less than 1024) except the FTP data channel (TCP port 20).
  • UDP Internet Control Message Protocol (ICMP) port unreachable scanning is one of the few UDP scans. UDP is a connectionless protocol, so it's harder to scan than TCP because UDP ports aren't required to respond to probes. Most implementations generate an ICMP port_unreachable error when a user (or intruder) sends a packet to a closed UDP port. Thus, a lack of a response indicates an active port.

You can perform port scans by using several tools that employ a combination of methods to detect attacks. You can run port scanning software from UNIX (e.g., System Administrator Tool for Analyzing Networks—SATAN, nmap) or NT systems. (Table 1, page 62, lists popular NT security monitoring tools.)

For example, I used WebTrends Security Analyzer to scan a group of four NT servers for TCP/UDP vulnerabilities. The Most Vulnerable Services Graph, which Screen 1 shows, reveals that four services are open to attack.

The Service Vulnerabilities table describes each risk in detail. For example, the FTP and HTTP services are vulnerabilities merely because they're running. The medium-risk vulnerability FTP service is susceptible because it has anonymous access enabled, and the HTTP service is high risk because attackers can exploit the $DATA hole in Microsoft's Internet Information Server (IIS). This vulnerability affects many IIS installations because an attacker can download the Active Server Pages (ASP) source by appending a ::$DATA string to the URL. This action can expose usernames and passwords that systems administrators have hard-coded within scripts (e.g., SQL and other administrator passwords).

After listing these vulnerabilities, many port scanners suggest possible fixes, such as shutting down the service or installing a hotfix. Although shutting off the service might not be an option, knowing that a vulnerability exists helps you balance the risk exposure with the benefit. Port scanners also remind systems administrators to turn off any service that they're not using.

Vulnerability Tests
Vulnerability testing is similar to port scanning, but you use vulnerability testing for specific OSs rather than TCP/IP communications protocols. Many security vulnerabilities are OS specific, so a good rule of secure management is to avoid advertising your servers' OSs. However, clues such as .asp file extensions, use of .htm rather than .html, and home pages named default.htm usually identify an IIS server to an attacker. In addition, many tools help attackers determine a system's OS, and to use these tools attackers need only the server's IP address.

Systems administrators are well advised to use software to examine their systems for OS vulnerabilities. For example, I used WebTrends Security Analyzer to scan four servers for OS vulnerabilities. The software generated a report that ranked the systems' weaknesses as high, medium, or low risk, as Screen 2 shows. Security Analyzer found that all the servers have vulnerabilities.

Knowing about these vulnerabilities is only half the battle. Most vulnerability testers will also tell you where you can get more information, such as pointers to Microsoft articles and links to hotfixes and service packs, and suggest corrective measures, such as instructions for updating the Registry to secure your systems. For example, Screen 3 shows a portion of the WebTrends Security Analyzer report that lists the suggested fixes for my systems' vulnerabilities.

New OS vulnerabilities, particularly NT weaknesses, appear all the time. So you can't run a vulnerability test once, fix the problems, and have a secure system. You need to use this tool frequently with an up-to-date database of OS vulnerabilities. In addition, many versions of port scanners and vulnerability testers can scan across a network—you can use them to test your network, and attackers can use them to probe your network for weaknesses.

Log Analyzers
A third rule of security management is to maintain system logs, especially for your servers and routers. Many sites diligently maintain logs and audit servers, but does anyone read the logs? Log files are usually large, and finding any serious trends in a multimegabyte text file is difficult. Log analyzers read log files and prepare a graphical or tabular representation of the data. Some NT log analyzer products run as standalone software, and others run in conjunction with NT's management capabilities. In addition, different log analyzers report different types of information, but most analyzers look at the following events:

  • Bandwidth utilization patterns
  • Bandwidth utilization top users
  • Incoming and outgoing protocol/service use (e.g., FTP, Web, Telnet)
  • Critical events and host addresses
  • Errors, warnings, and host addresses
  • VPN use statistics and external host addresses
  • Logon attempts and failures (and host addresses via remote access)
  • Top email senders' and recipients' host addresses (internal and external)
  • Top FTP download and upload host addresses
  • Top Telnet host addresses
  • Unsuccessful TCP connection setup requests
  • Packets and addresses blocked at the firewall and router

I used Computer Associates' SessionWall-3 to analyze my system logs. Screen 4 shows sample output as a graphical representation of an outgoing Web traffic log, including users, type of site visited, and number of visits. Screen 5 shows a display that CyberSafe's Log Analyst created to present information from the error log, such as failed logon and virus or Trojan horse events. This information is available in your log files, but it's easier to understand in graphical format.

Security Tools Resources
Internet Security With Windows NT
Author: Mark Joseph Edwards
Publisher: 29th Street Press
ISBN: 1882419626
Price: $49.95
600 pages

Maximum Security: A Hacker's Guide to Protecting Your Internet Site and Network, 2nd Edition
Author: Anonymous
Publisher: Sams
ISBN: 0672313413
Price: $49.99
829 pages

Internet Security: Professional Reference, 2nd Edition
Author: Derek Atkins et al.
Publisher: New Riders
ISBN: 156205760X
Price: $65.00
916 pages

Intrusion Detectors
Firewalls, port scanners, vulnerability tests, and log analyzers block unauthorized access, alert you to potential problems, and determine whether an attack has already occurred. Intrusion detectors tell you when unusual or suspicious activity is occurring in realtime. You can use intrusion detectors to identify the following common threats:
  • Denial of Service (DoS) attacks block access between a victim's network and the Internet by overloading system resources or usurping available bandwidth (e.g., WinNuke, TCP SYN flooding, ping flooding, Ping of Death, ICMP Bombing, Smurf, Teardrop, and Land).
  • Unauthorized access to restricted files or commands is an attempt by an attacker to read, write, or modify files or run commands for which the attacker doesn't have permission. This type of attack uses DNS, Finger, POP, and Sendmail buffer overflow; FTP hijacking; HTTP, IIS, or Microsoft Internet Explorer (IE) vulnerabilities; and password attacks on the UNIX root, NT administrator, or NetWare supervisor account.
  • Preattack probes are attempts by attackers to gain basic information about users, servers, or a network that they can use in subsequent break-in attempts (e.g., using tools such as SATAN or nmap, DNS zone transfers, traceroute or tracert, or SMTP name expansion—EXPN—and verify—VRFY—commands).
  • Suspicious activity can be any network traffic that takes place outside the expected, normal traffic patterns. Although this activity is often the result of error, these aberrant behaviors can identify undesired network activity; for example, IP unknown protocol events, use of IP routing, excessive FTP uploads, Trivial File Transfer Protocol (TFTP) traffic, NT connections between unrelated systems, access to server Registry files, excessive unsuccessful logon attempts to one or more account names, or attempts to delete, modify, deactivate, or flood log/audit files.
  • Hostile code insertion includes programs that can alter your system or network's behavior, such as virus, Trojan horse, and back-door programs; malicious applets; and packet sniffers.
  • Infrastructure attacks target the systems and facilities that make up the operational network (e.g., firewall modifications, router modifications, user account additions and modification, file permission and ACL modification, and DNS modification).

For more information about the previously listed attacks, go to Microsoft's Security Advisor page ( or CERT Coordination Center (

Intrusion detectors constantly monitor activity on your network and protected systems. If the software finds a behavioral pattern that matches a known attack signature, the intrusion detector sends a warning to the system manager via the software's interface, email, or pager. This interface lets a network administrator view realtime events that the software categorizes as high, medium, and low priority. Screen 6 shows the console from ISS's RealSecure intrusion detection software.

Intrusion detection software is becoming more popular as security incidents become more common and serious. As attacks grow in complexity and involve multiple sites, users would benefit if vendors' intrusion detection software could communicate across sites and share data about an ongoing attack. To address this challenge, the Internet Engineering Task Force (IETF) created the Intrusion Detection Exchange Format Working Group (idwg) to define data formats and protocols for sharing information between intrusion detection and response systems and network management systems. The IETF expects to release initial specifications by the end of 1999. (For more information about this development, go to idwg-charter.html.)

NT Security
Several port scanners, vulnerability testers, log analyzers, and intrusion detectors run on NT. These tools should be a part of your security toolkit and you should employ them regularly. (For more information about security tools, see "Security Tools Resources.") Although many NT security products are available, consider software that can detect and probe NT systems from non-NT platforms such as UNIX. Don't limit your thinking to just NT—the bad guys don't!