Peek into packets and spot traffic tie-ups with help from Microsoft's network analyzer
Network Monitor is a component of the Windows Server OSs and Microsoft Systems Management Server (SMS) that lets you monitor network traffic as it crosses the wire. By using Network Monitor, you can monitor network traffic in real time or capture and store packets for later analysis. You can use the information that Network Monitor captures to troubleshoot problems on LANs, WANs, and virtually any device that uses TCP/IP to communicate. Network Monitor has three primary uses:
- Troubleshooting network connectivity. This is the number-one reason to use Network Monitor. If you have two machines that have problems communicating with each other, you can use Network Monitor's Network Trace feature to help determine the problem's exact cause. You can also use Network Monitor to view each TCP/IP packet that travels between the two devices and the information contained within each packet.
- Assessing network performance. Network Monitor gives you a clear picture of current network utilization. If you suspect that you have a network performance bottleneck, you can use the information that Network Monitor provides—such as detailed network-utilization statistics and information about the network traffic source—to find the bottleneck. Although you typically won't use Network Monitor to initially identify a problem as network communicationsrelated, it's a great second-level troubleshooting tool that can help you further pinpoint a problem and displays much more detail than Performance Monitor does.
- Troubleshooting beaconing hardware devices. Before switched networks existed, you could use Network Monitor to track down problems with hardware devices on a network. You can still use Network Monitor to track fragmented or damaged packets sent out by faulty equipment, but to do so you'll probably need the full version of Network Monitor, which supports remote agents and the capture of packets on a network segment even when the traffic isn't directed to the machine that's running Network Monitor. (For more information about the two versions of Network Monitor, see the sidebar "Network Monitor Versions.") If you have a managed switch, you can use a combination of the managed-switch statistics and Network Monitor to obtain as clear a picture of the problem as possible when diagnosing faulty network hardware.
Installing Network Monitor
To use Network Monitor, you must have a NIC that supports promiscuous mode installed in the server that's running SMS or Network Monitor. (Most NICs support promiscuous mode.) Network Monitor isn't installed by default unless you explicitly selected it when you installed Windows Server 2003 or Windows 2000 Server. To install the version of Network Monitor that's included in Windows 2003 or Win2K Server, perform these steps:
- Open Control Panel (click Start, highlight Settings, and click Control Panel).
- Double-click Add or Remove Programs.
- Click Add/Remove Windows Components.
- Click Management and Monitoring Tools, then click Details.
- Select the Network Monitor Tools check box and click OK.
Starting Network Monitor
After you've installed Network Monitor, you're ready to start the utility. Click Start, Programs, Administrative Tools, Network Monitor. (Alternatively, you can run Network Monitor from the command line or use a batch file to automate packet captures.) You'll see the initial Network Monitor screen. To start capturing packets, click the Capture button. After Network Monitor starts capturing packets, the Network Monitor window will look similar to the window in Figure 1. As you can see, Network Monitor's main window consists of four panes that display different types of information.
Network utilization bar graphs. The first pane (pane 1, red frame) contains a bar graph that displays traffic statistics on your server. The first bar—% Network Utilization—is the most important one. If your server is on a shared segment with other computers and your network utilization exceeds roughly 35 percent, the server could have a serious network bottleneck. Ethernet uses the Carrier Sensing Multiple Access with Collision (CSMA/CD) protocol, which detects collisions. On a nonswitched Ethernet network, network utilization above 35 percent generates numerous collisions, which dramatically decrease throughput. If you're experiencing high network utilization, consider installing an Ethernet switch to increase throughput.
If your server is connected to a dedicated switch port, network utilization can go much higher without producing network delays. However, if your network utilization averages above 80 percent, consider installing a NIC with dual ports or upgrading the backbone to Gigabit Ethernet or 10 Gigabit Ethernet. If you have many broadcasts or multicasts, or both, per second (i.e., more than 50), you could have a beaconing NIC or just many computers that issue broadcasts. It's a good idea to gather network traffic statistics before you have a problem, so that you have a baseline by which to compare your current network traffic with historical network traffic patterns.
Network connections. The second pane (pane 2, blue frame) displays a list of devices with which the server is communicating. The names in the Network Address 1 column are either the names of Network Monitorsupported NICs that are in use on your network or unsupported NICs' Media Access Control (MAC) addresses. (To display a list of NICs on your network, select Options, Show Vendor Names.) The 1->2 column displays the number of packets sent to the device in Network Address 2, and the 1
Network statistics. The third pane (pane 3, green frame) displays statistics about the current network packet trap. If you plan to capture packets for longer than 1 minute, you might have to increase the capture-buffer size, otherwise you'll start to lose packets in the capture buffer. The default capture-buffer size is only 1MB, which can fill up almost instantly on a busy network. When the buffer is full, the oldest packets are discarded and replaced with new packets. To modify the buffer size, select Capture, Buffer Settings and change the buffer-size setting. Don't set a buffer size that's greater than the amount of available physical memory, or you might drop frames because of page-file swapping.
Detailed network packet information. The fourth pane (pane 4, yellow frame) displays detailed information about the number and type of frames sent to or received from each device. If an unusually large number of bytes are sent to or received from a particular device, the device might be faulty or simply transferring a large amount of data across the network.
You can set a capture filter by selecting Capture, Filter, then setting parameters for the capture filter. A filter lets you trap packets according to specific protocols, network addresses, or text-pattern matches. It also reduces the size of the capture buffer, which can be helpful on busy networks. If you're new to Network Monitor, I suggest that you don't use the capture filter until you're very familiar with the type of traffic on your network. It's easy to accidentally filter out the exact data you're looking for by setting a capture filter that's too restrictive. If you want to filter out traffic, I suggest using the display filter (which I explain a bit later) after you've completed your network capture to hide any unwanted packets.
Displaying Captured Packets
After you've captured the amount of data that you want, click the stop and view icon (circled in red in Figure 2) to stop and view the packet capture. Figure 2 shows the results of a packet capture. The capture summary displays several columns, which Table 1 defines.
Double-clicking a frame lets you view detailed information for that specific packet, as Figure 3, shows. The top pane displays a summary of captured packets. The middle pane displays detailed information about the packet that's highlighted in the top pane. The bottom pane displays the raw packet information in hexadecimal format. When you click the detailed information in the middle pane, the bottom pane highlights where the information is coming from in the packet. In the middle pane, you can click any plus (+) sign to display additional detail. The middle pane consists of the following information:
- Base frame properties—contains general information about the packet that Network Monitor tracks and that isn't contained in the captured packet. The information includes when the packet was captured, time interval since the previously captured frame, frame number, and frame length.
- Frame header information—contains the packet's destination and source MAC addresses, routing information, and the number of bytes that remain in the packet.
- IP header information—contains information about the version of IP in use (typically IP version 4—IPv4), header length, type of service, packet length, time to live—TTL (i.e., the number of router hops a packet can take before it's discarded), and source and destination IP addresses.
- TCP/UDP/Internet Control Message Protocol (ICMP) header information—contains information about the source and destination ports, sequence number, acknowledgment number, data offset, TCP flag information, window, checksum, and number of bytes remaining in the packet. The information in this section will vary depending on the protocol type.
- Data section—contains actual data. If the packet contains a higher-level protocol such as DNS or HTTP, Network Monitor will display additional information in the packet's data section.
Setting Display Filters
After you capture packets, you can use Network Monitor's display filter to display only packets that meet certain criteria. To set a filter, select Display, Filter and enter any necessary conditions. You can filter by address, protocol, or protocol property. For example, to set a filter that displays only undocumented header packets for HTTP packets, follow these steps:
- Select Display, Filter.
- Click the Expression box.
- Click the Property tab. In the Protocol:Property window, scroll down and double-click HTTP.
- Click Undocumented Header.
- In the Relation window, select exists and click OK twice.
TCP/IP Session Basics
The "lite" version of Network Monitor captures broadcast packets and network traffic that's sent to or received from the server on which Network Monitor is installed. If you've ever examined your firewall log, you'll see that Network Monitor captures similar information but in much greater detail. A Network Monitor packet capture can be a little intimidating the first time you inspect one. (The first time I reviewed a packet capture, my initial thought was, "What the heck am I looking at?") To ease the shock of interpreting a packet capture, it's helpful to know what to look for and what you're looking at. But before we learn how to read a packet capture, a basic understanding of a TCP/IP session is in order. At a very basic level, a TCP/IP session includes the following components:
- Establish a session—three-way handshake. A TCP/IP session begins with a handshake. The computer that requests the session sends a synchronize (SYN) packet to the target computer. The target computer responds with an acknowledgment (ACK) packet and sets the data window size. Then the computer that originally requested the session sends an ACK packet to the target computer to acknowledge the data window size.
- Data transfer. During the session, data is transferred between two computers, with the receiving computer sending an ACK packet with approximately every other packet it receives from the sender. Under typical circumstances, most packets are either ACK or push (PSH) packets. During the data-transfer session, the number of packets that can be sent without requiring an ACK packet might be modified according to the amount of network traffic and buffer space on the receiving computer. This modification is known as a "sliding window" because the amount of data transferred can "slide" before requiring an ACK packet.
- Close session—modified three-way handshake. During a graceful close, the sender (i.e., the computer that requests the session close) sends a finish (FIN) packet to indicate that the data transfer is complete. The receiver sends an ACK to the sender to acknowledge the receipt of the FIN packet, then sends a FIN packet back to the sender. The sender then sends an ACK packet back to the receiver. A session close can also be ungraceful. In this scenario, the sender transmits a packet to the receiver that the receiver doesn't acknowledge. The sender keeps resending the packet to the receiver until the maximum retry value is reached, at which time the session is aborted.
Reading Packet Captures
As you know, network traffic is broken up into packets before it's sent across the network. Most packets on your network are typically TCP; however, you can also have other types of packets, such as DNS, ICMP, NetBIOS over TCPIP (NBT), and UDP. Each packet contains a TCP flag in the packet's header section as well as other flags, which are listed below in the order in which they appear in the header.
- U (URG)—urgent; data is sent out of sequence.
- A (ACK)—acknowledgment that lets the sender know the packet was received. The packet receiver sends an ACK packet that includes the sender's sequence number plus the length of the data packet to verify that the packet information was properly received.
- P (PSH)—forces data delivery before the buffer is full. This flag is used primarily for interactive traffic.
- R (RST)—reset; the session was terminated abnormally. The presence of many packets with the RST flag could indicate a problem on your network.
- S (SYN)—used to start a session and agree on an initial sequence number; often used for Denial of Service (DoS) attacks. Hackers can attempt to initialize numerous sessions from an invalid IP address, thereby causing the server to respond with an ACK packet that's destined for the invalid IP address. Most firewalls implement countermeasures to guard against this type of attack.
- F (FIN)—graceful end to a TCP session. A FIN packet indicates that the sender is finished sending data.
Flags to watch. Most networks should have traffic that consists mostly of ACK and PSH packets. Unless the server is constantly establishing new sessions, you should see relatively few SYN packets. Servers that experience a large number of SYN packets could be under some type of DoS attack (aka a SYN flood). The number of FIN packets should be roughly equal to the number of SYN packets. You should see few RST packets. A high number of RST packets is a possible symptom of an intermittent network connection, beaconing NIC, high network traffic, faulty network switch or hub, or another type of hardware failure.
Port numbers. As you know, most TCP/IP packets identify a source and a destination port. When you read the packet information, make a note of the source and destination ports on the server. You should also become familiar with well-known ports and the ports that Windows uses. If you notice an unfamiliar port in a capture, it could be a symptom of hacking activity. Table 2 lists some commonly used ports.
After you run a packet capture, you can use the display filter to filter the packet capture for a specific protocol. For example, if you're troubleshooting an HTTP connection problem, use Network Monitor to capture the network packets, then set a filter only on HTTP packets (port 80) to troubleshoot specific HTTP communication problems.
Retransmissions. You can use Network Monitor to troubleshoot connectivity problems. Look for reset, or retransmission, packets (i.e., packets that contain the R flag in the packet header) that have increasing timeout lengths, with static sequence numbers in multiple packets. Retransmissions occur when a transmission between two computers is interrupted for any reason—for example, because of failed hardware or WAN delays. When a retransmission occurs, the timeout value is doubled until the retransmission is attempted 5 times (the default value). Figure 4 shows a capture that occurred when a network cable on the client was disconnected during an FTP file transfer. Frame numbers 834992 (left column in the top pane) have the same sequence value in each packet, which indicates a connectivity problem. Frames 11721385 also share the same sequence number. With normal FTP traffic, the sequence number in each packet should increase, not stay the same. Notice that it took a total of 10 packets to abort the FTP transmission—5 retransmission attempts on the FTP transfer itself, and 5 retransmission attempts to close the connection. Look at the timestamp of 77.012668 in frame 834. The time difference between packet 834 and 845 (77.668893 - 77.012668) is 0.656225. The time difference between packets 845 and 868 is 1.312449, which is twice as long as 0.656225. For each successive packet, the time values double until the maximum retry value (in this case, 5) is reached.
On a slow WAN link or other line with a large variance in connection speed, you might have to increase the maximum retry number. To change this value, on the server running Network Monitor, navigate to the registry subkey HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters and add the DWORD entry TcpMaxDataRetransmissions. Set the value to decimal and change the value to a number larger than 5.
A packet trace can tell you the "best" value for your environment. For example, if you're experiencing a timeout problem caused by multiple retransmissions, increase TcpMaxDataRetransmissions to a fairly high value (e.g., 20) and see how many connections can recover from retransmission packets. If you notice that most FTP sessions can recover within, say, 8 packet retransmissions, you can set TcpMaxDataRetransmissions on your server to 9. This value lets the server time out if the WAN is down but attempt a reconnection with the client if the connection is still alive.
The timeout value for any session is initially set to 3 seconds, then is adjusted based on the connection speed. Our sample connection was on a LAN, so the first connection retry time was relatively short—0.65 seconds. On a slow WAN link, increasing the maximum retransmissions can cause the server to take a long time before it times out. For a slow WAN connection that starts with an initial retry value of 5 seconds, how long would it take for a connection to time out, assuming that the maximum-retransmissions value was 6? The answer is 315 seconds (5 + 10 + 20 + 40 + 80 + 160), or more than 5 minutes.
Using Network Monitor to Head Off an AUTH Attack
One practical use for Network Monitor is to obtain detailed information about the packets traveling to and from your server when you suspect the server is under attack. For example, you might be familiar with the type of attack by which a spammer bombards an Exchange server with SMTP AUTH commands until the spammer successfully logs on to the server. By default, Microsoft Exchange Server 2003 and Microsoft Exchange 2000 Server let a mail server relay messages if the sender can authenticate with a valid username and password. A spammer obtains a valid user ID and password by performing a brute-force attack against your mail server or launching some type of attack against your network. You can turn on Exchange's Diagnostics Logging and set the maximum value for MSExchangeTransport Categories, which will show you the user ID that was used to authenticate to the server. However, the Microsoft Management Console (MMC) Event Viewer snap-in doesn't usually display the spammer's IP address. You can obtain the address by performing a packet trap, as follows:
- Install Network Monitor on the Exchange server and begin a packet capture. (Of course, you must wait until the spammer authenticates to your server to obtain the spammer's IP address.) You might want to increase the capture-buffer size to make sure you don't lose any captured packet information. After the spammer authenticates to the server, you can stop the packet capture.
- Set a filter to TCPDestination Port 25, as Figure 5 shows. To do so, select Display, Filter and highlight the line Protocol
- Find the Auth Login command. Examine the data in each SMTP packet until you reach a packet that contains Auth Login. In the top window, the Src Other Addr column displays the spammer's IP address. Although the IP address might be spoofed, you can at least block port 25 traffic that comes from this address to prevent the spammer from using it in the future. Better yet, disable Basic and Integrated Windows Authentication on any outfacing Exchange server to prevent users from authenticating to the mail server when sending mail.
- Find the username. In case you're wondering, the next command string in the TCP data field should have the username and password that was used to authenticate to the server. However, these values are Base64 encoded, so you'll need to use a Base64 coder/decoder to decode them. Many Base64 coders/decoders are available on the Web—such as the one at http://www.dillfrog.com/tools/base-64_encode. (For practice, try using the Dillfrog decoder to decode the user ID c3BhbW1lcg
Armed for Network Troubleshooting
Network Monitor is a handy network troubleshooting tool, but it requires some training and skill to obtain the greatest benefit. Become familiar with Network Monitor and the traffic on your network before you have an emergency, so that you can establish some network baselines and not have to fight a learning curve under stressful conditions. Network Monitor and other third-party network sniffers require expertise to quickly find and resolve problems. Get up to speed now to make the most of Network Monitor's capabilities.
Answers to decoding examples:
| MICROSOFT ARTICLES |
"How to Automate Network Captures with Network Monitor"
Dillfrog Base64 Encoder