A client company called me because users were having difficulty receiving mail from the Internet and sending mail to a remote Microsoft Exchange server. This client runs a combination of Exchange Server 2003 and Exchange 2000 Server platforms. The company has multiple locations with an Exchange server in each location. It has a front-end server to handle incoming mail and Microsoft Outlook Web Access (OWA). I had recently installed a Research In Motion (RIM) Blackberry Enterprise Server (BES) on the front-end server and feared that doing so had caused Internet Mail to stop working. I discovered later that BES didn't break the mail. In the weeks prior to the BES installation, the company needed to reboot the server to get mail flowing again, but this time the mail didn't start working after the reboot.
I used the Exchange System Manager (ESM) to examine the mail queues. The queue going from the front-end server to the main Exchange server was in a retry state. The front-end server and the client's Los Angeles Exchange Server are in the same routing group. I reviewed the Los Angeles server queues and one of the routing group connector queues was in a retry state to a remote Chicago Exchange server in a different routing group. I tried to force the connection on the front-end server to the Los Angeles Exchange server, but the queue would return to a retry state after a few minutes.
Whenever mail delivery problems occur between two Exchange Servers, it’s a good idea to turn up the diagnostic logging on servers that have problems communicating. I turned up the logging level to maximum on both the front end and main Exchange servers for the MSExchangeDSAccess and MSExchangeTransport services. To turn on diagnostic logging, right click the server, select Properties, click the Diagnostics Logging tab, then select MSExchangeDSAccess in the Services pane. Select all the categories and click the maximum radio button. Repeat these steps for the MSExchangeTransport Service. Messages in the Application Event Viewer will usually give you a clue about why the mail was backing up in the queues. Unfortunately, in this case I didn't find any Event Viewer messages explaining why mail delivery was failing.
On a hunch, I ran a full virus scan on both servers. The virus scan revealed several Netsky and Klez variants in the D:\program files\exchsrvr\mailroot\vsi 1\queue folder. Both of these servers are running Symantec Norton Antivirus Corporate (NAVC) 9.0 and Symantec Mail Security 4.5. One disturbing point is that I received an Access denied error message when I tried to delete or quarantine some of the viruses. Every time I scanned the queue folder, additional viruses showed up. Symantec recommends that you exclude real-time scans of the D:\program files\exchsrvr\mailroot folder when both NAVC and Symantec Mail Security are running on the same server. For more information about this configuration, open the following link http://service1.symantec.com/SUPPORT/ent-security.nsf/529c2f9adcf33a1088256e22005026f1/bcdac2b5d7f7249088256e9e00827e26?OpenDocument?=Symantec%20AntiVirus%20Corporate%20Edition&ver=9.0&src=ent&pcode=sav_ce&dtype=corp&svy=&prev=&miniver=savce_9.0.
I had already configured NAVC to exclude real-time scanning of the mailroot folder and the other suggested folders on all Exchange servers. Because viruses continued to show up in the queue folder, I assumed that the server was still receiving virus-infected messages. I stopped all Exchange services so mail would stop flowing into the queue folder and ran a manual scan using NAVC. This time, no new viruses appeared, but I still received the Access denied error message when I tried to quarantine or delete the viruses. Evidently, some process was conflicting with the process of deleting the viruses. I tried stopping all Symantec virus-scanning engines, but I still couldn't delete the viruses. I looked at the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run registry subkey and noticed that Symantec was loading a file called ccApp.exe. I used Windows Task Manager to end the ccApp.exe process and was finally able to delete the viruses in the queue folder. I scanned the queue folder again and it was clean. When I rebooted the server, mail started flowing again to all the sites. I had to perform this procedure on the front-end server and Los Angeles Exchange server to get mail working again. I contacted Symantec for help and received the following advice. When you install NAVC on any server running SMTP/Port 25 traffic, don't install the email scanning components that are optional with NAVC. To check whether you have the email scanning components installed with NAVC, start NAVC and click on the Configure menu. If anything besides the File System Real Time protection option appears you should reinstall NAVC without the email scanning components. Evidently the email scanning components of NAVC are not too robust, because they can break under a significant mail load. This makes sense because this problem occurs on larger clients that have heavy mail flow and multiple servers. With NAVC 9.0 you can modify the existing installation, but with NAVC 8.6 you must uninstall and reinstall NAVC to remove the email scanning components. After I made this change to NAVC mail flow has remained stable.
Sysinternals TCPView is a useful utility when you suspect that a computer has been hacked. You can download the tool from http://www.sysinternals.com/ntw2k/source/tcpview.shtml. TCPView lists all the UDP and TCP endpoints on your system. It also lists the IP addresses or names of the remote connections, the port numbers, and the connection state. You can use Sysinternals' Autoruns utility to identify any suspicious programs that are set to automatically load, then use TCPView to see whether the program has created a UDP or TCP connection to a remote host. Be aware that hackers will often spoof the IP address or compromise several machines to perform their hacks. Using TCPView to review the type of traffic, endpoints, and connection state is the first step to tracking down the source of the hack.