My consulting company recently received a call from a client company that was having problems with backup failures and poor server performance when sending and receiving email. When we arrived at the client site, we found the problem was more serious than a failed tape drive and slow server. I logged on to the server and noticed it was running extremely slow. The server showed a lot of drive activity and high CPU usage. I pressed Ctrl+Alt+Delete to open Windows Task Manager and sorted the processes by CPU usage. I noted that store.exe was taking up most of the CPU cycles. Microsoft Exchange 2000 Server and Windows 2000 Server were running on this machine. Could the problem be a corrupted Exchange Store? Large email volume? The organization wasn't a heavy email user and had only 15 users connected to the server.

I started the Exchange System Manager (ESM), which took awhile to load and looked at the configuration. I opened Administrative Groups, Admin_Group_Name, Servers, Server_Name, Protocols, Smtp, Default SMTP Virtual Server, Current Sessions and noticed six connections had been connected to the SMTP virtual server for more than 5 minutes. This finding was the first clue that something was very wrong on the server. Typically, an Exchange session lasts for a few seconds at most, unless the connection is sending or receiving an email message with a large attachment. I looked at the queues on the default SMTP virtual server and noticed that more than 50 queues were in various states of sending email or waiting for a retry. Obviously, someone was using the company's mail server as a relay. But how? As you know, Exchange 2000 isn't an open relay by default. The server was current with the latest versions (Win2K Service Pack 4—SP4—and Exchange SP3) and had the latest critical security updates. I reviewed the relay settings and used the open relay test from Open Relay DataBase (ORDB.org—http://www.ordb.org) to ensure that the relay was closed.

Whenever I tried to clear a connection to the default SMTP virtual server, the connection would reappear, usually with a different domain name but from the same IP scheme. I traced the IP addresses to a block allocated by an ISP in China. After concluding that the server was not an open relay, I decided that someone was probably authenticating to the server and sending spam. The backup was failing because it was trying to back up all the mail the spammer was attempting to send.

I opened the Microsoft Management Console (MMC) Active Directory Users and Computers snap-in and, with the client’s help, removed all the invalid users. I noticed that some users in the Administrators Group didn’t belong in the group, so I removed the unauthorized users. I first suspected that a former employee had “sold” the password to a spammer so that the spammer could use the mail server to relay messages (more about this later). I thought some malicious activity might be occurring on the server, so I checked the following run subkeys in the registry to determine whether any hacking programs were loaded:
· HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
· HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run
· HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersionPolicies\Explorer\Run

The subkeys turned out to be clean. I also ran a virus scan on the server, and the server was clean. From the surface, the server looked free from any hacking tools. In Part 2, I’ll discuss how I closed the hole and how to prevent this type of problem from recurring.

Tip Some ISPs now check for a domain's DNS record in an attempt to reduce the amount of spam received by their mail servers. If you’re making a change to or establishing new mail service for a domain, make sure that the ISP adds a DNS entry for the domain. A common symptom of not having the correct DNS records is the inability to send mail to certain domain names. Most DNS providers will automatically do add a DNS entry for the domain when establishing a mail exchange (MX) record for the domain and when entering a reverse Pointer (PTR) record, in case the mail server performs a reverse lookup for the domain. By ensuring that you have an MX record and reverse record for your domain, you should be able to reduce the sending problems you have with specific domains.

Reader Comments

I have seen this same issue with 2 different customers this year. I don't know if passwords were compromised or not, but I was forced to change the Exchange server settings so that no relaying was allowed (authenticated or not) except from within the internal subnet that the Exchange server resided on. I enabled stronger password policies on those two clients and forced the users to change their passwords. Neither customer has had a problem since, but I am curious to see how these guys were able to relay in the first place as well. I traced one of the relayers to a Chinese company that had a San Francisco office (marketing agency for some generic type of viagra). I called them and they claimed no knowledge of what was going on and tried to refer me to China. If possible I would like to be able to safely re-enable authenticated relaying (makes life much easier on the mobile users) but I am very reluctant to do so until the cause of the attempt is determined. Any information you could provide would be extremely helpful as your situation sounds very similar to mine.

Sounds interesting and familiar

It is quite a interested read. But where is part II? and When?

Awsome article!! Now I can't wait to read Part 2. Bring it on!! (smile) Thank you for including the URL to the Open Relay DataBase.

It's a very good article ...Pls keep posting these kind of articles. We will not get these practical issues in books.

I have had similar problems and have performed the same steps as you have mentioned. I am looking forward to part 2 to find out your resolution to the problem.

Most excellent.

very good

Can't wait for part2, this is like a spy novel!

I have experienced the same problem, where, my exchg 2000/2003 server was used as a relay, only authenticated relay was allowed. I would like to find out how they did it. I had to not allow relaying to stop the hijacking.

So where is part two? I am finding part one refered to in various newsletters.

Great article. I'm a security novice, though I've been managing my own network for years. It's helpful to get a better sense for how some of this illegitimate traffic wends its way through compromised systems. I rely on a LinkSys router to bar non-responding messages, and antivirus and message filtering software on workstations. My sense is that these defenses are quite effective, so I'm a happy camper.

WE WANT PART 2! WE WANT PART 2! (Exclaimed as knife and fork are pounded loudly upon the bench spilling grog through the cracks)

I had the very same problem, even though I was extremely vigilant and paranoid about open relays (running a different mail server, not Exchange). Despite open relay checks coming out clean, I was noticing spam being sent by those good folks in China, using multiple machines to hit my server. My final and best solution turned out to be banning the whole IP subnets from those locations. I'm looking forward to part 2 of the article.

When will Part 2 be published.

Can't wait for part 2, please don't delay

This is at & up to as far as I have ever got with checking strange happenings. So what happened next?

very interesting

So what was the reason?

I know you want eyeballs to the site, but frankly those that have the misfortune of being saddled with Exchange need to know what to do about it now, not next week. Then again if they read this column, they probably already know.

I like this Article.

i hope part 2 is out soon. a firewall with logging would have traced the problem a lot sooner.

I'm curious... Was it the authentication?

This just happened to me on a non MS platform. I used a network packet sniffer on my server and recorded the transactions. Turns out my web server was the culprit. First, some background. My mail server requires authentication to send mail. It will only open relay from my server which it resides (localhost). My web daemon resides on the same server as my mail daemon. I have DSL and a static IP address with domain names assigned. I was doing some enhancements to my web server specific to proxying. The SPAMMER rat-bastards (SRB for short) found it within a day and a half. They used a GET method to test my system, and a POST method (cgi forms) to send the message through my web server, which connected to my mail server (which will only open relay for only the localhost server). It was about 40k SPAM emails in less than a week before I could catch it. Yikes... Insult to injury, the SRB's used my web server to probe all open mail relays within my ISP and had them join the fun. Many of these servers within my domain allowed the whole ISP's domain to open relay from their mail server. I believe this is a way around mail relaying using DHCP instead of static addressing. There were a bunch of them (They all showed up in my logs). The clue came from my web logs where I noticed an entry: [10/Nov/2003:22:31:02 -0800] "GET http://www.helllabs.com.ua:80 HTTP/1.0" (BTW: This web site probes other sites and sells lists of open proxy's for just this purpose. If you have a misconfigured web proxy, I bet you are on these SRB's list) From the packet sniffer, I was able to record a test: (I removed the IP address and replaced it with '#') POST http://##.##.##.##:25/ HTTP/1.1 Content-type: application/octet-stream Content-length: 470 Host: ##.##.##.## HELO ##.##.##.## MAIL FROM: RCPT TO: DATA Message-ID: <080076058054054046054048046049051052046050051052058052058056048@##.##.##.##> To: From:jackbran4@hotmail.com Subject: the one message you need Date: Tue, 25 Nov 2003 12:20:28 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit 054054046054048046049056055046050048 . QUIT QUIT Three things are necessary for this to work. HTML 1.1 standard requires an URL and a host name. In order for the POST method to work, a "Content-Length" of the exact amount of characters which the web server will take. Notice the URL is remapped to port 25 (http://##.##.##.##:25), and the rest of the message starting from HELO, is an articulated conversation with a mail server that doesn't require authentication (localhost). The above was the test probe used to determine if my web server will proxy. If "tupperhorse1@aol.com" receives the message, they know my web server is misconfigured, and the SPAM will start again. The person sending this request to me came from an unsuspecting web surfer that hosted a trojan script on their PC that sent me probes as they surfed the web. Problem with putting the probe's domain in the firewall, was they were from a big dial-up ISP, and I use my site for business. Can't very well lock out the whole domain. Couple of things you can use to check your system. First telnet to your web server and issue the command: >telnet your.ip.address 80 >GET http://www.google.com:80 HTTP/1.0 If you see google's web site, you are screwed... They probably got you. Check the other web servers within your domain for the same thing. You could check your web logs (You do have web logs don't you?) for probes in the form of: GET http://some.ip.address:80/ HTTP/1.1 The "http://some.ip.address" is the giveaway. Non-proxying URL requests come in the form of: GET /pathname/somepage.html HTTP/1.1 Next, check the open relay policy of your mail server. Look for web servers that would fit within that policy. If your system allows relay from your ISP's domain, you might want to rethink that. I know there are alot of web servers and mail servers out there that are misconfigured. Good luck finding and fixing them. In closing, if you received SPAM from my system, sorry...

good article and timely