This batch-file script uses a whitelist to abolish spam while allowing IMF blacklisting
I wanted a way to create a whitelist that we could use with Exchange Intelligent Message Filter (IMF) on Exchange Server 2003 Service Pack 2 (SP2) without also losing my ability to use a blacklist. But if you convert the blacklist to a whitelist, you lose your blacklist functionality. To use the blacklist as a whitelist, the only method I could find through an Internet search was a solution (see http://forums.msexchange.org/m_1800395576/printable.htm) that causes items to be moved into the pickup folder where they're rerouted for delivery, whereas with traditional blacklisting, a piece of blacklisted mail is moved out of the delivery queue. Here are the steps for this solution:
- In Exchange System Manager (ESM) under Global Settings, Message Delivery, Properties, Sender Filtering, add the addresses you want to whitelist.
- Check only the Archive filtered messages and Accept messages without notifying sender of filtering check boxes.
- Open a command prompt and enter your mailroot folder drive: \Program Files\Exchsrvr\Mailroot\vsi n\ (n is the instance number of the SMTP virtual server).
- Create an NTFS junction point by using linkd.exe (from the Windows Server 2003 Resource Kit tools) or junction (formerly a Sysinternals utility, now offered by Microsoft at http://www.microsoft.com/technet/sysinternals/fileanddisk/junction.mspx) by running either of the following commands:
linkd filter /D pickupwhere /D deletes the source junction point (here, Filter) regardless of whether a link exists at the source.
junction filter pickupThe previous steps make a symbolic link to the Pickup and Filter directories. All messages filtered by using the sender filter are then stored in the Pickup directory and delivered to the correct Inbox.
This fix caused some problems, though. First, our corporate email servers would try to deliver duplicate messages to To: and CC: recipients that weren't part of our Exchange organization. And of course, this method eliminates my blacklist on which I depend to delete all those double–opt-in messages (i.e., sent to employees who have left the company or to employees who signed up for a nonapproved email subscription) that are hard to get rid of and very annoying. Lowering the spam confidence level (SCL) in IMF to a 1 reduced spam to less than one percent; however, lowering the SCL to 1 also resulted in a high number of false positives.
A Better Method
With the problem of false positives in mind, I wanted a way to filter the Exchsrvr\Mailroot\ vsi n \UCEArchive directory that IMF generates to hold messages that meet the SCL threshold. I played around with a couple of command-line options and finally came up with the idea of writing a script that uses a whitelist to filter valid email from the UceArchive directory. The script reads the whitelist (a text file) and validates messages by comparing the email addresses in the UceArchive directory with those in the whitelist. Then the script moves the valid email from the UceArchive directory to the Pickup directory and moves spam email from the UceArchive directory to a designated folder.
This solution offered the best of all worlds:
• I could lower the SCL to 1.
• The script would resend false positives to the Pickup directory.
• I got to keep my blacklist for all those double opt-ins.
Here's how to use my script-based solution:
- Create a BList directory in D:\Program Files\Exchsrvr\Mailroot\vsi 1\BList (i.e., the drive where the vsi 1 folder is located).
- Build your whitelist as a simple text file. My script, WList.bat in Listing 1, isn't case sensitive, and what you enter in the whitelist text file represents substrings, so be careful to enter correct email domain names, and precede each domain name with the "at" symbol (@). For example, specifying ford.com in the whitelist will also allow ashford.com. Therefore, if you want to whitelist ford.com, enter @ford.com in the whitelist. Figure 1 shows a sample whitelist file.
Because the script scans all text in the header and body of the email message, you can include in the whitelist file addresses in other fields, such as To: or Subject. For example, you could include in the whitelist file From:email@example.com. (In that case, you’d need to add all variations of the From: address, such as Bob
and Bob@bob.com. You could also add lines that contained known safe IP addresses to prevent forged headers from coming through.
- Move the text file to a PC with the same "vsi n" directory structure as your Exchange Server 2003 Service Pack 2 (SP2) Exchange server.
- Create the script. You can use my sample script, WList.bat, modify it to meet your own needs, or write your own. Because the UceArchive directory can't contain any of the script components, you'll need to copy the script as well as the resource kit sleep.exe utility, which the script requires, outside of this directory. (We run WList.bat 24 ´ 7 on our SMTP gateway, but you could also create a scheduled task to run it.) The beauty of this script is that using a For loop with the delims keyword and the Findstr command (which searches for patterns of text in files using regular expressions) with the /G: switch (which tells the script to get search strings from the specified file) lets you make your whitelist text file easy to maintain because you can make changes to whitelist.txt without stopping whitelist.bat. (You can find the complete syntax for Findstr at http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/findstr.mspx?mfr=true.) For easier housekeeping, my whitelist.txt entries are in alphabetical order.
In WList.bat, you can modify the number of seconds the program waits for the loop to start each time to meet your company’s service level agreement (SLA) for email delivery. Additionally, you can further filter the Blist directory by filtering on the SCL rating itself. This appears as ‘X-SCL: #’, where the # symbol is the SCL rating number.
WList.bat runs quickly. In initial testing, the script scanned more than 1,200 email messages and moved them in less than three seconds using a whitelist consisting of about 200 line items.
Because the server-side SCL rating can't be lower than the email-client–side SCL rating, lowering the rating to a 1 also lowers it on Microsoft Office Outlook 2003 clients. This lower rating might cause some false positives on the client side. However, these false positives will soon disappear as users approve domains and senders. You can avoid this problem altogether by only enabling IMF on an SMTP gateway. Then redirect all port 25 traffic to the gateway and let it handle all the processing.
After extensively testing my script on a Friday, I put it into production on our Exchange server. That following Monday morning, our CEO told my director that he thought our email was down because he received no spam on his BlackBerry device! Before we started using WList.bat, about 10 to 12 percent of spam sent to users' accounts made it into their mailboxes. Now employees get no spam at all because our spam-filtering method is based exclusively on whitelisting.