At COMDEX last November, Bill Gates spent a portion of his keynote address announcing a new product: Exchange Intelligent Message Filter (IMF). COMDEX might seem like an odd venue for an Exchange Server announcement until you consider that IMF is really a spam filter and that spam is a growing problem and annoyance to users, administrators, and pretty much everyone except the people who send it. After you know what IMF does, how it works, and how to deploy it, you'll realize that IMF might not solve your spam problem all by itself but can be a valuable adjunct to other spam-reduction measures.

IMF Requirements
IMF is implemented as an event sink that extends the behavior of the Exchange Server 2003 SMTP service. IMF runs only on Exchange 2003 because it requires some changes that Microsoft made to support the tagging of suspect messages with a spam confidence level (SCL)--more about that later. In a typical deployment, you put IMF on an Internet-facing bridgehead server so that it can scan incoming SMTP mail. Because IMF requires Exchange 2003, you can't put it on a standalone server as a front-end filter. The only other limitation is that IMF doesn't support Exchange clusters, but you probably aren't using clustered bridgeheads to receive mail from the Internet, so this limitation isn't likely to be a problem.

How IMF Works
IMF uses Microsoft SmartScreen Technology, filtering technology that Microsoft Research originally developed for use with Hotmail. Depending on whose estimate you use, somewhere between 65 percent and 85 percent of the mail arriving at major ISP servers is spam, so Microsoft obviously needs a powerful filter. However, given Hotmail's end-user base, the filter also needs to be simple to use. Several Microsoft products--Hotmail, IMF, the Outlook 2003 Junk E-mail Filter, and Microsoft Entourage 2004 for Mac--use SmartScreen. In all these products, it provides more-or-less- automatic filtering services that work without much user or administrator intervention.

One of the key differences between SmartScreen and other filters is that users and administrators can't train or adjust SmartScreen--it's a black box that uses a large set of filtering data. Microsoft's data-gathering approach is interesting: Users who sign up for a Hotmail feedback program are asked each day whether one particular message they've received is spam. The contents of the messages are fed into the corpus used to build the SmartScreen filter list. In January 2004, Microsoft Research reported that the corpus contained more than 10 million messages.

The contents of the corpus are used to build a filter file that positively and negatively weights key message phrases and words. To determine whether a message is spam, the filter uses the weights to calculate a weight for the subject and one for the message body. These weights are added together to give the message a total weight. The filter then performs more checks, which Microsoft hasn't completely described. (The company says that fully describing these checks would help spammers outwit them.) The message can gain or lose weight depending on the result of these tests. After IMF calculates the final weight, the filter converts that weight into an index from 1 to 10. This index is the aforementioned SCL.

IMF supports two filtering thresholds--the gateway threshold and the store threshold--that use the SCL. As its name suggests, the gateway threshold is on the IMF server. Messages whose SCL exceeds the threshold are subject to an action that you select: You can reject them and send a nondelivery report (NDR), delete them and send no NDR, archive them to a location that you specify, or ignore the SCL. It's important to note that IMF operates in conjunction with the Exchange 2003 transport-filtering mechanisms. IMF won't filter out any messages to or from any names, IP addresses, or domains that you've configured as trusted senders. IMF's filtering happens after the Exchange SMTP service has accepted a message. If the message SCL exceeds the gateway threshold, the selected action occurs. If the SCL is lower than the threshold, IMF passes the message on to the mailbox store that contains the recipient's mailbox, just as would happen if IMF weren't present.

At that point, the store threshold kicks in. If the user uses Microsoft Office Outlook 2003 or Outlook Web Access (OWA) 2003, the store compares the message SCL with the store threshold. If the SCL is lower than the threshold, the store checks the message against the user's Blocked Senders List. If the sender is on the list, the store processes the message according to the user's junk-mail-filtering setting--filing the message in the Junk E-mail folder or deleting it. If the sender isn't on the list and the SCL is below the store threshold, IMF delivers the message to the user's Inbox. If the message SCL exceeds the store threshold, the store files the message in the Junk E-mail folder or deletes it--unless the sender is on the user's Safe Senders List, in which case the message SCL is ignored.

Users who use earlier Outlook, earlier OWA, or non-Outlook clients don't benefit from IMF. Servers that run Exchange 2000 Server can't interpret the SCL; they deliver messages tagged with an SCL to Exchange 2000 mailbox clients as usual. Store threshold filtering depends on the presence of Safe and Blocked Senders lists, which exist for a mailbox only if it has been logged on to via Outlook 2003 or OWA 2003. If you use POP or IMAP clients, no store-level filtering will occur.

IMF has one additional wrinkle: If you have multiple Active Directory (AD) forests and your inbound SMTP bridgeheads in one forest can accept mail for recipients in the other forest, you need to enable cross-forest authentication if you want the SCL property to persist when messages move between forests. Cross-forest authentication is necessary because the SCL is an extended message property that's transferred only within an Exchange organization or between Exchange organizations that have an authenticated connector. The Microsoft Exchange Intelligent Message Filter Deployment Guide has a complete description of how to set up cross-forest authentication; in brief, you need to set up user accounts in each forest, give them the correct permissions, then specify the account to use for each end of the connector.

Obtaining and Installing IMF
Microsoft's original plan was to distribute IMF only to customers who had purchased a Software Assurance (SA) license. However, that strategy was unpopular, and after what I assume was a vigorous internal debate, Microsoft did an about-face and released IMF free to all Exchange 2003 customers. You can download IMF from Microsoft's Downloads for Exchange Server 2003 page at http://www.microsoft.com/exchange/downloads/2003.asp. Scroll down to the Security and Protection section and click the Exchange Intelligent Message Filter link. At the resulting page, download the filter and click the Microsoft Exchange Intelligent Message Filter Deployment Guide link. You should download and read the guide before attempting to deploy IMF.

The process of installing event sinks on an Exchange server isn't for the faint of heart; you have to put the sink in the correct place, then register it properly so that the SMTP core can find and call it. It's easy to make mistakes when doing this manually. Fortunately, IMF comes packaged as a Windows Installer (.msi) file that does all the work. All you have to do is launch the .msi file and tell Windows Installer whether you want to install IMF, the IMF management tools, or both. You can install the management tools on any machine on which Exchange System Manager (ESM) is installed, but as I mentioned earlier, you can install IMF only on an Exchange 2003 server.

Of course, if you implement IMF on one of your inbound SMTP bridgeheads, you must implement it on all of them. If you don't, some mail might be routed through unfiltered servers. (The Outlook 2003 Junk E-mail Filter might help here, but stopping the junk at the perimeter is better.)

Configuring IMF
Speaking of the IMF management tools, there are two. IMF installation adds a new Intelligent Message Filtering tab to the Message Delivery Properties dialog box and an Intelligent Message Filter node under the Protocols node for each SMTP virtual server in ESM. You access the Message Delivery Properties dialog box and its new Intelligent Message Filtering tab, which Figure 1 shows, from the Global Settings node in ESM. You can use this tab to set the gateway and store threshold values and to specify an action to take on messages that exceed the gateway threshold:

  • No action is the default action. Microsoft recommends that you take no action at first while you determine exactly how much spam you receive. You can set a threshold and No action, and the Performance Monitor's "The Total Messages Assigned an SCL Rating of X" counters will tell you how many messages hit a particular threshold value. You can add up the counter values to determine how many messages the filter caught.
  • The Archive action stores filtered messages in a folder named UCEArchive under the SMTP virtual server node, which is under the Mailroot node under the SMTP server. Archived messages are stored as plain text, so you can open and review them with Notepad, Outlook Express, or any other tool that can open text files. If the archive contains any legitimate messages, you can move them to the pickup directory and the SMTP service will reprocess them for delivery without further filtering. IMF doesn't provide any way to view the contents of the archives, but at least two free solutions exist; see http://www.e2ksecurity.com/archives/000835.html for details.
  • The Delete action throws the message away without warning; neither the recipient nor the sender receives any indication that any action has been taken. Use this action with caution because if you set the SCL too low, you'll probably lose legitimate messages.
  • The Reject action rejects the filtered message during the connection attempt and forces the sender's server to create an NDR and deliver it to the sender. IMF won't accept messages then send an NDR.

By default, these settings apply to all SMTP virtual servers on the IMF server. However, IMF should be enabled only for those virtual servers that you want to filter. For example, if you follow Microsoft's recommendation of using separate virtual servers for inbound and outbound traffic, you probably don't want to spend CPU cycles running IMF on the outbound virtual server. You enable and disable IMF filtering for a particular virtual server by using the second IMF management tool: the Intelligent Message Filter node that appears under each SMTP virtual server's Protocols node in ESM. When you open the Intelligent Message Filter node's properties dialog box, you see a list of virtual servers, each with a check box next to it. Make sure that only the boxes for the virtual servers on which you want filtering to be active are selected.

Advanced Tweaks
Of course, many of us would like to be able to tweak the IMF filtering corpus, but Microsoft isn't allowing that. The company also hasn't announced how it plans to make updates to the corpus available, but it will probably use a process similar to the one for the Outlook filter and Entourage filter: You'll download a hotfix from Windows Update. For now, the IMF deployment guide describes four registry entries that you can apply to tweak IMF's behavior. The first three tweaks require you to create a new subkey named HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\ContentFilter and add the following values to it:

  • ArchiveDir--To change the location of the archive directory, add a string value named ArchiveDir and use it to specify the full path (local paths only, please) to the archive directory. ArchiveDir applies to all virtual servers on the server, so messages aren't grouped by virtual server when you use this setting.
  • ArchiveSCL--To force the IMF to store filtered messages' SCL, add a REG_DWORD entry named ArchiveSCL and set its value to 1. Each filtered message will then have an X-SCL header.
  • CheckAuthSessions--As I mentioned earlier, messages coming from authenticated connections aren't filtered. However, you might want to filter messages from authenticated senders if you receive messages from users who are authenticated but might still be sending spam. This situation can occur at application service providers (ASPs) and universities and in other environments in which users aren't necessarily trustworthy. To enable filtering of messages sent by authenticated users, add a REG_DWORD entry named CheckAuthSessions and set the value to 1.

The fourth setting is applied in a different location.

  • Max Extended Rule Size--The lists of safe and blocked senders that each user creates is stored in an invisible message in the user mailbox. By default, Outlook and IMF allow about 510KB of data in the list, but the client and server must synchronize the lists, so you might want to keep them smaller. If so, create a REG_DWORD value named HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\Max Extended Rule Size, then make its value the maximum size (in bytes) that you want to allow.

IMF is a useful filtering solution, especially because it's free. In most environments, it will drop the volume of spam significantly. It's not a substitute for more sophisticated filters from the likes of NetIQ, Nemx Software, and Symantec--these filters and services allow more customization and additional filtering, quarantine, and management functionality. However, IMF works well in conjunction with these tools, and it's valuable in its own right for organizations that don't want to, or can't, spend money on a more powerful product. It's probably safe to assume that Microsoft will roll more spam-filtering functionality into IMF as time passes, and I'll be interested to see how well the product keeps up with the legions of evil spammers.