Solutions Snapshot
PROBLEM
You need to block access to thousands of unwanted Web sites without spending a lot of money.
SOLUTION
If you're already running ISA Server 2004 or 2006, you can use an inexpensive blacklist service and a couple of scripts to prevent access to inappropriate Web sites.
WHAT YOU NEED
ISA Server 2004 or 2006, blacklist service subscription, ImportBlacklist.vbs and ScheduledUpdate.bat scripts
DIFFICULTY
3 out of 5

Blacklisting services maintain lists of Web sites that contain pornography, hate speech, violence, hacking tools, or other prohibited content. You can subscribe to an inexpensive blacklisting service and import its list (typically updated each week) into ISA Server with a script. In fact, I've included a free script for doing this with this article. This might sound complicated, but don't worry, it's not hard to do. Let's walk through the steps together.

Step 1: Use ISA Server
Of course, you must have ISA Server 2004 or 2006, and your users' Web browsers must be configured to go through it for HTTP access to the Internet. This article assumes you've already got this set up, but if you don't, you can download a trial version of ISA Server from http://www.microsoft.com/isaserver. If you have Microsoft Windows Small Business Server (SBS) 2003 Premium Edition Service Pack 1 (SP1) or SBS 2003 Premium Release 2 (R2), it includes ISA Server 2004.

Step 2: Create a Domain Name Set
In ISA Server, you can use firewall policy rules to grant or deny access to a domain name set—that is, a list of DNS domains. The list can include a mixture of fully qualified domain names (e.g., www.windowsitpro.com) and domains with wildcards (e.g., *.microsoft.com). We need to create a domain name set to hold the hundreds of thousands of domains we wish to block. Let's call it Bad-Sites.

To create the Bad-Sites domain name set, open the Microsoft Management Console (MMC) ISA Server Management snap-in, expand the container list under your ISA Server, and click the Firewall Policy container to highlight it. Next, right-click the Firewall Policy container, select View, and select Task Pane (if it's not already selected). The task pane will appear on the right. In the task pane, click the Toolbox tab, then click the Network Objects category. Right-click Domain Name Sets and select New Domain Name Set. Enter the name Bad-Sites and click OK. At the top of the console window, click Apply to save the changes. Figure 1 shows the Bad-Sites domain name set on the Toolbox tab. The Bad-Sites list is currently empty, but we'll fill it with blacklisted domains in a moment.

Step 3: Create a Blocking Rule
Once you have your Bad-Sites list of unwanted domains, you'll block access to those sites with a rule in the firewall policy. This rule will come just before the rule that otherwise allows Internet access, which I'll assume already exists.

To create the rule that blocks requests to Bad-Sites, click the rule in your firewall policy that permits your users Internet access. Next, right-click the Firewall Policy container in the ISA Server Management console, select New, Access Rule, name the rule Site_Blocker, click Next, select Deny, click Next, accept the default for the rule to apply to all outbound traffic, and click Next. Click Add and add the Internal Network to the list of sources (expand the Networks folder to see the Internal Network object), click Next. Click Add and add the Bad-Sites domain name set for the destination (expand the Domain Name Sets folder to see the BadSites object), click Next. Accept the default All Users option, click Next, and click Finish.

Right-click your new Site_Blocker rule to move it up or down, if necessary, to place it just above your rule that allows users Internet access. Click Apply at the top of the console to save your changes. You can see the completed Site_Blocker rule in Figure 1, including the Bad-Sites set in the To column.

See “More Web Filtering” for guidance on blocking certain file types and using other ISA Server content-filtering features.

Step 4: Download a Blacklist
You now have a rule named Site_Blocker that prevents access to domains in the currently empty Bad-Sites list. This list must now be filled with the hundreds of thousands of domains that have undesirable content, and it must be updated at least weekly. But where can you get this information in a usable form? And how can you load it into your list?

Fortunately, free and inexpensive sources of blacklists are available on the Internet that can be imported into your Bad-Sites list. Perhaps the best known free blacklists are for the squidGuard (http://www.squidguard.org/blacklist) and DansGuardian (http://www.dansguardian.org) UNIX/Linux filters, but these blacklists work just fine with ISA Server too.

I prefer the inexpensive blacklist service at http://www.urlblacklist.com. As of this writing, a business can download an updated blacklist once per week for less than $190 a year with no per-user limits. Schools and individuals pay less. Because URLBlacklist.com is a commercial rather than free service, its blacklists are managed better and the service is more likely to still exist a year from now. You can download a small demo blacklist for free to try out the service.

When you download one of these blacklists, it will most likely be in a GNU zip (gzip)–compressed .tar file—that is, a file that ends with the .tar.gz extension. You can use graphical programs such as WinZip (http://www.winzip.com) to extract your blacklist text files, or you can get Windows versions of gunzip.exe and tar.exe for free from http://unxutils.sourceforge.net (notice that unxutils has only one i) or in the free Microsoft Windows Services for Unix (SFU) at http://www.microsoft.com/technet/interopmigration/unix/sfu.

To use gunzip.exe and tar.exe with the bigblacklist.tar.gz file downloaded from http://www.urlblacklist.com, move bigblacklist.tar.gz into a new folder and execute the following commands in a CMD shell to extract the blacklist files:

gunzip.exe bigblacklist.tar.gz
tar.exe -xf bigblacklist.tar

These commands will create a new folder that contains all the blacklist files. The files are placed into subfolders named after their contents. For example, one of these subfolders will be named porn, and in the porn folder you'll find a large text file named domains.

We'll use a script to import the porn domains text file into our blocked Bad-Sites list. Then, we'll use another script to automate downloading blacklist updates and importing them into Bad-Sites.

Step 5: Import Blacklist into Bad-Sites
Download the script named ImportBlacklist.vbs by clicking the 94079.zip link. Unzip the downloaded file and copy the two files it contains to your ISA Server's hard drive. (I'll explain the other file, ScheduledUpdate.bat, in a moment.)

The ImportBlacklist.vbs script imports a text file of domain names into a domain name set on ISA Server 2004 or 2006, either the Standard or Enterprise edition. Copy the porn\domains blacklist file to the folder on your ISA Server system that contains the ImportBlacklist.vbs script, then run the following command in a CMD shell (type the command all on one line) to fill your Bad-Sites list:

cscript.exe ImportBlacklist.vbs
 Bad-Sites domains 

To import domains from multiple files, merge them all together into one large file. For example, to append one file (domains1) to the end of another file (domains2), use the Type command as follows:

type domains1 >> domains2 

Alternatively, you could create multiple BadSites sets, one for each file to be imported, and add all these Bad-Sites sets to the destination in the Site_Blocker rule.

By default, the script deletes the contents of the domain name set first, then imports from the text file, so it's better to do your list management in the text file than in the domain name set itself. When the script finishes, refresh your ISA Server Management console to see the new contents of the Bad-Sites list (or close and reopen the console, which is often faster).

That's it! Now, when a user requests a file from a blocked domain, the user will get an error page instead. As long as the HTTP request is routed through ISA Server, this domain blocking works even when the user's browser isn't configured as a Web proxy client. (But it's better to configure all browsers as proxy clients.) And the performance penalty of ongoing domain blocking is relatively small because it's not regular expression pattern matching, it's just simple string comparisons against the user's requested URL. Very slick.

Step 6: Schedule Updates
Manually downloading blacklist updates and importing them into ISA Server is easy enough, but it can be tedious. Fortunately, it can be scripted. A scheduled batch script that uses a free Windows version of wget.exe (http://www.gnu.org/software/wget) can download the latest version of your favorite blacklist every week or night, then run gunzip.exe, tar.exe, and ImportBlacklist.vbs to update your ISA Server system hands-free.

Listing 1 shows a simple batch script named ScheduledUpdate.bat that performs these tasks. The script downloads a small demo blacklist from URLBlacklist.com and imports its porn list into an ISA Server domain name set named Bad-Sites using the ImportBlacklist.vbs script. In real life, you'll need to edit this script to download the full blacklist for which you've paid and to perform error-checking, logging, and/or administrator notification. Use the Scheduled Tasks applet in Control Panel to schedule the script.

Updating your blacklist is important because new bad sites are found every week. Scheduling this work is important because of the time it takes to import very large lists. On a server with a single 2.2GHz Pentium 4 CPU, for example, it takes less than 10 minutes to import 100,000 domains from a blacklist file, but that same machine requires three hours to import 500,000 domains. And during the import process, the CPU will be pegged at 100 percent. So, schedule the blacklist updates for off-peak hours, and run the ImportBlacklist.vbs script with the \belownormal option (as the last line of Listing 1 shows) to use a lower multitasking priority. Other ISA Server processes will have an easier time getting CPU cycles.

Note that you'll have to allow ISA Server HTTP access to the Internet for the batch script to run. Following the procedure in Step 3, create a rule that gives ISA Server access only to the blacklist download site. Set the source network to Local Host and the destination URL to the location of the blacklist to be downloaded.

Importing blacklists for domain blocking is just one example of ISA Server's scriptability. You can find lots of other scripts at sites such as http://www.isatools.org, http://www.isaserver.bm, and http://www.isascripts.org (my site), and Microsoft has an ISA Server software development kit (SDK) if you want to write your own. Using blacklists and scripts as we've done here won't be as scalable or full-featured as using a commercial content filter, but if you're on a budget, it might be good enough.

SOLUTION STEPS:

1. Use ISA Server.
2. Create a domain name set.
3. Create a blocking rule.
4. Download a blacklist.
5. Import blacklist into Bad-Sites domain name set.
6. Schedule updates.