NTFS Data Streams—Intermediate

NTFS can support multiple data streams in files. The stream that you're probably most accustomed to viewing and manipulating is the visible unnamed stream. However, you can create additional hidden named streams that contain usable data. When you understand how to manipulate both the visible and hidden streams, you have another tool for your scripting tool belt. You can use streams with many scripting languages, including Windows NT shell, VBScript, and Perl. Here's a look at the basics of NTFS data streams and several examples of how you can use them in your scripts.

Understanding the Basics
The easiest way to understand streams is to see them in action. Open a command prompt on your computer. Type the command

Convert drive: /fs:ntfs

to convert a FAT partition to NTFS. Because you're dealing with NTFS streams, you must have an NTFS partition to use them. If you try to create a stream on a non-NTFS partition, you receive the error message The filename, directory name, or volume label syntax is incorrect. (For more guidelines about using streams, see the sidebar "About NTFS Data Streams," page 2.)

Next, type the commands in Listing 1, page 2. This command-line entry creates the text file teststreams.txt and enters the specified text in the visible stream. When users open teststreams.txt, they see the text This is the default data stream that is visible to users.

To input text into a hidden stream associated with the text file you just created, enter the commands in Listing 2, page 2. In this code, you use the syntax Filename:StreamName to name the hidden stream. Stream names are case insensitive, but they can't include spaces. The single redirect (>) character clears the hidden stream of any old content, and the double redirect (>>) character adds new content to the existing content. Clearing the old content and appending the new content don't affect the contents of the visible stream. If you append a number to a hidden stream, you must place a space between that number and the > or >> character. Otherwise, the command will fail.

If you open teststreams.txt in WordPad or Notepad, you see the visible stream but not the NewStream stream. To view the hidden NewStream stream, you can use the More command:

More < C:\teststreams.txt:NewStream

Alternatively, you can use the For /f command to enumerate the lines in the hidden stream. At the command line, type

For /f "Tokens=*" %i in
Do @Echo %i

If you run the For /f command from a batch file instead of the command line, you must replace the single percent signs (%) with double percent signs (%%).

Using the Echo command each time you need to modify a stream can be tedious. An easier way to edit streams is to open them in WordPad or Notepad. To edit NewStream in WordPad, enter the command

Write C:\teststreams.txt:NewStream

You can then edit NewStream's content and save the changes. If you reopen teststreams.txt in WordPad or use the More command, you see the changed NewStream.

If you prefer to use Notepad for editing, you must initially create the hidden stream with a .txt extension on the stream name. For example, to create the hidden stream OpenSesame.txt in the file teststreams.txt, you first create the default stream with the Echo command in Listing 1. You then use the Echo command in Listing 3 to create the hidden stream. To open this file in Notepad, type


As I've just demonstrated, you can use WordPad or Notepad to create and edit streams in a file that you've created with the Echo command. If you want to completely avoid using the Echo command, you can use Notepad to initially create a file with streams. (WordPad doesn't offer this capability.) Type the command


Notepad responds that it can't find the file and asks if you want to create it. Click Yes. Notepad opens and lets you create the OpenSesame.txt stream. Save the file, exit Notepad, and look at the size of the file you just created. The size will be 0 bytes even though the stream has content.

You can use other scripting languages besides NT shell scripting to read and open streams. In Perl, you open and read a stream like a file, as Listing 4 shows. In VBScript, you use the File System Object (FSO) model's FileSystemObject object to open and read streams, as Listing 5 shows.

Now that you know how to create and edit data streams in files, let's look at several examples of how you can use hidden streams.

Password-Protecting a Script
Suppose you're having problems with administrators unintentionally launching scripts when they want only to open the scripts for editing. To avoid these accidental launchings, you want to set up the scripts so that administrators must enter a password (as an argument) at the command prompt to execute them. However, you don't want to hard-code the password in the script because other users could then open the script and input the password they find.

A better approach is to create a hidden stream containing the password for the script you want to protect from accidental launches. To see how this approach works, let's create the script C:\teststreams.bat and protect it with the password MyPassword. Follow these steps:

  1. Enter the code in Listing 6 at a command prompt.
  2. Run the code. You've now created the script C:\teststreams.bat and entered code in the hidden stream MyPassword. To view this code, type
    More < C:\teststreams.bat:MyPassword

    at the command line. The results will look similar to those in Figure 1.

  3. Open C:\teststreams.bat in WordPad, and type the command
    For /f "Tokens=*" %%i in
            (C:\teststreams.bat:%1) Do %%i

    into the visible stream of C:\teststreams.bat. Note the use of the %1 value instead of the stream name. In the Filename:StreamName syntax, you can use the %0 value to represent Filename and the %1 value to represent StreamName. Because %0 and %1 are assigned actual values when you execute C:\teststreams.bat from the command prompt, you don't need to use %%0 or %%1, even though the For command is in a batch file.

    If you want to make the code even more cryptic, you can use this For command:

    For /f "Tokens=*" (%%i in %0:%1)
            Do %%i

    However, if you're on a Windows 2000 system, don't use the %0 value in the For command. In Win2K, the %0 value doesn't function correctly because Win2K places double quotes around it.

    No matter which For command you use, it plays an important role: The For command parses the lines of the code in the hidden stream, then executes the hidden code, one line at a time, as if the hidden code were in a visible stream.

  4. Run the script by typing
    C:\teststreams.bat MyPassword

    at the command line. If you try to launch the script without the MyPassword argument, the script quietly fails to run. If you want to test for the existence of the MyPassword argument and warn the user if it's missing, you can add the code in Listing 7 to the visible steam in C:\teststreams.bat.

Because the code in a hidden stream executes one line at a time, you can't use GoTo commands and labels in hidden-stream code. If you want to place a complex script in a hidden stream, you can use the Echo command to send the code to a .bat file, run that file, then have the file delete itself. This process exposes the code for only a short time when the script is running.

I recommend that you don't store any sensitive scripts or information in a hidden stream because you can use the Win32 backup API functions (i.e., BackupRead and BackupWrite) to enumerate a file's hidden stream. In addition, shareware programs that detect and enumerate hidden streams are available.

Using a Stream as an Input File
Experienced systems administrators often use a text file to provide input to a script. The script then performs an action for each item listed in the text file. If that action needs to be performed on additional items, the systems administrators simply add those items to the input file. By following this best practice, systems administrators avoid having to modify a tested, debugged script, thereby avoiding the risk of introducing a typo or other error that might stop the script from working correctly.

A spin on this best practice is to use hidden streams instead of text files to provide input to scripts. For example, let's create the script C:\serverping .bat and place a list of servers to ping in that batch file's hidden stream.

  1. Enter the code in Listing 8 at a command prompt. Replace the server names (i.e., Server1, Server2, and Server3) with the names of the servers and workstations you want to ping in your network. You can easily add more entries to the list by appending additional server or workstation names.
  2. Run the adapted code to create the script C:\serverping.bat and to input the code into the hidden stream ServerList.txt.
  3. Open C:\serverping.bat in Notepad, and type the command
For /f "Tokens=*" %%i in
Do Ping %%i

into C:\serverping.bat's visible stream. This For command reads the servers into the script, one at a time.

  • Run the script by typing

    at the command line. After the script pings each server, it writes the results to the console screen.

    Sending Keystrokes to a Command
    Sometimes commands require keyboard input. If you use a command that requires keyboard input in an automated script, the script hangs because it waits for keyboard input that never comes. For example, if you use the Xcopy command to copy a file and create a new directory structure, the command prompts you to enter whether the copy you're requesting is a file (enter F) or directory (enter D). If you use this command in a script, the script fails unless you input either F or D. But placing one character in an input text file can be risky because people might not know its purpose and delete the file. And creating a temporary file to hold one character can be cumbersome. A hidden stream, however, can hold this keystroke information efficiently.

    For example, run the commands in Listing 9. These commands create the script %Temp%\teststreams.bat and enter specified code into the hidden stream Keystrokes.txt. Then, open %Temp%\teststreams.bat in Notepad and copy the commands in Listing 10 into the visible stream.

    Without the hidden stream, the commands in Listing 10 require you to manually input the letter F. With the hidden stream, the commands execute without this manual input.

    Storing Detailed File Properties
    In "Real-World Scripting: Create a High-Availability Environment for Your Scripts," August 2000, I encouraged systems administrators to document scripts in a level of detail typically not seen in code comments. The details might include the script's author, date approved, dependencies, run syntax, and input and output filenames and paths.

    You can store a script's details in a Microsoft Word or Microsoft Excel document. However, a more efficient approach is to store the details in the script's hidden stream. You can either create one hidden stream containing all the details or create a stream for each detail. When you use hidden streams, the script's details are easily accessible because they reside in the script.

    Using a Stream as a Temporary File
    One limitation in NT shell scripting is that environment variables can't store command results in the form of lists or arrays. If you want to store command results as lists or arrays in a .bat or .cmd file, you must output the results to a temporary file, then use the For command to parse the temporary file when you need to use that information.

    However, using temporary files can be a lot of work. For example, if your .bat file is complex, you might need to create many temporary files to store the command results because you must use a unique filename for each temporary file. You must also remember to delete all temporary files when you're finished with them.

    To eliminate this extra work, you can use streams as a temporary storage location. The script RunMeAgain.bat uses streams as a temporary file to demonstrate how to create and use hidden streams. The script creates two hidden streams that track the date when you last ran the script and how many times you've run the script on the current day. You'll find the script in the Code Library on the Win32 Scripting Journal Web site at http://www I included comments in the script to help you understand and run the code. I tested this script on Win2K Professional, NT Server 4.0 Service Pack 6a (SP6a), and NT Workstation 4.0 SP6a.

    If You Want More Examples
    The examples I've just shown you only scratch the surface of what you can do with streams. For another example, see my column "Real-World Scripting: Using Blat to Send Email Notification Messages," page 9. For more examples of how to use streams in Win2K, check out Dino Esposito, "A Programmer's Perspective on NTFS 2000, Part 1: Stream and Hard Link," which you can find in the Microsoft Developer Network (MSDN) Online Library at library/techart/ntfs5.htm.15845