Hierarchical Storage Management (HSM) has been around for almost 20 years in the mainframe environment. The basic premise of HSM centers on managing aging data. Now this storage strategy is available for Windows NT. One of the products leading the way in HSM for NT is Eastman Software's OPEN/stor. I evaluated version 2.0--version 2.05 is now available.
Understanding HSM
As you add files to a computer hard disk, you typically access the
information less and less, and the files spend more time sitting idle on your
hard disk. So why not move these files to an Iomega Jaz drive or similar device
and retrieve them as you need them? The answer is simple. Once you move these
files, you need a way to find them when you need them.
By placing a file marker when HSM moves the files offline, HSM solves this problem of finding archived files. If you want to locate a particular file, the HSM database finds the file and passes it to you. To the user, the file transfer and retrieval process is invisible. The goal of HSM is to maintain optimal use of storage space.
HSM is an application for controlling and managing data storage on various storage devices. As you see in Figure 1, page 92, these storage devices are shown in a hierarchical fashion, with the most expensive, fastest devices at the top and the cheapest, slowest devices at the bottom. This hierarchy consists of different layers with local fast hard disks at the top, intermediate hard disks as the first migration level, and optional magneto-optic (MO) and tape jukeboxes at the lower levels.
Once you establish some migration rules, HSM can migrate seldom-accessed files from the local hard disk to a migration volume on the HSM server. HSM leaves a pointer on the local hard disk to the original file, and NT Explorer or File Manager displays the filename. If you double-click the filename, you can retrieve the file almost as fast as you can retrieve it from a local hard disk (unless HSM is storing the file on an MO or tape device).
HSM is not a backup application. In fact, Eastman Software's OPEN/stor, the original PC HSM product, specifically states that it is not a backup system. OPEN/stor simply moves files from one device to another in a hierarchy of storage devices. The overall goal is to maintain defined levels of files on NT servers and NT workstations.
OPEN/stor is not the only HSM product on the market, but because of the complexity of terminology and setup, it's the only product I'm going to discuss in this article. Microsoft plans to include elements of OPEN/stor in NT 5.0, so OPEN/stor is a preeminent product at the moment. Windows NT Magazine will look at other HSM products in future issues.
Basic Terminology and Rules
Before you can install and use HSM, you need a thorough understanding of
storage concepts and the rules of migration. You also need to pre-plan your
environment and setup because once you install HSM, changing components and
devices can be painstaking.
Active vs. Inactive Data. Hard disks have a mix of active and inactive data. Eastman Software estimates that you can consider 20 percent of the files on a hard disk active, while 80 percent are inactive. Of course, not everyone agrees on the terms active and inactive because not everyone can agree on how long a file is not used before it is considered inactive.
With OPEN/stor, the terms active and inactive are operationally defined in an administrator context. For example, one administrator may deem a file inactive if no one has accessed the file in 60 days, but another administrator may set 30 days as the criterion. With OPEN/stor, you have to define the rules for what's active and what's inactive (setting these rules is an OPEN/stor configuration option).
Movement of Inactive Data. After you define what files are active and inactive, you have to decide when to move the inactive files. Typically, you determine this setting according to the amount of free space on a managed disk (i.e., a volume). OPEN/stor defaults to using very high percentages (a 97 percent critical level, a 95 percent acceptable level, and a 93 percent optimal level) when calculating the amount of free disk space to maintain.
This issue is not trivial because files occupying more than 60 percent of a hard disk can cause serious fragmentation and dramatically slow down the hard disk I/O. Replacing large files with small placeholders helps maintain hard disk volume size.
Movement of Files. Eastman Software refers to moving files from the primary storage to secondary storage as migration. When the movement occurs, OPEN/stor leaves a placeholder in the original file's place. These placeholders maintain file location information and serve as a link to the secondary storage of the migrated file. If you scan your hard disk with NT Explorer or an application such as Word, NT sees the placeholder as the original file. OPEN/stor 2.0 creates a 420byte placeholder file. In future builds, the placeholder file size will match the original file size so you can see the original file size.
You can use the following Registry setting to control this option (I like to see the 420byte file size because I then know the file is a migrated file):
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ AvFilter\Parameters\ ChangeFileSize
The value is a REG_DWORD. Setting the value to 0 will result in the 420byte placeholder file size. If you set this value to 1, you'll see the true file size and not the placeholder file size.
Recalling Files. When you click the associated placeholder file to open a migrated file, or use an application to open the migrated file, this process recalls the migrated file to its original location. This retrieval process is transparent to the user and system. When you recall a file, HSM removes the placeholder file. You can also schedule a recall job to move all or selected files to their original location.
Migration Rules
OPEN/stor deals with three levels (critical, acceptable, and optimal) of
primary volume management. These levels are a percentage of the total capacity
of a volume on an HSM server.
The critical level is the maximum storage level that you do not want to exceed. In other words, once you reach this critical level, OPEN/stor migrates files to the secondary storage. OPEN/stor continues the migration until you reach the next level (i.e., acceptable) of management. OPEN/stor's default critical level value is 97 percent. This high value helps minimize the risk of a sudden migration of large numbers of files. I think the default value is too high, and I feel more comfortable using an 85 percent critical level.
Setting the acceptable level is very important. OPEN/stor obtains this level automatically when a critical migration (i.e., when the occupancy percentage surpasses the critical level) occurs. The default OPEN/stor acceptable level is 95 percent. For most cases, I recommend setting the acceptable level to 80 percent or lower.
The third level is the optimal level. You can reach this level only by running special jobs known as sweep jobs. You can easily set up and schedule these jobs, but they are not automatic in OPEN/stor when you exceed critical storage volume. OPEN/stor's default value for this setting is 93 percent. In general, I feel comfortable with a 70 percent to 75 percent optimal level (keep in mind that these suggested levels are ideal but might not be practical).
During migration, I've seen CPU utilization jump into the upper 30-percent range. This spike in activity is transient and is the same regardless of the number of available processors. This type of activity shows that OPEN/stor is a well-threaded application that takes advantage of symmetric multiprocessing (SMP) machines.
Selecting OPEN/stor-Compatible Hardware
Before you install OPEN/stor, you need to understand how it works on your
system. You must have at least two SCSI hard disks, but three is suggested. You
install OPEN/stor on disk 1, mirror the database on disk 2, and create at least
three migration volumes on disk 3.
OPEN/stor will migrate files from a smaller volume to a larger volume, but not the other way around. For example, I have a jukebox that uses 1.3GB platters with 650MB per side. I created an extended partition on the hard disk (because the size needed to be less than 650MB) to be the basis for the migration volumes, and I created three 325MB logical disks (Migrate1, Migrate2, and Migrate3) on this extended partition. OPEN/stor immediately saw these migration volumes. Without them, OPEN/stor will not install.
Finally, you need to have a dedicated backup device. All devices must be on the OPEN/stor supported device list (to view this list, go to http://www.eastman software.com/hsm/devices.htm). Make certain that all controllers, backup devices, and libraries you plan to use are on the list. For example, the HP C1533A 4mm device is listed, but the new DDS-3 devices are not (version 2.05 supports these devices). The hardware choice is critical. If you plan to use multiple SCSI controllers (ostensibly, OPEN/stor can manage four, but I have used only two), try to use the same type and model.
Installing OPEN/stor
Initially, I treated the OPEN/stor installation process as being domain
aware and discovered it was not (i.e., I had to install OPEN/stor locally before
I could use it over the network--version 2.1 of the software is supposed to
correct these installation problems). However, I found that the following
installation procedure worked for me all the time.
Log on to an NT server locally (i.e., don't log on to a domain) using an Administrator account that has a password. Create an HSM user and password in User Manager for Domains (you must set up OPEN/stor on a server). For example, I set up OPEN/stor with the username openstor with a password HSM (you must have a password for this account). NT adds the user as a member of the local Administrator's group and Backup Operator's group. You also need to open User Manager, Policies, User Rights and click the Show Advanced User Rights checkbox so you can give this user the right to log on as a service.
Insert the OPEN/stor CD-ROM into the server, and click Setup. I typically use the custom installation features so I can choose settings and see what options are being set up. Imagine that you are installing OPEN/stor on a three-drive SCSI system (you must install OPEN/stor on a SCSI system because all communication is via winaspi). Load the OPEN/stor application on disk 1, the mirrored database on disk 2, and the three migration volumes on disk 3, as previously discussed. The total installation time will be about five minutes. During the installation, be sure you use the username and password (openstor and HSM in the example above) that you just created on the NT server.
The installer will then prompt you to reboot the server. At this point, you can reboot the server and log on to a domain account so you can configure OPEN/stor, which installs as a common application.
Configuring OPEN/stor
For OPEN/stor to run properly, you have to configure drives and tape
devices, you have to register the product, and you have to specify the amount of
storage. When you reboot the server and open OPEN/stor, the system prompts you
to configure the working set. Go to View on the toolbar, and choose Modify
Working Set, as you see in Screen 1. The software asks you to enter the name of
the OPEN/stor server (to establish the working server) and then connects when
you enter the HSM password. Next, you have to click the HSM Properties button
and configure OPEN/stor, as you see in Screen 2.
Clicking the HSM Properties button brings up the Status page, as you see in Screen 3. If you want to add storage (OPEN/stor supports 2GB of storage out of the box), you need to register the software and configure the capacity. To do so, you purchase options from Eastman Software. These add-ons are floppy disks that you use to insert the registration keys. I registered and added 20GB of additional storage (to accommodate my jukebox).
Before you can use OPEN/stor, you will want to install a dedicated backup device. On the HSM Properties window (which by default shows your migrate volumes), click the Devices tab and then click the Backup tab on this page to add the dedicated backup device. You use this device to back up the OPEN/stor databases. Likewise, you will want to add any tape or MO libraries at this time. To add a tape or MO library, click the Libraries tab under the Devices tab and click the Add/Remove button to add the library (OPEN/stor displays the library). Once you set this value, you can highlight the library and click the Properties button to view any robotic arms and drives.
I had some difficulty initially setting up my jukebox. OPEN/stor returned an error 4 message saying HSM services received an unknown error. On the HSM monitor screen, I had to click the second button from the left on the same toolbar as the Properties button, and stop and then restart the service. After I did so, I could set up the library. You don't have to worry about configuring the tapes, because OPEN/stor does this task transparently. The only major components left to configure are the NT Server or NT Workstation systems that you want to manage.
If you click the Servers tab on the HSM Properties window, you see the domain or workgroup name. If you highlight this name and press Enter, you see a list of all the systems on the network.
To add systems that you want to manage with OPEN/stor, you simply add the systems to the managed window, as you see in Screen 4. Before you do this addition, you must boot the system locally and add the HSM user (in my case openstor) to the local Administrator's group and Backup Operator's group, and give the user the right to log on as a service. Failure to do so results in an Error Code 128 error message. Once you have proper rights and have added the system, you have to select the disks (i.e., volumes) that you want to manage. At this point, OPEN/stor copies drivers to the remote system and asks you to reboot the remote system so that new services can start on that system. In addition, OPEN/stor places the avstor.$$$ file on all managed volumes during installation (the avstor.$$$ file is important for the uninstall procedure--for information about uninstalling OPEN/stor, see the sidebar, "Removing OPEN/stor 2.0," page 94).
Testing OPEN/stor
To test OPEN/stor, I set up a primary disk that was 100 percent full, as you
see in Screen 5. I then set the levels to 85 percent (critical), 75 percent
(acceptable), and 60 percent (optimal) on the Management tab of the Managed
Volume window (double-click the managed system, and then click the managed
drive). Obviously, the 100 percent level exceeded the critical level of 85
percent, and a file migration began immediately. Within about 15 minutes,
OPEN/stor had migrated files to the secondary storage, placing the disk at 75
percent capacity (the acceptable level). OPEN/stor had migrated a total of 3751
files occupying 228MB. If you don't set the file criteria properly, OPEN/stor
displays a message telling you it can't work properly. Finally, I conducted a
sweep of the primary disk. Unlike critical migrations, sweep jobs take volumes
to their optimal level. In addition, sweeps go past the necessary files and
pre-migrate files. A pre-migrated file is considered to be inactive but has not
been replaced with a placeholder. In about an hour, the amount of used space on
the drive went from 100 percent to 60 percent. In addition, I had complete
access to all the original files on the disk. From an administrative standpoint,
this reduction, with full function remaining, is very beneficial and powerful.
Creating sweep jobs for the managed volumes is a good idea. Scheduling and running sweeps will optimize future migrations.
You can obviously set up very stringent rules and run specific jobs with a scheduler. With the above setup and configuration, customizing OPEN/stor is easy. You can even be up and running with OPEN/stor in about an hour; just be certain that you understand the files you exclude from storage management and their syntax.
Opening migrated files on a managed system was also easy. The retrieval time on the first layer of secondary storage was very acceptable. Most users probably won't notice any delay in accessing ordinary files (however, this parameter is contingent on file size and type). Also, you can run defragmenting applications (I used both Diskeeper and Speed Disk) and Chkdsk on the managed systems without disrupting the placeholder function.
Is OPEN/stor Right for You?
OPEN/stor is a very attractive HSM package for NT. In many environments,
managing volumes makes sense. For example, systems specializing in photo-editing
can use OPEN/stor to migrate large files to secondary storage, thereby
maintaining primary storage effectiveness. Environments that generate a lot of
daily documents that you might not use again but that you need to maintain can
also benefit from this system.
The application is easy to use, powerful, and fully automatic once you have it set up (for information about some useful tricks to use with OPEN/stor, see the sidebar, "Important Factors to Know," page 94). With a primary backup application, you also have added file protection and easy restoration. Backing up and restoring 420byte files is much easier than doing the same with larger data files.
Once installed, OPEN/stor functioned as I expected with no disruption to the active managed systems. If your environment is conducive to disk management, OPEN/stor will clearly fit your needs. Its power and use is impressive, and Eastman Software's technical support was prompt and helpful. The online Help is also excellent.
| OPEN/stor 2.0 |
|
Contact: Eastman Software Storage Management * 800-962-8245 Web: http://www.eastmansoftware.com |
| Price: Free evaluation CD-ROM for up to 2GB of data or Registration Kit (starting at $2995) and Capacity License (starting at $650) for additional storage |
| System Requirements: Windows NT Server version 3.51 or 4.0, One or more host bus adapters for the secondary storage SCSI devices, 100MB server space for application installation and OPEN/stor databases, Up to 2GB secondary storage used for evaluation CD-ROM, CD-ROM drive, Supported tape backup device (optional) |
