Quest Software's Benchmark Factory 2.5 is a highly scalable workload generator for database, Web, mail, and file server applications. You can use the product to load-test an application for performance tuning, expose and diagnose application reliability problems, and determine the maximum throughput that a hardware configuration will support.
You license Benchmark Factory by application driver. Twelve drivers are available—File Server, Web Server, DB2, Informix, Microsoft SQL Server, Oracle 7, Oracle 8, Sybase, ODBC, Messaging API (MAPI), IMAP4, and POP3. The Professional edition includes the Benchmark Foundation Classes, which are benchmark components that you can use with Microsoft Visual C++ to write custom benchmarks. The license key that Benchmark Factory includes unlocks the features you have licensed. I used a key that unlocked all drivers and benchmarks but not the Benchmark Foundation Classes.
Installing Benchmark Factory
The minimum hardware requirement for Benchmark Factory Agent machines and the Visual Control Center (VCC) machine is a Pentium 90 processor, 32MB of RAM, and Windows 2000, Windows NT, or Windows 9x. Quest Software recommends a Pentium II processor running Win2K or NT 4.0, with 32MB of RAM for Agent systems and 64MB of RAM for the VCC machine. Benchmark Factory supports TCP/IP, the IPX and SPX protocols, and any network that supports Winsock.
I installed Benchmark Factory from a CD-ROM to a 450MHz Pentium II Compaq running NT 4.0 Service Pack 5 (SP5). After sharing the Netinstall directory that the installation process created, I ran Setup from the directory to install the Agent to a Win2K Professional-equipped client on the network.
Benchmark Factory has three primary components: the VCC, which runs benchmark tests and manages results; a runtime version of Sybase's Adaptive Server Anywhere, which stores benchmark results; and the Benchmark Factory Agent, which runs on one or more computers and generates a workload that simulates one or more users per Agent. You need to install Microsoft Internet Explorer (IE) 5.0 to operate the VCC. If you want Benchmark Factory to write test results to a Microsoft Excel spreadsheet, you need to install Excel on the VCC computer. The installation routine doesn't check for the presence of IE 5.0 or Excel, so you need to install those programs first to avoid problems with Benchmark Factory.
Benchmark Factory's installation routine is easy to follow and contains several standard installation scenarios and a Custom option. The installation process lets you choose locations in which to install Benchmark Factory and the results repository that Adaptive Server Anywhere accesses. If you need to collect detailed transaction statistics during testing, the repository can quickly grow to several hundred megabytes or more. Benchmark Factory uses ODBC to write to the repository, so you can use another database, such as SQL Server, in place of Adaptive Server Anywhere.
Benchmark Factory supports multiuser testing, which uses networked Agents that the VCC controls, and single-user testing, which runs directly from the VCC. To use Benchmark Factory, you create a project in which you choose a benchmark type (e.g., TPC-C or File Server). Each project contains at least one predefined test and is the framework for customizing how you want each test to run. Each test has one or more transactions, which are discrete test units (e.g., reading an email message, viewing a Web page). Transactions are static or dynamic. Static transactions reside in the benchmark's .dll file and have fixed functionality. You can define dynamic transactions (e.g., executing a SQL query), and you can use one or more static or dynamic transactions to create a multiuser or single-user test. A Benchmark Factory Profile points to the server you are testing, which Benchmark Factory refers to as the System Under Test (SUT).
When you run a multiuser test, Benchmark Factory passes the test parameters to one or more Agents, which run test transactions repeatedly during a time period that you specify. In contrast, the VCC runs single-user tests without using an Agent. The VCC runs a single-user workload a fixed number of times.
To run a test, place it in the Run Queue and schedule the test to run immediately or at a later time. Jobs in the Run Queue run one at a time in the order you place them in the queue. While a job is running, you can use the VCC to view the job's progress. When a job finishes, you can use the VCC to view the results or have Benchmark Factory export the results to an Excel spreadsheet.
I ran a TPC-C workload, a transaction-processing test that is one of Benchmark Factory's standard database benchmarks. The test has no connection to the Transaction Processing Performance Council's (TPC's) TPC-C test.
I used Benchmark Factory's New Project Wizard to base a project on the TPC-C workload. Then, I used the New Profile Wizard to give Benchmark Factory the information that the program needs to run the test against a specific server. Information that the program required for the TPC-C test included the server name, the database driver for the test to use (in this case, SQL Server), the database for the test to use, and the User ID (UID) and password for the Agents to use to authenticate their access to the database.
After the New Profile Wizard was completed, the New Project Wizard regained control and displayed the Scale Selection screen. Scale sets the test size. For database testing, Scale specifies the database size in rows and controls the number of rows that Benchmark Factory generates to fill the test database. Scale also controls the range of key values that the test queries.
The New Project Wizard checked the server for the necessary database objects and created jobs to delete, recreate, load, and index the tables. Figure 1 shows the VCC as these jobs were running. The Run Queue is in the screen's upper-right pane, a list of transactions and benchmarks appears in the screen's upper-left corner, and a Results summary appears in the pane at the bottom of the screen. After the database creation jobs finished, I had completed the basic preparation necessary to run a test.
Letting the New Project Wizard create database objects that you need for testing is a very handy feature of Benchmark Factory. However, when you create large databases, the wizard's database creation jobs can take a long time to run. To speed this process, Benchmark Factory has a Data Generator tool that creates each table's data in a flat-file format. Programs such as SQL Server's bulk copy program (bcp) can use the file to bulk-load the tables at high speed.
Creating a Benchmark Test
For each driver, Benchmark Factory includes at least one predefined test. The TPC-C workload contains the TPC-C Transaction Mix test. To customize how the test would run, I right-clicked the test in the VCC and selected Properties. The Properties screen of every benchmark test has six tabs—General, Transactions, User Load, Latency, Timing, and Options. The General tab lets you type a test description and shows when you last modified the test.
The Transactions tab lets you specify what percentage of the test will consist of each type of transaction. For example, if you simulate the workload of a company that processes most payments between 4 p.m. and midnight each day, you might set Benchmark Factory's simulated workload to include 30 percent payment-processing transactions during that shift. During simulation of another shift, you might set payment-processing transactions at 2 percent of the workload.
The User Load tab specifies how test iterations will run and how many virtual users will take part in each test iteration. By running multiple test iterations with different numbers of load-generating users, you can create a throughput curve for the application. This curve shows the maximum throughput (transactions per second—tps) the server will process, the transaction response time at each load level, and the degree to which performance decreases under increasing loads.
Latency dictates how long each virtual user waits between transactions, in effect determining how hard each virtual user will work the SUT. The Latency tab contains several options for setting fixed- or random-length delays before and after each transaction is executed.
On the Timing tab, you set the duration of each of the four benchmark iteration intervals. The test begins with Quiet Time, which is a waiting time before each iteration. Quiet Time lets server activity from the previous iteration stop before the next iteration begins. During Ramp Up, each virtual user begins running transactions that the test prescribes, but Benchmark Factory doesn't report those transactions as a part of the test results. During the Execution Time interval, Benchmark Factory measures and reports results from all transactions. During Ramp Down, virtual users continue to begin new transactions, but Benchmark Factory doesn't record the transactions. Ramp Down ensures that the SUT's workload continues unabated while the transactions that began during Execution Time are completed.
The Options tab, which Figure 2 shows, lets you tell Benchmark Factory the name of any script you want the program to run before each test iteration. On the Options tab, you can also tell Benchmark Factory what timing data to collect during a test, and you can set several other control options.
The ability to automatically capture Win2K and NT performance-monitoring counters during testing is one of Benchmark Factory's most useful and convenient features. When a bottleneck in your system's information flow occurs during testing, you need to know where the bottleneck occurs, such as in CPU utilization or within the I/O subsystem. To tune your system's performance and eliminate the bottleneck, you need to watch performance-monitoring counters. In Benchmark Factory, you can select specific counters that the program will automatically record during an iteration's Execution Time.
Benchmark Factory's performance-monitoring function would benefit from improvement. The program maintains a list of server-specific counters and organizes the counters according to test class (i.e., database, Web, mail, file server). When I need to run the same tests against a different server, I must select counters from the new server and add them to the product's original list. I'd prefer to have the ability to create named sets of server-independent performance counters (e.g., a set for monitoring SQL counters, a set for Exchange counters, a set for basic NT or Win2K counters). Within the profile that points to a specific SUT, I'd like to have the ability to select one or more sets of counters that Benchmark Factory would monitor whenever a test uses that profile. This feature would greatly reduce the effort necessary to monitor the specific counters that any particular test requires.
Running the Benchmark
To run a full test, I started the Benchmark Factory Agent program on each of the test machines and waited for them to connect to the VCC. I checked the TPC-C Transaction Mix test properties and dragged the test to the Run Queue screen, causing Benchmark Factory to display the Queue Job Properties screen. The General tab on this screen let me add a comment to document SUT configuration details for the test. Other tabs on the Queue Job Properties screen let me schedule the job to run at a specific or recurring time and instruct the program to use email or a pager number to notify someone when the test completed. Figure 3, page 140, shows the Agent Status view on the VCC while the test was running. The screen shows which Agents were participating in the test iteration, how many virtual users (Threads) each Agent simulated during that iteration, and a summary of throughput for each Agent.
After the test was completed, I accessed test results by right-clicking the test name in the VCC's upper-left pane and selecting Run History from the context menu. The Run History screen lists every test execution that the Results database records. I double-clicked an execution, and the program displayed a Windows Explorer-like report that listed information categories on the left and a display pane for the selected category on the right. I clicked the Excel Reporting button, and Benchmark Factory converted the report data to an Excel spreadsheet, which is the easiest way to view, print, and save test results. The spreadsheet has tabs for General test parameters, Client Configuration, Results Summary, and Realtime Summary (i.e., performance counter). The spreadsheet also has a tab for each iteration of the test; these tabs show Agent- and transaction-level results.
Proper benchmarking requires that you run a test several times, each time using a different user load. To compare the results from each load level, you need to restore test conditions to a known state before each iteration of the test. For a database test, you need to restore the database to the beginning state and restore the database server's cache to a uniform initial state. Although Benchmark Factory does not support this restoration, the program lets you run a command file or script before each iteration of a test. By creatively using Microsoft Windows NT Server 4.0 Resource Kit utilities, you can automate almost any task to restore the test server to a known initial state. For information about using resource kit utilities in benchmark tests, see "Restoring the Database During Benchmark Testing," October 2000.
Testing a Web Site
Benchmark Factory has several features (i.e., Web site spidering, record and playback, user sessions) that let you create complex Web site tests. The program spiders a Web site by tracing links, creating a transaction for each link that works properly, and reporting broken links. Spidering also lets you import and create test transactions from a live Web site. I tested spidering against large and small Web sites. When Benchmark Factory used a VCC that had 256MB of RAM to spider a large Web site, the program ran out of virtual memory and ended the test prematurely after locating about 1000 links. After I limited the spidering depth, the program successfully spidered the site, creating more than 800 transactions and discovering 11 broken links. The small Web site that I tested Benchmark Factory against required an initial logon, which the spidering function couldn't handle. Thus, the program couldn't traverse the site.
To use the record and playback features, you fill in Web application forms and move from page to page while Benchmark Factory records each action as a transaction within a user session. A user session is a list of transactions, including form input, that you can run as either a single-user test or as a transaction in a multiuser benchmark test.
I initiated record and playback features by selecting New from the User Sessions context menu. After I entered a starting URL, I found that the controls are similar to a tape recorder's controls. I clicked Record, and my subsequent Web browsing, including what I typed into Web forms, became new transactions. I clicked Stop to end recording, then Play to review the Web pages that made up the user session. Benchmark Factory let me drag the new transactions into other tests.
Documentation and Support
Benchmark Factory ships two printed manuals with the product. Benchmark Factory Getting Started includes installation instructions and a tutorial that walks new users through setting up and running each of the four basic benchmark types. However, several new users in the Windows 2000 Magazine Lab found the manual difficult to follow, and in some cases, out-of-date or incorrect. Benchmark Factory Reference Guide contains general information but highlights creating dynamic transactions to customize benchmarks and using the Benchmark Foundation Classes with Visual C++ to write your own custom benchmarks. This manual, revised in July 1998, needs updating.
Benchmark Factory also includes a large online Help file. This file is an excellent reference and includes much information that the printed manuals don't contain. Although the online Help file often answered my questions, its index didn't include references I expected to find. For example, when I wanted instructions to use the Web site spidering feature, I found a description of relevant parameters, but no overview about how to use the feature. The Benchmark Factory technical support staff was knowledgeable and responsive.
I tried to create a decision-support database-oriented test similar to TPC's TPC-D, TPC-H, or TPC-R test, but Bench- mark Factory couldn't do what I needed it to do. Benchmark Factory reports throughput rates for multiuser transaction-oriented tests but can't measure how long it takes for several virtual users to complete a fixed set of tasks. I wanted to test how long NCR's Teradata decision-support database system takes to process a set of concurrently running workloads. I had a set of query transactions that I wanted each of several virtual users to run in a random order, but found no way to do this in Benchmark Factory. Benchmark Factory could accommodate this type of test if the program extended the concept of a single-user test—which can run a user session a set number of times—to run as a multiuser test and provided a facility to let the transactions that make up a user session run in random order.
The Windows 2000 Magazine Lab commonly uses 50 or more computers as load-generating Agent systems on Benchmark Factory. You can use the VCC to see each Agent machine's status and to close the Agent software, but the mechanics of administering a large number of Agent machines is cumbersome without some additional level of automation. I use Altiris' LabExpert to control some or all of the Agent machines in a test. LabExpert's ability to reboot Agents and execute a local resident command (e.g., the resource kit's Logoff utility) on a group of Agent computers saves time in the Lab.
You can run the Benchmark Factory Agent as a Win2K or NT service and remotely start and stop the Agent service. By default, the Agent service runs under the local system account, which the Agent service can't use to authenticate access to other computers. As a result, the Agent service fails to run File Server tests and some mail tests. So, I configured my Agent machines to automatically log on to a particular domain user ID, and I routinely start the Agent from a Logon script for that user. Benchmark Factory needs to expand the tools it provides for Agent and Agent computer administration.
A Highly Scalable Tool
Benchmark Factory is a highly scalable tool that can help you address a broad range of problems that relate to performance and capacity planning. To simulate more complex applications, the optional Benchmark Foundation Classes library supports custom Visual C++-based benchmark development.
Benchmark Factory's standard tests aren't difficult to run. However, partly because of Benchmark Factory's dated documentation and the product's variety of features, you'll need time and practice to become proficient with the program's more advanced applications. Smaller organizations might consider Benchmark Factory expensive, but in today's e-commerce world, the cost of poor application performance and application downtime can be even higher.
|Benchmark Factory 2.5|
| Contact: Quest Software * 314-523-4400|
Price: Starts at $6000 for one test driver with 50 virtual users; additional test drivers are $3000 each, and adding 50 virtual users costs $600
Pros: Generates scalable workloads for database, Web, mail, and file server applications; includes many standard benchmark tests and support for writing custom tests; saves test results to an Excel spreadsheet
Cons: Needs better written documentation; lacks ability to perform decision-support database testing; lacks tools to manage raw results database; lacks tools to control Agent systems