Demand for increased capacity, speed, and resources plagues every systems administrator. Network response time can slow to a crawl just because you add a few workstations to your network. Unexpected side effects (e.g., slow response times, incompatibility with existing software, a system crash) can pop up after you introduce a new software product. When you deploy new hardware, you face issues such as whether the hardware will perform as advertised and whether everything in your network will work properly once you deploy it.
Dynameasure Enterprise 1.5 by Bluecurve replaces guesswork with hard data so that you can deploy new hardware and software applications with confidence. Bluecurve's software measures end-to-end system performance from the application layer by applying controlled stress to the system. Systems administrators can use this flexible tool for server benchmarking, capacity planning, stress testing, and network analysis. Bluecurve geared Dynameasure 1.0 (see John Enck, "Dynameasure by Bluecurve: Born to Measure," November 1996) toward performance testing a Microsoft SQL Server environment using client/server hardware and software components. Bluecurve's Dynameasure 1.5 product line includes Dynameasure for SQL, Dynameasure for File Services, and Dynameasure Enterprise.
Dynameasure for SQL applies stress to the network via simulated OnLine Transaction Processing (OLTP) and SQL decision support workloads. This test service is an updated version of the SQL test service that Bluecurve provided in version 1.0. Dynameasure for File Services, new in version 1.5, measures Windows NT file services performance. This test service lets you apply copy workloads, which simulate copying files to and from a server; backup and retrieval workloads, which simulate backing up a client machine to a server and retrieving files from a server; and network application workloads, which simulate loading apps onto a client machine from a server and uploading and downloading files. Dynameasure Enterprise provides both test services, but because the Lab reviewed the SQL component in Dynameasure 1.0, I'll primarily report on the new File Services component in this review.
Dynameasure for File Services lets you create what-if scenarios of real-world network file operations. From more than 50 predefined file operations that clients typically perform, you choose one or more operations and create a simulated workload to meet your test objectives. (You can even model new users to gauge how they will affect network performance.) Each file operation uses one or more data, text, image, or binary file to perform tasks such as copying images to the server, backing up a client machine, and loading a client program. Dynameasure uses your existing network client configuration to run motors that generate client load. The motors coexist with resident client applications, so you can test your everyday operating environment while end users keep working.
Putting controlled stress on the network and assessing performance let you pinpoint server, network, and client bottlenecks. You can (and should) customize the workloads to construct scenarios that provide meaningful information specific to your network. Tailoring the file operations and analyzing the results can help you decide whether you need more CPUs, memory, disk space, or network bandwidth.
Dynameasure Preliminaries
To run Dynameasure tests, you must conFigure four components: a control
client, a control server, at least one test client, and at
least one test server. The control client, running either Windows NT or
95, executes an agent program known as the Manager. The Manager is the interface
through which you set up, run, monitor, and analyze the tests. The control
server hosts databases that contain the test data and store the test results.
Test clients obtain the test parameters from and record their results to the
control server. For most tests, you will use several test client systems. Each
test client, running Win95 or NT, executes an operator program. Each operator
can start 25 motors (or 100 motors with an additional licensing agreement); each
motor generates the workload of one user. The final component, the test server,
is the focal point of the test: On the test server, you load the test data
set, the data that the test clients access during the test run.
Bluecurve (and the Windows NT Magazine Lab) recommends that you use separate computers for the control server and the test server. Dynameasure stores the transaction results from the tests in a database on the control server. With separate machines, the I/O load of updating the transaction results database does not influence the I/O load on the test data set.
Bluecurve supplies a well-written user manual that takes you through a logical progression of what the product is and how to use it. Screen shots and diagrams that describe the architecture and components of Dynameasure supplement the discussion of concepts and definitions. You can find complete instructions for the installation and use of the different components, even step-by-step instructions for running a File Services sample test. I highly recommend you thoroughly study Chapter 9, "Testing Strategies," before you try to use the product. The chapter details how to locate bottlenecks, the characteristics of workloads, and how to select the data set and tests to provide a meaningful performance evaluation. Another must-read chapter is Chapter 8, "Analyzing Test Results." Once you've run a test, this chapter will help you sort, filter, and view the results. You'll also find out how to print reports to help you zero in on network problems.
The software includes online Help files that follow Microsoft's format. The Dynameasure Demo CD-ROM contains technical data and a new user tutorial, which I found useful. The tutorial provides an excellent explanation of the software components, and it steps you through configuring and running a successful test.
On Your Mark, Get Set...
Bluecurve has improved Dynameasure's installation process in version 1.5.
The setup program walks you through the installation process for the control
server, the control client, and the test clients. If Dynameasure doesn't detect
the Microsoft Open Database Connectivity (ODBC) drivers, it will prompt you to
install the drivers from the installation CD-ROM onto the control server, the
control client, and every test client. On the control server, Dynameasure uses
Microsoft SQL Server to create, store, and access test data objects. If SQL
Server is not present on the control server, Dynameasure installs it. If you
plan to run Dynameasure for SQL, you must install SQL Server (or Oracle 7.3) on
the test server to support the test data set. If you plan to run only
Dynameasure for File Services, you do not need SQL Server on the test server.
(For details about how we set up the Lab's test environment to run Dynameasure,
see the sidebar, "The Lab's Testbed.")
After completing the installation, I started the control client's Manager. The Manager contains three components: the Dispatcher, which you use to select, manage, and monitor the tests; the Builder, which you use to build and maintain the test environment; and the Analyzer, which you use to manage and analyze test results. I invoked the Dispatcher, which displays the window shown in Screen 1. Test clients appear in the Dispatcher window after you load the operators onto the workstations and start them. Test servers appear in the Dispatcher window after you register them with the Manager. The color of the test client and test server icons represents resource status: Blue signals Ready, red signals Trouble, Not Reporting, and green signals Active Testing.
Before you can run a test, you must build the test data set to be transmitted across the network between the test server and the test clients. From the Manager, click Builder to display the window shown in Screen 2. In this window, you define the type of data set to build, specify where to store it, and execute the build. Because my goal was to review Dynameasure for File Services, I selected File Test Datasets as the type of data set I wanted to build. In Screen 2, you can see that the test will use a collection of data, text, image, and binary files.
Dynameasure supplies SQL and File Services data sets on separate CD-ROMs. The CD-ROMs provide preset scales, which you can load onto the test server using the Builder option of Manager. Scaling affects the amount of data loaded in the test data set; you can adjust the data set scale to fit your available disk space and satisfy your testing requirements. For the example in Screen 2, the scale 1.0 data set has size 593MB. Changing the scale to 0.01 reduces the size to about 5.9MB, and scaling to 10.0 increases the size to about 5.9GB. (Bluecurve's hard disk space requirements for File Services test servers are 10MB for scale 0.01 data sets, 610MB for scale 1.0 data sets, and 6.1GB for scale 10.0 data sets.)
Next, from the Dispatcher window, I clicked Add Test and selected the Copy All Bi-directional File Services test. This test will copy compressed and uncompressed data, text, and image files between the server and clients. The Queue section of the Dispatcher window displays the test you select. Double-clicking the test displays the File Test Specifications window, as shown in Screen 3.
On the Target tab, you select the test server and specify the path for the data set. On the Steps tab, you specify the number of steps (from one to six) for the test. The Steps option lets you gradually increase the number of motors (not exceeding your license agreement) that the test applies to the system. The Lab license allowed a maximum of 100 motors, so I chose six steps, with the number of motors increasing in this order: 10, 20, 40, 60, 80, 100. On the Times tab, you can designate the total run time for each step of the test. For the Lab's test, I set the time to 300 seconds. On the Settings tab, you can set additional execution parameters. For example, I set Think Time (the number of seconds between completing a transaction and starting the next transaction) to 5 seconds, as shown in Screen 3.
Dynameasure uses a mix of transactions for each test. On the Weights tab of the File Test Specifications window, you can adjust the weight of each type of transaction included in a test. If you set a transaction's weight to 0, Dynameasure will not execute the transaction during the test. Increasing a transaction's weight results in that transaction executing proportionately more times than the other transactions in the test. On the Details tab you can change transaction configuration parameters such as copy direction, block size, and file scale. On the Schedule tab, you can specify a time to run the tests, such as after workday hours or on the weekend. After you complete all the settings for your test, you save the test specifications to the queue.
Go!
To initiate the test in the queue, I clicked Start Queue in the Dispatcher
window. In the pretest phase, the operators running on the test clients generate
the motors. In the Clients section of Screen 4, you can see the blue squares
that represent the motors, displayed with the test client icons. During a test,
the motor icons change color to signal status: blue for Ready, yellow for
Testing Passive, green for Testing Active, gray for Dropped Out, and red for Not
Reporting.
In the bottom section of the Dispatcher window, you can monitor the test's progress. In Screen 4, you can see the phases for each step, the elapsed times, and which step is being tested. The Results tab charts the test results at the end of each step, as shown in Screen 5. This feature lets you closely monitor each stage of the test. I found this monitoring capability very helpful in two situations. In one situation, I saw that the network had reached the maximum Bytes Per Second in the first few steps. I stopped the test and reduced the number of motors for the next test run. In the second situation, I saw that the test was not going to reach the maximum Bytes Per Second, so I added more motors on the next test runs.
In Screen 5, you can see a few red motors, signaling that the operator and motors stopped reporting. I knew I had a problem when red became the most prevalent color on the screen. I started losing operators and motors in domino fashion. The test ran to completion, but the results were not what I expected. I ran the test a dozen times, with similar results (see the sidebar, " Analyzing Test Results," for details about Dynameasure's test analysis features). The results revealed that somewhere between 27 motors and 38 motors, the average response time per transaction more than doubled, the number of bytes transmitted per second across the network fell sharply, and the system could not support more than an average of 56 users. In a commercial business network, these results would imply that if the business expanded from 30 users to 40 users, file operations would slow considerably. I immediately thought this situation couldn't be right because I'd used the same network configuration to generate hundreds of client workloads using Dynameasure for SQL tests. I changed data set parameters such as scale and Think Time, but the results were much the same.
After spending more than enough time studying the situation, I called Bluecurve's technical support team. The representative was extremely helpful and stated that the Lab's network configuration should be able to handle a much bigger user workload. The representative suggested an approach right out of the user manual: Develop a plan first. Because Dynameasure Enterprise was a new release of familiar software, I had given the user manual a cursory glance, accepted the defaults for configuration, and run a test I picked at random. I had not defined what area of the network I wanted to stress (clients, server, bandwidth) or decided whether I was testing CPU usage, RAM, hard disk performance, or new hardware or software. Because the test results were not what I expected, I blamed the software. Instead, I should have read the manual and developed a strategy based on what kind of baseline I wanted to achieve. I simply wanted to measure network throughput under a workload of 100 motors. After I identified what my baseline was going to measure, I defined the performance testing strategy.
Bluecurve advised me to start with a small-scale data set and a minimal number of motors, and gradually increase the scale or the number of motors to find the network bottlenecks. So after reading the user manual, I proceeded to run the same File Services test: Copy All Bi-directional. I created a 0.01 scale data set, increased the Think Time to 10 seconds, changed the File Test Specifications to only 30 motors (one for each workstation), and conFigured only one step instead of six.
The test results did not improve: During pretest, operators and motors again turned red, signaling they were not reporting. However, now I had a better understanding of the information Dynameasure could give me. Besides monitoring the progress and results in the Dispatcher, I used other Dynameasure diagnostic tools. For example, Dynameasure logs every action the motor performs. I identified a red motor in the Dispatcher and then went to that machine and viewed Motor Details, as shown in Screen 6. From the Log, I determined that the motors were not being turned on during the pretest, which caused the Not Reporting status. I also observed that all the red client machines connected to the same repeater. The NT Event Log showed redirector errors for the problem machines.
Then the Miracle Occurs
I took the next logical action--I turned off the suspect client machines. I
was amazed with the next series of tests. The remaining 23 clients had no
problem running the test with 23 motors. Next, I gradually increased the number
of motors to the Lab's 100 limit. Screen 7 shows the results the Analyzer (the
third component in the Manager) displayed. The results show that the network
configuration could support 100 users applying the workload in the Copy All
Bi-directional test. I changed the Cogent S-1200 repeater that the problem
machines connected to, and all 30 operators started and displayed a blue Ready
status in the Dispatcher. All subsequent tests ran with 30 operators providing
100 motors.
As you can see, Dynameasure worked properly from the beginning. Had I followed a performance strategy and properly examined the test results, I would have saved myself days of anguish. Dynameasure found a hardware bottleneck in my network and let me correct it. Mastering Dynameasure takes time, but the tools and data resources make network analysis a breeze.
Screen 8 displays performance results for three client-to-server file operation tests that I ran on the Lab's network. The Motors per Step graph shows that the network can support at least 100 users. The Bytes per Second graph compares the transaction performance of individual file operations and charts how the network would react to a given number of users executing the same file operation. For example, the green line represents throughput performance for the Copy Large Data to Server test. After Step 3, system performance degrades. The Motors per Step graph shows that 40 motors completed the file operations in Step 3. Thus, adding more than 40 users to the network will decrease system performance.
The more I used Dynameasure 1.5, the more I liked it. Armed with the user manual's appendix, any systems administrator can select and modify tests to mimic typical user actions on the network. With customized tests, a systems administrator can identify where bottlenecks will occur, for which type of transactions performance will degrade, and the network's user capacity. This product definitely performs as advertised.
| Dynameasure 1.5 |
|
Contact: Bluecurve * 510-267-1500 Web: http://www.bluecurve.com Email: sales@bluecurve.com Price: $29,995, Dynameasure Enterprise (includes 1 Manager license, 100 test client motors for both SQL and File Services, 1 data set for SQL requests, and 1 data set for File Services); $2495, Dynameasure for File Services (includes 25 File Services test clients); $4995, Dynameasure for SQL (includes 25 SQL test client motors) System Requirements: Test server and control server: Pentium processor*, 32MB of RAM, NT 3.51 or higher, SQL Server 6.0 or higher or Oracle 7.3 Server Control client and test clients: Pentium or 486-type processor, 16MB of RAM, NT 3.51 or higher or Win95, 32-bit ODBC SQL client version 2.5 |
