The fundamentals of benchmarking are the three Rs: relevance, robustness, and repeatability. How many times have we seen the results of a benchmark and said, "What the heck is a Hellmark?" If the number is higher, is that better or worse? Who is Erastosthenes anyway, and what does a Sieve have to do with reality? Unless you're a theoretical mathematician, not much. We believe it's important to use real-world applications in everyday situations to test performance.
Windows NT brings new challenges to benchmarking: 32-bit programs, preemptive multitasking, threads, security, multi-CPU support cross-platform compatibility (Intel, PowerPC, MIPS, and Alpha), and more. The old 16-bit test suites for DOS/Windows won't cut it anymore. Simply porting them to Windows NT won't answer the burning questions: What does Windows NT uniquely offer to the computing industry? Is Windows NT scaleable? Is RISC faster than Intel on NT? Does it run on all platforms? These are the questions we are being asked. Fortunately, the answers are being worked on.
BAPCO has announced SYSmark for Windows NT, a workstation benchmark that rlms a set of Microsoft Test scripts against popular desktop applications: Word, Excel, PowerPoint (16-bit), Welcom Software's Texim Projec@ and Orcad MaxEDA (PCB design tool). In addition, BAPCO is working on a version that will test the file and print capabilities of Windows NT Server using the SYSmark clients.
On the database server side, the TPC Transaction Processing Council) benchmarks are the current leader. TPC-C tries to model a moderate-to-complex OLTP system based on an order entry system. Transactions are submitted from UNIX dumb terminals that interact with nine tables on the server database (approximately 25OGB) and are measured in transactions per minute (tpm). TPC-D benchmarks decision support by using a database stress test and complex queries.
Client Server Labs in Atlanta, Georgia, has announced an independent test suite that runs three simultaneous loads on Windows NT Server. The test is based on three benchmarks: a modified TPC-C using PCs instead of dumb terminals, TPC-D, and BAPCO (file serving). The relative performances of these three workloads are combined into a single measurement called an RPMark.
At Windows NT Magazine, our strategic direction is to adopt or create benchmarks that satisfy the three Rs. However, we don't want to invent these tools in a vacuum. We have invited representatives from the major chip vendors (Digital, Motorola, Intel, MIPS) and database vendors to work with us in developing these benchmarks. I invite you to participate with us in this challenge. Please let us know the types of questions you would like answered by benchmarks and the types of real-world measurements you would like to see used. You can email your comments to: firstname.lastname@example.org.