As many enterprises begin to consider migrating to Vista, recent news has added some drama to an already complex issue. In late November, Devil Mountain Software tested pre-release versions of Windows Vista SP1 and Windows XP SP3 with OfficeBench, a testing script that utilizes Microsoft Office to gauge system performance. During their tests, Devil Mountain found that Vista SP1 offered no performance improvements over Vista RC, but discovered that XP SP3 was more than 10 percent faster than XP SP2. Now here's the important bit: In the same battery of tests, XP SP3 was also found to be measurably faster than both Vista RC and Vista SP1.

Should Beta Software Be Benchmarked?

Microsoft Vista Product Manager Nick White commented on the test results, writing in his blog that benchmarks of pre-release software aren't the best indicators of software performance, as ongoing development work could invalidate any test. "Publishing benchmarks of the performance of Windows Vista SP1 now wouldn't be a worthwhile exercise for our customers, as the code is still in development," said White. "Further, tests like these only measure a very small set of Windows capabilities and so aren't representative of the user's overall day-to-day experience of working with Windows and running applications."

Director of Devil Mountain Software Randall C. Kennedy reacted strongly to the criticism, calling White's description of his testing as a "blatant mischaracterization" and saying that White's criticism was a "Microsoft hit piece ordered from on high." Kennedy argues that OfficeBench does a more rigorous battery of tests than White claims, and asserts that the Devil Mountain tests are a good indicator of overall system performance.

In Defense of OfficeBench

"OfficeBench opens up three applications \[Microsoft Word, Microsoft Excel and Microsoft PowerPoint\] and then loads documents, scrolls through pages and does extensive document formatting," explains Kennedy. "It reformats sections headers and subheads, pastes text between Word and Excel, creates a multi-slide presentation in PowerPoint, does a global search and replace, does a print-to-file in PowerPoint, then loads up IE and simulates some web searching."

For Microsoft, news of XP SP3 faring better in testing than Vista SP1 may have come at an inopportune time. Despite numerous Vista success stories,   corporate adoption of Vista is lagging behind that of Windows XP, and both hardware ISVs and consumers have pressured Microsoft to continue offering Windows XP on new computers.

Microsoft's View

A Microsoft PR representative declined to comment further on the story, but did point out a blog post by Microsoft MVP Robert McLaws for some additional perspective.

McLaws and Kennedy agree that enterprises basing their Vista migration plans solely on the promise of improved performance with SP1 will have little reason to upgrade. In his blog post defending Vista SP1, McLaws points to the success of XP SP1 and the networking improvements in Vista SP1 that Kennedy's test didn't cover. "As I recall, XP SP1 didn't feature any terribly big performance gains, it was all about fixing the stuff they didn't have time to fix before they shipped," McLaws writes. "Vista SP1 is mostly the same way, but has the added benefit of receiving the improvements from the Windows Server team as well, hence the networking improvements."

Most IT administrators would agree that performance improvements aren't their only concern, and Vista SP1 does introduce a host of important fixes that should improve upon the reliability and functionality of Vista. "Any organization fence-sitting \[and basing their migration to Vista\] on the performance issue won't get what they want...Vista SP1 is simply slower than XP SP3 at a lot of things," says Kennedy. "What Vista SP1 does offer is improved compatibility, lots of bug fixes, and improved reliability. Those are important benefits, and may be enough for some people to migrate to Vista. Improved performance \[over Windows XP\] just isn't one of those benefits."

Will the improved performance of Windows XP SP3 make you delay a migration to Windows Vista? Or do the planned fixes for Vista SP1 make your migration to Vista an easier job to handle? Add a comment to this post and let us know what you think.