Deficient IBM and Hitachi 120,000 mailbox configurations

You might think that configuration information published by hardware vendors reflect good technical practices. You might, but IBM and Hitachi both prove you wrong because their ESRP documents describing configurations to support 120,000 Exchange 2013 mailboxes are laughably bad.

Reading the Exchange Solution Reviewed Program (ESRP) page for Exchange 2013, I was taken by the claims advanced by IBM and Hitachi in documents that purport to describe storage solutions capable of supporting the predicted I/O workload generated by 120,000 Exchange 2013 mailboxes. My jaw dropped as I read the documents because both solutions are radically inadequate.

Let me explain why I should use such strong words. The IBM solution uses 24 servers in two Database Availability Groups. Nothing wrong here until you read:

“This solution utilizes Microsoft® Exchange 2013 Mailbox Resiliency with database availability groups (DAG). A two DAG solution comprised of 24 mailbox servers was created that supported a total of 120,000 mailboxes with a mailbox size of 2GB. Two XIV Gen3.2 frames were used, and the databases and copies were equally distributed across them. Each server hosted 5,000 users, and had 12 active databases, with 833 users per database. Within the DAG, there were two copies of every database; one local, and one on another server connected to a second XIV Gen3.2 storage array. This configuration can provide for both high-availability, and disaster-recovery scenarios.”

The problem stares us in the face: each database has just two copies. Why, I ask myself, would I want to deploy a solution that supports so many users in a configuration that exploits Exchange’s inbuilt high availability features without supplying sufficient capacity to protect the databases against failure? After all, a 12-node DAG has tons of potential to support more than two copies per database. In fact, you could have up to 12 copies (a tad excessive) of each database if you wanted to deploy sufficient hardware. Three (or even better, four) copies of each database would be much more reasonable for anyone seeking true high availability.

The core problem in IBM’s configuration is that the servers are deficient in memory and storage. In fact, they have just enough to be able to run the JetStress tests to validate that the configuration can support the theoretical I/O load that might be generated by 120,000 simulated users. Not enough storage capacity or RAM is available to support sufficient database copies to “provide for both high availability and disaster-recovery”.

Bad as the IBM configuration is, I think the Hitachi configuration is much worse. The introduction proudly proclaims:

This solution includes Exchange 2013 Mailbox Resiliency by using the database availability group (DAG) feature. This tested configuration uses twelve DAGs, each containing twenty four database copies and two servers (one simulated). The test configuration was capable of supporting 120,000 users with a 0.12 IOPS per user profile and a user mailbox size of 2 GB.

Once again we see an obvious problem. The IBM solution at least follows the principle that a large DAG is usually better than a small DAG because a larger number of DAG members allows for greater flexibility in database copies and placement. The Hitachi solution is ridiculous and demonstrates a total lack of knowledge about how DAGs work. Why anyone would think that deploying twelve two-member DAGs would be anything close to a good configuration for 120,000 mailboxes is beyond me. In fact, it’s a joke configuration that is designed with one purpose in mind and that’s to pass the theoretical challenge posted by JetStress.

Neither configuration is suited for a production system. Both are designed solely to test storage and lack the resources necessary to handle the full workload of multirole DAG member servers deployed into real-life environments. And both use storage-based protection (RAID) to justify minimum DAG-based protection against failure, which is fine if you want to depend on just two database copies... Even with the undoubted goodness that RAID and enterprise-class disks can provide over JBOD, two database copies are insufficient.

It’s a great idea for Microsoft to provide a single location where vendors can post their results for different Exchange 2013 storage configurations. All the flaws and benefits of the various approaches that can be taken in hardware planning for different scenarios are exposed in one place. But hardware vendors do themselves no good by posting results that are laughable, obviously deficient, and totally impractical in the real world, all of which goes to prove that you should never accept documents that describe vendor-supplied configurations at face value. Always validate the described configurations and results using your own knowledge and experience of how software works in production environments or by running your own tests. After all, anyone can download and run the JetStress tool.

If IBM and Hitachi really understood Exchange they wouldn’t have come up with these configurations. A few minutes playing with the Exchange 2013 server role calculator would have resulted in much better configurations. But maybe the IBM and Hitachi hardware wouldn’t have done so well using those configurations… 

Follow Tony @12Knocksinna

Discuss this Blog Entry 10

on Nov 20, 2013

@KWashington, I think you're referring to HP's 500,000 mailbox Exchange 2010 deployment (http://windowsitpro.com/blog/managing-500000-mailbox-exchange-2010-deployment). They have not yet moved to Exchange 2013, although I hear that work is being done to prepare for a migration in early 2014.

My view on hardware vendors is that the best reference architectures are those that stand up to the cold hard test of "would I be happy to deploy this configuration into production". If you can't answer that question positively, then the configuration is not realistic and the vendor is playing games. All vendors play games anyway; the question is how far away they move from reality.

on Nov 21, 2013

Maybe you are talking about sizing, rather than layout but:

Isn't deploying a multiple 2 node dags a good way of making sure that the 2nd server will always have a shadow copy of mails in case of a disaster (if all servers are in same AD site, and servers are spread over 2 locations and backups are used)

I also can't see a benefit of having a 3rd or 4th copy of a database when backups are being performed and there is 1 copy in each data center.

on Nov 21, 2013

@kjrow, I think a larger DAG is always better than a smaller DAG on the basis that the larger DAG is a lot more flexible when it comes to outages. Consider the situation when a server fails in a two-node DAG; you have now lost all redundancy because only one copy of a database is available. If the same happens in a four-node DAG, three nodes are available to host database copies and you can maintain redundancy. The DAG will take care of shadow copies of messages in transit using Safety Net... that is not as important as maintaining basic database redundancy, which is the horrible flaw affecting both these configurations.

on Nov 27, 2013

Tony, I appreciate your thoughts. But it’s worth clarifying two quick points: First, the base configuration that Microsoft recommends for Exchange is two copies of the database -- IBM, Hitachi, EMC, and others all configure for two copies. Second, ESRP was developed by Microsoft to provide a common testing framework for storage, not servers.

on Nov 27, 2013

@dbenj, I'm glad that you appreciate my thoughts, but I think you miss a core point of mine - that no one in their right mind would create a configuration like this to support such a large population of Exchange mailboxes. Two copies of a database is the basic minimum redundancy for databases - but I can't think of a single respected authority on Exchange high availability who would recommend two databases for a high availability configuration, which these purport to be. Also, I understand the purpose of ESRP, but that doesn't get away from the point that neither IBM nor Hitachi improve their credibility by describing these kind of nonsensical configurations. Yes, they work in the ESRP lab and tick the testing boxes, No, they are not suitable for deployment nor are they in any way, shape, or form (IMHO) suitable as a storage configuration for 120,000 Exchange 2013 mailboxes.

on Dec 9, 2013

Hardware vendors have been putting together bogus implementations for Exchange since Exchange 4.0. This is no different, and I’m not surprised that Tony Redmond calls them out on it – after all, he works for a competitor. However, that being said, at least Tony's team has put together a usable solution. These two implementations are written to get the most performance out of their hardware. I remember when “a leading SAN vendor of the time” first put together a “solution” for Exchange 5.5 – they put 20 disks together in RAID0 and got stellar performance from the solution. And anyone can see why in a heartbeat, so Microsoft never endorsed that solution or its results.

For load testing servers, these “solutions” make some sense – you want the storage to be able to handle anything the server delivers as quickly as possible. However, once you have proven your servers, you need to go back and actually build a usable solution, which as Tony notes, includes at least 3 database copies, which by definition requires more than two servers per DAG. Such fun …

on Dec 9, 2013

@WillMart, I do not work for a competitor. I used to work for HP but I left their employment nearly four years ago and have no involvement with HP now. I don't even own any of their stock! My purpose here is to demonstrate how a "solution" (from any vendor) can be crafted to meet the needs of a program. Of course, the vendor hopes that you will read between the lines and think that their solution is wonderful and capable of being deployed into production, whereas in fact it cannot (well, it can, but you'd be a fool to do so). Focusing on practical and workable designs that are capable of handling the slings and arrows of outrageous operations is a better approach than taking the output of marketing exercises to heart.

on Jan 23, 2014

@WillMart, what did you mean by "a leading SAN vendor of the time”? CPQ?

DWS
on Jan 29, 2014

Seems to me the real problem is calling these papers "solutions". It's Microsoft who is calling a Jetstress test framework a "solution program", yes?. The ESRP purpose is not a full on storage sizing guide or architecture. Look at ALL of the ESRPs, it is only for showing disks IOPS performance - that's the pass/fail criteria... It appears the standard for this test is 2 copies, not realistic true. But for this program, its the norm. I'm guessing vendors follow this standard for comparison / competitive reasons

on Jan 29, 2014

@DWS, good points. I guess I took IBM and Hitachi to task because their claims were so outlandish and so far off the mark for such a large user base. But you can find fault in every one of the vendor solutions, none of which would stand up in a production setting. Games vendors play...

Please or Register to post comments.

What's Tony Redmond's Exchange Unwashed Blog?

On-premises and cloud-based Microsoft Exchange Server and all the associated technology that runs alongside Microsoft's enterprise messaging server.

Contributors

Tony Redmond

Tony Redmond is a senior contributing editor for Windows IT Pro and the author of Microsoft Exchange Server 2010 Inside Out (Microsoft Press) and Microsoft Exchange Server 2013 Inside Out: Mailbox...
Blog Archive

Sponsored Introduction Continue on to (or wait seconds) ×