Sizing, storage, and more for Exchange virtual machines
The advantages of running Exchange 2007 in a virtualized environment are significant. Since many businesses are under pressure to cut costs associated with server hardware procurement, management, and maintenance, using virtualization allows IT departments to do more with less.
Virtualization also frees organizations from the limits imposed by available datacenter space and power resources. Exchange 2007 requires a minimum of four physical servers to operate in a highly available configuration. And it can require 10 or more physical servers to separate the five server roles onto different physical machines.
When you add Exchange’s dependency on domain controllers (DCs) into the mix, creating an environment resilient to server hardware, software, and component failure can require a significant amount of space and effort. As a result, companies are using virtualization to reduce the physical hardware configuration of an Exchange deployment.
Additionally, Exchange 2007 is more, not less, complicated than previous versions of Exchange. It requires more configuration, which increases deployment and pilot times. Therefore, starting the deployment within a virtual test-bed environment before heading to production is a great way to reduce risks.
Virtual test-bed environments let you set up an Exchange infrastructure that’s completely separate from your production environment, deploy various Exchange server roles in different configurations, and test mail flow and device access before you deploy into production. One of the most common mistakes in Exchange deployments is underestimating the number of different ways to configure Exchange, its impact on the AD infrastructure, and the complexity of the server roles. A virtual environment lets you practice and test your eventual architecture before deploying into a production environment—ultimately decreasing both deployment and piloting costs.
Five Roles Mean More Complexity
The real complexity of Exchange comes into play when you try to deploy it in a highly available fashion. Exchange 2007 ships with five server roles: mailbox, client access, hub transport, unified messaging, and edge services. The minimum highly available Exchange 2007 configuration requires the first three roles, which are the primary roles—thus a minimum of four dedicated physical servers.
The highly available configuration for the Exchange 2007 mailbox server role is stateful, requiring a minimum of two dedicated servers, with each server being able to fail to the other via Microsoft Clustering Services. The client access and hub transport roles can be installed together on a single physical server, but without virtualization they can’t be installed on an OS that has one of the clustered mailbox server roles. These two roles can be configured in a stateless high availability configuration and load balanced through either Windows Network Load Balancing or alternately through a physical load-balancing device.
In a physical server configuration, therefore, you would need one pair of servers supporting the combined hub transport and client access server roles, and one pair supporting two mailbox server roles. But if you virtualized the servers, you could reduce the number of physical servers from four to just two.
Proper planning and sizing is key to successful virtualization of your Exchange environment. Let’s take a look at what you need to consider.
Memory Requirements
Sizing processors, memory, and storage requirements for virtualizing Exchange 2007 is straightforward. Both Microsoft’s Hyper-V and VMware’s virtualization technologies consume 10 to 15 percent of the physical resources of the host machine. This percentage will decrease as virtualization technology gets further embedded into chipsets within the next 12 to 18 months and as servers continue to get faster. For the time being, however, 10 to 15 percent is a good rule of thumb.
If you follow Microsoft’s guidance on memory requirements for Exchange, the mailbox server role is the key consideration. Microsoft recommends you reserve 2GB of RAM for Exchange and 5MB to 10MB per user for an average mailbox user. I think 5MB is appropriate for servers that manage over 500 active mailboxes.
For a 1000-mailbox deployment, a physical configuration for each of the mailbox servers should safely be around 7GB. The hub transport/client access servers aren’t too memory intensive, so in a physical configuration, 4GB of RAM is typically sufficient.
Adding 7GB for the mailbox server plus 4GB for the hub transport/client access servers with 10 to 15 percent buffer equals approximately 12GB to 14GB of RAM for the virtual servers. In reality, for the average server managing 1,000 employees or more, the lower end of this range is sufficient as it’s unlikely that all 1,000 employees would be connected to the server concurrently.
Exchange 2007 is still not processor intensive unless you’re using full-text indexing and journaling. Even so, Exchange 2007 servers that are using full-text indexing as well as journaling typically stay under 40 percent of the processor utilization of a standard dual quad-processor server. So combining the roles doesn’t typically get close to the maximum thresholds unless there is a significant amount of concurrency as well as a significant amount of ongoing email storage activity.
Storage Considerations
Storage requirements for Exchange 2007 in virtual environments are comparable to those in physical environments. SAN storage is still preferable to DAS storage in larger implementations of Exchange, for a variety of reasons that are best left for another article.
A minimum of five disks should be allocated for the storage of mailboxes for a typical 1,000 mailbox account. A general rule of thumb when using 10K SAS drives is that each drive in a RAID-5 configuration should support 100 I/O operations per second (IOPS). In this minimum five-drive configuration, the storage solution would support 500 IOPS.
To correctly size Exchange, it’s important to understand how many IOPS each mailbox requires. There is a significant amount of debate regarding how many IOPS an individual user of a mailbox consumes. Other factors such as archiving and BlackBerry usage can increase the load on a server, as does the number of concurrent users. For Exchange 2007, the overall potential IOPS numbers can range between 0.3 IOPS per mailbox and 3 IOPS per mailbox for concurrent users, with heavy mobile usage being the biggest driving factor for increasing the I/O demand.
Unfortunately, many of the performance sizing tools for Exchange available from Microsoft and others don’t take into account the peaks and valleys that a typical storage solution encounters during regular usage. For instance, snapshots and backups can consume a significant amount of storage and many of the benchmarks available do not take these extra data protection activities into account.
In poor storage implementations, Exchange responsiveness can slow and even stop during the backup window. Therefore, it’s safest to be conservative in the number of IOPS per mailbox and not stray below 1.0.
Keep Exchange Virtual Machines Clean
In addition to the memory hit, the only other potential gotcha of virtualization involves the tendency by IT organizations to want to load additional virtual machines (VMs) on the same physical servers as the Exchange server. To many folks, it seems natural to combine collaborative applications such as Microsoft Office SharePoint Server, Office Communications Server, and Exchange 2007 on the same virtual server infrastructure and physical servers.
It’s difficult, however, to accurately size physical resource requirements for these applications because they each have dramatically different performance requirements. Since modeling the performance of different application characteristics is complex, time consuming, and error-prone, it’s best to keep Exchange virtual servers deployed on a physical host dedicated to Exchange.
Disaster Recovery and Business Continuity
Finally, many organizations are considering virtualization for use in email business continuity and disaster recovery scenarios. The ease of copying VMs to a remote site enables a relatively rapid “full fidelity” recovery of Exchange and makes not only the OSs, but the AD infrastructure and the storage servers available on a moment’s notice. Although replication technologies such as VMware’s VMotion aren’t recommended by Microsoft for use with Exchange 2007 as Microsoft hasn’t thoroughly tested them, these technologies should someday make site-to-site failover a simple reality.
In the meantime, customers are looking to storage-based replication solutions to mirror virtual servers and storage to secondary sites. Even when automated replication technologies for Exchange become available, in many cases a manual process will still be required to change critical DNS (e.g., MX) records and AD infrastructure to support the secondary site recovery in the event of a failure.
Lessons from Real-World Scenarios
Migrating from Exchange 2003 to Exchange 2007 carries with it a significant increase in datacenter power utilization. For example, two representative customers whom I’ve recently worked with on virtualization projects calculated that the additional servers required to support Exchange 2007 would drive power utilization in their datacenter from the pre-Exchange 2007 levels of 80 percent to 90 to 95 percent capacity. Because of the number of servers required and power limitations, it was impossible to add the servers necessary to migrate to Exchange 2007, while maintaining the current infrastructure.
By deploying a virtual infrastructure and leveraging their existing SANs, these companies were able to not only deploy Exchange 2007 but also reduce their power consumption through server consolidation. In addition, they now have a virtual server platform that can support their rapid growth projections over the next couple of years.
Both companies virtualized their Exchange 2007 system by using two quad-core processor servers, each with a minimum of 8GB of RAM, and redundant local hard drives running VMware ESX 3.5 as host machines. They mounted two Windows Server 2003 R2 64-bit OSs as guest machines from their SAN via iSCSI connections.
These two guest servers run as a mailbox server and a combined client access/hub transport server respectively. The physical servers use storage on their SAN to hold mailbox and public folder content, and also act as a quorum disk for the highly available mailbox server role.
Using the above configuration, each customer supports up to 500 mailboxes. Each has been running its Exchange infrastructure in production for over nine months.
What You Can Expect
Scalability claims vary around the number of mailboxes supported for Exchange 2007 running in a virtualized environment. In reality, Exchange 2007 exhibits the same storage resource constraints as prior releases of Exchange: Bottlenecks are typically due to I/O constraints on the storage solution.
In a typical 1,000-mailbox deployment running at a moderate email user configuration, with 10 percent or 100 mailboxes being accessed via higher impact BlackBerry devices, and supported with appropriate storage, the following profile is consistent for an Exchange 2007 mailbox server: • CPU utilization is low • Memory utilization is low to moderate • Storage utilization continues to be the key bottleneck, but is still moderate • Network traffic typically remains constant
Although this scenario isn’t indicative of every environment (since email loads and usage patterns vary widely), what is noteworthy is the low load level on the physical server and the available unused capacity. This is also the case with the client access and hub transport servers. Just as with the mailbox server, CPU utilization and memory utilization are typically very low, while storage I/O and network utilization remain low to moderate.
As for storage I/O and network utilization, which is moderate to high at peak usage in this configuration, they can both be addressed by adding additional storage drives to a SAN or managing the network more efficiently if these issues create a scalability problem.
As with any IT infrastructure, virtual servers must be managed. Loading too many VMs on a physical server will cause truly bizarre events and activities (such as the bouncing of client connections to Exchange forcing re-logins every minute or scheduled backups maxing out I/O, making the server inaccessible to new clients) to occur in network, storage, memory, and other physical resources when these are throttled arbitrarily by a virtual platform.
Appropriate sizing is a key requirement when deploying Exchange 2007 on a virtual platform, as are management tools. Lately I’ve been seeing Hyper-V achieve a slight edge over VMware on performance with Exchange, but customers should still choose their virtualization platform strategically based on their entire corporate virtualization roadmap.
One area that distinguishes the two platforms is management—VMware’s product is still much richer in capabilities in this area. To manage virtual and physical servers in the Hyper-V world, you will require either a separate Windows Server 2008 or Microsoft Systems Center Virtual Machine Manager.
Virtualizing Exchange Can Make a Difference
With the above caveats, virtualization is a safe and effective alternative platform for the successful operation of Exchange 2007. Quite a few companies are running full production deployments of Exchange 2007 in a virtualized environment. Although running on a virtual platform might not be suitable for all environments (such as smaller non-redundant Exchange architectures), it’s ideally suited and increasingly recommended for deploying Exchange 2007 in a highly available configuration.