The story of Exchange IOPS: How a crusade to make Exchange less of a storage hog enabled a successful cloud service

We probably realized it at the time (but thought it was OK) - Exchange 2003 was a storage hog. The software just loved to wallow in expensive storage and fully occupy the mind of pernickety Storage Area Networks (SANs). Everything was fine when large enterprises were footing the bill, but something had to be done to reduce the demand for storage. The subsequent project (or crusade in the minds of some) has lasted roughly 12 years and has been tremendously successful in transforming the ability of Exchange to run on low-cost storage. In fact, just the kind of disks that Office 365 has in abundance. 

The history of software is littered with examples of grand strategic changes that never amounted to much when implemented. The crusade to reduce the number of disk I/O operations generated per user per second (IOPS) is an example of how a big bet paid off.

If we look back to Exchange 2003, if you wanted to deploy the software to serve more than a few hundred mailboxes, you had to invest in expensive Storage Area Network (SAN) hardware. SAN technology got a bad reputation in places as being complex and prone to problems if it wasn’t configured just so. That feeling lingers today, but when a SAN was deployed, configured, and managed properly, it is capable of delivering great performance for a range of applications.

Unfortunately, a SAN costs. The storage smarts, high-speed connections, expensive disks built for reliability and robustness, and all the surrounding technology makes a SAN something that is expensive to sell, service, and operate. In short, the need for SAN-quality storage created a dependency that Exchange could live without.

A focused effort to reduce the IOPS footprint required by the Information Store started soon after Exchange 2003 appeared. As depicted in the table below, that effort has been extraordinarily successful in driving the IOPS per user from 1.0 to around 0.05 today (lower in Exchange 2016 if you use lagged database copies). As with all performance data, you have to remember that input parameters exert huge influence over results. In this instance, the data is based on an IOPS profile of 100 messages per day.

 

Exchange 2003

Exchange 2007

Exchange 2010

Exchange 2013

Exchange 2016

IOPS per user

1.0

0.32

0.1

0.07

0.044(*)

Drives

SAS 146 GB 15K

SAS 146 GB 15K

SAS 1 TB 7.2K

SAS/SATA 4 TB 7.2K

SAS/SATA 8 TB 7.2K

Disk redundancy

RAID-10

RAID5/10

JBOD

JBOD

JBOD

High Availability

Backup and Restore

LCR, CCR, and SCR/single database copy and failover

DAG/multiple database copies and failover

DAG/multiple database copies and failover

DAG/multiple database copies and failover

Backup

Streamed to tape

Server failover

Native data protection

Native data protection

Native data protection

Controller

SAN

Array

Array

Array

Array

Write cache protection

SAN

Battery

Battery

Battery

Battery

Average corporate mailbox

200-500 MB

250 MB – 1 GB

500 MB – 2 GB

1 GB – 5 GB

2 GB – 10 GB

 

 

 

(*) The official guidance from Microsoft is that Exchange 2016 and Exchange 2013 deliver the same IOPS performance. However, I checked the lower figure before publication and am happy that this can be achieved and indicates that continued progress is being made. For planning purposes, it's best to use the higher figure, which is the basis used by tools such as Microsoft's Exchange Server Role Sizing Calculator. The redoubtable Ross Smith IV would always prefer you to use 0.07.

Note: SATA or SAS drives can be used with Exchange 2013 or Exchange 2016. Microsoft prefers SAS and says in its discussion about the preferred architecture: “7.2K RPM serially attached SCSI (SAS) disks (while SATA disks are also available, the SAS equivalent provides better IO and a lower annualized failure rate).

Change has come about through changes in the software at the ESE database layer and the Information Store allied to fundamental architecture changes in the high availability space to deliver the Database Availability Group (DAG). Taken together with the dropping cost of storage and the growing popularity of JBOD, the net result is that the topic of storage for Exchange leads to a radically different discussion today.

Lots of expense is removed from the deployment equation when the need for high-end storage disappears. Driving storage cost so low creates the ability to run cloud services like Exchange Online at price points that would be impossible if SANs were required.

In fact, the crusade to shrink IOPS has not only proven to be the right call, it also made it possible for Microsoft to lay the foundation for the transfer of so much user data to Office 365 over the last five years to serve over 60 million active mailboxes as part of a commercial cloud business that’s now running at an $9.4 billion annual revenue run rate.

There’s no way that Microsoft could afford to provision 50 GB of storage per mailbox and charge the prices they currently offer for Office 365 plans. In fact, 50 GB is just the start, because there’s that expandable archive mailbox as well plus space for Recoverable Items, and so on. Put it this way, a single Office 365 user could easily fill a complete old-style 146 GB enterprise-class SAN drive.

As MVP Paul Cunningham points out in an article, good reasons still exist to use a SAN to provide storage to Exchange, especially if your company takes a storage-centric view rather than an application-centric view of how resources should be deployed. The good thing that arises from the design decision made by the Exchange developers twelve years ago is that choice now exists. SAN or JBOD – it’s up to you.

Follow Tony @12Knocksinna

Discuss this Blog Entry 1

on May 21, 2016

To my mind to planning for non-100% concurrency is a bit of a science. If you're positive you'll never go >3,000 concurrent users and you have historical data to prove that then sure I don't see why not. I'd personally make sure I have 10-20% more than I need to handle future growth and make sure I don't thin-provision the storage too much. Perform proper Jetstress testing to make sure you actually -do- get the IOPS you need once the hardware is deployed; just because the hardware says it'll provide X IOPS doesn't mean the full HW deployment really will once you start factoring in other environmental items.

Please or Register to post comments.

What's Tony Redmond's Exchange Unwashed Blog?

On-premises and cloud-based Microsoft Exchange Server and all the associated technology that runs alongside Microsoft's enterprise messaging server.

Contributors

Tony Redmond

Tony Redmond is a senior contributing editor for Windows IT Pro. His latest books are Office 365 for Exchange Professionals (eBook, May 2015) and Microsoft Exchange Server 2013 Inside Out: Mailbox...
Blog Archive

Sponsored Introduction Continue on to (or wait seconds) ×