Office 365: a kind of dogfood environment for on-premises Exchange

Microsoft says that we should keep an eye on Office 365 to know what's coming next in on-premises Exchange. That's a tall order because they do such a good job of keeping the Office 365 kimono firmly closed. But some clues are out there - that is, if we worry about this kind of thing at all. I'm not sure that I do. But here goes.

You might have been puzzled by the statement “If you want clues about what’s coming in the next version of Exchange Server, keep an eye on what’s happening in Office 365” contained in Microsoft VP Perry Clarke’s recent attempt to pacify the on-premises community. Where does one look for clues in Office 365? Does Microsoft say where the new bits are? And would you know a new feature if you saw it?

It’s a fact of software engineering life that developers have test-bed systems that are used to validate software builds. Famously, Exchange has had its “dogfood” environment for many years. To reflect today's platforms, the Exchange product group runs both on-premises and a cloud dogfood systems and are diligent at consuming its own dogfood by using these servers to host their mailboxes. The idea, of course, is that developers should be able to recognize problems in their code if bugs surface when the developers use new builds to get work done. Think of it as a massive debugging party.

I view the Office 365 datacenters as the dogfood environment for what eventually will appear in an on-premises Exchange update. This isn’t to say that Office 365 runs beta Exchange Online code because code is validated before it is introduced into “the service”. Rather, Office 365 runs code for weeks or months before the code is packaged and provided to on-premises customers in the form of a cumulative update (or service pack). I see lots of goodness here because a solid shake-down by running code used by millions of cloud mailboxes seems like an excellent way to find bugs. Apart from the joy of stressing code at the edge of the performance envelope, running code in a datacenter is certainly a better way than exclusively relying on the suite of automated tests that are used to test basic functionality and that new code has not introduced any regression, which is what used to happen.

Sometimes Office 365 is not a good test platform, largely because Microsoft doesn’t deploy code onto an Office 365 server in the way that on-premises installations do. For example, automated processes strip servers down to bare metal and then reinstall the operating system (complete with any hot fixes and required patches) and the latest version of applications before they go into production. There’s no notion of applying updates to servers like you might when updating a server from Exchange 2013 CU2 to CU3. I assume that wipe and reload is an easier operation to automate for datacenter operations because it always results in a known environment. And because Office 365 looks nothing like your standard on-premises environment, it follows that some processes executed in an on-premises environment such as the application of security patches do not occur in the same way, which is one of the reasons why the MS13-061 fiasco happened.

It’s also fair to say that Office 365 is a much more regimented implementation than anything you will find on a customer site. Third party software is not installed in the same way as in an on-premises deployment (instead, third party vendors like BlackBerry offer integrations with Office 365) and are therefore not tested against new Exchange on-premises code. The kind of integrations that customers do would not be tolerated and probably won’t work in a datacenter at scale because they are looser and less automated than the playbook Microsoft follows. It therefore follows that you cannot assume that code that works for Office 365 will function perfectly when introduced into the imperfect, non-Microsoft centric, and heterogeneous environments that exist in so many customer deployments. This is why bugs appear in on-premises deployments – even though APIs (like the EMS cmdlets and EWS) can be tested, specific conditions that arise in customer environments have not been tested for during development or not exercised in the datacenter.

Accepting that Office 365 is not a perfect match (or near a perfect match) for any known customer environment, a huge dividend can still be gained in terms of bug identification and elimination by running code in Office 365 for weeks before on-premises customers get hold of it. And processes developed by Microsoft for use in their datacenters do find their way into the on-premises product. Not all of the processes, but certainly good examples exist such as Managed Availability and workload management. Microsoft’s focus on driving I/O down so that JBOD storage is more than a viable option for Exchange has also benefited from the need to operate datacenters at scale and at a low price point.

Some goodness is present in the current arrangement. Office 365 customers get “evergreen” software (even if they don’t want it, or realize that they are getting it), on-premises customers should get higher quality software (a problem recently, but signs are that quality is improving), and Microsoft’s marketing fraternity are able to take the battle to Google because Office 365 can introduce new features at the same rapid cadence that seems to have become the norm for cloud-based email services.

Back to the EHLO post, you might wonder how to track Office 365. It’s difficult, because even though more recent code is running, features might not have been “lit up” (made visible to end users). Thus, at the time of writing, my Office 365 tenant domain is running version 820.5 (run the Get-OrganizationConfig cmdlet and check the value of the AdminDisplayVersion attribute) whereas Exchange 2013 CU3 is version 775.38. According to one of my Microsoft sources, a new Exchange build is produced five times weekly. Given that the version number running in Office 365 is some forty-five versions ahead of CU3, you can speculate that Exchange Online is about nine weeks ahead of its on-premises cousin when this snapshot was taken. The gap flexes over the development cycle but this is a reasonable indication of the gap that exists between the respective software builds. In this instance, Office 365 is running a variant of what will eventually be Exchange 2013 CU4 (or Exchange 2013 SP1), which is what you would expect.

Clues are visible and evidence can be gathered to show that Office 365 does indeed run newer software. But apart from the notes on the Office 365 "service upgrades blog", it is possible that you will see little evidence of new features until they surface in Outlook Web App or the Exchange Administration Center (I am too lazy to go looking for a new EMS cmdlet). No doubt the new features will light up and delight me in due course. Until then, I’m clueless.

Follow Tony @12Knocksinna

Discuss this Blog Entry 2

on Dec 10, 2013

Why on earth Microsoft tests this way?
Is it cost?
Is it lack of resources?
MBAs making decisions again?
Just build a solid Exchange 2013 On-Premises infrastructure in-house as MS had back in the Ex 2007 & Ex 2010 days and test every updates for Exchange On-Premises before sending it to Exchange On-Premises customers worldwide.
Sounds like Ex 2013 CU1 & CU2 issues are because of testing this way.
Nice logic, Porsche owns Volkswagen, now let's test breaks on Volkswagen and then put it on Porsche :-)

on Dec 10, 2013

I didn't say that Microsoft tests software using Office 365. I did say that the software running in the cloud are several builds ahead of what is delivered to on-premises customers, which is what you'd expect. It is sensible for Microsoft to run software for several weeks in Office 365 where it is under their control and can be carefully observed under tremendous load before they release to on-premises customers. As to the issues with CU1 and CU2, I think administrative oversights and testing errors are more to do with them than the way the software was run in the cloud.

Please or Register to post comments.

What's Tony Redmond's Exchange Unwashed Blog?

On-premises and cloud-based Microsoft Exchange Server and all the associated technology that runs alongside Microsoft's enterprise messaging server.

Contributors

Tony Redmond

Tony Redmond is a senior contributing editor for Windows IT Pro and the author of Microsoft Exchange Server 2010 Inside Out (Microsoft Press) and Microsoft Exchange Server 2013 Inside Out: Mailbox...
Blog Archive

Sponsored Introduction Continue on to (or wait seconds) ×