Microsoft fixes hybrid connectivity problem in Exchange 2013 CU6

Over a busy Labor Day weekend, the Exchange product group hand-crafted a fix for the hybrid connectivity problem that had appeared after the release of Exchange 2013 CU6. The fix is a script that updates an XML configuration file and restarts IIS. It's quick and simple to apply but questions still remain as to why this situation developed in the first place?

Microsoft has published KB2997355 to provide a script fix for the issue that plagued Exchange 2013 administrators when they found that they couldn’t use the Exchange Administration Center (EAC) to create or otherwise manage Exchange Online mailboxes after deploying Exchange 2013 CU6. Objects could still be managed using the Exchange Management Shell, not that this prospect afforded much comfort to those charged with the administration of hybrid environments.

To their credit, Microsoft quickly closed the embarrassing gap that had opened up between on-premises and cloud administration. The root cause of the problem was a mistake in the EAC configuration file, proving yet again that a small slip is sufficient to cause horrible problems in today’s complex software environment.

The script (Exchange2013-KB2997355-FixIt.ps1) updates the erroneous configuration in $ENV:ProgramFiles\Microsoft\Exchange Server\V15\ClientAccess\ecp\DDI\RemoteDomains.xaml by adding a missing “TargetDeliveryDomain” property. The script then restarts IIS to apply the fix, which should be rolled out to all Exchange 2013 CU6 servers that are involved in the administration of hybrid environments.

Update: MVP Michel de Rooij has pointed out a problem that might arise due to the way that the fix-up script references the $ENV: PowerShell variable. The script will work perfectly if Exchange is installed on the drive where Windows is located (often C:) but will run into problems if you have relocated the Exchange binaries to another drive. In this case, you will have to edit the script to amend the reference in line 2 to "$Env:ProgramFiles\Microsoft\Exchange Server\V15\ClientAccess\ecp\DDI" and replace it with the place where the Exchange binaries are located, or replace the line with:

$baseDirectory =  "$Env:ExchangeInstallPath" + "ClientAccess\ecp\DDI"

Ever-helpful, Michel has posted a revised script to the TechNet Gallery. His version is more sophisticated because it reads the system registry to locate the directory where Exchange 2013 is installed and then builds the correct location using that information.

The fix was developed over the Labor Day weekend, one of the major U.S. holidays. Creating new code to fix a problem when the majority of the U.S. was on holiday shouldn’t come as a surprise in the bright new world of cloud systems when developers are always on-hand to fix problems should the need arise, especially when those problems cause issues for highly visible components like hybrid connectivity.

We should be happy with the speed to fix while recognizing that the accelerated development cadence might just have been a contributing factor in this episode. I guess it’s symptomatic of the world in which we live.

Follow Tony @12Knocksinna

Discuss this Blog Entry 2

on Sep 1, 2014

Thanks Tony. In our environment we are still running exchange 2007 unfortunately (only for another couple of weeks) and we also have a hybrid exchange configuration with Office365. We are still on CU4 as we saw no point updating to CU5 which had the Hybrid bug and we decided to hold out until CU6.

So, in our environment if we wanted to deploy CU6, we would need to deploy the main CU6 update, followed by the script in 2997355 and the hotfix in 2997209. Is this correct? It is good to get quick fixes to the issues but unfortunately now we have 3 steps to the update process instead of one.

I guess alternatively we wait until our 2007 is decommissioned and just need to run CU6 and the script from 2997355.

on Sep 2, 2014

I think you're right. The first patch is to make sure that Exchange 2007 and 2013 play nice together and you don't have Exchange 2013 databases going offline for no good reason. The second is to ensure that you can manage Exchange Online from EAC running on an on-premises Exchange 2013 server. And yes, it is messy and unfortunate and shouldn't be this way.

Please or Register to post comments.

What's Tony Redmond's Exchange Unwashed Blog?

On-premises and cloud-based Microsoft Exchange Server and all the associated technology that runs alongside Microsoft's enterprise messaging server.


Tony Redmond

Tony Redmond is a senior contributing editor for Windows IT Pro. His latest books are Office 365 for Exchange Professionals (eBook, May 2015) and Microsoft Exchange Server 2013 Inside Out: Mailbox...
Blog Archive

Sponsored Introduction Continue on to (or wait seconds) ×