Over a busy Labor Day weekend, the Exchange product group hand-crafted a fix for the hybrid connectivity problem that had appeared after the release of Exchange 2013 CU6. The fix is a script that updates an XML configuration file and restarts IIS. It's quick and simple to apply but questions still remain as to why this situation developed in the first place?
Microsoft has published KB2997355 to provide a script fix for the issue that plagued administrators when they found that they couldn’t use the Exchange Administration Center (EAC) to create or otherwise manage Exchange Online mailboxes after deploying Exchange 2013 CU6. Objects could still be managed using the Exchange Management Shell, not that this prospect afforded much comfort to those charged with the administration of hybrid environments.
To their credit, Microsoft quickly closed the embarrassing gap that had opened up between on-premises and cloud administration. The root cause of the problem was a mistake in the EAC configuration file, proving yet again that a small slip is sufficient to cause horrible problems in today’s complex software environment.
The script (Exchange2013-KB2997355-FixIt.ps1) updates the erroneous configuration in $ENV:ProgramFiles\Microsoft\Exchange Server\V15\ClientAccess\ecp\DDI\RemoteDomains.xaml by adding a missing “TargetDeliveryDomain” property. The script then restarts IIS to apply the fix, which should be rolled out to all Exchange 2013 CU6 servers that are involved in the administration of hybrid environments.
Update: MVP Michel de Rooij has pointed out a problem that might arise due to the way that the fix-up script references the $ENV: PowerShell variable. The script will work perfectly if Exchange is installed on the drive where Windows is located (often C:) but will run into problems if you have relocated the Exchange binaries to another drive. In this case, you will have to edit the script to amend the reference in line 2 to "$Env:ProgramFiles\Microsoft\Exchange Server\V15\ClientAccess\ecp\DDI" and replace it with the place where the Exchange binaries are located, or replace the line with:
$baseDirectory = "$Env:ExchangeInstallPath" + "ClientAccess\ecp\DDI"
Ever-helpful, Michel has posted a revised script to the TechNet Gallery. His version is more sophisticated because it reads the system registry to locate the directory where Exchange 2013 is installed and then builds the correct location using that information.
The fix was developed over the Labor Day weekend, one of the major U.S. holidays. Creating new code to fix a problem when the majority of the U.S. was on holiday shouldn’t come as a surprise in the bright new world of cloud systems when developers are always on-hand to fix problems should the need arise, especially when those problems cause issues for highly visible components like hybrid connectivity.
We should be happy with the speed to fix while recognizing that the accelerated development cadence might just have been a contributing factor in this episode. I guess it’s symptomatic of the world in which we live.
Follow Tony @12Knocksinna