We are in the middle of a major upgrade project for a new client. The client has two VMware ESX host servers running a variety of virtual server guests, including two domain controllers (DCs), File Server, Exchange Server 2007, SQL Server 2005, Microsoft Office SharePoint Server (MOSS) 2007 and Terminal Server. All servers are running Windows Server 2008 as the base OS with a mix of x86 and x64. All the servers were prepped for an initial Exchange cut over last Friday. Earlier in the week, on Tuesday, we completed training on Exchange and SharePoint for the client and the servers ran flawlessly.
But starting on Wednesday, we noticed that something was seriously wrong with some of the servers. The symptoms included the following:
• On the SharePoint Server, one site was completely inaccessible.
• SharePoint performance on the working sites was very slow.
• On the Exchange Server, the receive connectors did not appear in the Exchange Management Console (EMC).
• On the Exchange Server, the databases took a long time to appear in the EMC, and when they did their status was Unknown .
• Performance on one of the DCs was very slow.
• If you mapped a drive to any share on the slow servers and ran the command dir /s the directory listing paused very half second. Normally, a directory listing scrolls by so fast that you can’t read it.
Both ESX hosts had 16GB of memory and each had two quad core 3GHz processors in them. Some of the virtual servers were Ok, but obviously something was very wrong with the Exchange and SharePoint servers and one of the DCs. The server performance was so slow, it essentially made them unusable. Since we were cutting over to the new Exchange Server on Friday, we had to quickly determine the cause of the problem. A review of the Event Viewer did not show any possible causes. The CPU, disk, and network utilization was very low and well within normal parameters. Yet the slowness continued. Since the servers were running on ESX, we fortunately had an image backup of each of these servers from the prior week. We decided to restore the Exchange Server image from tape and the server started properly working fast again! But when we rebooted the server it slowed down again. We did notice that three updates that were installed on the server went through a configuration process when the server was rebooted. This led us to believe that the server was slowing down because of one or more of the Server 2008 updates. The updates that were installed on this server included:
Exchange 2007 SP1 Post Rollup 3.
KB950050
KB955020
KB949189
KB952287
KB941693
KB948590
KB950582
KB950762
KB950974
KB951066
KB953733
KB953838
KB953839
KB947562
KB951072
KB951618
KB951978
KB951698
To narrow down the problem, we removed each patch one at a time (one of them couldn’t be removed) and rebooted the server. Then we tested the server to see if the server started performing fast again. After removing each patch one by one, the server was still unfortunately slow. On a hunch one of my employees Blandy Allred suggested that we try to remove Symantec Endpoint SR2 MP2 Client. As soon as he removed it, the server started working fast again! We reinstalled the Symantec Endpoint Client and the server remained fast. After Endpoint was reinstalled, we reapplied all the patches and the server slowed down again! But we uninstalled and reinstalled Endpoint and the server started working again. Evidently one or more of the patches conflicts with Symantec Endpoint. If Symantec Endpoint is installed and the patches are applied, the server will slow down. If the patches are applied and then Symantec Endpoint is installed, the server will remain fast.
The fix here is a relatively simple one. If you are running Server 2008 with Symantec Endpoint and the server is slow, uninstall and reinstall Symantec Endpoint and the server performance should return to normal. I’ve notified Symantec of this problem, and I'll keep you informed of any progress that is made on this issue. It took us three days to figure out what the problem was. Fortunately the fix is simple once you know the solution. This issue appears to happen on both x86 and x64 versions of Server 2008. (See also, "Windows Server 2008's Radical Features").
Hopefully this article will prevent you from hitting your head against the wall trying to find the solution!
