Microsoft declares war on Symantec Enterprise Vault and looks to bring back data into Exchange Online

No one likes when they feel that something of theirs is lost. Which is exactly how Microsoft seems to feel about items that have been moved out to Exchange mailboxes to third party archives such as Symantec Enterprise Vault. And now Microsoft is locked and loaded with double barrels to take that data home with the introduction of the Office 365 Import service. Put data on PSTs, collect PSTs on SATA drives, ship to a Microsoft datacenter, import to Azure, transfer to Office 365 and ingest into Exchange Online. It all sounds so easy. I wonder what the Symantec folk feel?

Last week I discussed a new data import service that Microsoft has introduced to allow Office 365 customers to send PSTs to Microsoft datacenters for ingestion into the cloud. This week I sat in on an archiving session given by Dheepak Ramaswamy at the Ignite conference and learned some of the thinking and detail around the new capability.

There’s no doubt that Microsoft has Symantec Enterprise Vault (EV) in their cross-hairs. For years, Microsoft has muttered that the “stubs” created by EV when content was exported from mailboxes to the EV database compromised the integrity of the Information Store and prevented customers making full use of the compliance features built into Exchange. That argument was less powerful when on-premises Exchange wasn’t so good at compliance and ran on expensive storage, meaning that it was a good thing to offload messages and documents into another repository that supported full-feature eDiscovery.

The world is different now. Exchange boasts a wide range of compliance features in both on-premises and online versions and the latter has massive low-cost storage available to hold literally as much data as you care to throw at the service. Microsoft views this as an opportunity to “bring data home”, much like a shepherd collects their lost sheep.

The procedure requires companies to collect PSTs on 3.5 inch SATA 4 TB drives (6 TB drives will be supported soon) following by a dispatch to the closest Office 365 datacenter. The PSTs, which are protected by BitLocker encryption of the drives and can be optionally encrypted before being placed on the drives, are then connected into racks and the content is uploaded in a storage container in the Azure blob store. No Azure subscription is required as this is created automatically to allow transfer to occur. In addition, you don’t have to pay for the Azure storage used to hold the PST data while it waits to be processed. The drives are returned to the sender after the PSTs have been uploaded to Azure.

Shipping and upload takes a couple of days, after which the tenant administrator can start off jobs to transfer the PST data to mailboxes. Microsoft has no access to the data while it is being processed. Everything on the drives is encrypted and the tenant retains control of the BitLocker key throughout.

The data ends up as a blob in Azure and is available for up to 90 days. During this period the tenant administrator can transfer information to Exchange Online as they wish, controlling the process by providing Office 365 with a mapping file to connect the PSTs that have been uploaded to Azure with target mailboxes.

During the transfer to Exchange, the PST data is validated and scanned to remove corrupt item (think of running the SCANPST utility several times). Very corrupt PSTs are dropped as there’s no point in introducing bad items into mailboxes. Azure and Office 365 are connected on a common Microsoft datacenter backbone so transfer rates are reasonable with an expected range of 250 GB to 1.3 TB per day. The exact rate depends on the number of target mailboxes. If you point a massive PST at one mailbox, you create a narrow pipe and transfer will be slower than if a job processes multiple PSTs, each targeted at a different mailbox.

Data can be directed to either primary or archive mailboxes. It’s best to use archive mailboxes for one simple reason: archive mailboxes are only ever accessed when clients are online so after the data is transferred to the archive, it is immediately available. By contrast, if you transfer data to primary mailboxes, that data needs to be synchronized to local clients for people who use Outlook and thus cause a “synchronization storm”.

As a tip, if you want items imported into the root of the mailbox, specify "/" (without the quotes) in the TargetRootFolder column of the mapping file. Otherwise everything will be imported under the "Imported" folder.

Although EV seems to be the prime target for now, Microsoft has signed up archive specialists TransVault and Nuix to provide tools to help companies generate PSTs from other third party archive systems.

For various reasons, the Office 365 import service is only supported for tenants based in the North American, European, and Asia-Pacific datacenters. Tenants in China, Australia, Brazil , and Japan will have to wait for the service to be introduced for those regions.

The Office 365 import service also supports a direct upload option to allow tenants who don’t have terabytes of PST data to upload PSTs over the network. Once uploaded, the same ingestion process is used to move data from Azure into Exchange Online.

Documentation for both the drive shipping and direct upload import options is now available on TechNet.

SharePoint migration to Office 365 will be enhanced soon when the Import service is upgraded in about a month to use a new API to create a content file and manifest from on-premises document libraries that can be shipped to a Microsoft datacenter to go through the same ingestion process. Of course, in this case, the target will be SharePoint Online document libraries rather than mailboxes.

Microsoft also announced that they will soon enable unlimited quotas for archive Exchange Online mailboxes. Up to now, a default quota of 100 GB was assigned to archive mailboxes. This amount could be increased but it required the user to request Microsoft to assign additional quota. The difference now is that Microsoft is going to remove the quota altogether and users will be able to stuff as much data as they want into their archive mailboxes. The race to the first 100 TB mailbox begins!

Seriously, removing the quota for archive mailboxes makes the process of data ingestion much easier all round. All of those massive PSTs that are about to be shipped to Microsoft on SATA drives will find a nice online home in Exchange Online, available for as long as the user wishes and indexed and discoverable for compliance purposes too.

Follow Tony @12Knocksinna

Discuss this Blog Entry 3

on May 10, 2015

This feature will be a boon for faster adoption of 365 however once there is a hammer, not everything is a nail. This approach will be beneficial for organisations that have small data sets and/or WAN connections that would otherwise rule them out of a direct migration. For these businesses, the ship to disk option will be a great feature that will significantly reduce the on-boarding cost of 365 which will hopefully mean higher adoption of what is a great platform.

Organisations with large enterprise archiving solutions will often be better served taking a direct approach as the three step process (export to PST, ship disks, import PST) will, in almost all situations of large data sets, take longer than using a third-party migration tool to send that data direct to the mailbox. My team are currently performing over 10 migrations to 365 and the average performance is about 40GB/h for archive data and, where customers have really healthy environments with a nice big pipe we have seen it reach as much as 70GB/h, which is similar to those figures being quoted and a one step, auditable process.

Another concern for large enterprises is how chain-of-custody at an object level is maintained with this approach, as in most situations customers how have an enterprise archiving solution generally have pretty strict compliance requirements. How is Microsoft going to enable customers to be able to demonstrate to the Courts that the item that was in the archive is exactly the same as the item that made it to 365. Equally important is what the approach will be for reporting on items that they were not able to ingest?

Regardless of the approach – ship to disk or direct migration. It is great to see Microsoft taking a lead position on this topic and recognising the requirement to enable ease of archive migration and storage and also glad that we have another approach available to provide those organisations that truly do suit a ship to disk approach.

on May 11, 2015

@Simon, you make some good points. The Office 365 import service is not for everyone and is not the right solution in all circumstances. As you point out, other solutions exist that incorporate different feature sets that might be a better choice in particular circumstances, especially when the time comes to find/collect user PSTs. However, this service is free and can be regarded as setting the base for the market. Microsoft also said that they will provide a detailed chain of custody report to customers in the near future - like everything else in Office 365, I guess we can expect ongoing and rapid evolution. The advent of this service will cause customers to consider the options that exist to move data to Office 365 and, I think, open some opportunities for other vendors and consultants to engage with customers at that point to work through the exact requirements and come up with the appropriate solution.

on Feb 9, 2016

Great post. However, large organisations with tens of thousands of users will struggle to identify owners and assemble drives full of PST files. We have recently won a contract with the US government via Dell to do just this. It might be worth checking out ultimatemigrator.com .... the rebranded PST Accelerator.

Thanks

stefan

Please or Register to post comments.

What's Tony Redmond's Exchange Unwashed Blog?

On-premises and cloud-based Microsoft Exchange Server and all the associated technology that runs alongside Microsoft's enterprise messaging server.

Contributors

Tony Redmond

Tony Redmond is a senior contributing editor for Windows IT Pro. His latest books are Office 365 for Exchange Professionals (eBook, May 2015) and Microsoft Exchange Server 2013 Inside Out: Mailbox...
Blog Archive

Sponsored Introduction Continue on to (or wait seconds) ×