SharePoint Online Migration API and Improvements Simon Bourdages Microsoft PM

Introduction Overview Moving content into O365 Current tech New Capabilities Things to come The Migration API and PowerShell Using the PowerShell Building Code for the API Tips and Lessons Learned

The Migration API A dedicated API created by Microsoft in order to provide the 3rd party vendors a dedicated way to bring large volumes of contents into SharePoint Online and OneDrive for Business. The Migration API is live since May 2015

Moving Content in O365 Current Tech The Migration API was built to bring large Amount of content into the O365 Cloud Requires the Use of Azure Timer Job based Best way to avoid CSOM throttling Scalable and in continuous improvement

Moving Content in O365 Current Tech Source Package Azure Temporary Storage SharePoint/ODB Final Destination File Share, SharePoint On-Prem and potentially any other data Source Create package for the API to be able to accept it Use the Power of ingestion of Azure to bring the content faster in Microsoft network Timer Job based import in a scalable way that will not hurt the service using back end resources

Moving Content in O365 Current Tech File and Folders Basic Metadata Permissions Versions

Moving Content in O365 New Capability File and Folders Basic Metadata Permissions Versions Encryption at Rest Alerts Taxonomy

Moving Content in O365 Things to come File and Folders Basic Metadata Permissions Versions Encryption at Rest Alerts Taxonomy Removing the need of a customer Azure Account TAP program available for Developers More Based on Usage and Requests

PowerShell

The Migration API and PowerShell What is What The PowerShell Commands PowerShell based Client side tool Performs the end to end content migration The Migration API CSOM based Call Requires Transformation of data Server side Service Requires the content to already be in Azure container

The Migration API and PowerShell Tooling responsibility Source Package Azure Temporary Storage SharePoint/ODB Final Destination File Share, SharePoint On-Prem and potentially any other data Source Create Package for the API to be able to accept it Use the Power of ingestion of Azure to bring the content faster in Microsoft network Timer Job Based import in a scalable way that will not hurt the service using back end resources Client Tool Azure API SPO Migration API

PowerShell The Commands New Convert Set Submit New-SPOMigrationPackage ConvertTo-SPOMigrationTargetedPackage Set-SPOMigrationPackageAzureSource Submit-SPOMigrationJob

PowerShell commands Flow File Share Server Exported Content Package Azure Temporary Storage SharePoint/ODB Final Destination New SharePoint On Premise Export-SPWeb From On-Premise Convert Set Submit Un Compressed

PowerShell Supported scenarios File and Folders Basic Metadata Permissions Versions File Share SP 2010, 2013, 2016

New-SPOMigrationPackage PowerShell New-SPOMigrationPackage The first command will read the list of content targeted by the source path and then will generate the appropriate XML to perform migration. Here are the required parameters to enter (others are optional) SourcefilesPath : pointing to your content you want to migrate OutputPackagePath : pointing to your Temporary folder Ex: New-SPOMigrationPackage -SourcePath C:\Users\Pictures\ -OutputPackagePath C:\Users\test\tempPackage

ConvertTo-SPOMigrationTargetedPackage PowerShell ConvertTo-SPOMigrationTargetedPackage The Second command will simply convert the xml generated in your temporary folder into a format that the destination will understand. This will be the final package. Here are the required parameters to enter (others are optional) TargetwebURL : point to the Web of the destination SourceFilesPath : pointing to your content you want to migrate SourcePackagePath : pointing to your Temporary package folder OutputPackagePath : pointing to your final package folder targetDocumentLibraryPath : This is the path to your destination library Credentials : SPO credential that has admin rights to the destination site Ex: ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl https://datamigration-my.sharepoint.com/personal/admin_datamigration_onmicrosoft_com/ -SourcePackagePath C:\Users\test\tempPackage -OutputPackagePath C:\Users\test\SPOpackage -targetDocumentLibraryPath Documents -sourcefilespath C:\Users\Pictures\ -Credentials (Get-Credential admin@datamigration.onmicrosoft.com)

Set-SPOMigrationPackageAzureSource PowerShell Set-SPOMigrationPackageAzureSource The third command will simply convert the XML generated into a format that the destination will understand. Here are the required parameters to enter (others are optional) SourceFilePath : Pointing to the content you want to migrate SourcePackagePath : Pointing to the final package folder AccountName : The name of the Azure storage account Accountkey : The key to the Azure storage account AzureQueueName : The name of the azure queue blob(optional but recommended) We also recommend to take the output of this command and store it in a variable for the next command. $temporaryVariable = Set-SPOMigrationPackageAzureSource…. Ex: $azureSource = Set-SPOMigrationPackageAzureSource -SourceFilesPath C:\Users\Pictures\ -SourcePackagePath C:\Users\test\SPOpackage -AccountName mystoragename -AccountKey 000000000000000000000000000000000000000000000000000000000000000000000000000000== -AzureQueueName migrationqueue

Ship drive requirements Max 6TB drives SATA II or III Format the drive ( not a requirement but still makes things easier) Prepare your container Install the Azure tool : http://go.microsoft.com/fwlink/?linkid=301900&clcid=0x409

Submit-SPOMigrationJob PowerShell Submit-SPOMigrationJob The last command will simply convert the XML generated into a format that the destination will understand. Here are the required parameters to enter (others are optional) TargetwebURL : Point to the Web of the destination MigrationPackageAzureLocations : The temporary variable from the previous command Credentials : SPO credential that has admin rights to the destination site Ex: Submit-SPOMigrationJob -TargetWebUrl https://datamigration-my.sharepoint.com/personal/admin_datamigration_onmicrosoft_com/ -MigrationPackageAzureLocations $azureSource -Credentials (Get-Credential admin@datamigration.onmicrosoft.com)

PowerShell Remarks for PowerShell Export-SPWeb Target a List Uncompressed New-SPOMigrationPackage Be careful of SPO limitations ConverTo-SPOMigrationTargetedPackage Using different output folder is recommended Set-SPOMigrationPackageAzureSource The Azure Containers will be required for the Submit-SPOMigrationJob Submit-SPOMigrationJob The destination list needs to exist

Migration API

Building code for the Migration API Uses PRIME package formatting Recommended to stay with what the PowerShell commands supports SPO Migration TAP program spomigtap@microsoft.com The Migration API is agnostic of the source

The Migration API calls [ClientNS.ClientCallableMethod]         public Guid CreateMigrationJob (             Guid gWebId,             string azureContainerSourceUri,             string azureContainerManifestUri,             string azureQueueReportUri,) [ClientNS.ClientCallableMethod]         public Guid CreateMigrationJobEncrypted(             Guid gWebId,             string azureContainerSourceUri,             string azureContainerManifestUri,             string azureQueueReportUri,             MigrationOption AES256CBCKey)

Migration Job Status [ClientNS.ClientCallableMethod]         public Guid SPMigrationJobState (Guid MigrationJobId)

Extra Requirement for Encryption The encryption method should follow the AES CBC 256 Standard The IV should be different for every file including the manifests in the package. should be stored as a metadata on each files. Name = [IV] Value = [Base64Encoded byte array of the IV]

Migration API Encrypted Queue Here is the JSON content in the queue message {"Label", "Encrypted"}, {"JobId", "[JobId value]"}, {"IV", ”[IV value in base64format]”}, {"Content", “[encrypted message in base64string]”} Once decrypted the messages will be the same as the API without encryption.

Migration API Logs [Date]Message [Date] [Action] message [Date] [item type] [SPObject name] [SPObject path] [SPObject ID] [Action] message Note that there may be other structures, and in some cases message will overlap into multiple lines, such as

Optimizations Parallel jobs Find the perfect package size Number of items Weight of metadata Weight in GB over the wire Single Report queue for every jobs Keeping every threads running No over provisioning

Please use Event Board to fill out a session evaluation. Questions? Please use Event Board to fill out a session evaluation. Thank you!

Sample Code $userName = "admin@contoso.onmicrosoft.com" $adminSite = "https://contoso-admin.sharepoint.com"  $sourceFilePath = "D:\data\documents\" $packagePath = "d:\data\documentPackage" $targetWebUrl = "https://contoso.sharepoint.com/sites/finance"  $spoPackagePath = "d:\data\documentPackageForSPO" $targetLibrary = "Shared Documents"  $azureAccountName = "acctName" $azureAccountKey = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000==" $azureQueueName = "contosoMigrationQueue"  $cred = Get-Credential $userName New-SPOMigrationPackage -SourceFilesPath $sourceFilePath -OutputPackagePath $packagePath  ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl $targetWebUrl -SourceFilesPath $sourceFilePath -SourcePackagePath $packagePath -OutputPackagePath $spoPackagePath -TargetDocumentLibraryPath $targetLibrary -Credentials $cred $azureSource = Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFilePath -SourcePackagePath $spoPackagePath -AccountName $azureAccountName -AccountKey $azureAccountKey -AzureQueueName $azureQueueName Submit-SPOMigrationJob -TargetWebUrl $targetWebUrl -MigrationPackageAzureLocations $azureSource -Credentials $cred