Microsoft Ignite 2015 4/23/2017 3:56 PM Advanced SSDT and DACFx Practical techniques for real world database development and deployment Darren Hall M359 © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Ignite 2015 4/23/2017 3:56 PM © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Advanced SSDT and DACFx Microsoft Ignite 2015 4/23/2017 3:56 PM Advanced SSDT and DACFx Importing existing databases using SSDT Automating deployments using the DAC Framework Customising build and deployment © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Meet the players Visual Studio SSDT SSMS SqlPackage DACFx SQL Client Microsoft Ignite 2015 4/23/2017 3:56 PM Meet the players Visual Studio SSDT SSMS SqlPackage DACFx SQL Client dacpac SQL DB © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SSDT and enterprise systems Microsoft Ignite 2015 4/23/2017 3:56 PM SSDT and enterprise systems © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Example enterprise database solution Microsoft Ignite 2015 4/23/2017 3:56 PM Example enterprise database solution RefDB “Global” databases. 1 each of these on a single SQL instance per environment Auth Route I/face BLOB Shard Shard Shard Horizontally partitioned “Shards” ~100 of these across multiple SQL instances © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Enterprise database schema Microsoft Ignite 2015 4/23/2017 3:56 PM Enterprise database schema © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Enterprise database schema Microsoft Ignite 2015 4/23/2017 3:56 PM Enterprise database schema By TheCulinaryGeek from Chicago, USA (Spaghetti with Meatballs 02 Uploaded by the wub) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)] © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SSMS – Extract Data-Tier Application Microsoft Ignite 2015 4/23/2017 3:56 PM SSMS – Extract Data-Tier Application Using SQL Server Management Studio to extract a dacpac resulted in 16% SUCCESS! © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Common issues when importing Microsoft Ignite 2015 4/23/2017 3:56 PM Common issues when importing Invalid objects Old objects referencing non-existent columns / encrypted objects / invalid synonyms / cross database references / circular references / unsupported objects for target platform Engage a Subject Matter Expert: Talk to your DBA to help clean up these issues Validation turned on Turn validation off for initial import! Use the latest version of the tools Import everything and fix issues within Visual Studio Lots of objects SSDT imports the first 1000 stored procedures into Procs1, the next into Procs2 etc You may need to add some folder structure to your project to keep developers sane. Having many 1000’s of objects affects SSDT performance © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Composite projects Reference external databases Microsoft Ignite 2015 4/23/2017 3:56 PM Composite projects Reference external databases Break up a larger database into logical units Separate by schema / functional area Decrease build time – build smaller pieces By User:Amada44 (Own work) [Public domain], via Wikimedia Commons © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Composite project references Microsoft Ignite 2015 4/23/2017 3:56 PM Composite project references dacpac My DB Project Artefacts Artifacts data-tier application SQL DB master msdb system database My Other DB Project Same Database Same Server, Different Database Different Server, Different Database project © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Composite projects in Visual Studio Microsoft Ignite 2015 4/23/2017 3:56 PM Demo Composite projects in Visual Studio © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Automating deployments Microsoft Ignite 2015 4/23/2017 3:56 PM Automating deployments © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Typical pipeline DEV CI QA UAT PROD Microsoft Ignite 2015 4/23/2017 3:56 PM Typical pipeline DEV CI QA UAT PROD © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Desired pipeline flow Changes are applied forwards DEV CI QA UAT PROD Microsoft Ignite 2015 4/23/2017 3:56 PM Desired pipeline flow Changes are applied forwards DEV CI QA UAT PROD © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Drift Measure of change to a database from a known state Microsoft Ignite 2015 4/23/2017 3:56 PM Drift Measure of change to a database from a known state By Jan Kåre Rafoss from Norway, original uploader was RX-Guru at de.wikipedia (Based on Image:Isachsen2004.jpg) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Drift Microsoft Ignite 2015 4/23/2017 3:56 PM © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Deployment automation options Microsoft Ignite 2015 4/23/2017 3:56 PM Deployment automation options Command line using SqlPackage.exe The Swiss army knife of database deployment tooling for dacpacs There are LOTS of options - Wrap calls in .cmd , use publish profiles / response files Outputs are files so can be limiting for automation / workflow PowerShell or .NET Flexible, more easily used in workflows Microsoft.SqlServer.Dac namespaces Have full access to DacServices methods and objects Lots of options – use DacDeployOptions or wrap in cmdlets or functions © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Extract a dacpac using SqlPackage.exe Microsoft Ignite 2015 4/23/2017 3:56 PM Extract a dacpac using SqlPackage.exe SqlPackage.exe ^ /Action:Extract ^ /TargetFile:D:\Documents\AdventureWorks2014.dacpac ^ /SourceDatabaseName:AdventureWorks2014 ^ /SourceServerName:.\SQL2014 ^ /p:DacMajorVersion=2 ^ /p:DacMinorVersion=0 Full description of SqlPackage.exe command line parameters here: https://msdn.microsoft.com/en-us/library/hh550080(v=vs.103).aspx © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Extract a dacpac using PowerShell Microsoft Ignite 2015 4/23/2017 3:56 PM Extract a dacpac using PowerShell { $SourceServerName='.\SQL2014' $SourceDatabaseName='AdventureWorks2014' $TargetFile='D:\Documents\AdventureWorks2014.dacpac‘ $DacFxLocation = 'C:\Program Files (x86)\Microsoft SQL Server\120\DAC\bin' Add-Type -Path "$DacFxLocation\Microsoft.SqlServer.Dac.dll" $DacService = New-Object Microsoft.SqlServer.Dac.DacServices "server=$SourceServerName" $ProgressEvent = Register-ObjectEvent -InputObject $DacService -EventName 'Message' ` -Action { Out-Host -InputObject $Event.SourceArgs[1].Message.Message } $DacService.Extract($TargetFile, $SourceDatabaseName, $SourceDatabaseName, “2.0.0.0") UnRegister-Event $ProgressEvent.Name } © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Comparing SqlPackage to DacServices Microsoft Ignite 2015 4/23/2017 3:56 PM Comparing SqlPackage to DacServices Command Line PowerShell or .NET SqlPackage.exe Microsoft.SqlServer.Dac.DacServices /Action:Extract Extract(…) /Action:Publish Deploy(…) /Action:DeployReport GenerateDeployReport(…) /Action:DriftReport GenerateDriftReport(…) /Action:Script GenerateCreateScript(…) /p:RegisterDataTierApplication=True Register(…) n/a Unregister(…) /Action:Export ExportBacPac(…) /Action:Import ImportBacPac(…) © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SqlPackage /Action:Extract using response file Microsoft Ignite 2015 4/23/2017 3:56 PM SqlPackage /Action:Extract using response file SqlPackage.exe ^ /Action:Extract ^ /sdn:%DatabaseName% ^ /ssn:%SqlServerName% ^ /tf:%DatabaseName%.dacpac ^ /p:VerifyExtraction=True ^ @extract.rsp ^ 2>%DatabaseName%.verify.txt if not exist %DatabaseName%.dacpac ( ^ /p:VerifyExtraction=False ^ @extract.rsp) © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Example Response file for /Action:Extract Microsoft Ignite 2015 4/23/2017 3:56 PM Example Response file for /Action:Extract /OverwriteFiles:True /p:DacMajorVersion=2 /p:DacMinorVersion=0 /p:IgnorePermissions=True /p:IgnoreUserLoginMappings=True © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Response file for /Action:DeployReport Microsoft Ignite 2015 4/23/2017 3:56 PM Response file for /Action:DeployReport /Action:DeployReport /OverwriteFiles:True /p:DacMajorVersion=2 /p:DacMinorVersion=0 /p:IgnorePermissions=True /p:IgnoreUserLoginMappings=True /p:CreateNewDatabase=False /p:IgnoreIndexOptions=True /p:IgnoreAnsiNulls=False /Variables:master="master" © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo PowerShell: Check for drift Microsoft Ignite 2015 4/23/2017 3:56 PM Demo PowerShell: Check for drift By Doka54 (Own work) [Public domain], via Wikimedia Commons © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Drift report warning! A Drift Report compares using default values Microsoft Ignite 2015 4/23/2017 3:56 PM Drift report warning! A Drift Report compares using default values This means a drift report will: Ignore Index Padding but detect changes to Index Options Ignore Fill Factor but detect change to other Table Options Ignore Ansi Nulls This might matter to you and/or your DBA! Consider using a DeployReport instead and explicitly specify the options you want to include/ignore © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Customisation using DACFx Microsoft Ignite 2015 4/23/2017 3:56 PM Customisation using DACFx © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

The Data-Tier Application Framework Microsoft Ignite 2015 4/23/2017 3:56 PM The Data-Tier Application Framework Microsoft.SqlServer.Dac.Model.TSqlModel This is the full model of the database schema. Example uses Code generation using T4 templates for DTOs/Validations etc. CRUD / audit trigger generation Microsoft.SqlServer.Dac.Deployment Namespace containing BuildContributor, DeploymentPlanModifier, DeploymentPlanExecutor classes Static Code Analysis during build Generate schema documentation during build Customise the behaviour of the deployment plan Reliability and repeatability Fit with existing processes Reduces resistance to change when introducing tooling Assist with migration to new processes Control Behaviours Hide confusing / dangerous options Add custom Static Code Analysis rules © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Customising build & deployment Microsoft Ignite 2015 4/23/2017 3:56 PM Customising build & deployment BuildContributor Executed when SQL project is built after project model has been validated. Can access the completed model, in addition to all properties of the Build task and any custom arguments DeploymentPlanModifier Executed when SQL project is deployed, as part of the deployment pipeline, after the deployment plan has been generated, but before the deployment plan is executed. Can modify the deployment plan by adding or removing steps. Can access deployment plan, comparison results, and source and target models DeploymentPlanExecutor Executed when the deployment plan is executed and provides read-only access to the deployment plan. The DeploymentPlanExecutor performs actions based on the deployment plan (https://msdn.microsoft.com/en-US/library/dn268597(v=vs.103).aspx) © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo DeploymentPlanModifier Microsoft Ignite 2015 4/23/2017 3:56 PM Demo DeploymentPlanModifier © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DACFx namespaces and classes Microsoft Ignite 2015 4/23/2017 3:56 PM DACFx namespaces and classes Dacpac automation Microsoft.SqlServer.Dac DacServices DacPackage DacDeployOptions Data-Tier Application management Microsoft.SqlServer.Management.Dac Microsoft.SqlServer.Management.Utility DeployedDac Utility Customise build and deployment Microsoft.SqlServer.Dac.Deployment Microsoft.SqlServer.Dac.Model Microsoft.SqlServer.TransactSql.ScriptDom DeploymentPlanModifier, DeploymentScriptDomStep TSqlModel TSqlScript © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Ignite 2015 4/23/2017 3:56 PM Summary SQL Server Data Tools provide a rich database development experience from source code to deployment The Data-Tier Application Framework enables powerful automation and customisation options Together SSDT and DACFx offer a compelling Database Lifecycle Management story © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

References SQL Server Data Tools Team Blog Microsoft Ignite 2015 4/23/2017 3:56 PM References SQL Server Data Tools Team Blog http://blogs.msdn.com/b/ssdt/ Download Latest SQL Server Data Tools https://msdn.microsoft.com/en-us/library/mt204009.aspx Build and Deployment Contributors https://msdn.microsoft.com/en-US/library/dn268597(v=vs.103).aspx DAC Extensions https://github.com/Microsoft/DACExtensions Microsoft and Database Lifecycle Management (DLM): The DacPac https://www.simple-talk.com/sql/database-delivery/microsoft-and-database-lifecycle-management-%28dlm%29-the-dacpac/ Dacpac Explorer https://github.com/GoEddie/DacpacExplorer Ed Elliot’s blog https://The.AgileSql.Club TSQL Smells https://github.com/davebally/TSQL-Smells Dave Ballantyne’s blog http://dataidol.com/davebally/ © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Related Ignite NZ Sessions 1 SSDT for Visual Studio: Bet you can’t tell what your database looked like 6 months ago [M222] Wed 11:55am Ballroom 1 5 Stop Coding on Pigs [M373] Fri 10:40am Ballroom 1 2 What's new in Visual Studio 2015 and ALM 2015 – Part 1 [M323] Wed 1:55pm NZ2 6 Torment Your Colleagues with the 'Roslyn' .NET Compiler Platform [M385] Fri 1:55pm NZ2 3 What's new in Visual Studio 2015 and ALM 2015 – Part 2 [M332] Wed 3:10pm NZ2 Find me later at… Hub Happy Hour Thu 5:30-6:30pm Closing drinks Fri 3:00-4:30pm 4 SQL Server Execution Plans and the Query Optimizer [M410] Thu 10:40am Ballroom 2

Resources Microsoft Virtual Academy TechNet & MSDN Flash 4/23/2017 Microsoft Virtual Academy Resources TechNet & MSDN Flash Free Online Learning http://aka.ms/mva Subscribe to our fortnightly newsletter http://aka.ms/technetnz http://aka.ms/msdnnz Sessions on Demand http://aka.ms/ch9nz © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Complete your session evaluation now and win! 4/23/2017 3:56 PM © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4/23/2017 3:56 PM © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.