New Features and Enhancements in SSIS 2016 RISSUG OCTOBER 5, 2016 Matt Batalon MCITP/MCSA email: mbatalon@outlook.com Twitter:@MattBatalon https://www.linkedin.com/in/matthewbatalon

About Me Presently a Senior SQL Server Developer for healthcare company in RI Using SQL Server since SQL Server 2000 SQL Server Certified 2008, 2012, 2014 (MCTS, MCITP, MCSA) President of the Rhode Island SQL Server Users Group (Official PASS Chapter) sqlsaturday.com (Providence December 10, 2016) at Bryant University Precons: December 9th, Grant Fritchey & Aarron Bertrand

Agenda: New SSIS Features in 2016 Too many to go over, some of the best ones (my opinion) AlwaysOn Support for SSIS catalog Automatically Tuning Buffer Size Incremental Package Deployment Always Encrypted New Roles for SSIS Custom Logging Control Flow parts Azure Support in SSIS SQL Server 2016 is a major upgrade. Too many features to go over.

Demo Environment All free! SQL Server 2016 Developer Edition SQL Server 2016 Management Studio Visual Studio Community 2015 SQL Server Data Tools 2016 AdventureWorksCTP32016 db WorldWideImporters db All free!

SQL Server Integration Services Started out as DTS in SQL Server 2000 Became SSIS in 2005 Largest upgrade in 2012 Project Deployment Model SSIDB Catalog More SSIS 2016 Features Some build upon features made available in 2012 and 2014

2016: SSISDB AlwaysOn Support Used for High Availability of your SSISDB Catalog DB “Technically” not supported in SQL Server versions 2012/2014 Could be implemented with some work arounds. Stop and think about how critical your SSIS deployed packages are. SSIS Catalog is extremely popular right now. Might be a useful feature if properly implemented. ensuring high availability for your projects, packages, logs and environments. Backup SSISDB: Backup Master Encryption Key to a file Backup DB Generate a Login Script for a specific user for a cleanup job, startup scripts and maintenance jobs. Restore SSISDB Enable Clr Create and Assymetric Key and login with an UNSAFE permission, in order to run clr stored procs the ssisdb needs. Past versions: Connect to every node and create the SSISDB catalog.  We need to create the catalog even on secondary nodes to create the other server-level objects (cleanup jobs, keys, accounts etc) that are used by SSIS. Delete the SSISDB databases on secondary nodes. Create an availability group, specifying SSISDB as the user database Specify secondary replicas. dm_hadr_availability_replica_states ars inner join sys.availability_groups ag

SSISDB with AlwaysOn 2012/2014 When a failover occurred, SSISDB would failover to the secondary node, which is now the primary When a package was executed on the new primary node, typically would generate and error: Error 15581: Please create a master key in the database or open the master key in the session before performing this operation This occurs because of the way SSISDB encrypts sensitive information (such as connection string passwords, and certain parameters. To resolve this, the database master key needs to be re-encrypted by the service master key after failover occurs. To resolve it, you’d need to re-encrypt the database master key by running this T-SQL script.

To enable AlwaysOn on SSIS 2016 Install a Windows Failover cluster, install SQL Server 2016 with Integration Services on each node and enable the Always On feature on each instance Create the SSIS catalog on the primary node Add the SSISDB to the Availability Group. Right-click the SSIS catalogs node and choose Enable AlwaysOn Support...

Enable AlwaysOn Support Once you do this, on the Primary Node, you can right click, and Enable AlwaysOnSupport.

Connect to the replicas

2016: AutoAdjustBufferSize Property New property added to the data flow task in 2016 Auto Adjusts the buffer size Eliminates some trial and error with DefaultMaxBufferRows and DefaultBufferSize Default Values are usually too low, forced intervention The data flow uses buffers to transfer and transform the data. Buffers are chunks of memory. The size of the buffer is determined by two properties: DefaultBufferMaxRows - with a default of 10,000 rows - andDefaultBufferSize - with a default of 10MB. Imagine you have to fill buckets of water and pass them along a line to put out a fire. If you have bigger buckets, you can transfer more water and the fire will be put out more quickly. But if your buckets are too big, it takes too long to fill them with water while the fire is raging on. So you have to find the optimal size of the buckets to balance the amount of water being passed and the time to fill those buckets. The same is true with data flow buffers: you have to find the optimal size of the buffers.

SSIS Data Type Conversions Example Source Row SSIS Data Type Conversions = DT_STR = 10 Bytes = DT_Date = 3 Bytes = DT_STR = 20 Bytes = DT_UI4 = 4 Bytes = DT_CY = 8 Bytes Total: 79 Bytes

A look inside the buffer… EstimatedRowSize * DefaultMaxBufferRows = EstimatedBufferSize 79Bytes * 10,000 = 790KB 790KB < 10MB – Up the default MaxRowSize Smaller the row size, the more rows fit into a buffer. The Goal is the have as many rows possible inside a single buffer

sp_spaceused <table name> bytes = 42,112*1024 = 43,112,688 row = 43,112,688/ 500,000 = 86 bytes per row

DefaultMaxBufferRows DefaultBufferSize / rowsize 10MB * 1,024 * 1,024 / 86 = 121,927

Automatic Performance Tuning All the MATH goes away! AutoAdjustBufferSize will adjust the buffer size accordingly to find the best pipeline performance Ignores DefaultMaxBufferSize, DefaultMaxRowSize Turned off by default

2016: Incremental Package Deployment SSIS 2012 introduced the SSIS catalog Project Deployment Model 1 or many SSIS packages could be created as a SSDT project, and deployed to the SSIS catalog project parameters, project connection managers for all packages in the project Easily switch environments, connections, parameters, etc..

One glaring omission…. You were forced to choose “project” or “package” model deployment. If you chose project model, force to deploy the entire project, not just one package Versioning : project has 30 packages, and I need to only change 1 package Work In Progress project package, bug fix in current project SSRS has this for a while. Based upon projects and solutions. Right click deploy.

2012/2014 SSIS Previous Versions

2016 SSIS Latest Version

2016: Always Encrypted Data Support in SSIS Very popular feature in SQL Server 2016 Enhanced data security for data not just at rest but also data in “motion” SSIS 2016 support for reading and writing data in an Always Encrypted Db Requires an additional connection property and Column Master Key, Column Encryption Key, local certificates(keys) and an ADO.NET connection column encryption setting=enabled First of all what is Always encrypted. Always Encrypted is a new feature in SQL Server 2016, which encrypts the data both at rest *and* in motion (and keeps it encrypted in memory). So this protects the data from rogue administrators, backup thieves, and man-in-the-middle attacks. Unlike TDE, as well, Always Encrypted allows you to encrypt only certain columns, rather than the entire database. The client library ensures that plaintext is only revealed within the application or middle tier, and nowhere in between the application and the database. Always Encrypted is the ADO.NET 4.6, so you will need to ensure .NET Framework 4.6 is installed on any machine that will run a client application that interfaces with Always Encrypted data. Demo Setup. a connection has been made with the additional connection property column encryption setting=enabled the column master key created in the database is imported into the local certificate store (more detail on this later on)

Always Encrypted cont. OLEDB Connection Managers do not support ADO.Net Connection Managers only supported When first trying to read/write data from and Always Encrypted Database, SSIS may throw an error Normally in SSIS you would use OLE DB connection managers to move data between sql tables.

2016: New SSIS roles Old: ssis_admin This role provides full administrative access to the SSIS Catalog database. New ssis_alwayson_monitor role to support AlwaysOn ssis_logreader SSIDB View access, can now view the reports in the SSIS catalog Previously, if you were not a ssis_admin, the reports would show no data when viewed In the SSIS catalog, a lot of information about the execution of the packages is automatically logged. There are a set of built-in reports helping you to find detailed logging results about those executions. However, in earlier versions you needed to be an SSIS admin in order to view the data inside the reports. If not, the reports would remain empty.

No Results

2016: Custom Logging Levels Determines how much information in logged to the SSIS catalog Something between None and Verbose Can be set as a server wide default Can minimize the data written to the SSISDB when packages execute Not to be confused with log providers inside a ssis package, like text files, sql tables, windows events…

Previous Levels Included None Basic (default- everything) Performance Verbose RuntimeLineage (new) Custom (new) None - logging is turned off. Basic - all events are logged, except custom and diagnostic events. The name is quite misleading, as a lot of information is logged. This is the default logging level of the SSIS catalog. Performance - only performance statistics and the OnWarning and OnEvent are logged. This logging level might log less information than the basic logging level, but it depends on the number of data flows and their complexity. Data flow performance information of this logging level is shown in the catalog report Execution Performance. Verbose - all events are logged. Only use this logging level for advanced troubleshooting or fine tuning. RuntimeLineage: new logging level in 2016. Collects the data required to track lineage in the data flow i.e. where the data came from and where did the data go. Catalog .event_message_context.

2016: Control Flow Parts Formerly name Control Flow Templates Can be viewed in the SSIS solution explorer Can contain an encapsulated Data flow task Re-Usable component in packages Can be changed once, and no need to edit each package Can’t be edited inside a package Once created, appear in SSIS toolbox

Moving Data to and from the Cloud AzCopy Azure Data Factory Azure Export/Import service Powershell FTP SSIS – New features in SSIS 2016 AzCopy: A popular command-line utility designed for high-performance uploading, downloading, and copying data to and from Microsoft Azure Blob Storage. Azure Import/Export Service: Used to transfer large amounts of file data to Azure Blob storage in situations where uploading over the network is prohibitively expensive or not feasible by sending one or more hard drives containing that data to an Azure data center. Azure Data Factory (ADF): With the latest ADF service update and Data Management Gateway release, you can copy from on-premises file system and SQL Server to Azure Blob.  See Azure Data Factory Update – New Data Stores and Move data to and from Azure Blob using Azure Data Factory and Move data to and from SQL Server on-premises or on IaaS (Azure VM) using Azure Data Factory.   Powershell: Work with Azure storage programmatically.

2016: Azure Support SSIS Azure Pack – Control Flow and Data Flow Sources Destinations Connections HDInsight Import Export Wizard (2016)

2016: Azure Pack Technically available in SSIS 2014 Still very new, separate download/install Custom SSIS components to move data to and from Azure and do much more DataFlow Azure Blob source is new in 2016 Along with HDInsight Pig task Azure Blob Source may have been added to the 2014 azure pack. Not highlighted in books online for 2014 azure pack.

Azure Blob Storage is cheap

Control Flow Data Flow

Azure Storage Connection Manager

Azure Storage Account Information

Import/Export Wizard Always has been a very useful tool to quickly import or export data as one off task. Now in 2016, has Azure blob source and destination support

Many More Updates OData Source V3 or V4 OData connection manager for ATOM and JSON data Excel 2013 and Excel 2016 connections HDFS File Connection Support Target Specific Server Versions ATOM = Web Feeds ( open data source).

Sources and References for this Presentation - Links What's New in Integration Services https://msdn.microsoft.com/en-us/library/bb522534.aspx MSSQL Tips https://www.mssqltips.com/sqlservertip/4362/whats-new-in-sql-server-integration-services-2016--part-1/ https://www.mssqltips.com/sqlservertip/4371/whats-new-in-integration-services-2016--part-2/ https://blogs.msdn.microsoft.com/mattm/2012/09/19/ssis-with-alwayson/