What Is The SSIS Catalog and Why Do I Care? Stan Geiger Sr. Product Manager – BI Tools Idera @MSBI_Stan | stan.geiger@idera.com

About Me Sr. Product Manager with Idera Geek Sync Presenter Performance Monitoring of Microsoft BI stack Backup and Recovery of Microsoft SQL Server Geek Sync Presenter Blog Contributor HSSUG presenter Over 25 years experience BI, Data Architect DBA Developer Data Analyst

Agenda Introduction SSIS Catalog Overview SSIS Project Deployment Execution and Logging SSISDB Practical Examples

SSIS Catalog Catalog Creation Catalog Objects Projects Packages Parameters Environments Companion database SSISDB New Repository Model introduced in SQL Server 2012. Instead of deploying packages we deploy projects. Previous options: Deploy to file system. Deploy to package store. (Which is nothing more than storing it on the file system under the SQL Server directory) Deploy to SQL Server (Stored in MSDB syssispackages) Previous Ways to deploy: Create deployment utility from the project. (Run it an install to file system or SQL Server) Use DTUTIL command line (run from command prompt to manage packages) Use SSMS You can upgrade packages to the new deployment model. (Upgrade Wizard available in Data Tools/Visual Studio) You lose some capabilities you previously had.  For example, configurations used to configure SSIS packages across multiple environments are no longer available.  Project parameters now replace configurations along with Environments, which live on the Integration Serves Catalog.  “DEVIN KNIGHT has a good blog on upgrading” Catalog Creation After install, right click on Integration Services and select “Create Catalog”. Follow the wizard. “Open SSMS and view properties”

SQL Server Database Engine Integration Services Catalog SSIS Catalog SQL Server Database Engine Integration Services Catalog Folder Projects Environments Parameters Environment References Packages Environment Variables Catalog consists of folders. Each folder can have one or more projects and environments. Packages live in projects. Projects Projects are deployed to the “Projects” folder. Projects consist of: Parameters (project, package level) Packages deployed with project References to environment variables Version history is stored at the time the project is deployed. By default settings, 10 versions back are maintained. Environments Allows you to run projects in multiple environments. Replaces need for configuration files. Example, test vs. prod. You can assign different values to parameters in your project based on environments. Environments can be assigned to projects or packages at execution time.

SSIS Catalog Open SSMS Show catalog properties. Clean logs Retention Period Default logging level Max number of versions Click on project and go to configure Show parameters Show references Show how parameter values are based on references to an environment Click on Environments Go to properties Go to variables Add environment example

SSIS Project Parameters Assigns values at time of execution Package parameters package scope Expressions allowed Project Parameters Shared among all packages in the project Literal values Parameters can be assigned at time of execution using environment variables. Parameters can be defined using expressions only at the package level. Package level parameters are scoped at the package level only. Project parameters are shared across all packaged in the project. Literal values only, cannot use expressions.

Deployment Project Based Versioning Parameters Environments Visual Studio SSMS isdeploymentwizard.exe Versioning Parameters Environments Environment variables Assign to parameters during package execution Configurable Will show Visual Studio deployment in example. “Open SSMS” Show project versions example. Parameters Environments Environment Variables

Deployment Environments Use with project parameters Multiple configurations (test, prod, dev, etc.)

Deployment Visual Studio Example Project level connections Parameters Parameterized connections Example Open Visual Studio Show project parameters Parameterized connection managers

Execution and Logging Execution Methods SSMS dtexec SQL Server Agent Stored Procedure (SSISDB database) Execution Methods Right click on package in SSMS and execute Use dtexec (calls stored procedures in SSISDB to create execution, set execution parameters and start execution) SQL Server Agent Job TSQL using stored procedures residing in SSISDB

Execution and Logging Logging Levels None Basic (default) Performance All events except custom and diagnostic events Performance OnError and OnWarning events Performance Statistics Verbose Kitchen Sink Customized (SSIS 2016) Previous to 2012 logging was only at the package level and had to be turned on and the events selected for logging, along with any custom messages created. The events were logged in the SYSSSISLOG table in MSDB. This option is still available in current versions of SQL Server. Basic: OnPreValidate OnPostValidate OnPreExecute OnPostExecute OnInformation OnWarning OnError Basic It captures important events (like errors and warnings), as well as enough progress information to display what the currently active task is, and how long each task has. Performance used when you are doing benchmarking and performance tuning for your packages. While it actually logs less, it captures a lot more events internally to analyze the performance of the data flow components. A bit more overhead during execution – packages run with Basic will actually run a little faster than Performance Verbose The Verbose logging level captures a lot of events (especially when using OLE DB connections, as they generate many DIAGNOSTIC events). Verbose should only be used when trying to debug or diagnose package failures.

Execution and Logging Examples “Go to SSMS” Execute by “right click” example. SQL Agent job example.

Execution and Logging Stored Procedure Example Execution Reporting Logging Dependent Default Reports Custom Reports “Open SSMS and show TSQL example” Execution reports are available in the catalog by default. Amount of data is dependent on the logging level. You can write your own reports as the data is available through the SSISDB. “Show reporting example in SSMS”

SSISDB Catalog Database Tables Views Stored Procedures Execution information Configuration information Views Stored Procedures Package Execution Administration Maintenance SSIDB catalog database contains all tables, views , stored procedures gear around: Execution information of packages Configuration of packages and projects Administration of projects and packages as well as execution of packages through stored procedures.

SSISDB executions - executables inside the package, parameter values and data taps. executables – all executable objects within a package. executable_statistics – data about all executable objects execution_component_phases – log referring to each data flow execution_data_statistics – total rows sent through the data flows execution_parameter_values – parameter values at time of execution These tables are centered around information regarding package execution. The views collapse and consolidate data. “Go to examples in SSMS” In execution_parameter_values When the value is 20, the parameter is a project parameter. When the value is 30, the parameter is a package parameter. When the value is 50, the parameter is one of the following. LOGGING_LEVEL DUMP_ON_ERROR DUMP_ON_EVENT DUMP_EVENT_CODE CALLER_INFO SYNCHRONIZED

SSISDB operations - information about each operation run in the SSIS catalog. validations– validation of projects which contains folder name and package name. event_messages – all messages passed during events operation_os_sys_info – cpu count, available physical memory and total physical memory at the time of operation event_message_context - conditions that are associated with execution event messages The operations tables contain information around all operations performed on the SSIS Catalog. Operations Object_type values 101 – deploy 200 – create execution, start execution 202 – stop execution 300 – validate project 301 – validate package Status (1), running (2), canceled (3), failed (4), pending (5), ended unexpectedly (6), succeeded (7), stopping (8), and completed (9). Event_message_context Context_type 10 Task State of a task when an error occurred. 20 Pipeline Error from a pipeline component: source, destination, or transformation component. 30 Sequence State of a sequence. 40 For Loop State of a For Loop. 50 Foreach Loop State of a Foreach Loop 60 Package State of the package when an error occurred. 70 Variable Variable value 80 Connection manager Properties of a connection manager.

Demo Questions? Stan Geiger Sr. Product Manager – BI Tools Idera @MSBI_Stan | stan.geiger@idera.com