SSIS Templates, Configurations & Variables Daniel Bowlin MCITP SSIS Templates, Configurations & Variables sqlbidan@gmail.com

A little about me SQL data warehouse developer with Time Warner Cable Founder of the Capital Area SQL Server User Group in Albany NY Certified in the MS SQL Server BI stack About 15 years working with SQL Server and 4 years specialized in the BI Stack I blog on www.BIDN.com

Opening Credits I first learned about management frameworks and templates from two sources: SQL Server 2008 Integration Services, Problem-Design-Solution Most code presented is from this book http://www.amazon.com/Microsoft-Server-2008-Integration-Services/dp/0470525762/ref=sr_1_1?s=books&ie=UTF8&qid=1367539127&sr=1-1&keywords=ssis+problem+design+solution Andy Leonard http://sqlblog.com/blogs/andy_leonard/archive/2011/09/16/designing-an-ssis-framework.aspx

What is a Management Framework? Standardized development methods Metadata (configuration) change management VERY useful for moving through environments Package management and auditing Reusability Package template Helps reduce development time

Management Framework Components Logging mechanism Package start & finish Individual task start & finish Package/task errors Variable value changes Optional ETL components Source vs. Target row count comparisons Standard deviation comparisons over time Additional ETL logging components

Logging Mechanisms A task that will log the start of a package using Execute SQL Task, a stored procedure, and some package values A task to log the completion of the package, same as start. Event handlers to log events in your packages execution

Logging Mechanisms Event handlers OnError logging for the package OnPreExecute for tasks OnPostExecute for tasks OnVariableValueChanged

Management Framework Components Common connections Database Mail Configuration management of Connectors Configurable variables Package template to pull it all together

Package Template A Package Template is the component that puts all the other components in one spot. Stored in (2008) Program Files\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems Accessed from an SSIS Project Right click Project>>Add>>New Item

Template Demo Review the components of template.

Package Configurations Package configurations are like any config file. They help define the parameters used to run a program/package. Typically Name – Value pairs Can be different in different environments Can be manipulated during package execution Parent Package configurations

Package Configurations Framework configurations contain 3 components An environment variable An XML File A SQL Server database table.

Package Configurations The environment variable is pretty simple it is just a name value pair pointing to your XML file. System Applet in Control Panel Advanced Settings >> Environment Variables >> System Variables >> New Name: SSIS_Config_File Value: Path to XML File

Package Configurations XML File contains the connection information to the database

Package Configurations Database contains a table with all the configuration values the package will need. Dbo. SSIS Configurations

Package Configurations The configuration table can contain property settings for: Variables Event Handlers Connection Managers Package properties Executable components

Package Configurations

Package Configurations A few notes The configuration entry pointing to the Environment variable should be at the top of your configurations. The environment is read from the computer that calls the task. Tests using a local copy of SSMS may produce misleading results

Package Configurations Parent Package Configurations Allow a parent package to control the any configuration value of a child package. Can direct the child package to behave differently depending on the value of the configuration variables passed from the parent.

Configurations Demo Demo configurations Add a new connection Make the new connection a configuration Run a sample package Change configuration Rerun sample package

Development Standards Every shop should have development standards and stick to them Naming conventions ProjectName_PackageName ProjectName_PackageName_VariableName Parent_VariableName Conn_DatabaseName (no server name) Deployment standards Promote to production standards Templates and configurations help keep things consistent. Annotate

Variable development & use Along with configurations, variables can build a lot of flexibility into your packages Any value that you think might change at any time in the future should be put in a variable and given a configuration. Date ranges (in days ago) Distribution lists

Variable development & use Variables are useful for Full queries Open query development Storing values retrieved from a db for use elsewhere in your package. In ETL processes I consider variables for row counts a necessity. Error handling management Error counters Max error count.

Variable development & use Variables can be used to manage program flow in precedence constraints. Variables are limited in their use only by your imagination, time for development, and flexibility needs. A cool thing about variables is that they can have configurations as well

Variable development & use Show a few samples of variable use

Review Management Framework & framework components Logging and Auditing Template packages Package Configurations Development standards Variable use

Questions ????