1. A Reluctant User’s Guide
Making Peace With SSIS
A Reluctant User’s Guide
Joshua Lynn

2. My Favorite Resources Andyleonard.blog www.mssqltips.com
Google => Stackoverflow.com

3. A Reluctant User’s Guide Part I – Getting Started
Making Peace With SSIS
A Reluctant User’s Guide
Part I – Getting Started
Joshua Lynn

4. Lessons from Robotics Engineering is Choosing
Every choice comes with pros & cons
Failure is not an option, it’s a requirement
Programmers are the 5th Player
You’re done playing when you deploy

5. SSIS Choices Lessons from Robotics Time Ease Weight Cost Speed Upkeep
NASA
Going to the Moon
SSIS Choices

6. SQL Server Integration Service What is SSIS?
Integration Services can extract and transform data from a wide variety of sources such as XML data files, flat files, and relational data sources, and then load the data into one or more destinations.
- Microsoft

7. Ok, But What Is SSIS Really?
Software tool for ETL and workflow
ETL = Extraction Translation and Loading of Data
Execution Environment
Where packages are run
Server or standalone
Development tools
Included with SQL server
No additional license required
The execution environment is independent of the SSIS server.
If you own SQL server you own SSIS
A SSIS package can be run without an SSIS server or SQL server – directly from Visual Studio or utilities
There are benefits to using SSIS as a server service but there are also benefits to using it standalone

8. Where does it fit in the SQL Server World
SSIS
(as server)
Data Sources & Destinations
Diagram is as a traditional deployment as a server
SSIS DB lives on SQL server and stores the package catalog & logging
Data sources and destinations can be different versions of SQL server or not even SQL
SSIS execution environment can interact with the OS: File system, executables, BAT & PS Scripts
Yes linked servers can make connections to other data sources and T-SQl can do some of the same things.

9. Why Should You Care?
A bridge between SQL Server db engine and the outside world
TSQL can only do so much
SSIS works differently then SQL (in memory by rows not sets)
You already own it
Security & Logging built in
It extends what you can do with SQL server out of the box
It keeps getting better
If you’ve tried it before it may be better now

10. Why is it so annoying? Development – A mind of its own
Deployment – Easy if you know what to do otherwise not so much
Execution Black Box – Getting better
Complicated History of product
DTS (old not to be confused with dtsx) VS SSIS
Dev tools: BIDS / SSDT-BI / SSDT
Documentation and books are weak
“Can’t I use {X} to do the same thing?” - Yes

11. Why I’m the wrong person to talk about SSIS
I’m not an SSIS Expert
Often Frustrated by SSIS
Can do many things in TSQL often faster to develop and process

12. Why I’m the right person to talk about SSIS
I’m not an SSIS Expert
Often Frustrated by SSIS
Can do many things in TSQL often faster to develop and process
A Personal Journey
My goal is to get you to think about SSIS in a way that can be useful to you without having to clear a high threshold to make it work
Present at PSSUG tell your journey

13. What Does It Do Better Than TSQL?
How I think about SSIS
Output files
Logistics
Manipulate external resources
Parallelism
What Does It Do Better Than TSQL?

14. How I think about SSIS Third party tools make it much better.
Visual Studio it can be a stand-alone tool for ad-hoc tasks with a UI
Like a SQL script but with more reach
There is no requirement to use all of it
Removes the need for xp_cmdshell or custom solutions using .net CLR
Allows integration between two worlds T-SQL and the OS /External applications
Security and logging

15. Why Did I Use SSIS
T-SQL great for data, bad at interacting with the rest of the world
SSIS bridged the T-SQL data manipulation world with external resources

16. Why Did I Use SSIS Are there other solutions? Yes
Security
Logging
Scheduling
Data driven & parameters
Extensible
Purpose built for ETL
Are they Better? Maybe, as always the answer is it depends
Cost (Buy vs Build)
Sustainability

17. Rules For Making Peace With SSIS
How I came to terms with SSIS

18. Rules For Making Peace With SSIS
Use SSIS for what its good at
ETL - Data In/Out
Maintenance: sure but there are other ways that may be better
Workflow to bridge data and OS/App functions
Data Import Wizard is SSIS
Maintenance Plans are SSIS

19. Rules For Making Peace With SSIS
Use what you need
You don’t have to use all of it
You don’t have to master all of it to gain a benefit
Use the right tool for the right job especially when there are multiple ways to do the same thing

20. Rules For Making Peace With SSIS
If there is a better way then use it
There are multiple ways to do the same thing as SSIS as well as within SSIS.
T-SQL and Sprocs can do the heavy lifting of ETL just fine
SSIS can also be used for orchestration of a process, the workflow not just the data flow

21. Rules For Making Peace With SSIS
Build static package first then make dynamic with variables

22. Rules For Making Peace With SSIS
Work with Visual Studio and SSDT not against
Dev environment has a mind of its own some times
Let it do the heavy lifting when you can

23. Rules For Making Peace With SSIS
Good Code Is Good Code
Build ETL package first
Build Workflow package to call ETL Package
Use parameters and variables

24. SSIS Development

25. Development Essentials
Install SSDT for Visual Studio
Configure VS for SSIS Development
Anatomy of SSIS Solution

26. How To Get Started SQL Server Dev Edition
SSMS – SQL Server Management Studio
Visual Studio 2015 or 2017 Community Edition
SSDT – SQL Server Data Tools (stand alone installer)
If something doesn’t work uninstall and try again
VS Takes a long time to install

27. SSDT – SQL Server Data Tools
SQL Server Data Tools is a modern development tool for building SQL Server relational databases, Azure SQL databases, Analysis Services (AS) data models, Integration Services (IS) packages, and Reporting Services (RS) reports. With SSDT, you can design and deploy any SQL Server content type with the same ease as you would develop an application in Visual Studio.
For most users, SQL Server Data Tools (SSDT) is installed during Visual Studio installation. Installing SSDT using the Visual Studio installer adds the base SSDT functionality, so you still need to run the SSDT standalone installer to get AS, IS, and RS tools.
SSDT is needed for the version of Visual Studio you are running
SSDT for VS 2015 has a release number of 17.x
SSDT for VS 2017 has a release number of 15.x
Functionality varies a bit by VS version but more by target server of the SSIS package

28. SSDT & Visual Studio 2019
SSIS now installed from Visual Studio Market Place
SSRS & SSAS as of VS 2017 installable from VS Market Place
Services/New-Delivery-Model-for-SQL-Server-Data-Tools-in-Visual- Studio/ba-p/479289

29. Blue back labels are standard Visual Studio items
Red backed are SSIS specific

30. To start over when the workspace gets messed up
Visual Studio Layout
To start over when the workspace gets messed up
(and it will)
Window Menu
Reset Window Layout

31. Where is the SSIS Toolbox ?!
SSIS Menu
SSIS Toolbox
Control Flow tab
Right Click
SSIS toolbox from context menu
NOTE: Context Changes Context Menu

32. Context Changes Context Menu
Control Flow
Data Flow
Design Surface
Item
Design Surface
Item
* Also True for Connection Managers

33. Anatomy of a SSIS Solution
Solutions, Projects, Packages, Oh my!

34. Target Server vs Project Version
Target Server = SSIS Platform Where the package is deployed and executes
Not = The version of the server you are connecting to for data I/O
ProjectVersion = XML Schema version of the dtsx in use,
Can upgrade but not downgrade to the best of my knowledge
VS15 & 17 may be the same ProjectVersion
* Connection Managers have their own configuration for each type of data source/destination

35. Setting Target Server NOTE:
Setting Target Server Changes Functionality
VS 2015 & 2017 with SSDT supports development with backwards compatibility to SQL Server 2012
This is for the SSIS runtime environment not the connection
Deployment to runtime environment must match Target server version
Target Server version effects functionality, tasks, and components

36. Anatomy of a package, project, and dev environment
Control Flow
Data flow
Parameters
Variables
Tasks
Sources
Destinations
Connections
Properties
These are the SSIS concepts you need to understand to get started in development
Security and Deployment are also needed

37. Control Flow vs Data Flow
Control Flow = Sequence of workflow
A simple ETL project would have one Control Flow Task that is the Data Flow to Extract data to a file destination
Double Click on Data Flow Task to open the Data Flow Tab for that task
OR Switch to Data Flow tab and select Data flow from dropdown
Data Flow =
Process of ETL for data

38. Sources, Destinations, & Connections
Connections are how SSIS interacts with external data
Sources & Destinations are SSIS’s adapters to the connection
Example:
Flat File connection (defines structure of flat file)
Flat File Source to read from file
Flat File Destination to write to a file

39. Sources, Destinations, & Connections
The connection defines what SSIS can do with the data
Example:
SQLOLEDB connection (like SQL client connection string)
Connection allows commands
OLEDB source and destinations can execute tsql, sprocs, or CRUD data from a table but behave differently as used in data flow
Not to be confused with

40. Dev Tip
If you start with a source with data SSIS will pick the SSIS data types
If you start with an empty destination SSIS will convert to the destination type for you
True for SQL source but not destination
True for text based files
Smei-true for excel source and destination
Also annotations and control flow labels

41. Demo 1: The useless Demo Extract from db to Flat file
SSIS Demo: 01 - The useless Demo
Why Useless Demo
All data type handling was implicit and simple
Output Matched source exactly
Static references
No Deployment
New SSIS project
Add Data Flow Task
Add OLE DB Source
New Connection Manager
Server Name XPS15-JOSHUA
Database = SSIS_Demo
Add Flat File Destination
Show without connecting workflow
Connect the output of the source
Double click flat file destination
Add new connection manager
C:\Temp\SSIS Demo\My First Output File From SSIS.txt
Check Column names in the first data row
Columns screen set column delimiter to tab
Ok out
Show target folder
Run package
Why is it useless:
All data type handling was implicit and simple
Output Matched source exactly
Static references

42. SSIS Data Types SSIS Data Types
flow/integration-services-data-types?view=sql-server-2017
_Type_Conversions
SSIS uses its own data types regardless of source or destination
Converting types is necessary sometimes as part of the data flow

43. Converting Data Demo Import Flat file (text) to DB with data types
SSIS Demo: 02 - IMP - NOAA Hourly TAB to Table
SSIS Demo: 03 - IMP - NOAA Hourly TAB to Table Convert Data Type
Notice:
Setting Data Type of source
Implicit vs Explicit Conversion
Replacing vs adding columns
Keeping the name for output
Derived vs Converted
SQL Task vs SSIS (Do you trust the Data Source)
Nulls as data in SSIS

44. If it can be named, name it with something meaningful
Dev Tip
If it can be named, name it with something meaningful
{Also annotations, control flow labels, and groups}
Also annotations and control flow labels

45. SSIS Variables & Parameters
External interface to package
Can not be altered inside package
Good for passing in values for runtime
Variables
Internal to package
Values can be expression evaluated at runtime
Values can change at runtime via task

46. SSIS Variables & ParametersVS
T-SQL with Variables & Parameters
{called from SSIS}
{Related but not the same}

47. SSIS Variables & Parameters
SSIS Demo: 04 - EXP - NOAA Hly to Flat File with Params.dtsx
Notice:
Parameters mapped to variables
Parameters set as defaults but can be set externally
Variables with expressions
Possibility of using an expression Task
Use of “\\” in path names to escape \
Expression for Connection String of Flat File Destination
Use of File System task with Variable
Ignore SSIS calling T-SQL with Parameters & Variables

48. Dev Tip
To Duplicate Project: Right Click on Package to duplicate Copy from context menu
Also annotations and control flow labels

49. Dev Tip To Duplicate Project: Right Click on SSIS Packages Folder
Paste from context menu
Notice change in in dtsx files
DTS:DTSID
Also annotations and control flow labels

50. Creating Templates Create package Save to:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Micros oft\SSIS\ProjectItems\DataTransformationProject\DataTransformat ionItems
Restart Visual Studio
Add New Item to the project

51. Recording & Renaming Output
Break Connection from source to next component
Clear output columns
Add back and alias output from source in order
Build new destination
Fix broke bits in between
Update Properties
{This is one way but not the only way}
SSIS Demo: 05 - Reorder Output (Copy 04)
Check Connection string

52. Importing Nulls to db 3 things for importing Nulls
Data Source set to treat missing data a NULL not default of Data Type
Preserve null in any data conversions or derived fields
Set db destination to treat NULLS as NULLS and not default
07 - IMP - NOAA Hourly CSV with Null Handeling

53. Execute SQL With SQL Parameters & Variables
Allows SQL to be called with variables from SSIS runtime
Use with T-SQL code or Stored Procedure Call
Can be used for SQL Data Sources

54. Execute SQL With SQL Parameters & Variables
query-parameters-to-variables-in-an-execute-sql-task?view=sql- server-2014

55. Execute SQL With SQL Parameters & Variables
query-parameters-to-variables-in-an-execute-sql-task?view=sql- server-2014

56. Execute SQL With SQL Parameters & Variables
query-parameters-to-variables-in-an-execute-sql-task?view=sql- server-2014

57. Execute SQL Task With SQL Parameters

58. Execute SQL With SQL Parameters & Variables
04 - EXP - NOAA Hly to Flat File with Params
Look at Data Flow OLE DB data source
Look at relationship between T-SQL Variables & SSIS Variables
INPUT and OUTPUT
Result set

59. Specifying Results from Stored Procedures
06 - EXP - NOAA Hly to Flat File with Stored Procedure Return
With Results syntax needed when return data structure can not be determined

60. Conditional Flow - Tasks
There is no IF, CASE, or other conditional task flow component
There are logical flow controls built in
08 - Controlling flow.dtsx

61. Controlling Flow - Data
Conditional Split Data Flow Component
Default pathway
Conditional CASE like additional Pathways
09 - IMP - NOAA Hourly TAB to UnPivot Table.dtsx

62. Calling Sub-Packages Ability to run a package from another package
Use Parameters

63. Looping Containers Allows for dynamic variation
For each member of a set
Until a condition is met
A bit more like programming
10 - For Each Loop from Record set - EXP to many files.dtsx

64. Script Task & Component
Good way to expand capabilities
Access to C# and .NET without SQLCLR
SCRIPT Component
Source
Destination
Transformation
11 - Script Transform - EXP - NOAA Hly to Flat File.dtsx

65. Dev Tip Debugging Errors and Warnings Output Window Execute package
Play button
Right Click package
Set Break Points
Execute Task
Doesn’t work in Data Flow
Errors and Warnings
Output Window

66. Dev Tip In line activities are terrific in SSIS Conditionals
UnPivot Transform
Derived Fields
Can only replace if same data type
How to add static filed
Conditionals
Variables
Split

67. Deployment Configurations Visual Studio Direct From SSMS PowerShell
Deployment Wizard {Native Domain Only}
Build project -> .ISPAC file
From SSMS
PowerShell
Sprocs & TSQL