1. Remote SQL Server Troubleshooting and Monitoring Using SSIS
Don St. Cyr
(don saint-seer)
@Machiavelli_too
DBADon on StackOverflow
Remote SQL Server Troubleshooting and Monitoring Using SSIS
Questions, don’t hesitate
How many SQL instances?
Simple application, simple Pkg’s, Simple process
Report Spreadsheet
3 SQL Files
Open C:\Temp
Mod_Stats PS!
Report.HTML
Excel Report short name

2. Sponsors
Resources:
Professional Microsoft SQL Server 2014 Integration Services
(Brian Knight, Devin Knight, Jessica M. Moss, Mike Davis, Chris Rock)
published by Wrox
and
StackOverflow.
Self Taught
Tips appreciated

3. All About Me!
DBA at ScriptPro Formerly worked as a contractor to the U.S Army and the DoD on their Force Management database. Also was a technical trainer at a hospital and a systems administrator for a school district.
Pharmacy Solutions
Robotic Dispensing Systems (Pill counting robot)
Pharmacy Workflow SW
Full pharmacy Mgmt SW
DB team of 4
3k-4k accounts
1k-1.3k accounts
1.3k-1.5k SQL Instance
Robust servers CS
But still need current and historical data ASAP
Build apply change index?
Stats off?
Disk issues?
Latency?

4. Why SSIS for Troubleshooting?
A project to learn on
Portability
Security
Automation
Graphical interface
Simple to work with
Great documentation and community
Preparation for future ETL/data mining projects
Easily interact with other tools (i.e. Powershell & CMD line)
Why not?
Goals:
Portable revolving set of data
DL from site
able reports
Compare site to site
Compare current VS old
New DBA, left up to me
Real Reasons
Add value

5. Project Infancy Rpt Name Query 2 Query 3 Query 7 Query 6 Query 5
Glenn Berry’s
“SQL Server Diagnostic Information Queries” (
One Package to Rule Them All!
Rpt Name
Query 2
Query 3
Query 7
Query 6
Query 5
Query 4
Query 8
Query 9
Query 10
Query 11
Modified versions of Glenn Berry’s Diagnostic Queries
Very good queries
Understand these, understand SQL Server
Thx Glenn
Scripts per day/month
All queries in 1 large package
Linear process
Head to Toe
The first version had separate queries

6. 2nd and 3rd Iterations Rpt Name Query 2 Query 3 Query 4 Query 5
More eyes on Rpt, more Queries
Bloated to
Added parallel streams to run quicker
Pkg still cumbersome
Deployment issues
Slow running
Broke Pkg into two
Important stuff first
Slow second
Maintenance trumping
Thinking to further Break things up
Part 2
Query 8
Query 9
Query 10
Query 11
Query 12
Query 13
Query 14
The next few versions had separate queries

7. Modular Stats The current version has 25-30 separate queries
Improvements Over the Old
Simple construction
Flexible arrangement
Individual self contained pieces can be exchanged/removed
Base system runs with or without any of the individual packages
*NOTE: Package name and text
output file need to be named the same
(see step 4 of Master Package slide)
Hurdles:
Create individual Pkg’s
Execute all the Pkg’s
Execute all Pkg’s securely
Plusses:
Decreased complexity
Flexibility
Targeted
Easier deployment
Specific queries to sites
High of 60 to 25-30
Some queries we thought would be great
The current version has separate queries

8. Nuts and Bolts
We create a Data_Feed table with the important information that we will need to run the packages. The PACKAGE names, the CLASS and SUBCLASS (to create the order), the SAVEDAYS (how many days to save the output), the DIRECTORY where the files get staged, a VALID_TIME and an INVALID_TIME to turn the packages off and on as needed.
SQL Performance Database Stats database
Data_Feed table
Package and output names
Class and Subclass (Dewey decimal system)
Save Days
Directory C:\Temp bad!
Valid/invalid times turn off and on
Show data_Feed SQL with update statements
This will be in the downloads

9. Master Package SP Counts the # of packages in the Data_Feed table
Selects top 1 based on Class then Subclass and then executes the package
Package exports to text files and saves them in the (Directory) folder
Deletes all the text files (using the Package name) older than X (SaveDays) days in the (Directory) folder & subfolders
If SaveDays = 0, the SP deletes all the txt files that match the Package name dot txt
Selects the next top 1 & cycles through steps 2-5 until the count is reached
Last package to run is Rollup which creates the dated folder and moves the files into it
Master Package SP finishes by deleting any empty folders
All packages get placed in the Pkg folder
C:\Temp bad!
All text files in C:\temp will get moved
SQL Agent executes the Master Package SP
Master Pkg iterates over the data in the table
Exec Valid Pkg’s in order
Output to text files
Save Days = 0
Pkg’s fully encrypted
Master_Pkg passes in PW
To keep PW safe Master Pkg also encrypted
Same process can be used for data feed project or any other project that iterates over SSIS pkg’s
Show Master Pkg SQL

10. SSIS Packages Make the individual SSIS packages however you would like
I strongly encourage encrypting the packages and using the password in the Master Package SP to execute
This prevents anyone else from adding in a package of their own for automatic execution
Ensure the SSIS package name is exactly the same as the out file name
It is important to consider data privacy and other confidential information, in our case HIPAA
We avoid all PHI & PII in our reports, this info is about server health only!
Currently all Pkg’s are fairly simple
Coworkers without SSIS experience
SSIS used to transform anything to anything
Pipe delimited output
Output has exact name of pkg to clean itself
60+ sites, 60 plus days, MB

11. Reports
Each days information is saved as individual text files and saved into a dated folder The folders can be pulled back to a system that has Excel installed and full use of Powershell I also have a report in HTML for sites where getting the data back is more difficult, but this report is currently in beta
Servers, no Excel
Powershell to create spreadsheet Charts and Graphs
Example at end
NO PHI, PII, no patient medical data
HTML report via Powershell too, view onsite
HTML rpt is good for DoD and other sites where we cannot remove any data

12. The Rollup Package Name Scope Data Type Value Expression
The Rollup package uses these variables:
Name
Scope
Data Type
Value
Expression
Arg
Rollup
String
/C robocopy C:\Temp C:\Temp\ *.txt /mov /is
"/C robocopy C:\\Temp " + @[User::Path] + " *.txt /mov /is"
Date
(DT_STR,4,1252)DATEPART( "yyyy" , getdate() ) + RIGHT("0" + (DT_STR,4,1252)DATEPART( "mm" , getdate() ), 2) + RIGHT("0" + (DT_STR,4,1252)DATEPART( "dd" , getdate() ), 2)
Folder
C:\Temp
@[User::Folder]
Path
C:\Temp\
Read from the slide
Rollup has two 2 tasks
File system task create directory
Execute proc task to move the files
Rollup creates the folder in year Month Day format
Rollup moves the text files into the newly created folder
Will also move any other text files too

13. The Rollup Package (continued)
The Rollup package uses two tasks: The File System Task configuration as shown here

14. The Rollup Package (continued)
And the Execute Process Task configuration here:

15. Demos
Data Feed table Master Package SP Invalid Time Save Days part 1 Save Days part 2 Save Days part 3 Report Generation

16. My Next Steps (or, How I’ll do it Differently)
In the Master Package, use CMD line only (for DoD and other Gov clients) [currently uses a mix of CMD line & Powershell]
Export the same data to tables rather than text files and then use a new report to query the desired data from the tables
Use the report (or a separate package) to create a single text file for offsite review
Have a continuously updating feed of information that I can roll into my own monitoring tool

17. Thank you sponsors!

18. Don St. Cyr (don saint-seer) Don. StCyr@gmail
Don St. Cyr (don DBADon on StackOverflow
Resources: Professional Microsoft SQL Server 2014 Integration Services (Brian Knight, Devin Knight, Jessica M. Moss, Mike Davis, Chris Rock) published by Wrox StackOverflow