SSIS Data Integration Data Warehouse Acceleration Fact and Dimension Table Patterns Jim Miller – President BI Tracks Consulting Product Presentation Mar. 23, 2019

What is Biml? XML Schema used to define SSIS, and other Microsoft ‘BI Stack’ components Use with BimlExpress to auto-generate SSIS Projects and Packages in Visual Studio (SSDT) Who has used Biml? Who is developing a Data Warehouse?

Why Use Biml to Build SSIS Packages? Code Reuse, Code Modularity Code Reuse = Efficiency & Reliability Code Modularity (Smaller Pieces ) promotes Code Reuse Resource Repository Defined Outside of SSIS Projects Connections, Parameters, Packages Standardization of SSIS Project/Package Development Framework Extensions, Package Design Styles Accelerated Change Management SSIS Package ‘diffs’ are simple text file comparisons All made possible by dealing with just XML code. All XML code is generated by encapsulated SELECT statements.

Demo 1 – Generating SSIS Packages in Visual Studio using BimlExpress BimlExpress will generate ‘pure biml tags’, or biml script (embedded C# code nuggets). Who does not want to create higher quality work in less time?

Biml is just XML, so it’s free Biml is just XML, so it’s free. BimlSnap Online, which creates Biml is free. BimlExpress which converts Biml into SSIS packages is a free VS add-in and it’s also free.

Biml and Biml Script Examples Bimlsnap logic is all in the database layer, Biml Script uses embedded .NET code nuggets

BimlSnap (and SSIS) Workflow and Databases bimlsnap_v2 (project config & metadata) Define Build Deploy Changes typically begin with a change to the database schema (metadata) SSIS_Data (runtime) SSIS Catalog SSISDB (runtime) log Log

Demo 2 – Select for XML Everything we do in BimlSnap to generate Biml is based on the “Select… for XML” technique.

BimlSnap Desktop - Components BimlSnap Desktop – Back-End DB, and Front-End App SQL Server Data Tools (SSDT) with BimlExpress Add-in from Varigence SSIS Run-time Framework (Optional) Download: http://www.bimlsnap.com/bimlsnap-desktop- download/ Open Source Databases Front-end Client SSIS Project to update Metadata

ELT vs. ETL Paradigm

BimlSnap - Metadata Management (v2) Built as a Data Mart using Type 1 and Type 2 Dimensions Extends Information Schema Views Front-end Client Includes SSIS Project to Refresh Metadata

BimlSnap – Runtime Framework Uses a free Back-end Database: (SSIS_Data) Extends Functionality of the SSIS Catalog Database (SSISDB) Integrates with BimlSnap Generated Packages to Provide: Project restart ability Project/Package row counts Project/Package runtimes Email alerts Package error logging Feeds a PowerBI Dashboard for Runtime Analytics

Demo 4 – Power BI Dashboard Power BI Project available upon request. Email info@bitracks.com

Demo 2 - BimlSnap Project Configuration Select a Template Email Alerts Project Re-runability Row Counts Select a Package Protection Level Encrypt Sensitive With User Key Do Not Save Sensitive

BimlSnap Package Configuration DataFlow Pattern Package Qualifier Source Connection Source Query Destination Connection Destination Table Truncate Option Identity Insert Options

SSIS/Biml Project Build Project Integration Connections Pattern Definition Package XML Project XML SP VS Package Configuration SP Package Template Parameters Framework Integration

Code Generators Code Generators Merge Statements Table Partitioning

Demo 3 & 4 - BimlSnap Project Walk-Through,and Package Creation BTW, all SQL code needed to generate SSIS packages is in the database.

BimlSnap Foot Print

Demo 5 – Select with HTML Output Documents all projects, packages in an conveniently linked HTML file

Links Blog: Accelerated ELT using BI Markup Language (Biml) Project Vulcan: https://vulcan.codeplex.com/ Varigence: https://www.varigence.com/ Biml Express (VS Add-in): https://www.varigence.com/BimlExpress BimlSnap Website: https://www.bimlsnap.com ELT vs. ETL: https://bitracks.wordpress.com/2018/02/07/elt-vs-etl-for-siss/ Presentation on YouTube: https://youtu.be/0u2w9OisAWI (jump to minute 6)