Populating a Data Warehouse Data Warehouse Acceleration Fact and Dimension Table Patterns Jim Miller – President, BI Tracks Consulting SQL Saturday – Louisville, TX July 21, 2018

What is Biml? XML Schema used to define SSIS, and other Microsoft ‘BI Stack’ components Use with BimlExpress to auto-generate SSIS Projects and Packages in Visual Studio (SSDT) Who has used Biml? Who is developing a Data Warehouse?

Why Use Biml to Build SSIS Packages? Code Reuse, Code Modularity Code Reuse = Efficiency & Reliability Code Modularity (Smaller Pieces ) promotes Code Reuse Resource Repository Defined Outside of SSIS Projects Connections, Parameters, Packages Standardization of SSIS Project/Package Development Framework Extensions, Package Design Styles Accelerated Change Management SSIS Package ‘diffs’ are simple text file comparisons All made possible by dealing with just XML code. All XML code is generated by encapsulated SELECT statements.

Demo 1 – Generating SSIS Packages in Visual Studio using BimlExpress BimlExpress will generate ‘pure biml tags’, or biml script (embedded C# code nuggets). Who does not want to create higher quality work in less time?

Biml is just XML, so it’s free Biml is just XML, so it’s free. BimlSnap Online, which creates Biml is free. BimlExpress which converts Biml into SSIS packages is a free VS add-in and it’s also free.

Biml and Biml Script Examples Bimlsnap logic is all in the database layer, Biml Script uses embedded .NET code nuggets

BimlSnap v2 Databases bimlsnap_mart_v2 (metadata) bimlsnap_v2 (project config) SSIS_Data (runtime) Sync Build Logs Changes typically begin with a change to the database schema (metadata)

Demo 2 – Select for XML Everything we do in BimlSnap to generate Biml is based on the “Select… for XML” technique.

BimlSnap Online and Desktop Use SQL Server Procedures, and Tables to generate Pure Biml code Includes a repository of re-usable ELT Patterns Common Data Warehouse patterns are currently available Integrated into an extensible ‘ELT Framework’ Visual Studio Designer is only used for ‘Unit testing’, not for GUI style development. This is the key paradigm shift from traditional SSIS Project/Package development. Results in more, smaller, modular packages – better engineering principle. Automatically handles the ‘run all’, and parallel grouping by assigning a sequence number.

BimlSnap Desktop - Components BimlSnap Desktop – Back-End DB, and Front-End App SQL Server Data Tools (SSDT) with BimlExpress Add-in from Varigence SSIS Run-time Framework (Optional) Snap_Mart v2 Metadata Solution (Optional) Download: http://www.bimlsnap.com/bimlsnap-desktop- download/ Back-end Databases Front-end Clients Optional SSIS Project to update/sync Metadata

Feature Matrix: http://www.bimlsnap.com/product-feature-matrix/ Changes typically begin with a change to the database schema (metadata) Metadata changes are then captured, providing new tables/columns in front-end programs Code Generators (e.g., MERGE statement) are updated using new metadata (placed in new Biml files) SSIS Packages are re-generated with new Biml

ELT vs. ETL – Best Approach?

BimlSnap - Metadata Management (v2) Stand alone application Has its own Data Mart using Type 1 and Type 2 Dimensions Uses a free Back-end Database: Front-end Client Extends Information Schema Views Includes Optional SSIS Project to Refresh Metadata

Demo 3 – Metadata Management Back end database SSIS Project to populate Front-end program to customize SSIS Project to ‘synchronize’ Sync

BimlSnap Desktop – Getting Started w/Metadata Steps Download Databases and Front-end products: http://www.bimlsnap.com/bimlsnap-desktop-download/ Run SQL Database Scripts (bimlsnap_v2, bimlsnap_mart_v2, SSIS_Data) Configure and Run SSIS Project: “Metadata Refresh v2” Use Metadata Front-end (SnapMart v2.3.exe) to designate databases for Metadata extracts Re-run SSIS Project: “Metadata Refresh v2” Update Dimension Column Types for: Business Keys (bk) ‘Type 2’ (t1) attributes if needed Run SSIS Project: “Metadata Sync v2” to ‘sync’ metadata with database: bimlsnap_v2 Only item we charge for is the “BimlSnap Desktop” Front-end, which is technically optional, but does make the product easier to use. Even this can be free if you occasionally tweet/blog on BimlSnap. Just email the link for your social media post to info@bitracks.com, and we’ll extend your subscription.

BimlSnap – Runtime Framework Uses a free Back-end Database: Extends Functionality of the SSIS Catalog Database (SSISDB) Integrates with BimlSnap Generated Packages to Provide: Project restart ability Project/Package row counts Project/Package runtimes Email alerts Package error logging

Demo 4 – Power BI Dashboard Power BI Project available upon request. Email info@bitracks.com

BimlSnap Project Configuration Select a Template Email Alerts Project Re-runability Row Counts Select a Package Protection Level Encrypt Sensitive With User Key Do Not Save Sensitive

BimlSnap Package Configuration DataFlow Pattern Package Qualifier Source Connection Source Query Destination Connection Destination Table Truncate Option Identity Insert Options

SSIS/Biml Project Build Project Integration Connections Pattern Definition Package XML Project XML SP VS Package Configuration SP Package Template Parameters Framework Integration

Code Generators Code Generators Merge Statements Table Partitioning

Demo 5 – Code Generators Merge Statements Partitioning Statements BTW, all SQL code needed to generate SSIS packages is in the database.

Demo 6 – Select for HTML Documents all projects, packages in an conveniently linked HTML file

Links Blog: Accelerated ELT using BI Markup Language (Biml) Project Vulcan: https://vulcan.codeplex.com/ Varigence: https://www.varigence.com/ Biml Express (VS Add-in): https://www.varigence.com/BimlExpress www.bimlsnap.com www.bitracks.com BimlSnap Online Video: https://youtu.be/PKQ3r0w_EO8 ELT vs. ETL: https://bitracks.wordpress.com/2018/02/07/elt-vs-etl-for-siss/ For a copy of this presentation, please email us at: info@bitracks.com