Populating a Data Warehouse Data Warehouse Acceleration Fact and Dimension Table Patterns Jim Miller – President, BI Tracks Consulting SQL Saturday – Houston, TX June 23, 2018

What is Biml? XML Schema used to define SSIS, and other Microsoft ‘BI Stack’ components Use with BimlExpress to auto-generate SSIS Projects and Packages in Visual Studio (SSDT) Who has used Biml? Who is developing a Data Warehouse?

Why Use Biml to Build SSIS Packages? Code Reuse , Code Reuse , Code Reuse Microsoft’s Project Vulcan was designed to bring Code Reusability to the MSBI Stack Resource Repository Defined Outside of SSIS Projects Connections, Parameters, Packages Standardization of SSIS Project/Package Development Framework Extensions, Package Design Styles Accelerated Change Management SSIS Package ‘diffs’ are simple text file comparisons All made possible by dealing with just XML code. All XML code is generated by encapsulated SELECT statements.

Demo 1 – Generating SSIS Packages in Visual Studio using BimlExpress BimlExpress will generate ‘pure biml tags’, or biml script (embedded C# code nuggets). Who does not want to create higher quality work in less time?

Biml is just XML, so it’s free Biml is just XML, so it’s free. BimlSnap Online, which creates Biml is free. BimlExpress which converts Biml into SSIS packages is a free VS add-in and it’s also free.

Biml and Biml Script Examples

Demo 2 – Select for XML Everything we do in BimlSnap to generate Biml is based on the “Select… for XML” technique.

BimlSnap Online and Desktop Use SQL Server Procedures, and Tables to generate Pure Biml code Includes a repository of re-usable ELT Patterns Common Data Warehouse patterns are currently available Integrated into an extensible ‘ELT Framework’ Visual Studio Designer is only used for ‘Unit testing’, not for GUI style development. This is the key paradigm shift from traditional SSIS Project/Package development. Results in more, smaller, modular packages – better engineering principle. Automatically handles the ‘run all’, and parallel grouping by assigning a sequence number.

BimlSnap Online - Components Website: https://bimlsnap.bitracks.net:2525/ SQL Server Data Tools (SSDT) with BimlExpress Add-in from Varigence SSIS Run-time Framework (Optional) Snap_Mart Metadata Solution (Optional) Download: http://www.bimlsnap.com/downloads/ Back-end SQL Server Database Front-end Client Optional SSIS Project to update Metadata Database Fastest way to ‘play’ with BimlSnap

BimlSnap Desktop - Components BimlSnap Desktop – Back-End DB, and Front-End App SQL Server Data Tools (SSDT) with BimlExpress Add-in from Varigence SSIS Run-time Framework (Optional) Snap_Mart v2 Metadata Solution (Optional) Download: http://www.bimlsnap.com/bimlsnap-desktop- download/ Back-end Databases Front-end Clients Optional SSIS Project to update Metadata Database

Feature Matrix: http://www.bimlsnap.com/product-feature-matrix/ Changes typically begin with a change to the database schema (metadata) Metadata changes are then captured, providing new tables/columns in front-end programs Code Generators (e.g., MERGE statement) are updated using new metadata (placed in new Biml files) SSIS Packages are re-generated with new Biml

BimlSnap - Metadata Management (v2) Stand alone application Has its own Data Mart using Type 1 and Type 2 Dimensions Uses a free Back-end Database: Front-end Client Extends Information Schema Views Includes Optional SSIS Project to Refresh Metadata

Demo 3 – Metadata Management Back end database SSIS Project to populate Front-end program to customize SSIS Project to ‘synchronize’ Sync

BimlSnap Workflow with Metadata Snap_Mart Metadata Upload Metadata Populate Changes typically begin with a change to the database schema (metadata) Metadata changes are then captured, providing new tables/columns in front-end programs Code Generators (e.g., MERGE statement) are updated using new metadata (placed in new Biml files) SSIS Packages are re-generated with new Biml Data Warehouse Biml Populate Changes

BimlSnap Online – Getting Started w/Metadata Steps Download SSIS Metadata and Framework Products: www.bimlsnap.com Run SQL Database Scripts (snap_mart, and optionally the ‘SSIS_Data’ run-time framework database) Configure and Run SSIS Project: “snap_mart Refresh” Use Metadata Front-end (SnapMart v1.2.exe) to designate databases for Metadata extracts Re-run SSIS Project: “snap_mart Refresh” Update Dimension Column Types for: Business Keys (bk) ‘Type 2’ (historical) attributes Export from local client and Import into www.bimlsnap.com

BimlSnap Desktop – Getting Started w/Metadata Steps Download Databases and Front-end products: http://www.bimlsnap.com/bimlsnap-desktop-download/ Run SQL Database Scripts (bimlsnap_v2, bimlsnap_mart_v2, SSIS_Data) Configure and Run SSIS Project: “snap_mart v2 Refresh” Use Metadata Front-end (SnapMart v2.exe) to designate databases for Metadata extracts Re-run SSIS Project: “snap_mart v2 Refresh” Update Dimension Column Types for: Business Keys (bk) ‘Type 2’ (t1) attributes if needed Configure and Build packages in the “Bimlsnap Desktop” application Only item we charge for is the “BimlSnap Desktop” Front-end, which is technically optional, but does make the product easier to use. Even this can be free if you occasionally tweet/blog on BimlSnap. Just email the link for your social media post to info@bitracks.com, and we’ll extend your subscription.

BimlSnap – Runtime Framework Uses a free Back-end Database: Extends Functionality of the SSIS Catalog Database (SSISDB) Integrates with BimlSnap Generated Packages to Provide: Project restart ability Project/Package row counts Project/Package runtimes Email alerts Package error logging

Demo 4 – Power BI Dashboard Power BI Project available upon request. Email info@bitracks.com

BimlSnap Project Configuration Select a Template Email Alerts Project Re-runability Row Counts Select a Package Protection Level Encrypt Sensitive With User Key Do Not Save Sensitive

BimlSnap Package Configuration DataFlow Pattern Package Qualifier Source Connection Source Query Destination Connection Destination Table Truncate Option Identity Insert Options

SSIS/Biml Project Build Project Integration Connections Pattern Definition Package XML Project XML SP VS Package Configuration SP Package Template Parameters Framework Integration

Code Generators Code Generators Merge Statements Table Partitioning

Demo 5 – Code Generators Merge Statements Partitioning Statements BTW, all SQL code needed to generate SSIS packages is in the database.

Demo 6 – Select for HTML Documents all projects, packages in an conveniently linked HTML file

Links Blog: Accelerated ELT using BI Markup Language (Biml) Project Vulcan: https://vulcan.codeplex.com/ Varigence: https://www.varigence.com/ Biml Express (VS Add-in): https://www.varigence.com/BimlExpress www.bimlsnap.com www.bitracks.com BimlSnap Online Video: https://youtu.be/PKQ3r0w_EO8 For a copy of this presentation, please email us at: info@bitracks.com