1. Populating a Data Warehouse
Data Warehouse Acceleration
Fact and Dimension Table Patterns
Jim Miller – President, BI Tracks Consulting
SQL Saturday – Houston, TX
June 23, 2018

2. What is Biml?
XML Schema used to define SSIS, and other Microsoft ‘BI Stack’ components
Use with BimlExpress to auto-generate SSIS Projects and Packages in Visual Studio (SSDT)
Who has used Biml? Who is developing a Data Warehouse?

3. Why Use Biml to Build SSIS Packages?
Code Reuse , Code Reuse , Code Reuse
Microsoft’s Project Vulcan was designed to bring Code Reusability to the MSBI Stack
Resource Repository Defined Outside of SSIS Projects
Connections, Parameters, Packages
Standardization of SSIS Project/Package Development
Framework Extensions, Package Design Styles
Accelerated Change Management
SSIS Package ‘diffs’ are simple text file comparisons
All made possible by dealing with just XML code. All XML code is generated by encapsulated SELECT statements.

4. Demo 1 – Generating SSIS Packages in Visual Studio using BimlExpress
BimlExpress will generate ‘pure biml tags’, or biml script (embedded C# code nuggets).
Who does not want to create higher quality work in less time?

5. Biml is just XML, so it’s free
Biml is just XML, so it’s free. BimlSnap Online, which creates Biml is free. BimlExpress which converts Biml into SSIS packages is a free VS add-in and it’s also free.

6. Biml and Biml Script Examples

7. Demo 2 – Select for XML
Everything we do in BimlSnap to generate Biml is based on the “Select… for XML” technique.

8. BimlSnap Online and Desktop
Use SQL Server Procedures, and Tables to generate Pure Biml code
Includes a repository of re-usable ELT Patterns
Common Data Warehouse patterns are currently available
Integrated into an extensible ‘ELT Framework’
Visual Studio Designer is only used for ‘Unit testing’, not for GUI style development. This is the key paradigm shift from traditional SSIS Project/Package development. Results in more, smaller, modular packages – better engineering principle. Automatically handles the ‘run all’, and parallel grouping by assigning a sequence number.

9. BimlSnap Online - Components
Website:
SQL Server Data Tools (SSDT) with BimlExpress Add-in from Varigence
SSIS Run-time Framework (Optional)
Snap_Mart Metadata Solution (Optional)
Download:
Back-end SQL Server Database
Front-end Client
Optional SSIS Project to update Metadata Database
Fastest way to ‘play’ with BimlSnap

10. BimlSnap Desktop - Components
BimlSnap Desktop – Back-End DB, and Front-End App
SQL Server Data Tools (SSDT) with BimlExpress Add-in from Varigence
SSIS Run-time Framework (Optional)
Snap_Mart v2 Metadata Solution (Optional)
Download: download/
Back-end Databases
Front-end Clients
Optional SSIS Project to update Metadata Database

11. Feature Matrix: http://www.bimlsnap.com/product-feature-matrix/
Changes typically begin with a change to the database schema (metadata)
Metadata changes are then captured, providing new tables/columns in front-end programs
Code Generators (e.g., MERGE statement) are updated using new metadata (placed in new Biml files)
SSIS Packages are re-generated with new Biml

12. BimlSnap - Metadata Management (v2)
Stand alone application
Has its own Data Mart using Type 1 and Type 2 Dimensions
Uses a free Back-end Database:
Front-end Client
Extends Information Schema Views
Includes Optional SSIS Project to Refresh Metadata

13. Demo 3 – Metadata Management
Back end database
SSIS Project to populate
Front-end program to customize
SSIS Project to ‘synchronize’
Sync

14. BimlSnap Workflow with Metadata
Snap_Mart
Metadata
Upload Metadata
Populate
Changes typically begin with a change to the database schema (metadata)
Metadata changes are then captured, providing new tables/columns in front-end programs
Code Generators (e.g., MERGE statement) are updated using new metadata (placed in new Biml files)
SSIS Packages are re-generated with new Biml
Data Warehouse
Biml
Populate
Changes

15. BimlSnap Online – Getting Started w/Metadata
Steps
Download SSIS Metadata and Framework Products:
Run SQL Database Scripts (snap_mart, and optionally the ‘SSIS_Data’ run-time framework database)
Configure and Run SSIS Project: “snap_mart Refresh”
Use Metadata Front-end (SnapMart v1.2.exe) to designate databases for Metadata extracts
Re-run SSIS Project: “snap_mart Refresh”
Update Dimension Column Types for:
Business Keys (bk)
‘Type 2’ (historical) attributes
Export from local client and Import into

16. BimlSnap Desktop – Getting Started w/Metadata
Steps
Download Databases and Front-end products:
Run SQL Database Scripts (bimlsnap_v2, bimlsnap_mart_v2, SSIS_Data)
Configure and Run SSIS Project: “snap_mart v2 Refresh”
Use Metadata Front-end (SnapMart v2.exe) to designate databases for Metadata extracts
Re-run SSIS Project: “snap_mart v2 Refresh”
Update Dimension Column Types for:
Business Keys (bk)
‘Type 2’ (t1) attributes if needed
Configure and Build packages in the “Bimlsnap Desktop” application
Only item we charge for is the “BimlSnap Desktop” Front-end, which is technically optional, but does make the product easier to use. Even this can be free if you occasionally tweet/blog on BimlSnap. Just the link for your social media post to and we’ll extend your subscription.

17. BimlSnap – Runtime Framework
Uses a free Back-end Database: Extends Functionality of the SSIS Catalog Database (SSISDB)
Integrates with BimlSnap Generated Packages to Provide:
Project restart ability
Project/Package row counts
Project/Package runtimes
alerts
Package error logging

18. Demo 4 – Power BI Dashboard
Power BI Project available upon request.

19. BimlSnap Project Configuration
Select a Template
Alerts
Project Re-runability
Row Counts
Select a Package Protection Level
Encrypt Sensitive With User Key
Do Not Save Sensitive

20. BimlSnap Package Configuration
DataFlow Pattern
Package Qualifier
Source Connection
Source Query
Destination Connection
Destination Table
Truncate Option
Identity Insert Options

21. SSIS/Biml Project Build
Project Integration
Connections
Pattern Definition
Package
XML
Project
XML SP VS
Package Configuration SP
Package Template
Parameters
Framework Integration

22. Code Generators
Code Generators
Merge Statements
Table Partitioning

23. Demo 5 – Code Generators Merge Statements Partitioning Statements
BTW, all SQL code needed to generate SSIS packages is in the database.

24. Demo 6 – Select for HTML
Documents all projects, packages in an conveniently linked HTML file

25. Links
Blog: Accelerated ELT using BI Markup Language (Biml) Project Vulcan: Varigence: Biml Express (VS Add-in): BimlSnap Online Video: For a copy of this presentation, please us at: