1. Andrzej Kukuła
Easy ETL with and SSIS

2. Sponsors

3. Session will begin very soon :)
Please complete the evaluation form from your pocket after the session. Your feedback will help us to improve future conferences and speakers will appreciate your feedback!
Enjoy the conference!

4. About me
20+ years of professional experience in IT Trainer, Consultant Architect of HA/DR Solutions, BI Solutions, Data Platform Solutions, Cloud Solutions Building proficiency in AI/ML and Data Science MCSE, MCITP, MCTS Certified Polish Data Community (former PLSSUG) – PASS Chapter Leader Conference and user group organizer,

5. Agenda
The ETL SSIS and its problems Introduction to Biml BimlScript and its features Code generation described Interesting functionalities Digressions Demos Summary

6. The ETL
ETL is a significant cost in DW project But it‘s not going to disappear tomorrow It is still the foundation of many big Enterprise Data Warehouses Thus the goal: simplify ETL to lower cost SSIS is very good ETL solution but...

7. SSIS and its problems
Manual package creation using visual editor + drag & drop
Slow, boring, error prone
Not adaptable to requirements (think about changing logic in 10/100/… packages)
Not developer-friendly
Not generic
Version control is hardly possible
No easy API
No code templates/design patterns/DRY
No CI/CD
Should we really spend our time fighting this?

8. Example package

9. Example DTSX
Is this reasonable amount of code to accomplish the task?…

10. Enter

11. What’s Biml? Business Intelligence Markup Language
A really easy XML-based language to describe BI assets
Connections, Tables, Views
SSIS Packages, SSIS Projects
Dimensions, Measure Groups, Cubes
and more…
Smart default values and behavior
Don’t bother with SSIS metadata (most of the time)
Easy to integrate with VC (TFS, Git, and more)
Easy to organize code into reusable libraries
Available for free in Visual Studio (SSDT) with BimlExpress (formerly BIDS Helper)
Demo!

12. Example Biml elements
Quite easy to remember, aren’t they?

13. Biml and BimlScript Biml = XML Domain Specific Language
BimlScript = Biml + C#/VB code nuggets
called template
Demo!

14. BimlScript We have full power of .NET at our disposal
Execute code during compilation
Read files
Access remote data
Even create database objects (e.g. semi-temporary tables)
Reference external .NET assemblies
Supports including and calling other Biml files
Extensive .NET Biml API built-in

15. BimlScript expansion
First, BimlScript is expanded to Biml Then Biml is compiled to DTSX

16. The process Biml + BimlScript source files C#/VB compiler
BimlScript expansion to Biml
Population of in-memory Object Model
Single, in-memory, expanded, compiled and merged Biml
Code generator …

17. Biml compilation Biml is compiled to DTSX + DTProj
The process is repeatable
Can be directly opened in SSDT (BIDS)
Execution using DTExec / SSISDB / SSDT
No additional runtime or runtime license required to execute package
Compilation is just automation of SSIS
The same rules apply
Objects must exist in database (so that code generator can create SSIS metadata and mappings)

18. RootNode The .NET object which represents all assets and metadata
Populated during parsing of Biml files
Useful in BimlScript
Allows creation of very dynamic solutions
RootNode.Connections
RootNode.Databases
RootNode.Tables
RootNode.Packages
and many many more

19. Code Reuse Can be used anywhere in Biml file
Expands file given in argument and return Biml string
Can specify additional arguments
Also CallBimlScriptWithOutput
Include (expand) specified Biml file (or .NET resource) before current file

20. Interesting features <Annotation></Annotation> ObjectTag
Lightweight metadata, can be associated with any asset
Documentation annotations
Interesting use case: SQL extended properties
ObjectTag
Property of all BimlScript .NET objects
Accessed programmatically from BimlScript
Very convenient and extensible object storage

21. Extending BimlScript object model
Contains plain C# code (no templating)
Can interact with Biml object model
Can use extended methods
Can define new classes
We can subclass original Biml classes to change/extend their behavior

22. More features Transformers and „Frameworks”
Ability to change Biml after it is already expanded
Can „reverse engineer” DTSX packages into Biml
It’s possible to write custom code generators
Different target runtime than DTSX

23. Source object types (digression)
Views in custom schema, e.g. BI.Person
Isolation
Doesn’t need to modify production objects
Can be in different database than production on the same instance (if absolutely no modification of production is allowed)
Abstraction
Rename columns
Convert data, create new columns
Model data – e.g. join with other sources
Filter data
Add extended properties
Security
Dedicated ETL user can be granted access only to views in this schema, doesn’t need access to underlying tables

24. Separation of duties External metadata
Business analysts don’t touch the packages
Programmers don’t model the business
Easy storage
MDS – Excel client!
Extended properties
custom

25. Complete demo
Complete ETL solution Storage of metadata (connections, mappings) outside of code Dynamic import of database table definitions Dynamic generation of stage tables Annotations and SQL extended properties as a metadata source Execute SQL during BimlScript expansion LogicalDisplayFolder as a metadata source Conditional logic Type conversion on the fly Nonstandard column mapping Master package Dynamic package project Dynamic rename of objects during ETL Package to create stage tables Tiers MSBuild
Demo!

26. How to benefit it? BI Project Decision Makers BI Architects
Faster initial delivery
Lower cost of change
Immediate Biml/Mist ROI
BI Architects
Reusable Design Patterns with Biml/BimlScript
One project compatible with SQL Server
Easily manage large BI code base, tasks, and issues using TFS
Plan for BI Continuous Integration/Continuous Delivery

27. How to benefit it? ETL developers BI consultants
Fast Biml learning curve
Generate your DTSX faster with Biml instead of drag’n’drop
Embrace DRY in BI development
Use proper version control to manage your sources
BI consultants
Increased productivity
Build your Biml/BimlScript code library and reuse it in different projects
No runtime license costs for customers

29. Questions?
@AndrzejKukula