1. ETL Automation using Biml
Sarju Patel
Lead Software Developer
One Call Care Management
10/06/2018

2. Agenda SSIS and problems with development What is Biml?
Biml and BimlScript
Biml coding options
Why automate ETL?
Demo
Learn more.

3. SSIS & Problems Manual drag & drop in Visual Studio or Bids.
Slow, boring, error prone
Not adaptable to requirement. (changing logic in multiple packages)
QA Testing.
Not developer-friendly
Not generic
Limited CI/CD and Version Control.
Nearly Impossible to impose Standards across multiple packages.

4. History of Biml
2007, Microsoft CSS (Customer Service and Support Started internal project called “Vulcan”.
Project Vulcan was a way to define BI objects in xml.
Source Code published on CodePlex.
User recognized power but no documentation or training resources.
In 2008, Microsoft Employee Scott Currie, who worked with Vulcan in CSS, found Varigence Inc.
Varigence published BIML as open language specification (not Vulcan but similar)
Addition of Biml functionality to BIDSHelper on CodePlex.
Separate BIMLExpress release.

5. What is Biml?
BIML – Business Intelligence Markup Language (Developed by Varigence)
It’s an XML dialect used to create SSIS, SSAS and tabular models etc.
Create Azure Data Factory Pipelines and Data Vault Components.
Compile into the EXACT same .dtsx packages that are created vis drag and drop in SSDT.
Biml is to SSIS packages as HTML is to Webpages.
Xml Primer

6. What is Biml?
Syntax:
<Biml xmlns="
<Packages>
<Package Name="SQLSaturday-Orlando2017"></Package>
<Package Name="SQLSaturday-Orlando2018"/>
</Packages>
</Biml>

7. What is Biml? Cont.

8. Biml and BimlScript Biml = XML Domain Specific Language
BimlScript = Biml + (C# or VB Script Nuggets)

9. Biml and BimlScript Cont.
Biml compile to .dtsx package
BimlScript is expended to Biml.
Single, in-memory, expanded, compiled and merged Biml
Biml compiled to .dtsx (ssis package)

10. Biml and BimlScript Cont.
We have full power of .NET
Execute code during compilation
Read files
Access remote data
Even create database objects
Compilation is just automation of SSIS
The same rules apply
Objects must exist in database to be able to create SSIS metadata and mappings

11. Delimiters <#@ #> Directives
Import, output, property, template language etc.
<# #>
Standard Control Block
Defines .NET code commonly referred to as Biml Script or Code Nuggets
<#= #>
Expression Control Block
Evaluates the .NET code, converts it to a string, then embeds it in the Biml file.
<#+ #>
Class Feature Control Block
Defines properties, methods, and files that are excluded from the Biml file but may be accessed by other BimlScript code nuggets.

12. Biml and BimlScript Cont.

13. Biml Coding Options

14. Why automate ETL?
Enforce standard patterns, naming conventions and best practices
Economy of scale and reusability
Lowers risk and cost of changes during the development stages of the project by lowering amount of required manual work and thus lowers the risk of human error.
Quick Delivery.

15. ETL Framework

16. Demo
1. Basic example 2. Embed small nuggets of (C# or VB) 3. Metadata Driven ETL framework. 4. Add more functionality in existing ETL framework.

17. Learn more
1. BimlScript.com 2. SQL Server Central – Stairway to Biml 3. Cathrine Wilhelmsen blog 4. Varigence.com

18. Questions?
Thank You