ETL Automation using Biml Sarju Patel Lead Software Developer One Call Care Management 10/06/2018

Agenda SSIS and problems with development What is Biml? Biml and BimlScript Biml coding options Why automate ETL? Demo Learn more.

SSIS & Problems Manual drag & drop in Visual Studio or Bids. Slow, boring, error prone Not adaptable to requirement. (changing logic in multiple packages) QA Testing. Not developer-friendly Not generic Limited CI/CD and Version Control. Nearly Impossible to impose Standards across multiple packages.

History of Biml 2007, Microsoft CSS (Customer Service and Support Started internal project called “Vulcan”. Project Vulcan was a way to define BI objects in xml. Source Code published on CodePlex. User recognized power but no documentation or training resources. In 2008, Microsoft Employee Scott Currie, who worked with Vulcan in CSS, found Varigence Inc. Varigence published BIML as open language specification (not Vulcan but similar) Addition of Biml functionality to BIDSHelper on CodePlex. Separate BIMLExpress release.

What is Biml? BIML – Business Intelligence Markup Language (Developed by Varigence) It’s an XML dialect used to create SSIS, SSAS and tabular models etc. Create Azure Data Factory Pipelines and Data Vault Components. Compile into the EXACT same .dtsx packages that are created vis drag and drop in SSDT. Biml is to SSIS packages as HTML is to Webpages. Xml Primer

What is Biml? Syntax: <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="SQLSaturday-Orlando2017"></Package> <Package Name="SQLSaturday-Orlando2018"/> </Packages> </Biml>

What is Biml? Cont.

Biml and BimlScript Biml = XML Domain Specific Language BimlScript = Biml + (C# or VB Script Nuggets)

Biml and BimlScript Cont. Biml compile to .dtsx package BimlScript is expended to Biml. Single, in-memory, expanded, compiled and merged Biml Biml compiled to .dtsx (ssis package)

Biml and BimlScript Cont. We have full power of .NET Execute code during compilation Read files Access remote data Even create database objects Compilation is just automation of SSIS The same rules apply Objects must exist in database to be able to create SSIS metadata and mappings

Delimiters <#@ #> Directives Import, output, property, template language etc. <# #> Standard Control Block Defines .NET code commonly referred to as Biml Script or Code Nuggets <#= #> Expression Control Block Evaluates the .NET code, converts it to a string, then embeds it in the Biml file. <#+ #> Class Feature Control Block Defines properties, methods, and files that are excluded from the Biml file but may be accessed by other BimlScript code nuggets.

Biml and BimlScript Cont.

Biml Coding Options https://varigence.com/Products

Why automate ETL? Enforce standard patterns, naming conventions and best practices Economy of scale and reusability Lowers risk and cost of changes during the development stages of the project by lowering amount of required manual work and thus lowers the risk of human error. Quick Delivery.

ETL Framework

Demo 1. Basic example 2. Embed small nuggets of (C# or VB) 3. Metadata Driven ETL framework. 4. Add more functionality in existing ETL framework.

Learn more 1. BimlScript.com 2. SQL Server Central – Stairway to Biml 3. Cathrine Wilhelmsen blog 4. Varigence.com

Questions? Thank You sarju401@gmail.com