12.12.2015 Easy ETL with 12.12.2015. Thank you to our AWESOME sponsors!

Slides:



Advertisements
Similar presentations
Oracle SQL Developer Data Modeler 3.0: Technical Overview March 2011.
Advertisements

Introduction to ETL Using Microsoft Tools By Dr. Gabriel.
Technical BI Project Lifecycle
© 2004 Visible Systems Corporation. All rights reserved. 1 (800) 6VISIBLE Holistic View of the Enterprise Business Development Operations.
DEV392: Extending SharePoint Products And Technologies Through Web Parts And ASP.NET Clint Covington, Program Manager Data And Developer Services - Office.
Building Enterprise Applications Using Visual Studio ®.NET Enterprise Architect.
1 An Introduction to Visual Basic Objectives Explain the history of programming languages Define the terminology used in object-oriented programming.
Microsoft Office Business Scorecard Manager 2005 : Programming Business Intelligence Features John Hooper Service Line Architect Microsoft Corporation.
Using Microsoft SharePoint to Develop Workflow and Business Process Automation Ted Perrotte National Practice Manager, Quilogy, Microsoft Office SharePoint.
Discover, Master, InfluenceSlide 1 SQL Server Compact Edition and the Entity Framework Rob Sanders Readify.
Chapter 5 Using SAS ® ETL Studio. Section 5.1 SAS ETL Studio Overview.
Create with SharePoint 2010 Jen Dodd Sr. Solutions Consultant
Building Public Facing Websites with SharePoint 2010 Prepared for ILTA’s SharePoint for Legal Symposium June 16 th, 2010 George Durzi Principal Consultant.
SPONSORS. Microsoft PowerPivot for SQL Server, Excel 2010, and SharePoint 2010 Michael Herman Syntergy, Inc.
SSIS Over DTS Sagayaraj Putti (139460). 5 September What is DTS?  Data Transformation Services (DTS)  DTS is a set of objects and utilities that.
DEV334 Creating Application Starting Points & Sharing Best Practices with Enterprise Templates Marc Gusmano Director of Emerging Technologies The Information.
SQL Server Integration Services (SSIS) Presented by Tarek Ghazali IT Technical Specialist Microsoft SQL Server (MVP) Microsoft Certified Technology Specialist.
WorkPlace Pro Utilities.
Microsoft Visual Basic 2005: Reloaded Second Edition
Lesley Bross, August 29, 2010 ArcGIS 10 add-in glossary.
OFC304 Excel 2003 Overview: XML Support Joseph Chirilov Program Manager.
SWE 316: Software Design and Architecture – Dr. Khalid Aljasser Objectives Lecture 11 : Frameworks SWE 316: Software Design and Architecture  To understand.
SEATTLE BI MEETUP BI & BIG FISH April 2 nd, 2014 Emre Motan.
Data File Access API : Under the Hood Simon Horwith CTO Etrilogy Ltd.
Using SAS® Information Map Studio
DTS Conversion to SSIS Conversion Best Practices Mike Davis
Data Management Console Synonym Editor
3 Copyright © 2009, Oracle. All rights reserved. Accessing Non-Oracle Sources.
Microsoft Office Excel 2007: Programming Business Intelligence Features John Hooper Service Line Architect Microsoft Corporation Jed Farr Senior Consultant.
Ms Dynamics Ax 2012 By Johnkrish. MSD Ax is a Customizable, Multi-language, Multi-Currency ERP Solution. Completely integrated & Web-enabled Supports.
Microsoft Visual Studio 2005 Tools for the Office System: Building Office Solutions Using Visual Studio 2005 Tools for Office Andrew Coates Developer Evangelist.
Please note that the session topic has changed
Metadata-driven Automatic Package Creation with Notes from the field.
Self-Service Data Integration with Power Query Stéphane Fréchette.
Developing SQL/Server database in Visual Studio Introducing SQL /Server Data Tools Peter Lu.Net Practices Director Principle Architect Nexient March 19.
Easy ETL with Andrzej Kukuła – Marcin Szeliga –
Copyright 2015 Varigence, Inc. Unit and Integration Testing in SSIS A New Approach Scott @varigence.
Biml Recipes: Automatically Create T-SQL Scripts for Common Tasks
Template Package  Presented by G.Nagaraju.  What is Template Package?  Why we use Template Package?  Where we use Template Package?  How we create.
Enterprise Library 3.0 Memi Lavi Solution Architect Microsoft Consulting Services Guy Burstein Senior Consultant Advantech – Microsoft Division.
Building Your ETL Framework with Biml Meagan Longoria March 19, 2016.
Advanced BIML topics Be a W.I.S.E. A.S.S. Me ! Self-employed BI consultant Author Trainer MCT
An Introduction to the magical world of BIML!
Introduction ITEC 420.
Building Enterprise Applications Using Visual Studio®
Managing, Storing, and Executing DTS Packages
BIML: Step by Step Julie Smith.
What’s new in SQL Server 2017 for BI?
Andrzej Kukuła Easy ETL with and SSIS.
Presented by: Warren Sifre
Populating a Data Warehouse
Populating a Data Warehouse
Populating a Data Warehouse
Simon Kingaby #SimonKingaby
Matt Masson Software Development Engineer Microsoft Corporation
Populating a Data Warehouse
Populating a Data Warehouse
Orchestration and data movement with Azure Data Factory v2
Populating a Data Warehouse
SSIS Deployment Smackdown!
SSIS Deployment Smackdown!
ETL Automation using Biml
Execute your Processes
Orchestration and data movement with Azure Data Factory v2
SSIS Data Integration Data Warehouse Acceleration
SSIS Data Integration Data Warehouse Acceleration
Mark Quirk Head of Technology Developer & Platform Group
9/8/ :03 PM © 2006 Microsoft Corporation. All rights reserved.
SSIS Data Integration Data Warehouse Acceleration
Optimizing the ETL Development process using BIML
Presentation transcript:

Easy ETL with

Thank you to our AWESOME sponsors!

Andrzej Kukuła  20 years of professional experience in IT  Microsoft Certified Solutions Expert  Biml Expert  System Architect  C#, F# Developer by heart  Leader of local PLSSUG/PASS chapter  Long-time Contributor to pymssql project  Passionate about many aspects of IT world 

Marcin Szeliga  Data Philosopher  BI Expert and Consultant  Data Platform Architect  20 years of experience with SQL Server  Ph.D. Candidate at Politechnika Śląska 

Agenda  The ETL  SSIS and its problems  Introduction to Biml  BimlScript and its features  Code generation described  Interesting functionalities  Digressions  Demos  Summary

The ETL  ETL is a significant cost in DW project  But it‘s not going to disappear tomorrow  It is still the foundation of many big Enterprise Data Warehouses  Thus the goal: simplify ETL to lower cost  SSIS is very good ETL solution  but...

SSIS and its problems  Manual package creation using visual editor + drag & drop  Slow, boring, error prone  Not adaptable to requirements (think about changing logic in 10/100/… packages)  Not developer-friendly  Not generic  Version control is hardly possible  No easy API  No code templates/design patterns/DRY  No CI/CD  Should we really spend our time fighting this?

Example Package

Example DTSX Is this reasonable amount of code to accomplish the task?…

Enter

What’s Biml?  Business Intelligence Markup Language  A really easy XML-based language to describe BI assets  Connections, Tables, Views  SSIS Packages, SSIS Projects  Dimensions, Measure Groups, Cubes  and more…  Smart default values and behavior  Don’t bother with SSIS metadata (most of the time)  Easy to integrate with VC (TFS, Git, and more)  Easy to organize code into reusable libraries  Available for free in SSDT-BI with BIDSHelper Demo!

Example Biml elements  Familiar and pretty easy to remember, aren’t they?

Biml and BimlScript  Biml = XML Domain Specific Language  BimlScript = Biml + C# code nuggets  called template Demo!

BimlScript  We have full power of.NET at our disposal  Execute code during compilation  Read files  Access remote data  Even create database objects (e.g. semi-temporary tables)  Reference external.NET assemblies  Supports including and calling other Biml files  Can reference external.NET assemblies  Extensive.NET Biml API built-in

BimlScript Expansion  First, BimlScript is expanded to Biml  Then Biml is compiled to DTSX

Biml Compilation  Biml is compiled to DTSX + DTProj  The process is repeatable  Can be directly opened in SSDT-BI (BIDS)  Execution using DTExec / SSISDB / SSDT  No additional runtime or runtime license required to execute package  Compilation is just automation of SSIS  The same rules apply  Objects must exist in database (so that code generator can create SSIS metadata and mappings)

The process Biml + BimlScript source files C#/VB compiler BimlScript expansion to Biml Population of in-memory Object Model Single, in-memory, expanded, compiled and merged Biml Code generator …

RootNode  The.NET object which represents all assets and metadata  Populated during parsing of Biml files  Useful in BimlScript  Allows creation of very dynamic solution  RootNode.Connections  RootNode.Databases  RootNode.Tables  RootNode.Packages  and many many more

Code Reuse  Can be used anywhere in Biml file  Expands file given in argument and return Biml string  Can specify additional arguments  Include (expand) specified Biml file (or.NET resource) before current file

Interesting features   Lightweight metadata, can be associated with any asset  Documentation annotations  Interesting use case: SQL extended properties  ObjectTag  Property of all BimlScript.NET objects  Accessed programmatically from BimlScript  Very convenient and extensible object storage

Extending BimlScript object model  Contains plain C# code (no templating)  Can interact with Biml object model  Can use extended methods  Can define new classes  We can subclass original Biml classes to change/extend their behavior

More features  Transformers and „Frameworks”  Ability to change Biml after it is already expanded  Can „reverse engineer” DTSX packages into Biml  It’s possible to write custom code generators  Different target runtime than DTSX

Source object types - digression  Views in custom schema, e.g. BI.Person  Isolation  Doesn’t need to modify production objects  Can be in different database than production on the same instance (if absolutely no modification of production is allowed)  Abstraction  Rename columns  Convert data, create new columns  Model data – e.g. join with other sources  Filter data  Add extended properties  Security  Dedicated ETL user can be granted access only to views in this schema, doesn’t need access to underlying tables

Separation of duties  External metadata  Business analysts don’t touch the packages  Programmers don’t model the business  Easy storage  MDS – Excel client!  Extended properties  custom

Complete demo  Complete ETL solution  Storage of metadata (connections, mappings) outside of code  Dynamic import of database table definitions  Dynamic generation of stage tables  Annotations and SQL extended properties as a metadata source  Execute SQL during BimlScript expansion  LogicalDisplayFolder as a metadata source  Conditional logic  Type conversion on the fly  Nonstandard column mapping  Master package  Dynamic package project  Dynamic rename of objects during ETL  Package to create stage tables  Tiers  MSBuild Demo!

How to benefit it?  BI Project Decision Makers  Faster initial delivery  Lower cost of change  Immediate Biml/Mist ROI  BI Architects  Reusable Design Patterns with Biml/BimlScript  One project compatible with SQL Server  Easily manage large BI code base, tasks, and issues using TFS  Plan for BI Continuous Integration/Continuous Delivery

How to benefit it?  ETL developers  Fast Biml learning curve  Generate your DTSX faster with Biml instead of drag’n’drop  Embrace DRY in BI development  Use proper version control to manage your sources  BI consultants  Increased productivity  Build your Biml/BimlScript code library and reuse it in different projects  No runtime license costs for customers

Thank you to our AWESOME sponsors!

Questions?