2. About Me – csnapp@captechconsulting.com @SnappSQL
MCSE and PMP certified IT Consultant with CapTech since and have over 14 years of Microsoft SQL Server experience
Computer Science degree from the University of Richmond
Masters degree in IT Management from the University of Virginia
Founded my own MLB Data Analytics Company
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

3. Topics We’ll Cover Explanation of ETL and ELT Strategies
Debate the characteristics
In Azure?
How to Implement an ELT Architecture
The Tactical Benefits:
Superior Traceability
Reduced Execution Times
Extensible Design Fits
Next Steps for Database Developers
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

4. Presentation Disclaimer
I’ve chosen to focus on my SQL Server and SSIS experiences but the concepts still apply to your database platform of choice
ELTL is assumed to be the same as ELT. The extra L takes place but feels unnecessary.
Sources can be anything, but let’s assume it’s tabular.
Targets are typically to support transactional systems or business intelligence, but let’s talk big data too!
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

5. Why the Debate? Landscape changes are causing the discussion:
Volume, Variety, Velocity, Veracity, Value
Analytic tools are changing the game too
Codebase complexities and delivery times need to be controlled
Key Difference: When and Where the transformation step is performed
Key Truth: Data Quality is always a concern
Key Takeaway: Business needs and technical capabilities still drive the data management decision
Offer data to analysts and let them link it in their BI tool
Consumers don’t care as long as its immediate and correct
Technology or methodology never absolves you of analysis
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

6. Background Level Set - ETL
Data from disparate data sources
Extract
Data in tool, in flight; modifications done in memory
Transform
Data to destination structure
Load
Image source:
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

7. Conventional ETL Design Let’s Debate?
Graphical view of data pipelines
Traditionally batch oriented, parallel and scheduled
Offer functionality not available in the RDBMS
Requires specialized developer skills
Business rules, cleansings, validations, filtering, joins, lookups all programmed into the tool
Focus on a single destination model and work backwards
Leverages a proprietary engine, potentially separate server
Performance dependent on component configurations, order operations, memory to data volume ratio
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

8. Background Level Set - ELT
Data from disparate data sources
Extract
Data in its original form to a staging area of the target server
Load
Data with semantic layer and MERGE to destination
Transform
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

9. Conventional ELT Design Let’s Debate?
Considered to be a more modern approach
Allows for more real-time processing
Fewer volume and structure concerns
Extract data fast, limiting source system strain
Reuse stage/processing to load multiple target structures
Consolidate area where business and data quality rules exist
Lays a foundation for a Data Lake, populating multiple destinations
Leverage a RDBMS’s transaction engine to work with data locally
Can necessitate more drive space & CPU, but reduces other hardware needs
Maintaining large repository is not quite so simple
Image source:
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

10. But What About Azure? Integration Runtimes Data Factory DF v2 Polybase
SSIS on VM
AML
T-SQL
Hive, Spark
Data Transfer Units a major factor
Scaling destination pricing tier throttles performance
Scale Azure SQLDB to DTUs per MB of bandwidth
SQL DW offers Massively Parallel Processing architecture
Best to use Polybase, leverage T-SQL on the DW
Many “distributed” design considerations
Loads from Hadoop/Data Lake? Consider Spark, Hive
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

11. My Proposed ELT Style Architecture
Stage
Use SSIS to truncate stage table and do 1:1 data flow task
Simultaneously copy of all source data about the transaction that changed
Semantic Views
SELECT statement which applies all business rules, joins, transformations from the stage area
Output columns are format of a single target table
T-SQL Merge
Source is the view - Destination is the target table - Join on Natural Keys
Can handle Type 1 or Type 2 deltas
Key Benefits:
Faster development cycles and execution times
Decreased cost and complexity of code maintenance
Flexible to fit many scenarios
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

12. Superior Traceability
Stage is a 1:1 copy of the source
Table names, Column names, Data types
Process used by multiple data flows
Views contain all the code
No longer tracing dozens of ETL tool components
Code is more readable and reviews are faster
Repeatable design patterns
Limit variability of implementation
Object Oriented methodology
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

13. Reduced Execution Time
Leverage TRUNCATE on stage
Remove usage of blocking components
No waste of transaction log
None of the components below cause wait times or require proper use of caching
Connections are opened/closed quickly
Use MERGE command to move data locally
Stages execute in parallel
Fast at determining if/what needs to change and much faster than OLEDB command or slowly changing dimension components
Source queries are small, fast and leverage indexes
Only stage data about data that needs refreshed
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

14. Extensible Design Fits
Adaptable to the most common scenarios
Some difference in “what to stage”
Order of target loads matters
Code reusability
Dynamically generate the MERGE command
Mapping table to translate vernacular between systems
Frameworks Too!
Use an Execution Log to track all executing processes
Consistent parameterization and lookup of date ranges
Re-startability
Data Validation & Retry Errors process
Target Structure
3rd Normal Form Transactional
Star Schema Reporting Mart
Load Frequency
One Time Migration 
On Going Refresh
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

15. Next Steps for Database Developers
Consider
Scaling back on using complex ETL tools to house ETL logic
Hardware sizing – more drive space and virtual memory on target server
Leverage
Visual Studio’s Database Projects to manage your objects
Data Analysts and Source to Target documentation
Implement
Build queries that replicate existing ETL processes
Pilot this process against a traditional approach
Copyright © 2018 CapTech Ventures, Inc. All rights reserved.

16. Questions?