1. The Science of Success: Building Faith in a Data Warehouse
Todd McDermid
The Science of Success: Building Faith in a Data Warehouse

2. SQLSaturday Sponsors! Titanium & Global Partner Gold Silver Bronze
Without the generosity of these sponsors, this event would not be possible! Please, stop by the vendor booths and thank them.

3. Todd McDermid
Sr. Business Intelligence Solution Architect at Absolute Software
10 years in BI
7 time Microsoft MVP (not current)
Creator of SSIS Dimension Merge component

4. What is Success?
Meeting or Exceeding Expectations

5. Expectations
Who determines expectations? How is the DW measured against them? Who decides when the DW has met them? Who can change the expectations… whenever they want?
Not You

6. The Good…

7. They Want to Trust You Single Version of the Truth
At best, each stakeholder only trusts their own data
You’re here to help other stakeholders trust them too
Analytic capability
You can give them analysis they need

8. The Bad…

9. You’re an Ignorant Witch
ETL is wizardry to business users
No amount of explanation is going to be remembered
Business data is foreign to you
You have no idea what “looks right”

10. The Ugly…

11. How can you say your Data Warehouse is right if someone else has to tell you when it’s wrong?
You are fighting with your hands tied behind your back

12. The Common (and Wrong) Solution

13. One-Time Validation
Subject-matter experts (SMEs) define success at outset. Once development is complete, SMEs validate.
Problems with this:
Predefined validation targets get out of date
Attempts at flexible validation targets tend to be non-specific
Being correct once doesn’t inspire lasting confidence

14. A Real Solution

15. Real Solution Requirements
Has to:
Validate frequently
Verify specific, unchanging (past) truths
Balance to “current” figures
Be “accessible” to SMEs

16. Additional Benefits
Allows you to tell business users when the DW is wrong Improves communication between you and SMEs about the definition of success Allows for success definition to change – and makes that change measurable

17. Continuous Data Warehouse System Testing

18. Two Testing Styles Static Dynamic Compares DW to unchanging data
Provides guarantees that reporting on older data remain correct
Will detect DW changes that cause differences in older data that SMEs don’t often review
Will detect unanticipated changes to source systems that affect the past
Tests become outdated
Compares DW to another “live” system
Provides guarantees that the DW balances to systems of record for current time periods
Should detect new scenarios and processes that the business did not communicate to BI team

19. Static Testing Two parts:
Static data (CSV, …) manually compiled from any source:
Not necessarily a direct export!
Data Warehouse data dynamically constructed from a query

20. Static Testing Scenarios
During initial development
Baseline result sets expected from requirements
Over regular use
Ensuring quarter-end reporting remains consistent
Fix bugs using Test-Driven Development (TDD)
Construct a failing test first
Keep test forever to prevent regressions

21. Dynamic Testing Two parts:
Data Warehouse data dynamically constructed from a query
Source system data dynamically constructed form a query

22. Dynamic Testing Scenarios
Ensure the DW balances with source systems
Internal table integrity
Many DWs do not define referential integrity
Otherwise unenforceable integrity constraints
Type 2 dimensions “have no date gaps” and/or “don’t have overlapping effective date ranges”

23. Frequent Validation Static Testing Dynamic Testing DW Query
Source Query
CSV
DW Query
Data Warehouse
Export
Source Data
PDF
Paper Napkin

24. Absolute Example

25. Two Testing Styles (Review)
Static
Dynamic
Compares DW to unchanging data
Provides guarantees that reporting on older data remain correct
Will detect DW changes that cause differences in older data that SMEs don’t often review
Will detect unanticipated changes to source systems that affect the past
Tests become outdated
Compares DW to another “live” system
Provides guarantees that the DW balances to systems of record for current time periods
Should detect new scenarios and processes that the business did not communicate to BI team

26. Third Testing Style: Statistical/Reasonability
What happens if the source system is wrong?
Or the business process changes significantly, breaking source system reporting, and DW reporting?
Compare DW data with itself:
Do metrics reasonably fit within past variances, trends, and correlations?

27. SQLSaturday Sponsors! Titanium & Global Partner Gold Silver Bronze
Without the generosity of these sponsors, this event would not be possible! Please, stop by the vendor booths and thank them.

28. The Science of Success: Building Faith in a Data Warehouse
Thank You!
Todd McDermid
The Science of Success: Building Faith in a Data Warehouse