1. The Science of Success: Building Faith in a Data Warehouse
Todd McDermid
The Science of Success: Building Faith in a Data Warehouse

2. SQLSaturday Sponsors!
Without the generosity of these sponsors, this event would not be possible! Please, stop by the vendor booths and thank them.

3. What is Success?
Meeting or Exceeding Expectations

4. Expectations
Who determines expectations? How is the DW measured against them? Who decides when the DW has met them? Who can change the expectations… whenever they want?
Not You

5. The Good…

6. They Want to Trust You Single Version of the Truth
At best, each stakeholder only trusts their own data
You’re here to help other stakeholders trust them too
Analytic capability
You can give them analysis they need

7. The Bad…

8. You’re an Ignorant Witch
ETL is witchcraft to business users
No amount of explanation is going to be remembered
Business data is foreign to you
You have no idea what “looks right”

9. The Ugly…

10. How can you say your Data Warehouse is right if someone else has to tell you when it’s wrong?
You are fighting with your hands tied behind your back

11. The Common (and Wrong) Solution

12. One-Time Validation
Subject-matter experts (SMEs) define success at outset. Once development is complete, SMEs validate.
Problems with this:
Predefined validation targets get out of date
Attempts at flexible validation targets tend to be non-specific
Being correct once doesn’t inspire lasting confidence

13. A Real Solution

14. Real Solution Requirements
Has to:
Validate frequently
Verify specific, unchanging (past) truths
Balance to “current” figures
Be “accessible” to SMEs

15. Additional Benefits
Allows you to tell business users when the DW is wrong Improves communication between you and SMEs about the definition of success Allows for success definition to change – and makes that change measurable

16. Continuous Data Warehouse System Testing

17. Two Testing Styles Static Dynamic Compares DW to unchanging data
Provides guarantees that reporting on older data remain correct
Will detect DW changes that cause differences in older data that SMEs don’t often review
Will detect unanticipated changes to source systems that affect the past
 Tests can become outdated
Compares DW to another “live” system
Provides guarantees that the DW balances to systems of record for current time periods
Should detect new scenarios and processes that the business did not communicate to BI team
 Tests can be extremely difficult or limited depending on “live” system capabilities

18. Static Testing Two parts:
Static data (CSV, …) manually compiled from any source:
Not necessarily a direct export!
Data Warehouse data dynamically constructed from a query

19. Static Testing Scenarios
During initial development
Baseline result sets expected from requirements
Over regular use
Ensuring quarter-end reporting remains consistent
Fix bugs using Test-Driven Development (TDD)
Construct a failing test first
Keep test forever to prevent regressions

20. Dynamic Testing Two parts:
Live system data dynamically resulting from a query
Data Warehouse data dynamically resulting from a query

21. Dynamic Testing Scenarios
Ensure the DW balances with source systems
Internal table integrity
Many DWs do not define referential integrity
Otherwise unenforceable integrity constraints
Type 2 dimensions “have no date gaps” and/or “don’t have overlapping effective date ranges”

22. Automate Execution Static Testing Dynamic Testing DW Query
Source Query
CSV
DW Query
Data Warehouse
Export
Source Data
PDF
Paper Napkin

23. Examples

24. Two Testing Styles (Review)
Static
Dynamic
Compares DW to unchanging data
Provides guarantees that reporting on older data remain correct
Will detect DW changes that cause differences in older data that SMEs don’t often review
Will detect unanticipated changes to source systems that affect the past
 Tests can become outdated
Compares DW to another “live” system
Provides guarantees that the DW balances to systems of record for current time periods
Should detect new scenarios and processes that the business did not communicate to BI team
 Tests can be extremely difficult or limited depending on “live” system capabilities

25. Third Testing Style: Statistical/Reasonability
What happens if the source system is wrong?
Or the business process changes significantly, breaking source system reporting, and DW reporting?
Compare DW data with itself:
Do metrics reasonably fit within past variances, trends, and correlations?

26. SQLSaturday Sponsors!
Without the generosity of these sponsors, this event would not be possible! Please, stop by the vendor booths and thank them.

27. The Science of Success: Building Faith in a Data Warehouse
Thank You!
Todd McDermid
The Science of Success: Building Faith in a Data Warehouse