1. Data Quality in the BI Life Cycle
Robert Blaas
Data Quality in the BI Life Cycle

2. Thanks to our sponsors

3. About Me rwb1912@gmail.com @rwb1912 nz.linkedin.com/in/rblaas

4. Agenda
Effects of Bad Data
Defining Data Quality & Data Quality Management
Governance and Process
How Can DQS Assist

5. Support Decision Making
Data & Information
Support Decision Making
Information Economy
Information = Context around data
Information shaped into Dashboards and Reports
Competitive EDGE
FASTER and Greater Appetite for information
Cut corners
Data Quality takes a back seat

6. Confident Decision Making ?
Ask Audience – How confident are you in your decision making process?
Let me tell you a story (Manufacturing & Insurance)

7. Some Statistics

8. Effects of Bad Data
Defining Data Quality & Data Quality Management
Governance and Process
How Can DQS Assist

9. Data Quality
Is a measure, or set of measures, that give an organization an indication of the level of confidence it can have in the data that is used in it’s operational and strategic decision making process.

10. Data Quality Management
Are the set of processes by which we manipulate the organizations data to increase its quality.
Regulatory (Banking, Privacy  personal details must be correct)
Inaccurate data can lead to business failures, e.g. wrong customer address leads to wrong shipping.
Data quality = OLTP delivery ( providing users with consistent and correct data for day to day business processing, e.g. being able to process orders, service, delivery)
Data quality = Decision support (correct, reliable and consistent information because underlying data is known to be correct and consistent)
Data Quality  should form an integral part of data governance

11. Why Check For Data Quality
Impacts Decision Making Process
Impacts Profitability
Impacts Brand
Regulatory Requirements

12. Common Causes of DQ Issues
Data Merging
Broken Rules
Data Entry
Inconsistent Sources
Data Transmission
Timeliness
Data Merging -> data merged is at risk of being incorrect
Data Transferred -> corruption during file transport, transformation during transmission
Timeliness - > Data must arrive on time
Data Entry -> especially in legacy environments where no control or relational db
MDS -> one version of the truth (single source) sadly this is a utopia.
Incorrect BR -> We often forget this, the need to validate the rule being implemented in calculation or transformation

13. What to Check For Consistency Accuracy Completeness Validity
Conformity
Duplicates
Completeness = Is all the relevant data available
Consistency = is data consistent throughout (always male/female, m/f, 0/1, true/false)
Validity = Does the data fall within accepted domains.
Accuracy = How accurate is the data, if we are measuring temperature to what level is temperature measured, with what variance or margin of error
Conformity = Does is it conform to the accepted business rule
Duplicates = Is the value duplicated, if so what represents the true value (2 customers but with different addresses)

14. Effects of Bad Data
Defining Data Quality & Data Quality Management
Governance & Process
How Can DQS Assist

15. People
Technology
Process

16. Data Quality through Governance
Taking Ownership
Cultural Shift
Collaborative Effort
Integral Part of Process
aking Ownership is crucial . Everyone Owns the Data/Information so everyone is responsible for the quality
Executive Buy In !!
Business Buy In
Cultural Shift
Must form Integral part of Governance Process

17. Data Quality Assurance Process
Monitor
Assess
Action
Communicate
ASSESS – Tools & Processes to asses issues
COMMINICATE – Corporate Platform to communicate, open, collaborative, tools  DATA STEWARDS (GOTO PEOPLE)
ACTION – Tools
MONITOR – Process, Continuous ongoing
LIFE CYCLE – NOT ONE OFF

18. DQ in IT Project Life Cycle
Analysis & Design
Profiling
Business Rules
Establish Domains
Test Cases
Initiation
Profiling
Production
DQS Projects
Monitoring
Action
Development & UAT
Domains
DQS Projects
Dashboards/Reports

19. Effects of Bad Data
Effects of Bad Data
Effects of Bad Data
Effects of Bad Data
Defining Data Quality & Data Quality Management
Defining Data Quality & Data Quality Management
Defining Data Quality & Data Quality Management
Defining Data Quality & Data Quality Management
Governance & Process
Governance & Process
Governance & Process
Governance & Process
How Can DQS Assist
How Can DQS Assist
How Can DQS Assist
How Can DQS Assist

20. What Does DQS Provide Knowledge Base Through Discovery
Reusable Knowledge Base
Semantic Layering
Monitoring
Business Interface
Reusable KB – build a data domain that can be reused across the organisation
Semantic – data is mapped to domains which are given semantic meaning to be tested
Discovery – KB can be built and expanded through data discovery
Extension KB = expand KB by linking through to 3rd party kb providers such as marketing companies for addresses.
Can work in conjunction with MDS

21. Functions of DQS Data Assessment KB Matching Match & Consolidate
Build Knowledge Base
Data Assessment
KB Matching
Knowledge
Base
Use Knowledge Base
Match & Consolidate
Data Cleansing

22. Categorized Reference Data Categorized Reference Data Services
Architecture
DQ Clients
MS DQ
Domains Store
Azure Market Place
Categorized Reference Data
Categorized Reference Data Services
DQS UI
Knowledge Discovery and Management
DQ Server
3rd Party
Reference Data Services
Reference Data Sets
Interactive DQ Projects
RD Services API
(Browse, Set, Validate…)
Reference Data API
(Browse, Get, Update…)
Data Exploration
DQ Engine
Cleansing
Knowledge Discovery
Data Profiling & Exploration
Reference Data
Matching
SSIS DQ Component
DQ Projects Store
Common Knowledge Store
Knowledge Base Store
MS Data Domains
Local Data Domains
DQ Active Projects
Published KBs

23. Time for a Demo

24. How we can assist Extensive DWH / BI Implementations
Information Modelling
SSIS Developments
MDS deployments
DQS deployments

25. Thank for attending Singapore SQLSaturday#646!