Steve Simon MVP SQL Server BI http://www.infogoldusa.com A dive into Data Quality Services SQL Server 2014 Boston Ma September 27 ,2014 Steve Simon MVP SQL Server BI http://www.infogoldusa.com

Steve Simon is SQL Server MVP and a Senior Business Intelligence Development Engineer with Atrion Networking Corporation, Providence RI USA. He has been involved with database design and analysis for over 29 years. Steve has presented numerous papers at PASS summits over the years including PASS Europe, in addition to numerous presentations at SQL Saturday events, the Amsterdam and Copenhagen and other local user groups. He a PASS Virtual Chapter Regional Mentor.

Our business challenge

Lack of conformity Products, manufacturers and descriptions can be added to the database table in many ways. Leaves these ’descriptions’ as unreliable for use within query predicates. With SQL Server 2008 we had data profiling task which advised of a problem, however had no intelligent solutions.

OLAP solutions seemed to be the least flexible. Lack of conformity Weekly modifications to core data that should be correctly entered the first time. OLAP solutions seemed to be the least flexible. Reprocessing of cubes and the time expended can mount up.

Enter Data Quality Services

What is Data Quality Services? A set of tools which allow data stewards to improve data quality (Domain experts). Produces a result set with suggested improvements. Does not change the original source data set.

Why should we use Data Quality Services? You can get Subject Matter Expert(SME)input. Manually define, match and cleanse (Man & Machine) . Computer cleansing of your data. How? Programmatically, then manually approve. The system learns.

Why should we use Data Quality Services? Can integrate with third party data. Can integrate with other data processing e.g. SSIS.

How to use Data Quality Services List of basic steps: Create / Refine / Use a knowledge base. Perform a data quality evaluation. Generate output.

List of components DQS Server. DQS Client.

Three main activities

When do we use Data Quality Services? Issue Detail Completeness Is all the data there? Conformity Is all the data in the correct format? (capitals ?) Consistency Do values represent the same meaning? (Data Mining and Human check) Accuracy Do data objects represent real world values? Validity Do data values fall within acceptable ranges? Duplication Are there multiple copies of the same data?

Courtesy Elad Ziklik

Requirements SQL Server 2012 Post Install BI Edition Must run DQS Installation Post SQL install Requirements Enterprise Edition Do Master Data Services Integration

Installing the server portion

After installation in SSMS

After having installed the DQS Server portion of the application, you get a few new databases and new security roles.

There is no API at present, so we must work with the ‘native’ client. Designed for single write (at this moment) There is no API at present, so we must work with the ‘native’ client.

Being a single write system.. While editing KB it is locked. Others cannot use. Complex KB.. Split up and import later. Each person can create their own entities or domains. KB ’pieces’ can be imported.

Mean while back at the ranch ! ‘The challenge’

Cleansing Issues Are duplicates wrong here?

Demo

SSIS

Data quality can be problematic.Human problems. Take Aways Data quality can be problematic.Human problems. DQS like any mining model is able to ’learn’. DQS learns data patterns and can detect data duplication? Human intervension decreases as models mature.

DQS knowledge bases can be used with SSIS. Take Aways Better quality data reduces data processing times. E.g. Cubes effective and efficient. DQS knowledge bases can be used with SSIS. Output correct/problematic can be sent to data stewards for validation and verification. Less frustration from end users and decision makers.

At the end of the day it all boils down to More importantly, why all of us can definitely benefit from a..

Steve Simon MVP SQL Server BI http://www.infogoldusa.com A dive into Data Quality Services SQL Server 2014 Boston Ma September 27 ,2014 Steve Simon MVP SQL Server BI http://www.infogoldusa.com