November 10 th, 2011 DQS MATCHING G ADI P ELEG, S ENIOR P ROGRAM M ANAGER SQL S ERVER D ATA Q UALITY S ERVICES Microsoft SQL Server 2012

Agenda Matching Project What is record matching? Data Issues DQS Matching Process DQS Data Matching Principles Matching Policy

Record matching is the task of identifying records that match the same real world entity.

The Cost of Duplicate Data …a few examples… Direct marketing communications are doubled up unnecessarily. Product shipments and customer-site based services could be sent to the wrong address due to an incorrect duplicate record being used. Your sales reporting may be inaccurate due to an over- inflated number of customers. Inaccurate sales analysis due to sales being split between multiple records that represent the same customer, resulting in an undervaluing of some key customers.

Where do Duplicate Records come from? Poorly designed softwareNo verification of existing records upon entry Formatting & abbreviations "Doctor Robert Smith" Vs. "Dr. Bob Smith". Data validationHuman errors can creep into the system when fields’ input is not validated Company merging and acquisitions Merging systems may result in duplicates in the merged data. Change of attributesThe same person may appear to not exist in the database if some of the attributes were changed (e.g., address, name etc.)

…Data Issues… There are different ways to represent the same person or address in a database: Data is ‘fuzzy’ in nature (spelling mistakes, abbreviations etc.).

How Data Issues Affects Matching? Matching Results Matching Results Reasoning The Data

Integrated Profiling Progress Notifications Status Connect Build Use DQ Projects Knowledge Management Knowledge Base Sample Data Sample Data

1. Prepare Matching Policy Leverage a KB with existing knowledge Design matching policy rules Each Rule weighs single or multiple domains Tune your policy with your source data 2. Matching Project Map your source to the Relevant KB Run matching Review results and reject mismatches Export survivors And matching results DQS Matching Experience

Identifies exact and approximate matches, enabling removal of duplicate data. Enables creating a matching policy interactively using a computer-assisted process. Ensures that values that are equivalent, but were entered in a different format or style, are in fact rendered uniform.

A matching policy is prepared in the knowledge base. A matching policy consists of matching rules that assess how well one record matches to another. Specify in the rule whether records’ values have to be an exact match, similar, or prerequisite. Train your policy by running and tuning each rule separately.

Identify the attributes in your data that are most significant for matching. Create domains/composite domains based on your data structure. Define matching rules.

Similarity, select Similar if field values can be similar. Select Exact if field values must be identical. Weight, determines the contribution of each domain in the rule to the overall matching score for two records. Prerequisite validates whether field values return a 100% match; else the records are not considered a match. Minimum matching score is the threshold at or above which two records are considered to be a match.

Domains of type ‘Date’, ‘Integer’ or ‘Decimal’ can be matched using the ‘Similar’ property by assigning a tolerance either in percentage or integer. Field values that fall within the defined tolerance are considered a match.

The Matching Results tab displays statistics for the current and previous run of a matching rule. Restore the previous rule.

Home Team Song Artist

The DQS matching system uses the knowledge accumulated in the knowledge base to propose matching candidates. This knowledge includes: Synonyms, Syntax Errors and their Leading Value (by domain) Domain Values and their synonyms and syntax errors are used by the matching system to find identical or similar records. Term-Based Relations (TBR) TBR improves consistency of data attributes values by transforming data values to a single form using user-defined term relations. In matching, TBRs are only applied in-memory for boosting matching accuracy. Nulls and Equivalents (“Unknown”, “99999”…) Manage values that represent missing data by linking to the ‘DQS_Null’ value to assure that they are considered as a match.

String 1String 2Similarity ScoreCharacter BeforeAfter 175 CLEARBROOK ROAD P.O. BOX E. 42ND STREET1834 E. 42ND. ST DE KALB AVE, NE1721 DE KALB AVE NE , S. GARFIELD AVE., BLDG. 1-B14538 S GARFIELD AVE BLDG 1B ,. - #704, SJ Technoville BD, SJ Technoville BD #, - Example:

Export - export both matching results (clusters) and survivors (unique records). A Matching project is performed in three steps: Mapping - map source columns to domains. Matching - run matching and view the results; it includes additional functionality such as: Reject records Filter results by ‘Matched’ & ‘Unmatched’ and by matching score. Display clusters in two different methods (overlapping and non- overlapping )

In Overlapping clusters a record may appear more than once in various clustered results. This structure may be harder to read since the same record exists in multiple clusters. In Non-Overlapping clusters, the system unifies clusters containing the same record. This structure is easier to read as you won't repeat the same observation twice. Overlapping Clusters (A~B), (B~C) Non-Overlapping Cluster (A~B~C)

Overlapping Clusters Non-Overlapping Clusters

Check the Rejected box to move the records out of the proposed cluster upon moving to the next page in the activity. Unlike the Cleansing Data Project where records move between tabs instantly, the rejected records are not removed from the clusters on the user interface. DQS Client User Interface Exported Matching Results

Matching and Survivorship results can be exported to a SQL table, Excel or CSV file for further analysis or consumption.

The Story Contoso airport receives passenger details from different airlines; the data contains duplicate passengers information which need to be identified and removed. The Story Contoso airport receives passenger details from different airlines; the data contains duplicate passengers information which need to be identified and removed. Exercise Description In this exercise you will : Prepare a Matching Policy and tune the matching rules. Create a Matching Project and run a matching process to identify duplicate passengers. Export the matching and survivors results. Exercise Description In this exercise you will : Prepare a Matching Policy and tune the matching rules. Create a Matching Project and run a matching process to identify duplicate passengers. Export the matching and survivors results.

Resources Sessions On-Demand & CommunityMicrosoft Certification & Training Resources Resources for IT ProfessionalsResources for Developers Connect. Share. Discuss.

DQS Blog Tips, tricks and guidance on best practices for using DQS – courtesy of the DQS team DQS Blog Tips, tricks and guidance on best practices for using DQS – courtesy of the DQS team DQS Movies A set of getting started movies for an easy introduction to DQS DQS Movies A set of getting started movies for an easy introduction to DQS DQS Forum Come participate in DQS related discussions in our DQS forum on MSDN DQS Forum Come participate in DQS related discussions in our DQS forum on MSDN Available Here blogs.msdn.com/b/dqs Available Here