Dirty Data Data Cleansing Xxxxxx DSCI 5240 December 4, 2012

Introduction Real data is dirty Why clean? – Eliminate duplicates – Smaller database – Accurate statistics The problem – Merge/Purge of large databases

Preview Data Cleansing Solutions Real World Data OCAR’s Data Conclusion

Data Cleansing Solutions Sorted-Neighborhood Method Equational Theory Transitive Closure

Sorted-Neighborhood Method Three phases – 1. create keys – 2. sort the data – 3. merge Three passes using different key – Multi-pass method

Sorted-Neighborhood Method Key selection First NameLast NameAddressIDKey SalStolfo123 First Street STLSAL123FRST456 SalStolfo123 First Street STLSAL123FRST456 SalStolpho123 First Street STLSAL123FRST456 SalStiles123 Forest Street STLSAL123FRST456

Sorted-Neighborhood Method Sort using the key selected First NameLast NameAddressIDKey SalStolfo123 First Street STLSAL123FRST456 SalStolfo123 First Street STLSAL123FRST456 SalStolpho123 First Street STLSAL123FRST456 SalStiles123 Forest Street STLSAL123FRST456

Sorted-Neighborhood Method A ‘window size’ is created for merging First NameLast NameAddressIDKey SalStolfo123 First Street STLSAL123FRST456 SalStolfo123 First Street STLSAL123FRST456 SalStolpho123 First Street STLSAL123FRST456 SalStiles123 Forest Street STLSAL123FRST456

Merge Phase - Equational Theory A set of equation rules that defines equivalence A type of clustering function (pattern recognition) Rules may require an expert

Merge Phase - Equational Theory English rules: Given two records, r1 and r2. IF (the last names of r1 equals the last name of r2, AND the first names differ slightly, AND the address of r1 equals the address of r2) THEN R1 is equivalent to r2

Merge Phase - Equational Theory Results SSNName (First, Initial, Last)Address Lisa Boardman144 Wars St Lisa Brown144 Ward St Ramon Bonilla38 Ward St Raymond Bonilla38 Ward St. 0Diana D. Ambrosion40 Brik Church Av. 0Diana A. Dambrosion40 Brick Church Av. 0Colette Johnen th St. apt.5a5 0John Colette th St. ap Ivette A Keegan23 Florida Av Yvette A Kegan23 Florida St. r1 r2

Merge Phase - Transitive Closure Applied to a single pass sorted-neighborhood method Improvement of accuracy Decreases processing time and cost

Merge Phase - Transitive Closure English rules: Given three records a, b and c. IF (a is similar to b AND b is similar to c) THEN a is similar to c

Real World Data State of Washington Department of Social and Health Services Office of Children Administrative Research (OCAR) of the Department of Social and Health Services

OCAR’s Data 6,000,000 records Grows by 50,000 per month 19 fields – First and last name – Birthdate – SSN – Case number – Worker ID – Gender – Race – Service ID – Service dates – Payments

OCAR’s Data - Problems Names misspelled Missing birthdates Missing or wrong SSN Multiple case numbers Ghost records

OCAR’s Data - Goals To answer: – “How many children are in foster care?” – “How long do children stay in foster care?” – “How many different homes do children typically stay in?”

OCAR’s Data - Cleaning 128,438 records sampled (one service office) Consulted with expert 1 24 rules established Used sorted-neighborhood multi-pass methods Applied equational theory Keys – 1. Last name, First name, SSN, and Case number – 2. First name, Last name, SSN, and Case number – 3. Case number, First name, Last name, and SSN 1 Timothy Clark, OCAR Computer Information Consultant

OCAR’s Data - Results Identified 8,504 individuals in sample 45.8% correctly classified 86.0% where correctly merged Multi-pass sorted-neighborhood confirmed

Review Multi-pass sorted-neighborhood method Equational method OCAR’s data

Conclusions Sort-neighborhood method can be expensive – During the sorting phase Process time improved accuracy – Multiple times – Small windows – Computation of the transitive closure

Sources Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem; Mauricio A. Hernandez and Salvatore J. Stolfo; Department of Computer Science, Columbia University, New York, NY Haiguang Li, 2011 class presentation