Report on Data Cleaning Framework Shahbaz Hassan Wasti

Benchmarks/metrics for Data Cleaning Techniques It is very difficult to measure the accuracy of different data cleaning techniques. Benchmarks and metrics to compare and evaluate different data cleaning techniques do not exist Research is being conducted to define standard benchmarks/metrics to evaluate data cleaning techniques Dasu et al. has proposed Statistical Distortion metric to evaluate the effect of data cleaning on the data. It gives how much data has been distorted after cleaning from the origin.

Common Practices to evaluate Data Cleaning Techniques I have studied the different experiments presented in several research papers to compare and evaluate data cleaning technique. It is observed that following common practices have been used by the researchers to compare one technique with other baselines are Number of errors cleaned in the dirty data by using the “ground truth” Ground Truth is prepared with the help of expert while manually cleaning the sample data Scalability is measured by time taken in cleaning with respect to the noise percentage in the data Precision and recall metric to estimate the possible erros Precision = (# corrected changed values/ all changes) Recall = (corrected changed values/ all erros) F-measure = 2 x (precision x recall)/ (precision + reacall) Validating sample cleaned data with the help of crowd

Experiment using Bayeswipe Techinque To evaluate the Bayesian cleaning method presented in the last meeting, I have downloaded tool from authors website Bayeswipe is developed to clean typographic, missing and substitution errors To explore the Bayeswipe I have selected two datasets Car sales data downloaded from authors website with ground truth and dirty sample University of Education Admission Dataset extracted from University of Education database

Statistics of Cars Dataset Total tuples in sample dirty dataset is 9124 All the attributes are categorical and contains string data Columns Dirty Records Distinct Values Dirty Data Ground Truth Model 42 225 198 Make 47 53 18 Type 40 34 12 Year 41 27 Condition 24 2 Wheelbase 3 5 Doors 4 Engine 71

Results of Car dataset after applying Bayeswipe The dataset was processed several times with same errors in the data It was surprising that Variations were observed in the results after every processing To compare the results, I have placed the ground truth dataset, dirty dataset and clean dataset in same spreadsheet It was observed that some records were cleaned correctly and some were ignored The algorithm also wrongly changed the correct data It was observed that most of the wrong correction made in the attributes with either alphanumeric or numeric data present in it. I have prepared the results summary of two processing

First Run Results Attribute Dirty Clean Not Cleaned Wrong Clean Distinct Values in GT Distinct Values in Dirty Data Model 42 18 24 22 198 225 Make 47 26 21 53 Type 40 16 3 12 34 Year 41 28 13 146 27 Condition 19 2 Wheelbase 15 7 5 Doors 14 4 Engine 23 71

First Run Results

Second Run Results Attribute Dirty Clean Not Cleaned Wrong Clean Distinct Values in GT Distinct Values in Dirty Data Model 42 25 17 28 198 225 Make 47 32 15 1 18 53 Type 40 26 14 8 12 34 Year 41 19 22 122 27 Condition 24 5 2 Wheelbase 21 13 3 Doors 9 4 Engine 71

Second Run Results

First & Second Run Result Variation

Statistics of University of Education Dataset (UE Dataset) Total 1000 clean records were randomly extracted from the database These clean records will be treated as ground truth The cardinality and degree of the dataset can be increased for further experiments I have chosen UE dataset because the availability of ground truth data. All the columns are of string type and categorical data Data has following columns Shift Category Campus Program Requirement Level City Admission Year Last Examination Year

UE Dataset Typo errors manually introduced in following column Columns Dirty Records Distinct Values Dirty Data Ground Truth Shift 110 9 2 Category 44 19 7 Campus 125 78 13 Program 127 57 23

First Run on UE Dataset Despite running the Bayeswipe on UE Dataset no record were cleaned by BayesWipe Columns Dirty Clean Not Cleaned Wrong Clean Distinct Values in GT Distinct Values in Dirty Data Shift 110 9 2 Category 44 19 7 Campus 125 78 13 Program 127 57 23

My Focus I will continue my experiments with Bayeswipe on some open dataset available on UCI machine learning repository. I have only checked typo errors in the above experiments Next I will prepare dataset for missing values The main problem in using open dataset is the availability of ground truth. I am working on the source and error model of the above technique to find out the reasons of inconsistency in results

Wrong Correction/Overcorrection Example Back