Why clean data? Data quality is important.

Cleaning data Makes the data fit for purpose/plausible Reduces the negative impact of errors Improves the data quality Improves the quality of the outputs

What to look for Non-response  an item non response  eg missing data Erroneous data  Can negatively affect data and resulting quality Suspicious data

Data gathering problems Manual entry (input error)  For example, switching numbers around, missing numbers, inputting two responses into one field Duplicates  For example, submit button hit more than once Measurement errors  For example, using inches instead of cm, reading scales incorrectly Non uniform standards for content and format  For example, people using different units - some giving index finger lengths in cm, some in mm.

Process of cleaning data Detect Resolve Treat

Detect Identify erroneous or suspicious data  Graph or sort data - look at outliers I have a student who throws ten dice and records the number of sixes. They recorded: (2, 0, 3, 12, 2, 0, 1, 1, 3, 1, 4).  What is wrong?  What do you think is the cause of it?

Detect Consider the data points  3, 4, 7, 4, 8, 3, 9, 5, 7, 6, 92  “92” is suspicious - an outlier Outliers:  are potentially legitimate (correct)  can be data or model glitches  can be a data miners dream, for example, a highly profitable customer Outlier - “departure from the expected”

Resolve Deciding if erroneous or suspicious data should be corrected or amended Deciding on the action to “treat” the data

Treat Leave as is Change  Impute: determine replacement value Replacement value is obtained from a similar record from the “clean” respondents from the data at hand Remove