1. In collaboration with Werner Nutt Free University of Bozen-Bolzano Data Quality Simon Razniewski

2. 10.8.2011 - EURACData Quality2 Introduction Simon Razniewski PhD Student at the FUB –Data quality –Data completeness Werner Nutt Professor in Computer Science at the FUB Focus in research and teaching: –Data management, data modelling –Data integration –Incomplete information

3. 3 Why data quality? Data are the basis for (scientific) conclusions about the world Conclusions only as good as the data they are based on Low-quality data  low-quality conclusions 10.8.2011 - EURACData Quality

4. 4 Some effects of erroneous data are funny Man invited for pre-natal check 10.8.2011 - EURACData Quality

5. 5 Some data errors are long-living Spinach contains much iron 100g of spinach contain 35mg of iron Gustav v. Bunge 1890 100g spinach contain only 3,5mg of iron 10.8.2011 - EURACData Quality

6. 6 Some data errors are mysterious Student records in in Georgia (USA), 2009 19.000 students leave their school to change to another … but arrive nowhere ? 10.8.2011 - EURACData Quality

7. 7 Overview What are data used for? –Data model the real world What can go wrong? –Wrong, outdated, missing data What can one do for –Correctness –Currency –Completeness of data? 10.8.2011 - EURACData Quality

8. 8 Data model the real world We analyze the data (instead of the real world) and draw (scientific) conclusions  data determines our conclusions Real world: Students, teachers, classes Database: Tables HOB Bozen Class 2A Paul Anna Maria Diego 10.8.2011 - EURACData Quality

9. 9 Questions about students „How many students are there in the class 2A of the HOB Bozen?“ „What is the average age of the students of this class?“ „How many students play an instrument?“ 10.8.2011 - EURACData Quality

10. 10 Table „Students“ NameDate of birthSchoolClass Paul7.4.1995HOB Bolzen2A Anna3.8.1959HOB Bozen1A Diego?HOB Meran2A What is the average age of the students of the class 2A of the HOB Bozen? 10.8.2011 - EURACData Quality

11. 11 Many things can go wrong NameDate of birthSchoolClass Paul7.4.1995HOB Bolzen2A Anna3.8.1959HOB Bozen1A Diego?HOB Meran2A 10.8.2011 - EURACData Quality What is the average age of the students of the class 2A of the HOB Bozen?

12. 12 Typos NameDate of birthSchoolClass Paul7.4.1995HOB Bolzen2A Anna3.8.1959HOB Bozen1A Diego?HOB Meran2A date of birth of Anna school of Paul 10.8.2011 - EURACData Quality

13. 13 Factual errors NameDate of birthSchoolClass Paul7.4.1995HOB Bolzen2A Anna3.8.1959HOB Bozen1A Diego?HOB Meran2A school of Diego (“HOB Meran“ instead of “HOB Bozen“) 10.8.2011 - EURACData Quality

14. 14 Outdated entries NameDate of birthSchoolClass Paul7.4.1995HOB Bolzen2A Anna3.8.1959HOB Bozen1A Diego?HOB Meran2A class of Anna (“1A“ instead of “2A“) 10.8.2011 - EURACData Quality

15. 15 Missing values NameDate of birthSchoolClass Paul7.4.1995HOB Bolzen2A Anna3.8.1959HOB Bozen1A Diego?HOB Meran2A date of birth of Diego (“Null value“) 10.8.2011 - EURACData Quality

16. 16 Missing records NameDate of birthSchoolClass Paul7.4.1995HOB Bolzen2A Anna3.8.1959HOB Bozen1A Diego?HOB Meran2A Maria 12.10.1995HOB Bozen2A the record about Maria is missing 10.8.2011 - EURACData Quality

17. 17 Missing concepts NameDate of birthSchoolClass Paul7.4.1995HOB Bolzen2A Anna3.8.1959HOB Bozen1A Diego?HOB Meran2A no possibility to store information about music instruments Instrument Cello ? ? 10.8.2011 - EURACData Quality

18. Datenqualität18 What can be done? There is a distinction between different dimensions of data quality The most important ones are Correctness Does the data match the real world? Timeliness Is the data up-to-date? Completeness Are all aspects of the domain of interest captured? Further: Comprehensibility, accessability, … 10.8.2011 - EURAC

19. Datenqualität19 Dimension 1: Correctness IT-techniques: 1.Detecting typos or statistical outliers students born in 1959 2.Recognizing duplicates Mohammad Al Zaïn = Muhamad Alzain 3.Rules for logical consistency no student can visit two schools at the same time Organisation: Special treatment of core data: Master data management For example: students, teachers, schools 10.8.2011 - EURAC

20. Datenqualität20 Dimension 2: Timeliness By workflow organisation: Bind workflows onto the IT system  Timeliness is guaranteed Example: an enrolment is only valid if it is recorded in the database Trough data about the currency of the data (metadata)  Timeliness can be estimated Example: “All dropouts until 31th of March are recorded“ 10.8.2011 - EURAC

21. Datenqualität21 Dimension 3/1: Completeness of values Can be enforced by the IT system Risk: nonsensical entries Alternative solution: enforce input of less values  Record reasons for missing values E.g. “Not applicable” or “Unknown” 10.8.2011 - EURAC

22. Datenqualität22 Dimension 3/3: Conceptual completeness Solid design is important, but not everything can be foreseen Flexible IT: Schema changes if necessary –Space for comments, additional information Otherwise: Other fields will be abused Example: Gasworks in the USA Warning of dogs for meter-readers … later they send bills AddressMountain Road 102 (Beware of dog) 10.8.2011 - EURAC

23. Datenqualität23 Dimension 3/2: Table completeness Events are completely recorded, if they are bound to the IT system Example: Sales in a supermarket In general, this binding is not possible  only parts of the database tables are complete But: Completeness is only necessary for specific uses Example: school statistics from ASTAT  Research 10.8.2011 - EURAC

24. Partial table completeness Common scenario: We have –Some, but not all data complete –Questions (‘‘queries“) over data Problems: –Do we have the data that is needed to answer the queries? If not: –What more data do we need? 10.8.2011 - EURACData Quality24

25. An (intuitive) example Suppose we have data about all students from –Italian schools –German schools, except of the primary school ‘‘Andreas Hofer“ –Ladin schools, except of the high school “Gherdëna“ Can we correctly answer questions about the italian students in South Tyrol?  Yes, because we have all data about students from italian schools 10.8.2011 - EURACData Quality25

26. An (intuitive) example (2) Suppose we have data about all students from –Italian schools –German schools, except of the primary school ‘‘Andreas Hofer“ –Ladin schools, except of the high school “Gherdëna“ Can we answer questions about the high-school students in South Tyrol?  No, because data from the “Gherdëna“ high school is missing  We could bug them to submit their data (but maybe the secretary is on holiday)  We could ask someone else for the data, e.g., the local district administration 10.8.2011 - EURACData Quality26

27. Our research How can one describe that data is complete to a certain extent? How can one find out whether the data one has is sufficient for a certain use? How can one find out which data is necessary to serve a certain use? 10.8.2011 - EURACData Quality27

28. Formal example How many students attend an Italian school? SELECT count(*) FROM student, school WHERE student.school = school.name AND school.language = ‘italian‘; Suppose, we have all Italian students. Can we answer this query completely? 10.8.2011 - EURACData Quality28

29. How can we formalize table completeness? “We have all students from italian schools“ We imagine: an ideal database that contains complete information about the world Completeness statements refer to this ideal database: => All ideal students from Italian schools occur among our real students 10.8.2011 - EURACData Quality29

30. How can we assert (partial) completeness of tables? “We have all students from italian schools“ Table completeness assertion: real.student CONTAINS( SELECT ideal.student.* FROM ideal.student, ideal.school WHERE ideal.student.school = ideal.school.name AND ideal.school.language = ‘Italian‘) Table completeness assertions constitute a logical theory about real and ideal database 10.8.2011 - EURACData Quality30

31. What does it mean that our query is complete? Consider two versions: “Real query““Ideal query“SELECT count(*) FROM real.student, real.schoolFROM ideal.student, ideal.school WHERE real.student.school = real.school.name ideal.student.school = ideal.school.name AND AND real.school.language = ‘italian‘; ideal.school.language = ‘italian‘; Our query is complete if the real and the ideal query return the same results (Can be expressed in logic, too) => Reasoning 10.8.2011 - EURACData Quality31

32. Our results so far Formalization General reasoning procedures for –Single block SQL queries –With comparisons –Group By –Aggregate functions min, max, count, sum Complexity analysis (sometimes high!) Architecture for reasoning system “Inverse reasoning” (see later slide) This is a start, many things are still missing 10.8.2011 - EURACData Quality32

33. Reasoning with schema information To draw interesting inferences, we need to take into account –Keys –Foreign keys –Finite domains ~> Reasoning becomes more complicated (Current research) 10.8.2011 - EURACData Quality33

34. Inverse reasoning So far: Given: Assertions about table completeness Question: Can query Q be answered completely? Also interesting: Given: query Q Question: which are the minimal completeness assertions that assure completeness of Q? Can be answered by applying our inference methods backwards 10.8.2011 - EURACData Quality34

35. Perspective: Probabilistic completeness management Our theory so far: Boolean statements: complete/not complete In practice, it is often sufficient to know “With probability < p, we make an error < ε“ Probabilistic assertions: “With 90% probability, we are not missing more than 5 students“ => Probabilistic inferences 10.8.2011 - EURACData Quality35

36. 36 Conclusion Data quality has several dimensions –Correctness, timeliness, completeness Our current interest –How can one describe which data are complete –How can one find out which queries can be answered completely? –If not, which additional data is needed? Perspective: Probabilistic completeness management 10.8.2011 - EURACData Quality