1. Tried and Tested? Evaluating Data Entry Consistency in the Component Model Prototype Brit Hopmann, July 1, 2008

2. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Presentation Background & aim consistency check Images & data Analysis & results (in 3 steps) Problems & possible solutions Conclusion & discussion data entry

3. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Background & aim consistency check Background Problems with data entry ambiguity Consequences: decreased recall and precision Aim Check and improve data entry consistency Solutions for existing problems? Changes? data entry

4. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Images

5. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Data 5 images 25 features (+ 2 discarded) 6 respondents Analysis in 3 steps Categorizing the answers (per image / per feature) Analysis per image and per feature Problems and possible solutions Analysis

6. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Analysis – step 1 example: answers for image4

7. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Analysis – step 1 Categorizing the answers (per image / per feature) 3 categories category consistencytimespercentage no problem5745,6% indecisive1612,8% problem5241,6% There were many problematic descriptions.

8. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Analysis – step 2 Summary per watermark (5 images) None of the images was described unanimously. imageno prob≈problemper image image 11528fine image 21339fine image 310312problematic image 4113 indecisive ( ≈) image 58512problematic total571652

9. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Analysis – step 2 Summary per feature (25 features + 2 discarded) Described consistently5-7 features*  3 of 5 images not consistently described10-8 features* * Depending on quality of definitions

10. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Analysis – step 3 How problematic are specific features? 1.Is or might the feature be problematic with other watermarks? 2.What caused the problem and can it be solved easily? -> image quality, definition, subjective description 3.What is the distinctive value of a particular feature? 4.Is there a possible solution? 5.Which problems will remain problematic after having applied a specific solution?

11. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Analysis – step 3 What could contribute to a solution? Grouping Allow assigning multiple features Drawback: increased decision effort Extra tool For instance graphical indicator of angle Drawback: exceptions such as rotated scans Clarify definitions For instance description of additional elements Relatively straightforward to apply Otherwise

12. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Crucial Question Which features will still pose a problem with a particular solution applied?

13. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Conclusion Problems that will eventually remain Eyes: alignment, distance, position Horns: endings, orientation, inside

14. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Conclusion Remaining problems Ears: endings, roundness, width Eyes: shape Head: shape In-between horns: shape Nose: concomitance with other elements, shape

15. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Discussion Features that evoke subjective descriptions (shape!) cause problems Thinking about possible changes in the model Clarify certain issues: clear definitions, instruction Expand model: extra tool, grouping Simplify model: adjust or take out features data Reconsider role of component model within Bernstein

16. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested? Discussion Questions and Comments?

17. Koninklijke Bibliotheek – National Library of the Netherlands Tried and Tested?