1. The Fish4Knowledge Project Disclosing Computer Vision Errors to End-Users Emma Beauxis-Aussalet, Lynda Hardman, Jacco Van Ossenbruggen, Jiyin He, Elvira Arslanova, Tiziano Perrucci 12 December 2014CWI Scientific Meeting1

2. Monitoring Fish Population 2 We count fish to study ecosystems We count fish to study ecosystems

3. Monitoring Fish Population 3 Why use Computer Vision? It can count fish from each species It supports short- to long-term research It is not intrusive, and cost-effective

4. Monitoring Fish Population 4 But… new practices need to be introduced, and scientific validity needs to be assessed. But… new practices need to be introduced, and scientific validity needs to be assessed. Why use Computer Vision? It can count fish from each species It supports short- to long-term research It is not intrusive, and cost-effective

5. Detecting Fish 5 Collect examples of fish (Ground-Truth) Collect examples of fish (Ground-Truth)

6. 6 Collect examples of fish (Ground-Truth) Collect examples of fish (Ground-Truth) Construct fish models Detecting Fish

7. 7 Collect examples of fish (Ground-Truth) Collect examples of fish (Ground-Truth) Construct fish models Classify fish species as the most similar model Classify fish species as the most similar model Detecting Fish

8. Motivations for HCI Research 8 Support uncertainty-aware data analysis What are the uncertainty factors? How to inform ecologists about each factor? How to support user assessment of end-results? Here the Octopus appeared. (½Φ )-(π√⅞) How precise is this?

9. 12 December 2014CWI Scientific Meeting What are the uncertainty factors? 9

10. Interactions of Uncertainty Factors 10 Computer Vision Errors

11. Interactions of Uncertainty Factors 11 Some fish are misclassified Computer Vision Errors

12. Interactions of Uncertainty Factors 12 Poor ground-truth yields poor models Poor ground-truth yields poor models Ground-Truth Quality Computer Vision Errors

13. Interactions of Uncertainty Factors 13 Poor images yield more errors Poor images yield more errors Ground-Truth Quality Computer Vision Errors Image Quality

14. Interactions of Uncertainty Factors 14 Typhoons yield poor images? What confidence intervals? Typhoons yield poor images? What confidence intervals? Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

15. Interactions of Uncertainty Factors 15 Missing videos? Number of Video Samples Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

16. Interactions of Uncertainty Factors 16 Some species swim in & out the field of view Some species swim in & out the field of view Number of Video Samples Duplicated Individuals Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

17. Interactions of Uncertainty Factors 17 Fields of view target specific species Fields of view target specific species Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

18. Interactions of Uncertainty Factors 18 Fields of view target specific species Fields of view target specific species and shift overtime Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

19. Interactions of Uncertainty Factors 19 Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

20. Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality Conveying Uncertainty Factors 20 Confusion Matrices Confusion Matrices Computer Vision Errors

21. Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Image Quality Conveying Uncertainty Factors 21 Confusion Matrices Confusion Matrices Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Logistic Regression Logistic Regression

22. 12 December 2014CWI Scientific Meeting Conveying Computer Vision Errors with Confusion Matrices 22 Here the Octopus appeared. (½Φ )-(π√⅞) How precise is this?

23. 12 December 2014CWI Scientific Meeting Conveying Computer Vision Errors with Confusion Matrices 23 Here the Octopus appeared. (½Φ )-(π√⅞) How precise is this? Without torture, no science. Russian Proverb Without torture, no science. Russian Proverb

24. State-of-the-Art 24 Confusion Matrix Confusion Matrix

25. State-of-the-Art 25 Typical Visualization Typical Visualization

26. State-of-the-Art 26

27. State-of-the-Art 27

28. State-of-the-Art 28 Diagonals are correct fish (TP). The rest are errors. Diagonals are correct fish (TP). The rest are errors.

29. State-of-the-Art 29 Columns are missed fish. (FN) Columns are missed fish. (FN)

30. State-of-the-Art 30 Rows are added fish. (FP) Rows are added fish. (FP)

31. State-of-the-Art 31 Rows & Columns are cumulated Rows & Columns are cumulated

32. State-of-the-Art 32 Advanced measurements are repeated Advanced measurements are repeated

33. Proposed Metric & Visualization 33 3 basic concepts: correct, added, missed fish 2 Main sources of errors Number & Proportions of error Simple metric for extrapolations

34. Proposed Metric & Visualization 34 3 basic concepts: correct, added, missed fish 2 Main sources of errors Number & Proportions of error Simple metric for extrapolations

35. Proposed Metric & Visualization NumbersProportions

36. Proposed Metric & Visualization NumbersProportions Improve Ground-Truth? Improve Ground-Truth?

37. Proposed Metric & Visualization NumbersProportions Improve algorithm? Improve algorithm? Improve Ground-Truth? Improve Ground-Truth?

38. 38 Issues Tackled

39. 39 Issues Tackled Large number of TN conceals uncertainty Large number of TN conceals uncertainty

40. 40 Issues Tackled Large number of TN conceals uncertainty Large number of TN conceals uncertainty Information is lost about errors interdependence Information is lost about errors interdependence

41. 41 Issues Tackled Information is lost about errors interdependence Information is lost about errors interdependence FP for one class are FN for another FP for one class are FN for another Large number of TN conceals uncertainty Large number of TN conceals uncertainty

42. 42 Issues Tackled Information is lost about errors interdependence Information is lost about errors interdependence Large number of TN conceals uncertainty Large number of TN conceals uncertainty Class proportions can vary

43. 43 Issues Tackled Information is lost about errors interdependence Information is lost about errors interdependence Large number of TN conceals uncertainty Large number of TN conceals uncertainty Class proportions can vary Using one single type of curve can hide differences Using one single type of curve can hide differences

44. 12 December 2014CWI Scientific Meeting Conveying Computer Vision Biases with Logistic Regression 44 in collaboration with Bas Boom Without torture, no science. Russian Proverb Without torture, no science. Russian Proverb

45. Logistic Regression Method 45

46. Logistic Regression Method 46

47. Logistic Regression Method 47

48. Logistic Regression Method 48

49. Logistic Regression Method 49

50. Logistic Regression Method 50

51. Visualization of Errors 51 Detail of the chances of biases

52. Achievements & Limitations BeforeAfter

53. Achievements & Limitations Fish counts are improved Fish counts are improved BeforeAfter

54. Achievements & Limitations BeforeAfter Fish counts are improved Fish counts are improved

55. Achievements & Limitations BeforeAfter Fish counts are improved Fish counts are improved

56. Achievements & Limitations Biases are reduced Biases are reduced BeforeAfter Fish counts are improved Fish counts are improved

57. Achievements & Limitations But beware further variations of class proportions! But beware further variations of class proportions! BeforeAfter Biases are reduced Biases are reduced Fish counts are improved Fish counts are improved

58. Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Image Quality Future Work 58 Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Integrate Computer Vision errors into ecologists’ statistical framework Integrate Computer Vision errors into ecologists’ statistical framework

59. Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Image Quality Future Work 59 Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Integrate Computer Vision errors into ecologists’ statistical framework Integrate Computer Vision errors into ecologists’ statistical framework User studies with our visualizations

60. Number of Video Samples Ground-Truth Quality Image Quality Future Work 60 Integrate Computer Vision errors into ecologists’ statistical framework Integrate Computer Vision errors into ecologists’ statistical framework Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output User studies with our visualizations Duplicated Individuals Field of View Develop measurement methods

61. Online Demo: http://f4k.project.cwi.nl 12 December 2014CWI Scientific Meeting61 emma@cwi.nl