The Fish4Knowledge Project Disclosing Computer Vision Errors to End-Users Emma Beauxis-Aussalet, Lynda Hardman, Jacco Van Ossenbruggen, Jiyin He, Elvira.

Slides:

Advertisements

Similar presentations

Chapter 4 Pattern Recognition Concepts: Introduction & ROC Analysis.

Advertisements

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

Accuracy Assessment of Thematic Maps

Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Model Evaluation Metrics for Performance Evaluation

CS 8751 ML & KDDEvaluating Hypotheses1 Sample error, true error Confidence intervals for observed hypothesis error Estimators Binomial distribution, Normal.

Supervised classification performance (prediction) assessment Dr. Huiru Zheng Dr. Franscisco Azuaje School of Computing and Mathematics Faculty of Engineering.

Statistics Micro Mini Threats to Your Experiment!

Statistical Concepts (continued) Concepts to cover or review today: –Population parameter –Sample statistics –Mean –Standard deviation –Coefficient of.

1 The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller MI – 25. oktober 2007 Kresten Toftgaard Andersen.

Performance Evaluation in Computer Vision Kyungnam Kim Computer Vision Lab, University of Maryland, College Park.

Statistics: The Science of Learning from Data Data Collection Data Analysis Interpretation Prediction  Take Action W.E. Deming “The value of statistics.

Jeremy Wyatt Thanks to Gavin Brown

Experimental Evaluation

Edge Detection Evaluation in Boundary Detection Framework Feng Ge Computer Science and Engineering, USC.

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 07: Cost-Sensitive Measures.

Accuracy Assessment. 2 Because it is not practical to test every pixel in the classification image, a representative sample of reference points in the.

Evaluating Classifiers

Chapter 4 Pattern Recognition Concepts continued.

1 Evaluating Model Performance Lantz Ch 10 Wk 5, Part 2 Right – Graphing is often used to evaluate results from different variations of an algorithm. Depending.

Estimation Bias, Standard Error and Sampling Distribution Estimation Bias, Standard Error and Sampling Distribution Topic 9.

Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =

PARAMETRIC STATISTICAL INFERENCE

Validating Wykoff's Model, Take 2: Equivalence tests and spatial analysis in a design- unbiased analytical framework Robert Froese, Ph.D., R.P.F. School.

Evaluating What’s Been Learned. Cross-Validation Foundation is a simple idea – “ holdout ” – holds out a certain amount for testing and uses rest for.

Experiments in Machine Learning COMP24111 lecture 5 Accuracy (%) A BC D Learning algorithm.

Accuracy Assessment Having produced a map with classification is only 50% of the work, we need to quantify how good the map is. This step is called the.

PROCESSING, ANALYSIS & INTERPRETATION OF DATA

Introducing the Separability Matrix for ECOC coding

Remote Sensing Classification Accuracy

Evaluating Results of Learning Blaž Zupan

Copyright © 2010, 2007, 2004 Pearson Education, Inc. All Rights Reserved. Section 7-5 Estimating a Population Variance.

Statistical planning and Sample size determination.

Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain.

Bab /57 Bab 4 Classification: Basic Concepts, Decision Trees & Model Evaluation Part 2 Model Overfitting & Classifier Evaluation.

Practical Issues of Classification Underfitting and Overfitting –Training errors –Generalization (test) errors Missing Values Costs of Classification.

Evaluating Predictive Models Niels Peek Department of Medical Informatics Academic Medical Center University of Amsterdam.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

Chapter 6: Analyzing and Interpreting Quantitative Data

December 21, 2015Prof. Vivek Kumar1 Six Sigma Tools for Measure.

Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC

Evaluating Classification Performance

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Genetic Algorithms (in 1 Slide) l GA: based on an analogy to biological evolution l Each.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

IPDET Module 9: Choosing the Sampling Strategy. IPDET © Introduction Introduction to Sampling Types of Samples: Random and Nonrandom Determining.

ROC curve estimation. Index Introduction to ROC ROC curve Area under ROC curve Visualization using ROC curve.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Accuracy Assessment Accuracy Assessment Error Matrix Sampling Method

Date of download: 6/3/2016 Copyright © 2016 SPIE. All rights reserved. A framework for improved pedestrian detection performance through blind image distortion.

Accuracy Assessment of Thematic Maps THEMATIC ACCURACY.

Experience Report: System Log Analysis for Anomaly Detection

AP Biology Intro to Statistics

CSI5388: A Critique of our Evaluation Practices in Machine Learning

Evaluating Results of Learning

Accuracy Assessment of Thematic Maps

ECE 471/571 – Lecture 18 Classifier Fusion 04/12/17.

Dipartimento di Ingegneria «Enzo Ferrari»,

Chapter 7 – K-Nearest-Neighbor

POSC 202A: Lecture Lecture: Substantive Significance, Relationship between Variables 1.

Our Data Science Roadmap

Experiments in Machine Learning

CSCI N317 Computation for Scientific Applications Unit Weka

Chapter 11 Practical Methodology

Model Evaluation and Selection

Data Mining Class Imbalance

False discovery rate estimation

Our Data Science Roadmap

COSC 4368 Intro Supervised Learning Organization

LARGE CROWD COUNTING by MIRTES CORREA RET 2018

Presentation transcript:

The Fish4Knowledge Project Disclosing Computer Vision Errors to End-Users Emma Beauxis-Aussalet, Lynda Hardman, Jacco Van Ossenbruggen, Jiyin He, Elvira Arslanova, Tiziano Perrucci 12 December 2014CWI Scientific Meeting1

Monitoring Fish Population 2 We count fish to study ecosystems We count fish to study ecosystems

Monitoring Fish Population 3 Why use Computer Vision? It can count fish from each species It supports short- to long-term research It is not intrusive, and cost-effective

Monitoring Fish Population 4 But… new practices need to be introduced, and scientific validity needs to be assessed. But… new practices need to be introduced, and scientific validity needs to be assessed. Why use Computer Vision? It can count fish from each species It supports short- to long-term research It is not intrusive, and cost-effective

Detecting Fish 5 Collect examples of fish (Ground-Truth) Collect examples of fish (Ground-Truth)

6 Collect examples of fish (Ground-Truth) Collect examples of fish (Ground-Truth) Construct fish models Detecting Fish

7 Collect examples of fish (Ground-Truth) Collect examples of fish (Ground-Truth) Construct fish models Classify fish species as the most similar model Classify fish species as the most similar model Detecting Fish

Motivations for HCI Research 8 Support uncertainty-aware data analysis What are the uncertainty factors? How to inform ecologists about each factor? How to support user assessment of end-results? Here the Octopus appeared. (½Φ )-(π√⅞) How precise is this?

12 December 2014CWI Scientific Meeting What are the uncertainty factors? 9

Interactions of Uncertainty Factors 10 Computer Vision Errors

Interactions of Uncertainty Factors 11 Some fish are misclassified Computer Vision Errors

Interactions of Uncertainty Factors 12 Poor ground-truth yields poor models Poor ground-truth yields poor models Ground-Truth Quality Computer Vision Errors

Interactions of Uncertainty Factors 13 Poor images yield more errors Poor images yield more errors Ground-Truth Quality Computer Vision Errors Image Quality

Interactions of Uncertainty Factors 14 Typhoons yield poor images? What confidence intervals? Typhoons yield poor images? What confidence intervals? Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

Interactions of Uncertainty Factors 15 Missing videos? Number of Video Samples Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

Interactions of Uncertainty Factors 16 Some species swim in & out the field of view Some species swim in & out the field of view Number of Video Samples Duplicated Individuals Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

Interactions of Uncertainty Factors 17 Fields of view target specific species Fields of view target specific species Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

Interactions of Uncertainty Factors 18 Fields of view target specific species Fields of view target specific species and shift overtime Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

Interactions of Uncertainty Factors 19 Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality

Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Biases & Noise in Specific Output Biases & Noise in Specific Output Image Quality Conveying Uncertainty Factors 20 Confusion Matrices Confusion Matrices Computer Vision Errors

Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Image Quality Conveying Uncertainty Factors 21 Confusion Matrices Confusion Matrices Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Logistic Regression Logistic Regression

12 December 2014CWI Scientific Meeting Conveying Computer Vision Errors with Confusion Matrices 22 Here the Octopus appeared. (½Φ )-(π√⅞) How precise is this?

12 December 2014CWI Scientific Meeting Conveying Computer Vision Errors with Confusion Matrices 23 Here the Octopus appeared. (½Φ )-(π√⅞) How precise is this? Without torture, no science. Russian Proverb Without torture, no science. Russian Proverb

State-of-the-Art 24 Confusion Matrix Confusion Matrix

State-of-the-Art 25 Typical Visualization Typical Visualization

State-of-the-Art 26

State-of-the-Art 27

State-of-the-Art 28 Diagonals are correct fish (TP). The rest are errors. Diagonals are correct fish (TP). The rest are errors.

State-of-the-Art 29 Columns are missed fish. (FN) Columns are missed fish. (FN)

State-of-the-Art 30 Rows are added fish. (FP) Rows are added fish. (FP)

State-of-the-Art 31 Rows & Columns are cumulated Rows & Columns are cumulated

State-of-the-Art 32 Advanced measurements are repeated Advanced measurements are repeated

Proposed Metric & Visualization 33 3 basic concepts: correct, added, missed fish 2 Main sources of errors Number & Proportions of error Simple metric for extrapolations

Proposed Metric & Visualization 34 3 basic concepts: correct, added, missed fish 2 Main sources of errors Number & Proportions of error Simple metric for extrapolations

Proposed Metric & Visualization NumbersProportions

Proposed Metric & Visualization NumbersProportions Improve Ground-Truth? Improve Ground-Truth?

Proposed Metric & Visualization NumbersProportions Improve algorithm? Improve algorithm? Improve Ground-Truth? Improve Ground-Truth?

38 Issues Tackled

39 Issues Tackled Large number of TN conceals uncertainty Large number of TN conceals uncertainty

40 Issues Tackled Large number of TN conceals uncertainty Large number of TN conceals uncertainty Information is lost about errors interdependence Information is lost about errors interdependence

41 Issues Tackled Information is lost about errors interdependence Information is lost about errors interdependence FP for one class are FN for another FP for one class are FN for another Large number of TN conceals uncertainty Large number of TN conceals uncertainty

42 Issues Tackled Information is lost about errors interdependence Information is lost about errors interdependence Large number of TN conceals uncertainty Large number of TN conceals uncertainty Class proportions can vary

43 Issues Tackled Information is lost about errors interdependence Information is lost about errors interdependence Large number of TN conceals uncertainty Large number of TN conceals uncertainty Class proportions can vary Using one single type of curve can hide differences Using one single type of curve can hide differences

12 December 2014CWI Scientific Meeting Conveying Computer Vision Biases with Logistic Regression 44 in collaboration with Bas Boom Without torture, no science. Russian Proverb Without torture, no science. Russian Proverb

Logistic Regression Method 45

Logistic Regression Method 46

Logistic Regression Method 47

Logistic Regression Method 48

Logistic Regression Method 49

Logistic Regression Method 50

Visualization of Errors 51 Detail of the chances of biases

Achievements & Limitations BeforeAfter

Achievements & Limitations Fish counts are improved Fish counts are improved BeforeAfter

Achievements & Limitations BeforeAfter Fish counts are improved Fish counts are improved

Achievements & Limitations BeforeAfter Fish counts are improved Fish counts are improved

Achievements & Limitations Biases are reduced Biases are reduced BeforeAfter Fish counts are improved Fish counts are improved

Achievements & Limitations But beware further variations of class proportions! But beware further variations of class proportions! BeforeAfter Biases are reduced Biases are reduced Fish counts are improved Fish counts are improved

Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Image Quality Future Work 58 Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Integrate Computer Vision errors into ecologists’ statistical framework Integrate Computer Vision errors into ecologists’ statistical framework

Number of Video Samples Duplicated Individuals Field of View Ground-Truth Quality Image Quality Future Work 59 Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output Integrate Computer Vision errors into ecologists’ statistical framework Integrate Computer Vision errors into ecologists’ statistical framework User studies with our visualizations

Number of Video Samples Ground-Truth Quality Image Quality Future Work 60 Integrate Computer Vision errors into ecologists’ statistical framework Integrate Computer Vision errors into ecologists’ statistical framework Computer Vision Errors Biases & Noise in Specific Output Biases & Noise in Specific Output User studies with our visualizations Duplicated Individuals Field of View Develop measurement methods

Online Demo: 12 December 2014CWI Scientific Meeting61