Diagnosing Error in Object Detectors Department of Computer Science University of Illinois at Urbana-Champaign (UIUC) Derek Hoiem Yodsawalai Chodpathumwan.

Slides:

Advertisements

Similar presentations

Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley.

Advertisements

Latent SVMs for Human Detection with a Locally Affine Deformation Field Ľubor Ladický 1 Phil Torr 2 Andrew Zisserman 1 1 University of Oxford 2 Oxford.

Semantic Contours from Inverse Detectors Bharath Hariharan et.al. (ICCV-11)

Regionlets for Generic Object Detection

Learning Shared Body Plans Ian Endres University of Illinois work with Derek Hoiem, Vivek Srikumar and Ming-Wei Chang.

Road-Sign Detection and Recognition Based on Support Vector Machines Saturnino, Sergio et al. Yunjia Man ECG 782 Dr. Brendan.

Ľubor Ladický1 Phil Torr2 Andrew Zisserman1

1 Part 1: Classical Image Classification Methods Kai Yu Dept. of Media Analytics NEC Laboratories America Andrew Ng Computer Science Dept. Stanford University.

Lecture 31: Modern object recognition

1 Building a Dictionary of Image Fragments Zicheng Liao Ali Farhadi Yang Wang Ian Endres David Forsyth Department of Computer Science, University of Illinois.

Robust Object Tracking via Sparsity-based Collaborative Model

Global spatial layout: spatial pyramid matching Spatial weighting the features Beyond bags of features: Adding spatial information.

Advisers: Prof. C.V. Jawahar Prof. A. P.Zisserman 3rd August 2011

Object Category Detection: Sliding Windows Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 04/10/12.

Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs Roozbeh Mottaghi 1, Sanja Fidler 2, Jian Yao 2, Raquel Urtasun 2, Devi Parikh 3 1 UCLA.

Enhancing Exemplar SVMs using Part Level Transfer Regularization 1.

Large-Scale Object Recognition with Weak Supervision

Groups of Adjacent Contour Segments for Object Detection Vittorio Ferrari Loic Fevrier Frederic Jurie Cordelia Schmid.

Student: Yao-Sheng Wang Advisor: Prof. Sheng-Jyh Wang ARTICULATED HUMAN DETECTION 1 Department of Electronics Engineering National Chiao Tung University.

Recognition: A machine learning approach

Recognition using Regions CVPR Outline Introduction Overview of the Approach Experimental Results Conclusion.

Quantifying and Transferring Contextual Information in Object Detection Professor: S. J. Wang Student : Y. S. Wang 1.

Good morning, everyone, thank you for coming to my presentation.

Statistical Recognition Slides adapted from Fei-Fei Li, Rob Fergus, Antonio Torralba, and Kristen Grauman.

1 Image Recognition - I. Global appearance patterns Slides by K. Grauman, B. Leibe.

1 Learning to Detect Objects in Images via a Sparse, Part-Based Representation S. Agarwal, A. Awan and D. Roth IEEE Transactions on Pattern Analysis and.

Object Detection using Histograms of Oriented Gradients

Object Recognition: Conceptual Issues Slides adapted from Fei-Fei Li, Rob Fergus, Antonio Torralba, and K. Grauman.

Object Recognition: Conceptual Issues Slides adapted from Fei-Fei Li, Rob Fergus, Antonio Torralba, and K. Grauman.

Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based.

Spatial Pyramid Pooling in Deep Convolutional

Lecture 29: Recent work in recognition CS4670: Computer Vision Noah Snavely.

Generic object detection with deformable part-based models

Object Detection Sliding Window Based Approach Context Helps

Computer Vision CS 776 Spring 2014 Recognition Machine Learning Prof. Alex Berg.

“Secret” of Object Detection Zheng Wu (Summer intern in MSRNE) Sep. 3, 2010 Joint work with Ce Liu (MSRNE) William T. Freeman (MIT) Adam Kalai (MSRNE)

Language and images Abhijit Sarkar. Noun Verbs Adjective Adverbs.

Last part: datasets and object collections. CMU/MIT frontal facesvasc.ri.cmu.edu/idb/html/face/frontal_images cbcl.mit.edu/software-datasets/FaceData2.html.

Mining and Analysis of Control Structure Variant Clones Guo Qiao.

1 Action Classification: An Integration of Randomization and Discrimination in A Dense Feature Representation Computer Science Department, Stanford University.

Marco Pedersoli, Jordi Gonzàlez, Xu Hu, and Xavier Roca

Learning Collections of Parts for Object Recognition and Transfer Learning University of Illinois at Urbana- Champaign.

Object Detection with Discriminatively Trained Part Based Models

Lecture 31: Modern recognition CS4670 / 5670: Computer Vision Noah Snavely.

Deformable Part Models (DPM) Felzenswalb, Girshick, McAllester & Ramanan (2010) Slides drawn from a tutorial By R. Girshick AP 12% 27% 36% 45% 49% 2005.

MSRI workshop, January 2005 Object Recognition Collected databases of objects on uniform background (no occlusions, no clutter) Mostly focus on viewpoint.

A Model for Learning the Semantics of Pictures V. Lavrenko, R. Manmatha, J. Jeon Center for Intelligent Information Retrieval Computer Science Department,

Figure 2-1. Two different renderings (categorizations) of corn yield data. Analyzing Precision Ag Data – text figures © 2002, Joseph K. Berry—permission.

Stable Multi-Target Tracking in Real-Time Surveillance Video

Visual Categorization With Bags of Keypoints Original Authors: G. Csurka, C.R. Dance, L. Fan, J. Willamowski, C. Bray ECCV Workshop on Statistical Learning.

Training and Evaluating of Object Bank Models Presenter ： Changyu Liu Advisor ： Prof. Alex Interest ： Multimedia Analysis May 16 th, 2013.

Object detection, deep learning, and R-CNNs

LIGO- G Z Analyzing Event Data Lee Samuel Finn Penn State University Reference: T030017, T

CS 1699: Intro to Computer Vision Detection II: Deformable Part Models Prof. Adriana Kovashka University of Pittsburgh November 12, 2015.

Pictorial Structures and Distance Transforms Computer Vision CS 543 / ECE 549 University of Illinois Ian Endres 03/31/11.

Category Independent Region Proposals Ian Endres and Derek Hoiem University of Illinois at Urbana-Champaign.

Recognition Using Visual Phrases

Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation.

Object Recognition by Integrating Multiple Image Segmentations Caroline Pantofaru, Cordelia Schmid, Martial Hebert ECCV 2008 E.

Combining Evolutionary Information Extracted From Frequency Profiles With Sequence-based Kernels For Protein Remote Homology Detection Name: ZhuFangzhi.

Rich feature hierarchies for accurate object detection and semantic segmentation 2014 IEEE Conference on Computer Vision and Pattern Recognition Ross Girshick,

Cascade for Fast Detection

Object detection with deformable part-based models

Lit part of blue dress and shadowed part of white dress are the same color

Object detection, deep learning, and R-CNNs

Recap: Advanced Feature Encoding

Cheng-Ming Huang, Wen-Hung Liao Department of Computer Science

“The Truth About Cats And Dogs”

Multiple Feature Learning for Action Classification

Using statistics to evaluate your test Gerard Seinhorst

Presentation transcript:

Diagnosing Error in Object Detectors Department of Computer Science University of Illinois at Urbana-Champaign (UIUC) Derek Hoiem Yodsawalai Chodpathumwan Qieyun Dai Work supported in part by NSF awards IIS and IIS , ONR MURI Grant N , and a research award from Google

Object detection is a collection of problems Distance Shape Occlusion Viewpoint Intra-class Variation for “Airplane”

Object detection is a collection of problems Localization Error Background Dissimilar Categories Similar Categories Confusing Distractors for “Airplane”

How to evaluate object detectors? Average Precision (AP) – Good summary statistic for quick comparison – Not a good driver of research We propose tools to evaluate – where detectors fail – potential impact of particular improvements Typical evaluation through comparison of AP numbers figs from Felzenszwalb et al. 2010

Detectors Analyzed as Examples on VOC 2007 Deformable Parts Model (DPM) Felzenszwalb et al (v4) Sliding window Mixture of HOG templates with latent HOG parts Multiple Kernel Learning (MKL) Vedaldi et al Jumping window Various spatial pyramid bag of words features combined with MKL x x x

Top false positives: Airplane (DPM) Other Objects 11% Background 27% Similar Objects 33% Bird, Boat, Car Localization 29% Impact of Removing/Fixing FPs AP = 0.36

Top false positives: Dog (DPM) Similar Objects 50% Person, Cat, Horse Background 23% Localization 17% Impact of Removing/Fixing FPs Other Objects 10% AP = 0.03

Top false positives: Dog (MKL) Similar Objects 74% Cow, Person, Sheep, Horse Background 4% Localization 17% Other Objects 5% AP = 0.17 Impact of Removing/Fixing FPs Top 5 FP

Summary of False Positive Analysis DPM v4 (FGMR 2010) MKL (Vedaldi et al. 2009)

Analysis of object characteristics Additional annotations for seven categories: occlusion level, parts visible, sides visible Occlusion Level

Normalized Average Precision Average precision is sensitive to number of positive examples Normalized average precision: replace variable N j with fixed N Number of object examples in subset j

Object characteristics: Aeroplane

Occlusion : poor robustness to occlusion, but little impact on overall performance Easier (None)Harder (Heavy)

Size : strong preference for average to above average sized airplanes Object characteristics: Aeroplane Easier Harder X-SmallSmall X-Large Medium Large

Aspect Ratio : 2-3x better at detecting wide (side) views than tall views Object characteristics: Aeroplane TallX-Tall Medium WideX-Wide Easier (Wide) Harder (Tall)

Sides/Parts : best performance = direct side view with all parts visible Object characteristics: Aeroplane Easier (Side) Harder (Non-Side)

Summarizing Detector Performance Avg. Performance of Best Case Avg. Performance of Worst Case Avg. Overall Performance DPM (v4): Sensitivity and Impact

DPM (FGMR 2010) MKL (Vedaldi et al. 2009) occlusiontruncsize view part_vis aspect Sensitivity Impact Summarizing Detector Performance Best, Average, Worst Case

DPM (FGMR 2010) MKL (Vedaldi et al. 2009) occlusiontruncsize view part_vis aspect Occlusion: high sensitivity, low potential impact Summarizing Detector Performance Best, Average, Worst Case

DPM (FGMR 2010) MKL (Vedaldi et al. 2009) occlusiontruncsize view part_vis aspect MKL more sensitive to size Summarizing Detector Performance Best, Average, Worst Case

DPM (FGMR 2010) MKL (Vedaldi et al. 2009) occlusiontruncsize view part_vis aspect DPM more sensitive to aspect Summarizing Detector Performance Best, Average, Worst Case

Conclusions Most errors that detectors make are reasonable – Localization error and confusion with similar objects – Misdetection of occluded or small objects Large improvements in specific areas (e.g., remove all background FPs or robustness to occlusion) has small impact in overall AP – More specific analysis should be standard Our code and annotations are available online – Automatic generation of analysis summary from standard annotations

Thank you! Similar Objects 74% Cow, Person, Sheep, Horse Background 4% Localization 17% Other Objects 5% AP = 0.17 Impact of Removing/Fixing FPs Top 5 FP Top Dog False Positives