Quantifying and Transferring Contextual Information in Object Detection Professor: S. J. Wang Student : Y. S. Wang 1

Outline Background Goal Difficulties in Usage of Contextual Information Provided solutions Another method: TAS Experimental Results and Discussion Conclusion and Future Direction 2

Background (I) Only the properties of target object used in the detection task in the past. ◦ Problem: Intolerable number of false positive 3

Background (I) Only the properties of target object used in the detection task in the past. ◦ Problem: Intolerable number of false positive 4

Background (II) What else??? Contextual information! 5

Goal Establish a model to efficiently utilize the contextual information to boost the performance of detection accuracy. 6

Difficulties (I) Diversity of Contextual Information ◦ There are may different types of context often co-existing with different degrees of relevance to the detection for the target object(s) in different images. ◦ Terminology:  Things (e.g. cars and people)  Stuffs (e.g. roads and sky)  Scene (e.g. what happen in the image) ◦ Thing-Thing, Thing-Stuff, Stuff-Stuff and Scene-Thing 7

Difficulties (II) Ambiguity of Contextual Information ◦ Contextual information can be ambiguous and unreliable, thus may not always have a positive effect on object detection. ◦ Ex: Crowded Scene with constant movement and occlusion among multiple objects. 8

Difficulties (III) Lack of Data for Context Learning ◦ Not enough training data :  Over-fitting problem  Wrong degree of relevance ◦ Ex: The contextual information of people on top of sofa can be more useful than people on top of grass. 9

Training Data Preparation & Notation Representation 10 Base Detector (HOG) Training Image Candidate windows Positive sample: Red Negative sample: Green

Provided Solutions A polar geometric descriptor for contextual representation. A maximum margin context model (MMC) for quantifying context. A context transfer learning model for context learning with limited data. 11

Polar Geometric Descriptor Instead of traditional annotation based descriptor, here we use polar geometric descriptor to describe two kind of contextual information (Thing-Thing, Thing-Stuff). 12 r :orientation b+1 :radial bins r*b+1 :patches 0.5σ, σ and 2σ :bin length Feature :HOG Patch representation: Bag of Words method using K-means with K = 100

Provided Solutions A polar geometric descriptor for contextual representation. A maximum margin context model (MMC) for quantifying context. A context transfer learning model for context learning with limited data. 13

Quantifying Context (I) Quantifying Context (I) 14 Risk function:

Quantifying Context (II) Quantifying Context (II) Goal = Minimize the Risk function 15 Minimize L equal to fulfill the following constraint Hard to be solved, could be replaced by

Quantifying Context (III) Maximum Margin Context Model 16 Add some extra variables and constraints

Provided Solutions A polar geometric descriptor for contextual representation. A maximum margin context model (MMC) for quantifying context. A context transfer learning model for context learning with limited data. 17

Context Transfer Learning Context Transfer Learning Two Cases: ◦ Similar contextual information  Ex: Cars and motorbikes ◦ Little in common in both appearance and context, but similar level of assistance provided by contextual information.  Ex: People and bikes 18

TMMC-I: Transferring Discriminant Contextual Information TMMC-I: Transferring Discriminant Contextual Information Similar context provide the assistance on the learning of w. 19

TMMC-I: Transferring Discriminant Contextual Information TMMC-I: Transferring Discriminant Contextual Information New Constraint: 20 Modified optimization function:

TMMC-II: Transferring the Weight of Prior Detection Score Similar level of assistance, same weight 21

TMMC-II: Transferring the Weight of Prior Detection Score 22 New Constraint: Modified optimization function:

Another Method: TAS 23

Another Method: TAS (I) 24 Steps: 1.Segmenting image into regions. 2.Use base-detector to get the candidate patches. 3.Establish the relationships between candidate patches and regions. 4.Use the relationships to judge there is a target object in the patch or not.

Another Method: TAS (II) Region clusters: 25

Another Method: TAS (III) Examples of experiment: 26

Experimental Result and Discussion Use four data sets for testing: ◦ VOC 2005 ◦ VOC 2007 ◦ I-LIDS ◦ FORECOURT 27

Experimental Result and Discussion 28

Experimental Result and Discussion 29

Experimental Result and Discussion Context Transfer Learning Models: 30

Experimental Result and Discussion Context Transfer Learning Models: 31

Conclusion and Future Direction In this paper, the author proposes a contextual information model to quantify and select useful context information to boost the detection performance. What can we do next? ◦ HOG feature not suits for stuff (e.g. sky, road) ◦ Automatic selection between TMMC-I, TMMC-II ◦ Automatic selection between target object category and source category 32

Reference Wei-Shi Zheng, Member, IEEE, Shaogang Gong, and Tao Xiang, ”Quantifying and Transferring Contextual Information in Object Detection ”, PAMI accepted. Geremy Heitz, Daphne Koller, “ Learning Spatial Context: Using Stuff to Find Things”, ECCV Youtube Search “Hard-Margin SVM” 33