Daozheng Chen 1, Mustafa Bilgic 2, Lise Getoor 1, David Jacobs 1, Lilyana Mihalkova 1, Tom Yeh 1 1 Department of Computer Science, University of Maryland, College Park 2 Department of Computer Science, Illinois Institute of Technology Active Inference for Retrieval in Camera Networks

Problem Search camera network videos to retrieve frames containing specified individuals.

Time query

Time query

Related Work Person re-identification [Wang et al. ’07] Graphical Models for camera networks [Loy et al. ’09] Tracking over camera networks [Song et al. ’07] Active Learning [Settles ’09]

Our Contributions Map video frames in a camera network onto a graphical model and use a collective classification algorithm to predict frame states and perform frame retrieval. Apply active inference to direct human attention to portions of the videos which are most likely to have the biggest performance improvement.

Graphical Structures

Collective Classification

Active Inference

Active Inference

Outline Graphical model construction Iterative classification algorithm Active inference Experiment Conclusion

Graphical Model Construction Temporal neighbors (TN) Frames from the previous and next k time steps within the same camera. Positively correlated spatial neighbors (PSN) Correlation of the labels of two camera is greater than some threshold. Negatively correlated spatial neighbors (NSN) Correlation of the labels of two camera is less than some threshold.

Temporal Neighbors (k = 1)

Positively correlated spatial neighbors

Negatively correlated spatial neighbors

Graphical Structures

Outline Graphical model construction Iterative Classification Algorithm Active Inference Experiment Conclusion

The Iterative Classification Algorithm (ICA) Local Models (LM). The label of a frame is only dependent on its features. Relational Models (RM). The label of a frame is dependent on its features and its neighbors’ current labels First apply the local model for initialization, and then use the relational model iteratively until predicted labels converge. [Sen et al. ’08]

The Iterative Classification Algorithm (ICA) Local Models (LM). Logistic regression as the classifier. Cosine similarity based on signatures using bag-of-feature model as features F q = [f q1,f q2,…,f qn ] F = [f 1,f 2,…,f n ] COS(F q,F)

The Iterative Classification Algorithm (ICA) Relational Models (RM). Logistic regression as the classifier. Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f 21,f 22,…,f 2n ]

The Iterative Classification Algorithm (ICA) Relational Models (RM). Logistic regression as the classifier. Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f 21,f 22,…,f 2n ] F TN = [f TN1,f TN2 ]

The Iterative Classification Algorithm (ICA) Relational Models (RM). Logistic regression as the classifier. Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f 21,f 22,…,f 2n ] F TN = [f TN1,f TN2 ] F PSN = [f PSN1,f PSN2 ]

The Iterative Classification Algorithm (ICA) Relational Models (RM). Logistic regression as the classifier. Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f 21,f 22,…,f 2n ] F TN = [f TN1,f TN2 ] F PSN = [f PSN1,f PSN2 ] F NSN = [f NSN1,f NSN2 ]

The Iterative Classification Algorithm (ICA) Relational Models (RM). Logistic regression as the classifier. Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f 21,f 22,…,f 2n ] F TN = [f TN1,f TN2 ] F PSN = [f PSN1,f PSN2 ] F NSN = [f NSN1,f NSN2 ] F RM

Outline Graphical model construction Iterative Classification Algorithm Active Inference Experiment Conclusion

Active Inference The retrieval algorithm can request the correct labels for some frames at inference time. [Rattigan et al. ’07] Subsequent inference using ICA is based on these corrected labels. Common methods for selecting frames to label: Random (RND). Uniform (UNI). Most certain to be relevant (MR). Most uncertain (UNC) Reflect and Correct. [Bilgic and Getoor. ’09]

Reflect and Correct (RAC) [Bilgic and Getoor. ’TKDD09]

Adaptive RAC (MLI)

Outline Graphical model construction Iterative Classification Algorithm Active Inference Experimental Evaluation Conclusion

Dataset [Ding et al. ’10]

Queries

Region of Interests Background subtraction to determine region of interest in the frame. Densely sample key points in the regions Use color histogram in RGB space to describe the region spanned by a key point Quantized the descriptor according to learned 500 code words. Produce a single signature for a video frame.

Spatial Topology

Methods for Comparison Active inference based on LM using RND, UNI, MR, UNC, MLI Active inference based on RM using RND, UNI, MR, UNC, MLI Average accuracy and Average 11-average precision as measurement

Results UNC-LM has the best performance when results are based on LM. RM always perform better than LM does under the same sampling method. UNC-RM and MLI always perform better. MLI never perform worse than MLI does.

Outline Graphical model construction Iterative Classification Algorithm Active Inference Experiment Conclusion

Using a graphical model provides significant performance improvements in frame retrieval. A simple method that captures the frame uncertainty has an advantage over other baseline methods. Our adaptation of RAC has overall better performance.

Questions?