模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Matching Tracking Sequences Across Widely Separated Cameras Yinghao Cai Kaiqi Huang, Tieniu Tan Presented by Zhuoshi Wei National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences 10/13/2008

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Outline Experimental Results Motivations Hierarchical appearance matching

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Motivations:

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Motivations: The field of view of a single camera is limited. Multiple camerasMultiple cameras are used to achieve wide area surveillance. How to automatically analyze and fuse the information gathered from multiple cameras???

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Problem Description: Images taken at Camera 1Images taken at Camera 2 establish correspondences between observations across cameras

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Problem description: Objective: To establish correspondences between observations across cameras. Difficulties: Low resolution Illumination variations Pose variations Different camera parameters Segmentation errors and partial occlusions.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Motivations Traditional methods: snapshot-based appearance matching: Match objects by a single image Sequence matching Camera 1: Camera 2: To alleviate the uncertainties due to segmentation errors and partial occlusions

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Hierarchical Appearance Matching sequence-based Focus on sequence-based appearance matching to alleviate the uncertainties due to segmentation errors and partial occlusions. Training stage: Obtain the appearance model for the moving object of the whole sequence. Testing stage: Sequentially accumulating the posterior likelihood by Bayesian inference.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Training stage: Partition the blob into regular patches for localization of color components.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Dominant color representation Each patch is represented by its dominant colors similar to [Piccardi, CVPR05]. someColors within a distance threshold are regarded as a single color which provides robustness to illumination changes to some extent.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Dominant Color Representation: Figure ： An example of using dominant colors to represent an image. (a) Original blob, (b) Object mask, (c) Rendered image by dominant colors. (a) (b) (c)

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Appearance model a single frame over the sequence Instead of building the appearance model of the whole sequence by a single frame, we obtain the appearance model by accumulating consistent hypotheses over the sequence. The appearance model of object over the sequence is obtained by concatenating feature vectors with the majority of vote over the sequence.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences The training process

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Each patch in one frame is matched against its corresponding patch in another frame by a similarity measure: Where: Num of pixels in the i-th patch of moving object “a” Num of dominant colors =1 frequency of the m-th dominant color

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Match across cameras At the training stage, we obtain the appearance model for each object as the most frequently occurring color patches over the sequence. For each object “a”, we obtain a class-specific appearance model under camera c1: Find out observations under another camera c2 which maximize:

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Testing stage The recognition score is calculated by accumulating the posterior likelihood over the sequence:

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Experimental Setup Figure (a) The layout of the camera system, (b) Views from two widely separated cameras.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Each column contains the same person under two disjoint views. Dataset: 42 people.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Experimental results: Figure (a) Rank Matching Performance. Rank i performance is the rate that the correct person is in the top i of the retrieved list. (b) Rank 1 Performance with different numbers of frames integrated at the testing stage.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Query example under camera 1: Top 5 matches ordered from left to right under camera 2

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Conclusions: threeSequences of images are compared on three matching layers:  Layer1: matching between feature vectors of the object over the sequence.  Layer2: matching between a class-specific appearance model and a single frame of an unknown object.  Layer3: matching over the whole sequences.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Conclusions: Presented a solution to the problem of matching sequences across different cameras. The color spatial information is preserved in its representation through partition. By incorporating multiple snapshots of the same object, the influence of the segmentation errors and partial occlusions is alleviated. Can be extended to the problem of camera handover and “query by example” in surveillance applications.

模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences