Multi-camera Video Surveillance: Detection, Occlusion Handling, Tracking and Event Recognition Oytun Akman

Overview Surveillance Systems Single Camera Configuration Moving Object Detection Tracking Event Recognition Multi-camera Configuration Occlusion Handling

Single Camera Configuration Moving Object Detection (MOD) Tracking Event Recognition

Single Camera Configuration Moving Object Detection (MOD) Input Image - Background Image = Foreground Mask

Single Camera Configuration Moving Object Detection (MOD) Frame Differencing (M. Piccardi, 1996) Eigenbackground Subtraction (N. Oliver, 1999) Parzen Window (KDE) Based MOD (A. Elgammal, 1999) Mixture of Gaussians Based MOD (W. E. Grimson, 1999)

Single Camera Configuration MOD – Frame Differencing Foreground mask detection Background model update

Single Camera Configuration MOD – Eigenbackground Subtraction Principal Component Analysis (PCA) Reduce the data dimension Capture the major variance Reduced data represents the background model (http://web.media.mit.edu/~tristan/phd/dissertation/chapter5.html)

Single Camera Configuration MOD – Parzen Window Based Nonparametrically estimating the probability of observing pixel intensity values, based on the sample intensities

Single Camera Configuration MOD – Mixture of Gaussians Based Based on modeling each pixel by mixture of K Gaussian distributions Probability of observing pixel value xN at time N, where (assuming that R,G,B are independent)

Single Camera Configuration MOD - Simulation Results

Single Camera Configuration Tracking Object Association Mean-shift Tracker (D. Comaniciu, 2003) Cam-shift Tracker (G. R. Bradski, 1998) Pyramidal Kanade-Lucas-Tomasi Tracker (KLT) (J. Y. Bouguet, 1999) (A constant velocity Kalman filter is associated with each tracker)

Single Camera Configuration Tracking – Object Association Oi(t) = OJ(t+1) if Bounding box overlapping D(Oi(t), OJ(t+1)) < thresholdmd, D() is a distance metric between color histograms of objects Kullback-Leibler divergence Bhattacharya coefficient

Single Camera Configuration Tracking – Mean-shift Tracker Similarity function between the target model q and the candidate model p(y) is where p and q are m-bin color histograms (http://www.lisif.jussieu.fr/~belaroussi/face_track/CamshiftApproach.htm)

Single Camera Configuration Tracking - Mean-shift Tracker - Simulation Result

Single Camera Configuration Tracking – Cam-shift Tracker Backprojection image (probability distribution image) calculated Mean-shift algorithm is used to find mode of probability distribution image around the previous target location

Single Camera Configuration Tracking – Cam-shift Tracker - Simulation Result

Single Camera Configuration Tracking – Pyramidal KLT Optical flow d=[dx dy] of the good feature point (corner) is found by minimizing the error function (http://www.suri.it.okayama-u.ac.jp/research/2001/s-takahashi/s-takahashi.html)

Single Camera Configuration Tracking - Pyramidal KLT - Simulation Results

Single Camera Configuration Event Recognition - Hidden Markov Models (HMM) GM - HMMs, trained by proper object trajectories, are used to model the traffic flow (F. Porikli, 2004) (F. Bashir, 2005) m :starting frame number in which the object enters the FOV n :end frame number in which the object leaves the FOV

Single Camera Configuration Event Recognition – Simulation Result

Multi-camera Configuration Background Modeling Occlusion Handling Tracking Event Recognition

Multi-camera Configuration Background Modeling Three background modeling algorithms Foreground Detection by Unanimity Foreground Detection by Weighted Voting Mixture of Multivariate Gaussians Background Model

Multi-camera Configuration Background Modeling Common field-of-view must be defined to specify the region in which the cameras will cooperate

Multi-camera Configuration Background Modeling - Unanimity If (x is foreground) && (xI is foreground) foreground

Multi-camera Configuration Background Modeling – Weighted Voting and are the coefficients to adjust the contributions of the cameras. Generally, the contribution for the first camera (reference camera with better positioning) is larger than the second one, and

Multi-camera Configuration Background Modeling – Weighted Voting

Multi-camera Configuration Background Modeling – Mixture of Multivariate Gaussians Each pixel modeled by mixture of K multivariate Gaussian distributions where

Multi-camera Configuration Background Modeling – Mixture of Multivariate Gaussians Input image Mixture of Multivariate Gaussians Single camera MOG

Multi-camera Configuration Background Modeling - Conclusions Projections errors due to the planar-object assumption Erroneous foreground masks False segmentation results Cameras must be mounted on high altitudes compared to object heights Background modeling by unanimity False segmented regions are eliminated Any camera failure failure in final mask Solved by weighted voting In multivariate MOG method missed vehicles in single camera MOG method can be segmented

Multi-camera Configuration Occlusion Handling Primary issue of surveillance systems False foreground segmentation results Tracking failures Difficult to solve by using single-camera configuration Occlusion-free view generation by using multiple cameras Utilization of 3D information Presence of different points of views

Multi-camera Configuration Occlusion Handling – Block Diagram

Multi-camera Configuration Occlusion Handling - Background Subtraction Foreground masks are obtained using background subtraction

Multi-camera Configuration Occlusion Handling – Oversegmentation Foreground mask is oversegmented using “Recursive Shortest Spanning Tree” (RSST) and K-means algorithms RSST K-means

Multi-camera Configuration Occlusion Handling – Top-view Generation

Multi-camera Configuration Occlusion Handling – Top-view Generation Corresponding match of a segment is found by comparing the color histograms of the target segment and candidate segments on the epipolar line RSST K-means

Multi-camera Configuration Occlusion Handling – Clustering Segments are grouped using “shortest spanning tree” algorithm using the weight function RSST K-means

Multi-camera Configuration Occlusion Handling – Clustering After cutting the edges greater than certain threshold RSST K-means

Multi-camera Configuration Occlusion Handling – Conclusions Successful results for partially occluded objects Under strong occlusion Epipolar matching fails Objects are oversegmented or undersegmented Problem is solved if one of the cameras can see the object without occlusion RSST and K-means have similar results K-means has better real time performance

Multi-camera Configuration Tracking – Kalman Filters Advantage: continuous and correct tracking as long as one of the cameras is able to view the object Tracking is performed in both of the views by using Kalman filters 2D state model: State transition model: Observation model:

Multi-camera Configuration Tracking – Object Matching Objects in different views are related to each other via homography

Multi-camera Configuration Tracking - Example

Multi-camera Configuration Tracking - Example

Multi-camera Configuration Tracking – Simulation Results Multi-camera Tracking

Multi-camera Configuration Tracking – Simulation Results Single-camera Tracking Single-camera Tracking

Multi-camera Configuration Tracking – Simulation Results Multi-camera Tracking

Multi-camera Configuration Tracking – Simulation Results Single-camera Tracking Single-camera Tracking

Multi-camera Configuration Event Recognition - Trajectories Extracted trajectories from both of the views are concatenated to obtain a multi-view trajectory

Multi-camera Configuration Event Recognition – Training

Multi-camera Configuration Event Recognition – Viterbi Distances of Training Samples Object ID Viterbi Distance to GM_HMM_1 Viterbi Distance to GM_HMM_2 Viterbi Distance to GM_HMM_1+2 1 10.0797 9.90183 19.7285 2 10.2908 10.1049 20.1867 3 10.2266 10.1233 20.1006 4 10.577 10.6716 21.2304 5 9.99018 9.84572 19.6763 6 10.0584 9.85901 19.6572 7 10.0608 9.88434 19.7496 8 10.2821 10.2472 20.3949 9 10.0773 9.8764 19.7181 10 10.3629 10.2508 20.3399 11 10.0322 9.86696 19.6382 12 10.0695 9.92222 19.7072 13 10.1321 9.95447 19.7818 14 10.2666 10.139 20.2119 15 10.2661 10.0629 20.0147 16 10.038 9.92932 19.6548 17 10.126 9.98202 19.7991 18 10.2134 10.108 19.9983 19 10.8046 10.5149 21.3008 20 10.4454 10.2919 20.333 21 10.111 9.90018 19.6983 22 10.1791 9.9294 19.9025 23 10.0511 10.0658 20.1564 24 10.1007 10.2248 20.248 25 10.3782 9.8986 19.9865 26 10.0308 9.9264 19.8682 27 10.0816 10.2139 20.1286

Multi-camera Configuration Event Recognition – Simulation Results with Abnormal Data Average distance to GM_HMM_1 : 10.20 Average distance to GM_HMM_2 : 10.06 Average distance to GM_HMM_1+2: 20.04 Object ID Viterbi Distance to GM_HMM_1 Viterbi Distance to GM_HMM_2 Viterbi Distance to GM_HMM_1+2 28 20.4058 19.9481 45.1818 29 21.2409 19.7736 45.034 30 26.9917 24.7016 55.2278 31 10.7213 10.5773 21.2099 32 10.4648 10.5105 22.1852 33 10.1611 9.97222 19.7785

Multi-camera Configuration Tracking & Event Recognition - Conclusions Successful results Correct initial segmentation Other tracker algorithms can be used Event recognition GM_HMM1+2 classifies the test data better

Thank you...

Summary - Surveillance Single camera configuration Moving object detection Frame differencing Eigen background Parzen window (KDE) Mixture of Gaussians Tracking Object association Mean-shift tracker Cam-shift tracker Pyramidal Kanade-Lucas-Tomasi tracker (KLT) Event recognition Multi-camera configuration Background modeling Foreground detection by unanimity Foreground detection by weighted voting Mixture of multivariate Gaussian distributions Occlusion handling