Robust Multi-Pedestrian Tracking in Thermal-Visible Surveillance Videos Alex Leykin and Riad Hammoud

Goal Create a pedestrian tracker that operates in: 1.Varying illumination conditions 2.Crowded environment To achieve it we create a fusion pedestrian tracker that uses input from: 1.IR camera 2.RGB camera Our approach consists of two stages: adaptive fusion background model Bayesian tracker blob array

Related Work Fusion background model: Y.Owechko, S.Medasani, and N.Srinivasa “Classifier swarms for human detection in infrared imagery”, OTCBVS 2004 M.Yasuno, N.Yasuda, andM.Aoki “Pedestrian detection and tracking in far infrared images” OTCBVS 2004 C. Dai, Y. Zheng, X. Li “Layered Representation for Pedestrian Detection and Tracking in Infrared Imagery” OTCBVS 2005 J.Davis, V.Sharma “Fusion-based Background Subtraction Using Contour Saliency”, OTCBVS 2005 Bayesian formulation: J. Deutscher, B. North, B. Bascle and A. Blake “Tracking through singularities and discontinuities by random sampling”, ICCV 1999 A. Elgammal and L. S. Davis, “Probabilistic Framework for Segmenting People Under Occlusion”, ICCV M. Isard, J. MacCormick, “BraMBLe: a Bayesian multiple-blob tracker”, ICCV 2001 T. Zhao, R. Nevatia “Tracking Multiple Humans in Crowded Environment”, CVPR 2004

Background Model Two stacks of codeword values (codebooks) codebook codeword Color μ RGB I low I hi Thermal t high t low

Adaptive Background Update If there is no match create new codeword Else update the codeword with new pixel information If >1 matches then merge matching codewords Remove the codeword if it had not appeared for a prolonged period of time Discard infrequent codewords Exclude p from update if it corresponds to a currently tracked body I(p) > I low I(p) < I high (RGB(p)∙ μ RGB ) < T RGB t(p)/t high > T t1 t(p)/t low > T t2 Match pixel p to the codebook b

Subtraction Results Color model only Combined color and thermal model

Tracking Location of each pedestrian is estimated probabilistically based on: Current image Model of pedestrians Model of obstacles The goal of our tracking system is to find the candidate state x` (a set of bodies along with their parameters) which, given the last known state x, will best fit the current observation z P(x’| z, x) = P(z|x’) · P(x’|x) observation likelihood state prior probability

Tracking – Accepting the State x’ and x  candidate and current states P(x)  stationary distribution of Markov chain m t  proposal distribution Candidate proposal state x’ is drawn with probability m t (x’|x) and then accept it with the probability α(x, x’)

Tracking: Priors N(h μ, h σ 2 ) and N(w μ,w σ 2 )  body width and height U(x) R and U(y) R  body coordinates are weighted uniformly within the rectangular region R of the floor map. d(w t, w t−1 ) and d(h t, h t−1 )  variation from the previous size d(x t, x’ t−1 ) and d(y, y’ t−1 )  variation from Kalman predicted position N(μ door, σ door )  distance to the closest door (for new bodies) Constraints on the body parameters: Temporal continuity:

Tracking Likelihoods: Distance weight plane Problem: blob trackers ignore blob position in 3D (see Zhao and Nevatia CVPR 2004) Solution: employ “distance weight plane” D xy = |P xyz, C xyz | where P and C are world coordinates of the camera and reference point correspondingly and

Tracking Likelihoods: Z-buffer 0 = background, 1=furthermost body, 2 = next closest body, etc

Tracking: Likelihoods Implementation of z-buffer (Z) and distance weight plane (D) allows to compute multiple-body configuration with one computationally efficient step. Let I - set of all blob pixels O - set of body pixels Then Color observation likelihood is based on the Bhattacharya distance between candidate and observed color histograms

Tracking: Jump-Diffuse Transitions Add a new body Delete a body Recover a recently deleted body Change body dimensions Change body position

Tracking: Anisotropic Weighted Mean Shift t Classic Mean-ShiftOur Mean-Shift t-1t H

Results

SequenceFramesPeoplePeople missed Frames missed False hits Frames in false hits Identity switches %

Conclusions A method to fuse visible and thermal inputs for background model creation: robust to illumination changes adaptive computationally efficient (30fps+) A novel formulation of priors in MCMC particle filter:

Future Work Extend binary background mask with foreground probability values Incorporate these probabilities into appearance- based fitness equation for particle filter-based tracker Utilize tracklet stitching to decrease the number of broken paths

Thank you!