Robust Multi-Pedestrian Tracking in Thermal-Visible Surveillance Videos Alex Leykin, Yang Ran, and Riad Hammoud

Goal Create a pedestrian tracker that operates in: 1. Varying illumination conditions 2. Crowded environment To achieve it we create a fusion pedestrian tracker that uses input from: 1. IR camera 2. RGB camera Our approach consists of three stages: BG SubtractionBayesian trackerPedestrian Classifier

Related Work Fusion background model:  Y.Owechko, S.Medasani, and N.Srinivasa “Classifier swarms for human detection in infrared imagery”, OTCBVS 2004  M.Yasuno, N.Yasuda, andM.Aoki “Pedestrian detection and tracking in far infrared images” OTCBVS 2004  C. Dai, Y. Zheng, X. Li “Layered Representation for Pedestrian Detection and Tracking in Infrared Imagery” OTCBVS 2005  J.Davis, V.Sharma “Fusion-based Background Subtraction Using Contour Saliency”, OTCBVS 2005 Bayesian formulation:  J. Deutscher, B. North, B. Bascle and A. Blake “Tracking through singularities and discontinuities by random sampling”, ICCV 1999  A. Elgammal and L. S. Davis, “Probabilistic Framework for Segmenting People Under Occlusion”, ICCV  M. Isard, J. MacCormick, “BraMBLe: a Bayesian multiple-blob tracker”, ICCV 2001  T. Zhao, R. Nevatia “Tracking Multiple Humans in Crowded Environment”, CVPR 2004

Background Model Two stacks of codeword values (codebooks) Color μ RGB I low I hi Thermal t high t low codeword codebook

Adaptive Background Update  If there is no match create new codeword  Else update the codeword with new pixel information  If >1 matches then merge matching codewords I(p) > I low I(p) < I high (RGB(p)∙ μ RGB ) < T RGB t(p)/t high > T t1 t(p)/t low > T t2  Match pixel p to the codebook b

Subtraction Results Color model only Combined color and thermal model

Tracking Location of each pedestrian is estimated probabilistically based on:  Current image  Model of pedestrians  Model of obstacles The goal of our tracking system is to find the candidate state x` (a set of bodies along with their parameters) which, given the last known state x, will best fit the current observation z P(x’| z, x) = P(z|x’) · P(x’|x) observation likelihood state prior probability

Tracking – Accepting the State x’ and x  candidate and current states P(x)  stationary distribution of Markov chain m t  proposal distribution Candidate proposal state x’ is drawn with probability m t (x’|x) and then accept it with the probability α(x, x’)

Tracking: Priors N(h μ, h σ 2 ) and N(w μ,w σ 2 )  body width and height U(x) R and U(y) R  body coordinates are weighted uniformly within the rectangular region R of the floor map. d(w t, w t−1 ) and d(h t, h t−1 )  variation from the previous size d(x t, x’ t−1 ) and d(y, y’ t−1 )  variation from Kalman predicted position N(μ door, σ door )  distance to the closest door (for new bodies) Constraints on the body parameters: Temporal continuity:

Tracking Likelihoods: Distance weight plane Problem: blob trackers ignore blob position in 3D (see Zhao and Nevatia CVPR 2004) Solution: employ “distance weight plane” D xy = |P xyz, C xyz | where P and C are world coordinates of the camera and reference point correspondingly and

Tracking Likelihoods: Z-buffer 0 = background, 1=furthermost body, 2 = next closest body, etc

Tracking: Likelihoods Implementation of z-buffer (Z) and distance weight plane (D) allows to compute multiple-body configuration with one computationally efficient step. Let I - set of all blob pixels O - set of body pixels Then Color observation likelihood is based on the Bhattacharya distance between candidate and observed color histograms

Tracking: Jump-Diffuse Transitions  Add a new body  Delete a body  Recover a recently deleted body  Change body dimensions  Change body position (optimize with mean shift)

Tracking: Anisotropic Weighted Mean Shift Classic Mean-ShiftOur Mean-Shift t-1t H t

Tracking Results Sequence number FramesPeople People missed False hits Identity switches %

Finding Gait in Spatio-temporal Space  Periodic Pattern Grouping Theory:  A two-dimensional pattern that repeats along one dimension is called a frieze pattern in the mathematics and geometry literature  Group theory provides a powerful tool for analyzing such patterns  Mapping gait into repetitive texture  Translational symmetry: Class P 4  Detection: verifying spatio-temporal texture  Localization: extract orientation (trajectory), frequency (period), representative motif (signature) Symmetries of the gait patterns

Classifying Pedestrians X-t Image Extract Lattice Signature Results Finding Gait in Spatio-temporal Space Details in Y. Ran, I. Weiss, Q. Zheng, and L. S. Davis. Pedestrian detection via periodic motion analysis. IJCV 2007

Pedestrian Classification

 Trace a single horizontal scan line in time  Get a plot  Perform 1D line fitting Double Helical Signature

Classification Results

Tracking results

Pedestrian Detection

Contributions  Robust to illumination changes  Resolving track initialization ambiguity with MCMC  Non-unique body-blob correspondence  Gait detector runs in real time

Future Work  Extend binary background mask with foreground probability values  Incorporate these probabilities into appearance-based fitness equation for particle filter-based tracker  Utilize tracklet stitching (via particle tracker) to decrease the number of broken paths

Aknowledgements Organizers of OTCBVS Benchmark Dataset Collection

Thank you! alexleykin.zapto.org