1. C. Canton1, J.R. Casas1, A.M.Tekalp2, M.Pardàs1
MLMI 2005, Edinburgh
Projective Kalman Filter: Multiocular Tracking of 3D Locations Towards Scene Understanding
C. Canton1, J.R. Casas1, A.M.Tekalp2, M.Pardàs1
1 Technical University of Catalonia, Barcelona, Spain
2 Koç University, Istanbul, Turkey

2. Outline Introduction Problem statement & Objective
Projective Kalman Filter (PKF)
Data scenario and formulation
Data association problem on P3→P2
Results & Performance
Conclusions & Future Research
Questions

3. Introduction
Tracking 3D locations within the SmartRoom scenario towards scene understanding can provide useful information (tracking of persons, head,…)

4. Problem statement
Standard approaches to track 3D locations from its 2D projections on N calibrated cameras involve:
2D feature selection over the N images
Kalman tracking
3D location estimation
Drawbacks:
Two disjoint problems
Data from N cameras is regarded as one single observation
Occlusion is handled in the estimation process but not in the tracking
Correspondence search among views
Initialization

5. Objective
Define a filtering scheme to track a 3D location from its N projections
2D feature selection over the N images
Joint 3D location estimation and tracking
Improvements:
Unified framework
Projective nature of N observations is taken into account
Joint 3D/2D occlusion detection scheme
Correspondence search among views
Initialization

6. Example

7. Kalman Filter (KF) Model
When estimating a state sR3 of a discrete time process governed by the linear stochastic difference equation
with a measuremement zR2xN that is
Kalman filter provides the optimal solution under the conditions:
Relations between hidden and observed data are linear
w[t] and v[t] have normal distribution
Projection is non-linear when seen as a morphism :R3→N2
Occlusions make this hypothesis unfeasible

8. Projective Kalman Filter (I)
Motivation:
Track a 3D location in Euclidean coordinates taking advantage of projective geometry
Model non-linearity between the hidden state s[t] and the observed data z[t] tacking into consideration the projective nature of the observations
Handle non-Gaussian impulsive noise: detect occlusion and disregard occluded data
Kalman theory can be applied to track 3D locations (with a Newtonian dynamic model) taking its projections as input data.

9. Projective Kalman Filter (II) Modelling non-linearity
Tackling projection non-linearity through observation matrix H:
An adaptive design of H[t] based on a compensation of the non-linearity from the prediction of the estimated state resolves the conflict (z=1).
During Kalman filter evolution, when applying H to the state vector s[t] coordinates might not be in the image plane (z1).

10. Projective Kalman Filter (III) Noise model
Observation noise covariance matrix R[t] controls how reliable is an observation. An adaptive approach to handle Gaussian noise and occlusions would be:
where:
Criterium to set the parameter βk from the PKF scheme: DATA ASSOCIATION & OCCLUSION DETECTION

11. Data association on P3→P2 (I)
Twofold objective:
Determine the spatial correspondence of two projections generated by the same 3D feature at two consecutive time instants in the same image
Detect an occlusion in a given view and modify R[t+1] accordingly

12. Data association on P3→P2 (II)
State Estimation Extrapolation
Data Bounding
Projection & Data Association
Occlusion Detection

13. Results Two types of data: Data specifications:
Synthetic: Exact algorithm evaluation and performance purposes
Real: Practial usage of this technique within a SmartRoom scenario to track the head of present people
Data specifications:
4 Calibrated cameras
768x576 pixels, 25 fps

14. Results on Synthetic Data (I)
First scenario: Gaussian noise
PKF outperforms KF by ~35%.
Interest Region

15. Results on Synthetic Data (II)
Second scenario: Gaussian and impulsive noise (occlusions)
PKF outperforms KF when occlusions are present
Influence of occlusions is reduced by the data association process
Interest Region

16. Results on Real Data (I)
Applied to track 2 people inside the SmartRoom at UPC towards scene understanding applications
Input 2D data is the top of non-overlapped foreground regions
When the 2 people are close, KF loses track but PKF keep it properly

17. Results on Real Data (II)

18. Conclusions & Future Work
New scheme to track 3D locations from multiple views embeding Kalman theory and projective geometry
Model multiple projections of a 3D location into a tracking loop
Occlusion detection combining 2D/3D data
Comparable computational complexity between PKF and KF
Real-time performance
Future Work:
Comparison with Particle Filtering tracking schemes
Apply this technique to body tracking into a SmartRoom

19. The End
Thank you!!!!