1. CSE 252C, Fall 2006UCSD1 Tracking objects across cameras by incrementally learning inter- camera colour calibration and patterns of activity (ECCV 2006) Andrew Gilbert and Richard Bowden University of Surrey

2. CSE 252C, Fall 2006UCSD2 Inter Camera Tracking As the number of cameras in a network increase, the user’s ability is limited in being able to manage such vast amounts of information. We would like to be able to automatically track objects across a camera network  Tracking an object throughout each cameras field of view (FOV)  Successful object “handover” between cameras in network We would like to be able to track objects across cameras without any explicit geometric or color calibration

3. CSE 252C, Fall 2006UCSD3 Proposed Algorithm Using an incremental learning method, the inter- camera color variations and the probability distribution of spacio-temperal links between cameras are modeled. Requires no color or spatial pre-calibration No batch processing We want the algorithm to work immediately, as more data becomes available will improve performance and is adaptable to changes in the cameras’ environments

4. CSE 252C, Fall 2006UCSD4 Test Environment Four non-overlapping color cameras in an office building.

5. CSE 252C, Fall 2006UCSD5 Intra-Camera Tracking The static background color distribution is modeled. A Gaussian mixture model on a per-pixel basis is used to form the foreground vs. background pixel segmentation, learned using an online approximation to expectation- maximization. Shadows are identified and removed by relaxing a models constraint on intensity by not chromaticity, and the foreground object is formed using connected component analysis on the resulting binary segmentation. Objects are linked temporally with a Kalman filter to provide movement trajectories within each camera. (Kalman Filter overview later!)

6. CSE 252C, Fall 2006UCSD6 Object Description Once foreground objects have been identified, an object descriptor is formed. Color histogram is used to describe the color fingerprint of the object  Spatially invariant  Simple and efficient to compute  Through some quantization, it also provides some invariance to changes in color appearance Similarity between objects is given by histogram descriptor intersection

7. CSE 252C, Fall 2006UCSD7 CCCM Consensus-Colour Conversion of Munsell colour space Breaks RGB color into 11 basic colors Each basic color represents a perceptual color category established through a physiological study of how humans categorize color. Works best without color-calibration of cameras in network Provides consistent inter camera without calibration, relying on the perceptual consistency of the color  If an object is perceived as red in both images, then CCCM will provide a consistent result With calibration, quantized RGB performs best (we’ll use this later).

8. CSE 252C, Fall 2006UCSD8 Color Descriptor

9. CSE 252C, Fall 2006UCSD9 Building Temporal links between Cameras Assume that objects follow similar routes inter camera and repetitions form consistent trends across data. The temporal inter camera links can be used to link camera regions together, yielding a probabilistic distribution of objects movement between cameras. As number of cameras in network increases, the number of possible links increases exponentially The majority of the possible links are invalid because they correspond to impossible routes We want a solution to distinguish between valid and invalid links  A solution that

10. CSE 252C, Fall 2006UCSD10 Building Temporal Links between Cameras Within each cameras field of view, the tracking algorithm forms color descriptor for object, the median histogram of the object recorded over the entire trajectory within a single camera Each new object is compared to previous objects within a given time window, T. Add the other equations here…. JLG!!!

11. CSE 252C, Fall 2006UCSD11 Region Links An example probability distribution showing a distinct link between two regions

12. CSE 252C, Fall 2006UCSD12 Incremental Block subdivision and recombination System based on rectangular subdivision Initially there are one block for each of the four cameras. If the maximum peak of the distribution is found to exceed the noise floor, this indicates a possible correlation between the blocks.  If correlation exists, block is subdivided into four equal sized blocks. Previous data and new data is used to form new links between newly formed sub-blocks  If no correlation, the links are thrown away to minimize maintenanced Blocks which are found to have similar distributions to neighbors are combined together  Reduces number of blocks and links maintained, and increases accuracy

13. CSE 252C, Fall 2006UCSD13 Incremental Block subdivision and recombination

14. CSE 252C, Fall 2006UCSD14 Calculating Posterior Appearance Distributions Given an object which disappears in region y we can model its reappearance probability over time as: Where the weight at time t is given as: This probability is used to weight the observation likelihood obtained through color similarity to obtain a posterior probability of a match.

15. CSE 252C, Fall 2006UCSD15 Modeling Color Variations The CCCM color quantization assumes a similar color response between cameras. Use CCCM as initial color descriptor, in parallel form color transformation matrices between cameras The tracked people are used as the calibration objects, and the transformation matrix is formed incrementally to model color changes between cameras

16. CSE 252C, Fall 2006UCSD16 Modeling Color Variations 12 34 Six transformations (with inverses) provide the twelve transformations needed to transform objects between the four cameras This allows less coarse quantization (RGB) to be used with improved performance

17. CSE 252C, Fall 2006UCSD17 Modeling Color Variations There is a color transformation matrix that transforms the color space of one camera into that of another camera With each tracked object, find this transformation matrix (using SVD) and then average it to the aggregate color transformation matrix Initially set to the Identity matrix

18. CSE 252C, Fall 2006UCSD18 Results Data used consisted of 10,000 objects tracked over a period of 72 hours of continuous operation Evaluation performed using unseen ground-truth 20 minute sequence with 300 instances of people tracked for more than 1 second.

19. CSE 252C, Fall 2006UCSD19 Results

20. CSE 252C, Fall 2006UCSD20 Results

21. CSE 252C, Fall 2006UCSD21 Results

22. CSE 252C, Fall 2006UCSD22 Conclusions Derived main entry / exit areas in a camera probabilistically using incremental learning Simultaneously the inter-camera color variations are learned These allow people to be tracked between spatially separated uncalibrated cameras with up to 81% accuracy No a priori information used and learning is unsupervised Fulfills three ideals of working immediately, improving performance as more data is accumulated, and can adapt to changes

23. CSE 252C, Fall 2006UCSD23 Let’s break it! Adding / taking off layers of clothing of different colors would fool the object descriptor. Take irregular paths. The link probabilities are built following the average paths. Move really slowly, so that you blend in with the background. (?) Anything else?... (invisible cloak)

24. CSE 252C, Fall 2006UCSD24 Results

25. CSE 252C, Fall 2006UCSD25 Kalman Filter Rudolph E. Kalman in 1960 published a recursive solution to the discrete-data linear filtering problem. Set of mathematical equations that implement a predictor-corrector estimator that is optimal in the sense that it minimizes the estimated error covariance Used extensively in computer graphics / vision for tracking Greg Welch, Gary Bishop (www.cs.unc.edu/~welch/kalman)

26. CSE 252C, Fall 2006UCSD26 Kalman Filter Trying to estimate the state of a discrete-time controlled process that is governed by the linear stochastic difference equation: With a measurement

27. CSE 252C, Fall 2006UCSD27 Kalman Filter Kalman Filter estimates system state at some time and then obtains feedback in terms of (noisy) measurement Equations for Kalman Filter fall into two categories:  (1) Time Update Equations Responsible for projecting forward in time the current state and error covariance estimates to obtain a priori estimates for the next time step  (2) Measurement Update Equations Measurement feedback - incorporates new measurements into the a priori estimate to obtain an improved a posteriori estimate

28. CSE 252C, Fall 2006UCSD28 Kalman Filter Time update = predictor Measurement update = corrector

29. CSE 252C, Fall 2006UCSD29 Kalman Filter – Time Update P is the error covariance estimate

30. CSE 252C, Fall 2006UCSD30 Kalman Filter – Measurement Update K is the kalman gain matrix

31. CSE 252C, Fall 2006UCSD31 Kalman Filter

32. CSE 252C, Fall 2006UCSD32 Extended Kalman Filter (EKF) The Kalman filter is estimating the state of a stochastic process that is linear. Many applications are non-linear!! Nothing to fear we have the EKF! EKF = a Kalman filter that linearizes about the current mean and covariance

33. CSE 252C, Fall 2006UCSD33 Kalman Filter Because the predicted state is based only on the previous state, it is computationally and memory-usage efficient and there are many implementations available OpenCV has implementation. Oh, and the state being tracked is the centroid of the objects…

34. CSE 252C, Fall 2006UCSD34 Questions Examples didn’t show occlusion during inter-camera tracking. How well does this system work with occlusion and tracking multiple people simultaneously? What other descriptors in addition to color histograms can be used to describe the tracked objects? Are there better performing intra-camera tracking techniques besides Kalman filter?