1. Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis

2. Nathan Jacobs2 Visual Surveillance Observe people and vehicles –Where are they? –Where have they been? –Where are they going? Answering these questions requires object tracking.

3. Nathan Jacobs3 Probabilistic Tracking Tracking is commonly cast in a Bayesian framework to estimate object shape and location –Initial estimate = combination of image data likelihood and initialization prior –Updated estimate = combination of image data likelihood and state prediction prior Likelihood functions are the focus of most tracking work –Color histograms, templates Our focus is on the prior terms

4. Nathan Jacobs4 Quotes from yesterday “Initialization of tracking is important but not addressed here.” “Our object model assumes a well calibrated camera and a flat-ground plane.” “The prior term is a tricky thing to design.”

5. Nathan Jacobs5 Passive Vision : The Big Picture Learn strong scene-specific priors by watching the same scene for a long time –Made easier because the cameras are static –Should be learned online Priors can be used to improve anomaly detection and tracking algorithms

6. Nathan Jacobs6 Scene-specific Motion Priors

7. Nathan Jacobs7 Unusual Traffic Motion Video segment with anomalous motion (an ambulance using the median to pass stopped cars). False color sequence highlighting anomalous motions.

8. Nathan Jacobs8 Online Prior Learning for Tracking Online learning and use of motion priors: reduces the number of particles needed increases the number of objects that can be tracked. Frames Objects

9. Nathan Jacobs9 What else can we model? Watching for a long time allows us to build models of –Pixel intensity –Image derivatives –Image motion patterns We now transition to features based on the shape of foreground objects

10. Nathan Jacobs10 An Example Video

11. Nathan Jacobs11 Generating Examples Shapes Current Frame Foreground Mask Shape Descriptor Background Image For a long time: 1.Detect foreground objects 2.Generate a shape descriptor for each object 3.Add shape descriptor to training set

12. Nathan Jacobs12 Shape Descriptor Currently using a simple shape model –A 20-dimensional feature vector –Each dimension is the distance from center to edge of object Other shape models are possible

13. Nathan Jacobs13 Two Shape Model Types Both models are PCA subspaces Global model –Subspace is location independent –Distribution estimate is location dependent Local model –Subspace and distribution are both location dependent

14. Nathan Jacobs14 A location-independent Shape Basis Training Set Generate shape subspace using PCA on all shapes in training set. First Principle Component (~size) Second Principle Component (~orientation)

15. Nathan Jacobs15 Location-dependent Coefficients First Principle Component (~size) Second Principle Component (~orientation)

16. Nathan Jacobs16 Location-specific Shape Subspaces Generate a shape subspace using shapes found in a small region of the image. Location-specific mean shapes

17. Nathan Jacobs17 Location-specific Shape Subspaces Shape subspaces are location dependent. Much smaller variations in some regions. First PC VariationsSecond PC Variations

18. Nathan Jacobs18 Shapes in (Shape) Space

19. Nathan Jacobs19 An Example from PETS First Principle Component (~size) Second Principle Component (~aspect)

20. Nathan Jacobs20 Location-dependent Mean Shapes Mean Shapes

21. Nathan Jacobs21 Location-dependent Subspaces First PC VariationsSecond PC Variations

22. Nathan Jacobs22 Object Initialization for Tracking Object initialization is a crucial step of any tracking algorithm Use shape priors to determine object boundaries –Combines image information and shape prior –Penalize unlikely shapes –More accurate than image information alone Major point: strong priors make simple methods work

23. Nathan Jacobs23 Object Boundary Detection Goal is to determine object boundaries to improve tracking initialization Algorithm –Find candidates using background subtraction –Initialize each candidate with a location- specific mean shape –Optimize shape by gradient descent in PCA shape subspace (penalize object overlap) Image data term: sum of per-pixel foreground probability inside shape Shape prior term: sum of absolute value of PCA coefficients

24. Nathan Jacobs24 Segmentation Results Subspace onlySubspace and Prior Global shape model Local shape model

25. Nathan Jacobs25 Summary 1.Static cameras give strong priors. 2.Unsupervised training of a localized shape prior is possible. 3.Localized shape priors can be used to improve object initialization for tracking.

26. Background

27. Nathan Jacobs27 Efficient Segmentation Use gradient descent in low-dimensional shape subspace Gradient estimation –For each underlying shape parameter Sum along two edges of polygon –For PC components and object location Weighted combination of polygon edge scores

28. Nathan Jacobs28 Choice of Support Region

29. Nathan Jacobs29 Choosing Constants for Updating Prior Models Threshold.999990.99990.9990.990.9 The best learning rate depends on scene, application, time-of-day, weather, image location. Slow updateFast update Current Frame VSSN 2006

30. Nathan Jacobs30 Segmentation Energy Function Minimize Penalty on size Per-pixel foreground likelihood Shape penalty based on prior (sum of PCA coefficients)