1. Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair) Dr Robert Schalkoff Dr Brian Dean

2. Tracking Overview Tracker Tasks Feature Descriptors Object Model Update / Learning Mechanism Tracking Framework Color Gradients Texture Shape Motion Template Contour Active Appearance Probability Densities Mean Shift Pixel-wise Classification Optical Flow Filtering techniques No Update Adaboost Expectation Maximization Re-weighting Strategy Object Detection Manual Segmentation Feature Points

3. Approach Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. Contour Extraction: Contour is extracted using a discrete implementation of level sets Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. Update Mechanism: The parameters of all the Gaussians are updated based on tracked data Results

4. Tracking Framework Bayesian Formulation: Image data of all frames Contour at time tPreviously seen contours Assuming conditional independence among pixels, Feature vector

5. Object Modeling f1f1 f2f2 ? Gaussian Mixture Model (GMM): Strength Image: >0 for Foreground <0 for Background y

6. Strength Image GMMLinear ClassifierSingle Gaussian

7. Strength Image (contd…) … Linear Classifier Single Gaussian Individual Fragments Final StrengthStrength Without Spatial Information

8. Topics Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. Contour Extraction: Contour is extracted using a discrete implementation of level sets Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. Update Mechanism: The parameters of all the Gaussians are updated based on tracked data Results

9. Contour Extraction Implicit representation of growing region Likelihood term (Strength image) Regularization term Energy Functional: (strength image) (frontier) > 0 Inside < 0 Outside

10. Contour Extraction (contd…) (Region to be shrunk) (Region already grown) (Region to be grown) (Region that need not be considered)

11. Contour Extraction (contd…) such that x x’ x such that Dilation Contraction

12. Contour Extraction (contd…) Expand Remove interior points Contract Remove exterior points

13. Contour Extraction (contd…) Likelihood Final Region

14. Topics Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. Contour Extraction: Contour is extracted using a discrete implementation of level sets Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. Update Mechanism: The parameters of all the Gaussians are updated based on tracked data Results

15. Region Segmentation Mode-seeking region growing algorithm: do { Pick a seed point that is not associated to any fragment Grow the fragment from the seed point based on the similarity of the pixel and its neighbor’s appearance Stop growing the fragment if no more similar pixels are present in the neighborhood of the fragment } until all pixels are assigned Seed point: Eigen values of 3x3 RGB covariance matrix where

16. Region Segmentation (contd…) Pick the minimum element in S. Create a region to hold the pixel and add the neighbors in a fixed window. Compute Mean μ j and Covariance Σ j of the region. Likelihood: Grow the region as before with two additional steps:  Update μ j, and Σ j, as a new pixel is added  Remove the corresponding element in S if a pixel is added Continue above steps if S is not empty. Initial region Mahalanobis distance Configurable parameter

17. Region Segmentation (contd…) Region Growing Graph-BasedMean-Shift

18. Region Segmentation (contd…) Region Growing Graph-BasedMean-Shift

19. Topics Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. Contour Extraction: Contour is extracted using a discrete implementation of level sets Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. Update Mechanism: The parameters of all the Gaussians are updated based on tracked data Results

20. Update Mechanism f1f1 f2f2 Update parameters of existing fragments Detect fragment occlusion Find new fragments Initial Frame Initial ModelFragment Association

21. Update Mechanism (contd…) Initial Model (function of past and current values) Weight computed by comparing Mahalanobis distance Updating parameters of existing fragments:

22. Update Mechanism (contd…) Occluded fragments: If a fragment is associated with less than 0.2% of the image pixels, then the fragment is declared as occluded. Finding new fragments: Helps in handling self-occlusion

23. Spatial Alignment The spatial parameters are updated using the motion vectors from Joint Lucas- Kanade approach Lucas-Kanade Joint Lucas-Kanade

24. Algorithm summary Initial frame: The user marks the object to be tracked. The target object and background scene are segmented based on their appearance similarity. The target object and background scene are modeled using a mixture of Gaussians where each Gaussian correspond to a fragment in the joint feature-spatial space Subsequent frames: Update the spatial parameters of GMM using the motion vectors of Joint Lucas-Kanade Each pixel is classified into either foreground or background by generating a strength map using the Gaussian mixture model (GMM) of the object and background. The strength map is integrated into a discrete level set formulation to obtain accurate contour of the object. Using the tracked data, the appearance parameters of the GMM are updated.

25. Topics Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. Contour Extraction: Extract contour using a discrete implementation of level sets Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. Update Mechanism: The parameters of all the Gaussians are updated based on tracked data Results

26. Experimental Results Elmo Sequence Monkey Sequence

27. Experimental Results (Contd…) Person SequenceFish Sequence

28. Experimental Results: Self-Occlusion Without Self-Occlusion Module With Self-Occlusion Module

29. Conclusion A tracking framework based on modeling the object as mixture of Gaussians is proposed An efficient discrete implementation of level sets is employed to extract contour. A mode-seeking region growing algorithm is used to segment the image. A simple re-weighting strategy is proposed to update the parameters of Gaussians. Future Directions: Incorporate shape priors. Utilize the extracted shapes to learn more robust priors. An offline or online evaluation mechanism during the initialization phase. Adding global information into the region segmentation process. Automating the object detection and initialization.

30. Questions ?

31. Thank you !