Tracking Using A Highly Deformable Object Model Nilanjan Ray Department of Computing Science University of Alberta

Overview of Presentation Tracking deformable objects –Motivations: desirable properties of a deformable object model –An example application (mouse heart tracking) Some technical background –Level set function and its application in image processing –Non-parametric probability density function (pdf) estimation –Similarity/dissimilarity measures for pdfs Proposed tracking technique Results, comparisons and demos Ongoing investigations –Incorporating color cues, and other features –Adding constraints on object shape –Application in morphing (?) –Incorporating object motion information (??) Summary Acknowledgements

Tracking Deformable Objects Desirable properties of deformable models: –Adapt with deformations (sometimes drastic deformations, depending on applications) –Ability to learn object and background: Ability to separate foreground and background Ability to recognize object from one image frame to the next, in an image sequence Show cine MRI video

Some Existing Deformable Models Deformable models: –Highly deformable Examples: snake or active contour, B-spline snakes, … Good deformation, but poor recognition (learning) ability –Not-so-deformable Examples –Active shape and appearance models –G-snake –… Good recognition (learning) capability, but of course poor deformation ability So, how about good deformation and good recognition capabilities?

Technical Background : Level Set Function A level set function represents a contour or a front geometrically Consider a single-valued function  (x, y) over the image domain; intersection of the x-y plane and  represents a contour: (X(x, y), Y(x, y)) is the point on the curve that is closest to the (x, y) point Matlab demo (lev_demo.m)

Applications of Level Set: Image Segmentation Matlab segmentation demo (yezzi.m) Vessel segmentation Brain reconstruction Virtual endoscopy Trachea fly through …tons out there Show videos

Level Set Applications: Image Denoising Two example videos Show video

Level Set Applications: Robotics Finding shortest path Show video

Level Set Applications: Computer Graphics Morphing Simulation Animation …. Go to for amazing videos

More Applications of Level Set Methods Go to plications/Menu_Expanded_Applications.h tml

Technical Background: Non-Parametric Density Estimation I(x, y) is the image intensity at (x, y)  i is the standard deviation of the Gaussian kernel C is a normalization factor that forces H(i) to integrate to unity Normalized image intensity histogram:

Technical Background: Similarity and Dissimilarity Measures for PDFs Kullback-Leibler (KL) divergence (a dissimilarity measure): Bhattacharya coefficient (a similarity measure): P(z) and Q(z) are two PDFs being compared

Proposed Method: Tracking Deformable Object Deformable Object model (due to Leventon [1]): –From the first frame learn the joint pdf of level set function and image intensity (image feature) Tracking: –From second frame onward search for similar joint pdf [1] M. Leventon, Statistical Models for Medical Image Analysis, Ph.D. Thesis, MIT, 2000.

Deformable Object Model Joint probability density estimation with Gaussian kernels: J(x, y) is the image intensity at (x, y) point on the first image frame  (x, y) is the value of level set function at (x, y) on the first image frame C is a normalization factor We learn Q on the first video frame given the object contour (represented by the level set function) Level set function value: l Image intensity: i

Proposed Object Tracking On the second (or subsequent) frame compute the density: Match the densities P and Q by KL-divergence: Minimize KL-divergence by varying the level set function  (x, y) I(x, y) is the image intensity at (x, y) on the second/subsequent frame  (x, y) is the level set function at on the second/subsequent frame Note that here only P is a function of  (x, y)

Minimizing KL-divergence In order to minimize KL-divergence we use Calculus of variations After applying Calculus of variations the rule of update (gradient descent rule) for the level set function becomes: t : iteration number  t : timestep size

Minimizing KL-divergence: Implementation There is a compact way of expressing the update rule: Where g 1 is a convolution kernel: is a function defined simply as: convolution

Minimizing KL-divergence: A Stable Implementation The previous implementation is called explicit scheme and is unstable for large time steps; if small time step is used then the convergence will be extremely slow One remedy is a semi-implicit scheme of numerical implementation: Where g is a convolution kernel: In this numerical scheme  t can be large and still the solution will be convergent; So very quick convergence is achieved in this scheme is a function defined simply as:

Results: Tracking Cardiac Motion A few cine MRI frames and delineated boundaries on them Show videos

Numerical Results and Comparison Pratt’s FOMSegmentation Score Slow Seq.Rapid Seq.Slow Seq.Rapid Seq. GVF Snake Method Proposed Method Comparison of mean performance measures Sequence with slow heart motion Sequence with rapid heart motion

Extensions: Tracking Objects in Color Video If we want to learn joint distribution of level set function and color channels (say, r, g, b), then non-parametric density estimation suffers from: –Slowness –Curse of dimensionality Another important theme is combine edge information and region information of objects One remedy sometimes is to take a linear combination of r, g, and b channels –Fisher’s linear discriminant can be used to learn the coefficients of linear combination A demo

Extensions: Adding Object Shape Constraint Can we constrain the object shape in this computational framework? Minimize: where

Application in Computer Graphics: Morphing (J 1,  1 )(I 2,  2 ) Initial object Shape and intensity/texture Final object Shape and intensity/texture Morphing: generate realistic intermediate tuples (I t,  t ) (I 1,  1 ),(I 2,  2 ), …..

Morphing: Formulation Generate intermediate shapes, i.e., level set function  t (say, via interpolation): Next, generate intermediate intensity I t by maximizing: Once again we get a similar PDE for I t

Morphing: Preliminary Results Show videos

Summary Highly deformable object tracking: Variational minimization of KL-divergence leading to fast and stable partial differential equations Several exciting extensions Application in morphing

Acknowledgements Baidyanath Saha CIMS lab and Prof. Hong Zhang Prof. Dipti P. Mukherjee, Indian Statistical Institute Department of Computing Science, UofA