1. Computer Vision, Robotics, Machine Learning and Control Lab
René Vidal
Center for Imaging Science, Whitaker Institute of Biomedical Engineering, Johns Hopkins University
Add d15.avi

2. Static scene reconstruction
Image Segmentation
Penguin/Ice/Water
Face Recognition
Man/Women
Video Segmentation
Tiger/Water/Bushes
3-D Reconstruction
One of the reasons we are interested in GPCA is because there are various problems in computer vision that have to do with the simultaneous estimation of multiple models from visual data.
Consider for example segmenting an image into different regions based on intensity, texture of motion information. Consider also the recognition of various static and dynamic processes such as human faces or human gaits from visual data.
Although in this talk I will only consider the first class of problems, it turns out that, at least from a mathematical perspective, all the above problems can be converted into following generalization of principal component analysis, which we conveniently refer to as GPCA

3. Dynamic scene reconstruction
Temporal segmentation
Guest/Host/Both
Dynamic texture recognit
Water/Steam/Foliage/Fire
Spatial segmentation
Car/Background
Gait/Posture recognition
Running/Walking/Limping
One of the reasons we are interested in GPCA is because there are various problems in computer vision that have to do with the simultaneous estimation of multiple models from visual data.
Consider for example segmenting an image into different regions based on intensity, texture of motion information. Consider also the recognition of various static and dynamic processes such as human faces or human gaits from visual data.
Although in this talk I will only consider the first class of problems, it turns out that, at least from a mathematical perspective, all the above problems can be converted into following generalization of principal component analysis, which we conveniently refer to as GPCA

4. Why are these problems challenging?
“Chicken-and-egg” problems
Given segmentation, estimate models
Given models, segment the data
Easy for humans, difficult for a computer
Given a set of data points lying on a collection of linear subspaces, without knowing which point corresponds to which subspace and without even knowing the number of subspaces, estimate the number of subspaces, a basis for each subspace and the segmentation of the data.
As stated, the GPCA problem is one of those challenging CHICKEN-AND-EGG problems in the following sense. If we knew the segmentation of the data points, and hence the number of subspaces, then we could solve the problem by applying standard PCA to each group. On the other hand, if we had a basis for each subspace, we could easily assign each point to each cluster. The main challenge is that we neither know the segmentation of the data points nor a basis for each subspace, and the questions is how to estimate everything from the data only.
Previous geometric approaches to GPCA have been proposed in the context of motion segmentation, and under the assumption that the subspaces do NOT intersect. Such approaches are based on first clustering the data, using for example K-means or spectral clustering techniques, and then applying standard PCA to each group.
Statistical methods, on the other hand, model the data as a mixture model whose parameters are the mixing proportions and the basis for each subspace. The estimation of the mixture is a (usually non-convex) optimization problem that can be solved with various techniques. One of the most popular ones is the expectation maximization algorithm (EM) which alternates between the clustering of the data and the estimation of the subspaces.
Unfortunately, the convergence of such iterative algorithms depends on good initialization. At present, initialization is mostly done either at random, using another iterative algorithm such as K-means, or else in an ad-hoc fashion depending on the particular application.
It is here where our main motivation comes into the picture. We would like to know if we can find a way of initializing iterative algorithms in an algebraic fashion. For example, if you look at the two lines in this picture, there has to be a way of estimating those two lines directly from data in a one shot solution, at least in the ABSENCE of noise.

5. What is our approach? x = A v y C w b = D p ( x ) j
Representation: hybrid dynamical models
Computational solution: Generalized Principal Component Analysis (GPCA) algebraic geometry, differential geometry, projective geometry and linear algebra
dynamics x t + 1 = A v y C w
images
appearance
running
walking
Theorem: GPCA [Vidal et al. 2004] b i = D p n ( x ) j z

6. Texture-based image segmentation
Our approach
Human

7. Texture-based image segmentation

8. Motion-based video segmentation

9. Motion-based video segmentation
Sequence A Sequence B Sequence C
Percentage of correct classification

10. Temporal video segmentation
Segmenting N=30 frames of a sequence containing n=3 scenes
Host
Guest
Both
Image intensities are output of linear system
Apply GPCA to fit n=3 observability subspaces
dynamics
appearance
images x t + 1 = A v y C w

11. Temporal video segmentation
Segmenting N=60 frames of a sequence containing n=3 scenes
Burning wheel
Burnt car with people
Burning car
Image intensities are output of linear system
Apply GPCA to fit n=3 observability subspaces
dynamics
appearance
images x t + 1 = A v y C w

12. Segmentation of dynamic textures
Output of a linear dynamical model (Soatto et al. 2001)
Dynamic texture segmentation using level-sets (Doretto et al. 2003)
dynamics x t + 1 = A v y C w
images
appearance

13. Segmentation of dynamic textures
One dynamic texture lives in the observability subspace
Multiple textures live in multiple subspaces
Apply PCA to image intensities
Apply GPCA to projected data
images

14. Segmentation of moving dynamic textures
Time varying model
Segmentation of multiple moving subspaces
Apply PCA to intensities in a moving time window
Apply GPCA to projected data
dynamics
images
appearance

15. Segmentation of moving dynamic textures
Static textures: optical flow is computed from brightness constancy constraint (BCC)
Dynamic textures: optical flow computed from dynamic texture constancy constraint (DTCC)

16. MRI-based heart motion analysis
Segmentation: extract static and dynamic landmarks from high-resolution images
Static (chest wall, aorta, spinal cord, sternum)
Dynamic (heart)
Registration: match real-time low-resolution images to high-quality images
Rigid (chest wall, aorta, spinal cord, sternum)
Non-rigid (heart)
Temporal alignment (using cardiac cycle)
Dynamical
To synthesize MRI data with ECG data
ECG gating

17. Dynamical Chest and Heart Segmentation
Original
Image
Morphological
Operations
Dynamic
Segmentation
Rene
Segmentation
By Intensity

18. Heart Segmentation by Dynamic Textures
nash

19. Chest Segmentation by Dynamic Textures
nash

20. DTI-based spinal cord injury detection
Diffusion tensor imaging: at each voxel have
Intensity of the voxel
Orientation of the voxel: tensor/ellipsoid
Can reconstruct fibers in the tissue
Spinal cord injury detection: regions where fibers no longer exist

21. DTI-based spinal cord injury detection
Axial segmentation of fibers in white matter
Injury detection

22. Ongoing and future research
Computer Vision & Graphics
Multiview motion segmentation
Cue integration (motion and texture) for segmentation
Shape recognition (faces)
Recognition of human motion and human activities
Animation of articulated bodies
Synthesis of dynamic textures
Biomedical Applications
Heart motion analysis
Spinal cord injury detection
Locomotion, posture and gait analysis
Medical robotics: ERC-CISST Center for Computer Integrated Surgical Systems and Technology
Vision, Learning and Statistical Geometry
Robust GPCA
Model selection
Relations to Kernel PCA
Estimating mixtures of dynamical models
Estimating manifolds from sample data points
Biometrics: Perception Human Crowd Activities