1. Learning Theory II: Modeling and Segmentation of Multivariate Mixed Data (BME 580.692, CS 600.462)
René Vidal
Time/Place: T-Th 4.30pm-6pm, Hodson 301
Office Hours: Mondays 5-6, 308B Clark Hall
Title: Segmentation of Dynamic Scenes and Textures
Abstract: Dynamic scenes are video sequences containing multiple objects moving in front of dynamic backgrounds, e.g. a bird floating on water. One can model such scenes as the output of a collection of dynamical models exhibiting discontinuous behavior both in space, due to the presence of multiple moving objects, and in time, due to the appearance and disappearance of objects. Segmentation of dynamic scenes is then equivalent to the identification of this mixture of dynamical models from the image data. Unfortunately, although the identification of a single dynamical model is a well understood problem, the identification of multiple hybrid dynamical models is not. Even in the case of static data, e.g. a set of points living in multiple subspaces, data segmentation is usually thought of as a "chicken-and-egg" problem. This is because in order to estimate a mixture of models one needs to first segment the data and in order to segment the data one needs to know the model parameters. Therefore, static data segmentation is usually solved by alternating data clustering and model fitting using, e.g., the Expectation Maximization (EM) algorithm.
Our recent work on Generalized Principal Component Analysis (GPCA) has shown that the "chicken-and-egg" dilemma can be tackled using algebraic geometric techniques. In the case of data living in a collection of (static) subspaces, one can segment the data by fitting a set of polynomials to all data points (without first clustering the data) and then differentiating these polynomials to obtain the model parameters for each group. In this talk, we will present ongoing work addressing the extension of GPCA to time-series data living in a collection of multiple moving subspaces. The approach combines classical GPCA with newly developed recursive hybrid system identification algorithms. We will also present applications of DGPCA in image/video segmentation, 3-D motion segmentation, dynamic texture segmentation, and heart motion analysis.

2. Course overview Linear and Nonlinear Dimensionality Reduction
Principal component analysis
Unsupervised Learning
Iterative methods for central and subspace clustering
Algebraic methods for central and subspace clustering
Applications in Computer Vision
3-D motion segmentation
Spatial and temporal video segmentation
Estimation and Segmentation of Hybrid Dynamical Models
Identification of hybrid systems

3. Linear dimensionality reduction
Principal Component Analysis (PCA)
Applications: data compression, regression, image analysis (eigenfaces), pattern recognition
As we all know, Principal Component Analysis problem refers to the problem of estimating a SINGLE subspace from sample data points. Although there are various ways of solving PCA, a simple solution consist of building a matrix with all the data points, computing its SVD, and then extracting a basis for the subspace from the columns of the U matrix, and the dimension of the subspace from the rank of the U matrix.
There is no question that PCA is one of the most popular techniques for dimensionality reduction in various engineering disciplines. In computer vision, in particular, a successful application has been found in face recognition under the name of eigenfaces.

4. Generalized PCA (GPCA)
Extensions of PCA
Probabilistic PCA (Tipping-Bishop ’99)
Identify subspace from noisy data
Gaussian noise: standard PCA
Noise in exponential family (Collins et al.’01)
Nonlinear PCA (Scholkopf-Smola-Muller ’98)
Identify a nonlinear manifold from sample points
Embed data in a higher dimensional space and apply standard PCA
What embedding should be used?
There have been various attempts to generalize PCA in different directions.
For example, Probabilistic PCA considers the case of noisy data and tries to estimate THE subspace in a maximum likelihood sense. For the case of noise in the exponential family, Collins has shown that this can be done using convex optimization techniques.
Other extensions considered the case of data lying on manifold, the so-called NonLinear PCA or Kernel PCA. This problem is usually solved by embedding the data into a higher dimensional space and then assuming that the embedded data DOES live in a linear subspace. Of course the correct embedding to use depends on the problem at hand, and learning the embedding is a current a topic of research in the machine learning community.
A third extension considers the case of identifying multiple subspaces at the same time. It is this case extension the one I will talk about in this talk under the name of Generalized PCA.
Mixtures of PCA (Tipping-Bishop ’99)
Identify a collection of subspaces from sample points
Generalized PCA (GPCA)

5. Nonlinear dimensionality reduction
Often data lie on a manifold
Unfolding the manifold: LLE, Isomap
Example on hand gestures

6. Applications of NLR Faces under different expression
Lips under different expressions

7. Data segmentation and clustering
Given a set of points, separate them into multiple groups
Discriminative methods: learn boundary
Generative methods: learn mixture model, using, e.g. Expectation Maximization

8. Generalized Principal Component Analysis
Polynomials can be expressed linearly in terms of a set of coefficients by using a polynomial embedding called Veronese map
Veronese map

9. Clustering data on non Euclidean spaces
Mixtures of linear spaces
Mixtures of algebraic varieties
Mixtures of Lie groups
“Chicken-and-egg” problems
Given segmentation, estimate models
Given models, segment the data
Initialization?
Need to combine
Algebra/geometry, dynamics and statistics

10. Applications of GPCA in vision and control
Geometry
Vanishing points
Image compression
Segmentation
Intensity (black-white)
Texture
Motion (2-D, 3-D)
Scene (host-guest)
Recognition
Faces (Eigenfaces)
Man - Woman
Human Gaits
Dynamic Textures
Water-steam
Biomedical imaging
Hybrid systems identification
One of the reasons we are interested in GPCA is because there are various problems in computer vision that have to do with the simultaneous estimation of multiple models from visual data.
Consider for example segmenting an image into different regions based on intensity, texture of motion information. Consider also the recognition of various static and dynamic processes such as human faces or human gaits from visual data.
Although in this talk I will only consider the first class of problems, it turns out that, at least from a mathematical perspective, all the above problems can be converted into following generalization of principal component analysis, which we conveniently refer to as GPCA

11. Segmentation problems in dynamic vision
Segmentation of video and dynamic textures
Segmentation of rigid-body motions

12. What are hybrid systems?
Previous work on hybrid systems
Modeling, analysis, stability
Control: reachability analysis, optimal control
Verification: safety
In applications, one also needs to worry about observability and identifiability
Modeling of a UAV, dynamic textures, human gaits

13. Identification of hybrid systems
Challenging “chicken-and-egg” problem
Given switching times, can estimate model parameters
Given the model parameters, estimate hybrid state
Given all above, estimate switching parameters
Iterate
Difficulties
Very sensitive to initialization
Needs a minimum dwell time
Does not use all data
Given input/output data, identify
Number of discrete states
Model parameters of linear systems
Hybrid state (continuous & discrete)
Switching parameters (partition of state space)

14. References

15. Vision Lab @ Johns Hopkins University
For more information,
Vision Johns Hopkins University
Thank You!