1. Optical Flow Estimation and Segmentation of Moving Dynamic Textures
René Vidal and Avinash Ravichandran
Center for Imaging Science
Johns Hopkins University

2. Dynamic textures Extract a set of features from the video sequence
Spatial filters
ICA/PCA
Wavelets
Intensities of all pixels
Model spatiotemporal evolution of features as the output of a linear dynamical system (LDS): Soatto et al. ‘01
images
Copyright © JHU Vision Lab

3. Learning the dynamic texture parameters
Model is a LDS driven by IID white Gaussian noise
Bilinear problem, can do EM
Optimal solution: subspace identification (Bart de Moor)
A suboptimal solution is obtained using the SVD by noticing that in the absence of noise
images
Copyright © JHU Vision Lab

4. Previous work on dynamic textures
Static camera
Modeling and synthesis: Shöld et al. ’00, Soatto et al ’01, Doretto ’03, Yuan et al. ‘04
Moving camera
Fitzgibbon ’01: Stochastic rigidity
Recognition: Saisan et al. ’01
Segmentation: Doretto ’03
Copyright © JHU Vision Lab

5. Paper contributions
Can we recover the rigid motion of a camera looking at a moving dynamic texture?
Can we segment a scene containing multiple dynamic textures?
Can we segment a scene containing multiple moving dynamic textures?
DTCC: Dynamic Texture Constancy Constraint
GPCA: Generalized Principal Component Analysis
GPCA + DTCC
Copyright © JHU Vision Lab

6. Modeling moving dynamic textures
A time invariant model cannot capture camera motion
A rigid scene is a 1st order LDS
A = 1, zt = 1, It = C = constant
Camera motion can be modeled with a time varying LDS
A models nonrigid motion
C models rigid motion
Identification of time varying LDS
Difficult open problem
Approximate solution: assume time invariant LDS in a temporal window, and estimate (A,C(t)) by shifting time window
dynamics
appearance
images
Copyright © JHU Vision Lab

7. Optical flow of a moving dynamic texture
Static textures: optical flow from brightness constancy constraint (BCC)
Dynamic textures: optical flow from dynamic texture constancy constraint (DTCC)
Copyright © JHU Vision Lab

8. Segmenting nonmoving dynamic textures
One dynamic texture lives in the observability subspace
Multiple textures live in multiple subspaces
Can cluster the data using GPCA
Fit p(z) to all data points
Cluster data from derivatives of p(z)
water
steam b i = D p n ( x ) j z
Copyright © JHU Vision Lab

9. Segmenting nonmoving dynamic textures
Goal: recognize multiple types of arrhythmias using heart MRI images
Model: visual dynamics are modeled as multiple dynamic textures
Heart motion: nonrigid
Chest motion: respiration
Copyright © JHU Vision Lab

10. Optical flow of multiple moving dynamic textures
Apply PCA to frames in a moving time window
Apply GPCA to projected data to segment sequence
Apply DTCC to each group to obtain optical flow
C1(t2)
C2(t2) … …
PCA
PCA
GPCA
GPCA
C1(t1)
C2(t1)
DTCC
Improve me
[u1,v1]
[u2,v2]
Copyright © JHU Vision Lab

11. Optical flow of multiple moving dynamic textures
Copyright © JHU Vision Lab

12. Conclusions and open problems
Paper Contributions
Can model moving dynamic textures using time varying linear dynamical models
Can estimate optical flow of moving dynamic textures using DTCC
Can segment dynamic textures using GPCA
Open problems
Identification of time varying linear dynamical models, with C(t) evolving due to perspective projection of a rigid-body motion
Copyright © JHU Vision Lab

13. Thanks

14. Improving quality of optical flow
Copyright © JHU Vision Lab