Optical Flow Estimation and Segmentation of Moving Dynamic Textures René Vidal and Avinash Ravichandran Center for Imaging Science Johns Hopkins University

Dynamic textures Extract a set of features from the video sequence Spatial filters ICA/PCA Wavelets Intensities of all pixels Model spatiotemporal evolution of features as the output of a linear dynamical system (LDS): Soatto et al. ‘01 images Copyright © JHU Vision Lab

Learning the dynamic texture parameters Model is a LDS driven by IID white Gaussian noise Bilinear problem, can do EM Optimal solution: subspace identification (Bart de Moor) A suboptimal solution is obtained using the SVD by noticing that in the absence of noise images Copyright © JHU Vision Lab

Previous work on dynamic textures Static camera Modeling and synthesis: Shöld et al. ’00, Soatto et al ’01, Doretto ’03, Yuan et al. ‘04 Moving camera Fitzgibbon ’01: Stochastic rigidity Recognition: Saisan et al. ’01 Segmentation: Doretto ’03 Copyright © JHU Vision Lab

Paper contributions Can we recover the rigid motion of a camera looking at a moving dynamic texture? Can we segment a scene containing multiple dynamic textures? Can we segment a scene containing multiple moving dynamic textures? DTCC: Dynamic Texture Constancy Constraint GPCA: Generalized Principal Component Analysis GPCA + DTCC Copyright © JHU Vision Lab

Modeling moving dynamic textures A time invariant model cannot capture camera motion A rigid scene is a 1st order LDS A = 1, zt = 1, It = C = constant Camera motion can be modeled with a time varying LDS A models nonrigid motion C models rigid motion Identification of time varying LDS Difficult open problem Approximate solution: assume time invariant LDS in a temporal window, and estimate (A,C(t)) by shifting time window dynamics appearance images Copyright © JHU Vision Lab

Optical flow of a moving dynamic texture Static textures: optical flow from brightness constancy constraint (BCC) Dynamic textures: optical flow from dynamic texture constancy constraint (DTCC) Copyright © JHU Vision Lab

Segmenting nonmoving dynamic textures One dynamic texture lives in the observability subspace Multiple textures live in multiple subspaces Can cluster the data using GPCA Fit p(z) to all data points Cluster data from derivatives of p(z) water steam b i = D p n ( x ) j z Copyright © JHU Vision Lab

Segmenting nonmoving dynamic textures Goal: recognize multiple types of arrhythmias using heart MRI images Model: visual dynamics are modeled as multiple dynamic textures Heart motion: nonrigid Chest motion: respiration Copyright © JHU Vision Lab

Optical flow of multiple moving dynamic textures Apply PCA to frames in a moving time window Apply GPCA to projected data to segment sequence Apply DTCC to each group to obtain optical flow C1(t2) C2(t2) … … PCA PCA GPCA GPCA C1(t1) C2(t1) DTCC Improve me [u1,v1] [u2,v2] Copyright © JHU Vision Lab

Optical flow of multiple moving dynamic textures Copyright © JHU Vision Lab

Conclusions and open problems Paper Contributions Can model moving dynamic textures using time varying linear dynamical models Can estimate optical flow of moving dynamic textures using DTCC Can segment dynamic textures using GPCA Open problems Identification of time varying linear dynamical models, with C(t) evolving due to perspective projection of a rigid-body motion Copyright © JHU Vision Lab

Thanks

Improving quality of optical flow Copyright © JHU Vision Lab