1. Optimization & Learning for Registration of Moving Dynamic Textures Junzhou Huang 1, Xiaolei Huang 2, Dimitris Metaxas 1 Rutgers University 1, Lehigh University 2

2. Outline  Background  Goals & Problems  Related Work  Proposed Method  Experimental Results  Discussion & Conclusion

3. Background  Dynamic Textures (DT) static camera, exhibiting certain stationary properties  Moving Dynamic Textures (MDT) dynamic textures captured by a moving camera DT [Kwatra et al. SIGGRAPH’03] MDT [Fitzgibbbon ICCV’01]

4. Background  Video registration Required by many video analysis applications  Traditional assumption Static, rigid, brightness constancy Bergen et al. ECCV’92, Black et al. ICCV’93  Relaxing rigidity assumption Dynamic textures Fitzgibbon, ICCV’01; Doretto et al. IJCV’03; Yuan et al. ECCV’04; Chan et al. NIPS’05; Vidal et al. CVPR’05; Lin et al. PAMI’07; Rav-Acha et al. Dynamic Vision Workshop at ICCV’05; Vidal et al. ICCV’07

5. Our Goal  Registration of Moving Dynamic Textures Recover the camera motion and register image frames in the MDT image sequence Translation to the leftTranslation to the right

6. Complex Optimization Problem  Complex optimization W.r.t. camera motion, dynamic texture model Chicken-and-Egg Problem  Challenges About the mean images About Linear Dynamic System (LDS) model About the camera motion

7. Related Works  Fitzgibbon, ICCV’01 Pioneering attempt Stochastic rigidity Non-linear optimization  Vidal et al. CVPR’05 Time varying LDS model Static assumption in small time windows Simple and general framework but often under- estimate motion

8. Formulation  Registration of MDT I(t), the video frame, camera motion parameters y 0, the desired average image of the video y(t), appearance of DT x(t), dynamics of DT

9. Generative Model Generative image model for a MDT

10. First Observation  Good registration A good registration according to the accurate camera motion should simplify the dynamic texture model while preserving all useful information Used by Fitzgibbon, ICCV’01, Minimizing the entropy function of an auto regressive process Used by Vidal, CVPR’05, optimizing time varying LDS model by optimizing piecewise LDS model

11. Second Observation  Good registration A good registration according to the accurate camera motion should lead to a sharp average image whose statistics of derivative filters are similar to those of the input image frames.  Statistics of derivative filters in images Student-t distribution/heavy-tailed image priors Huang et al. CVPR’99, Roth et al. CVPR’05

12. Prior Models  The Average Image Prior  The Motion Prior  The Dynamics Prior

13. Average Image Priors  Student-t distribution Model parameters / contrastive divergence method (a) Before registration, (b) In the middle of registration (c) After registration

14. Motion / Dynamics Priors  Gaussian Perturbation (Motion) Uncertainty in the motion is modeled by a Gaussian perturbation about the mean estimation M 0 with the covariance matrix S ( a diagonal matrix) Motivated by the work [Pickup et al. NIPS’06]  GPDM / MAR model (Dynamic) Marginalizing over all possible mappings between appearance and dynamics Motivated by the work [Wang et al. NIPS’05], [Moon et al. CVPR’06]

15. Joint Optimization  Generative image model  Optimization Final marginal likelihood Scaled conjugate gradients algorithm (SCG)

16. Procedures  Obtaining image derivative prior model  Dividing the long sequence into many short image sequences  Initialization for video registration  Performing model optimization with the proposed prior models until model convergence.  With estimated y0, Y and X, the camera motion is then obtained iteratively by Maximum Likelihood estimation using SCG optimization

17. Obtaining Data  Three DT video sequences DT data [Kwatra et al. SIGGRAPH’03]  Synthesized MDT video sequence 60 frames each, no motion from 1 st to 20 th frame and from 41 st to 60 th Camera motion with speed [1, 0] from 21 st to 40 th

18. Grass MDT Video  The average image (a) One frame, (b) the average image after registration, (c) average image before registration

19. Grass MDT Video  The statistics of derivative filter responses

20. Evaluation / Comparison  False Estimation Fraction  Comparison with two classical methods Hybrid method, [Bergen et al. ECCV’92] [Black et al. ICCV’93] Vidal’method, [Vidal et al. CVPR’05]

21. Waterfall MDT Video  Motion estimation (a)Ground truth, (b) by hybrid method, (c) by Vidal’s, (d) by our method

22. Waterfall MDT Video  The average Image and its statistics The average image and its derivative filter response distribution after registration by: (a) our method, (b) Vidal’s method, (c) hybrid method

23. FEF Comparison  On three synthesized MDT video

24. Experiment on real MDT Video  Moving flower bed video  554 frames total  Ground truth motion 110 pixels  Estimation 104.52 pixels ( FEF 4.98%)

25. Conclusions  Proposed: Powerful priors for MDT registration  Solution for: Camera motion, Average image of video, Dynamic texture model  What have we learned? Correct registration simplifies DT model while preserving useful information Better registration leads to sharper average image

26. Thank you !

27. Future work  More complex camera motion  Different metrics for performance evaluation  Multiple dynamic texture segmentation

28. Experiment on real MDT Video  Moving flower bed video  Our method 554 frames total Ground truth motion 110 pixels Estimation 104.52 pixels ( FEF 4.98%)  Vidal’s method 250 frames [Vidal et al. CVPR’05] Ground truth motion 85 pixels Estimation 60 pixels (FEF 29.41%)