1. Periodic Motion Detection via Approximate Sequence Alignment Ivan Laptev*, Serge Belongie**, Patrick Perez* *IRISA/INRIA, Rennes, France **Univ. of California, San Diego, USA May 9-10, 2005

2. Motivation Dominant motion estimation Works very well for a panning camera and static backgrounds Problems for scenes with motion parallax and multiple motions

3. Motivation Dominant motion estimation Works very well for a panning camera and static backgrounds Problems for scenes with motion parallax and multiple motions

4. Motivation Assumptions about background motion e.g. Homography:  Target == Outlier Assumptions about target motion Detect motion of specific type (Target == Inlier) - more difficult than for dominant motion - may have advantages in complex scenes Here: the type of target motion --- periodic

5. Periodic motion Periodic views can be approximately treated as stereopairs

6. Periodic motion Periodic views can be approximately treated as stereopairs Fundamental matrix is generally time-dependent  Periodic motion estimation ~ sequence alignment

7. Sequence alignment Generally hard problem Unknown positions and motions of cameras Unknown temporal offset Possible time warping Prior work treats special cases Caspi and Irani “Spatio-temporal alignment of sequences”, PAMI 2002 Rao et.al. “View-invariant alignment and matching of video sequences”, ICCV 2003 Tuytelaars and Van Gool “Synchronizing video sequences”, CVPR 2004 Useful in Reconstruction of dynamic scenes Recognition of dynamic scenes

8. Assumptions Constant translation Assume the camera is translating with velocity relatively to the object  For sequences corresponding points are related by  All corresponding periodic points are on the same epipolar line Points with non-constant motion Rejects trivial case of pure translation Can be detected by maximizing local variation of space-time gradients (Laptev and Lindeberg, IJCV 2005)

9. Space-time interest points c1 c2 c3 c4 Second-moment matrix Local maxima of H over (x,y,t) Detected points Points with similar neighborhoods

10. Periodic motion detection 1.Corresponding points have similar descriptors 2. Same period for all features 3. Spatial arrangement of features across periods satisfy epipolar constraint:  Use RANSAC to estimate F and p

11. Periodic motion detection Original space-time features RANSAC estimation of F,p

12. Periodic motion detection Original space-time features RANSAC estimation of F,p

13. Periodic motion detection Original space-time features RANSAC estimation of F,p

14. Periodic motion detection Original space-time features RANSAC estimation of F,p

15. Periodic motion segmentation Assume periodic objects are planar  Periodic points can be related by a dynamic homography: linear in time

16. Periodic motion segmentation Assume periodic objects are planar  Periodic points can be related by a dynamic homography:  RANSAC estimation of H and p linear in time

17. Object-centered stabilization

18. Segmentation Disparity estimation Graph-cut segmentation

19. Segmentation

21. Conclusion Direct method for periodic motion detection and segmentation Constant translation assumption might be relaxed by tracking the modeling H and F as non-linear matrix functions Extension to non-periodic motion recognition via sequence alignment using corresponding space- time points

22. Segmentation