Periodic Motion Detection via Approximate Sequence Alignment Ivan Laptev*, Serge Belongie**, Patrick Perez* *IRISA/INRIA, Rennes, France **Univ. of California, San Diego, USA May 9-10, 2005
Motivation Dominant motion estimation Works very well for a panning camera and static backgrounds Problems for scenes with motion parallax and multiple motions
Motivation Dominant motion estimation Works very well for a panning camera and static backgrounds Problems for scenes with motion parallax and multiple motions
Motivation Assumptions about background motion e.g. Homography: Target == Outlier Assumptions about target motion Detect motion of specific type (Target == Inlier) - more difficult than for dominant motion - may have advantages in complex scenes Here: the type of target motion --- periodic
Periodic motion Periodic views can be approximately treated as stereopairs
Periodic motion Periodic views can be approximately treated as stereopairs Fundamental matrix is generally time-dependent Periodic motion estimation ~ sequence alignment
Sequence alignment Generally hard problem Unknown positions and motions of cameras Unknown temporal offset Possible time warping Prior work treats special cases Caspi and Irani “Spatio-temporal alignment of sequences”, PAMI 2002 Rao et.al. “View-invariant alignment and matching of video sequences”, ICCV 2003 Tuytelaars and Van Gool “Synchronizing video sequences”, CVPR 2004 Useful in Reconstruction of dynamic scenes Recognition of dynamic scenes
Assumptions Constant translation Assume the camera is translating with velocity relatively to the object For sequences corresponding points are related by All corresponding periodic points are on the same epipolar line Points with non-constant motion Rejects trivial case of pure translation Can be detected by maximizing local variation of space-time gradients (Laptev and Lindeberg, IJCV 2005)
Space-time interest points c1 c2 c3 c4 Second-moment matrix Local maxima of H over (x,y,t) Detected points Points with similar neighborhoods
Periodic motion detection 1.Corresponding points have similar descriptors 2. Same period for all features 3. Spatial arrangement of features across periods satisfy epipolar constraint: Use RANSAC to estimate F and p
Periodic motion detection Original space-time features RANSAC estimation of F,p
Periodic motion detection Original space-time features RANSAC estimation of F,p
Periodic motion detection Original space-time features RANSAC estimation of F,p
Periodic motion detection Original space-time features RANSAC estimation of F,p
Periodic motion segmentation Assume periodic objects are planar Periodic points can be related by a dynamic homography: linear in time
Periodic motion segmentation Assume periodic objects are planar Periodic points can be related by a dynamic homography: RANSAC estimation of H and p linear in time
Object-centered stabilization
Segmentation Disparity estimation Graph-cut segmentation
Segmentation
Conclusion Direct method for periodic motion detection and segmentation Constant translation assumption might be relaxed by tracking the modeling H and F as non-linear matrix functions Extension to non-periodic motion recognition via sequence alignment using corresponding space- time points
Segmentation