1. IRISA / INRIA Rennes ilaptev@irisa.fr Computational Vision and Active Perception Laboratory (CVAP) KTH (Royal Institute of Technology) http://www.nada.kth.se/~laptev Local Motion Features for Video interpretation Ivan Laptev

2. Ivan Laptev (2007) IRISA / INRIA Rennes France  No global assumptions about the scene Complex BG motion Common problems: Changes in appearance Common methods: Camera stabilization Segmentation Tracking ? ? Motivation Goal: Interpretation of dynamic scenes … camera motion… non-rigid object motion … complex background motion

3. Ivan Laptev (2007) IRISA / INRIA Rennes France Space-time No global assumptions  Consider local spatio-temporal neighborhoods

4. Ivan Laptev (2007) IRISA / INRIA Rennes France Space-time No global assumptions  Consider local spatio-temporal neighborhoods

5. Ivan Laptev (2007) IRISA / INRIA Rennes France Applications: preview Sequence alignment Periodic motion detection Action recognition

6. Ivan Laptev (2007) IRISA / INRIA Rennes France  How to find informative neighborhoods?  How to deal with transformations in the data?  How to use obtained features in applications? Questions  How to describe the neighborhoods?  How to use obtained features in applications? (ICCV’03)  How to find informative neighborhoods? (ICCV’03)  How to deal with transformations in the data? (ICPR’04)  How to describe the neighborhoods? (SCMVP’04) (ICPR’04) (ICCV’05)

7. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICPR’04)  How to describe the neighborhoods? (SCMVP’04) (ICPR’04) (ICCV’05)

8. Ivan Laptev (2007) IRISA / INRIA Rennes France Space-Time interest points What neighborhoods to consider? Distinctive neighborhoods High image variation in space and time   Look at the distribution of the gradient Gaussian derivative of Second-moment matrix Original image sequence Space-time Gaussian with covariance Space-time gradient Definitions:

9. Ivan Laptev (2007) IRISA / INRIA Rennes France defines second order approximation for the local distribution of within neighborhood Space-Time interest points Properties of : Large eigenvalues of  can be detected by the local maxima of H over (x,y,t): (similar to Harris operator [Harris and Stephens, 1988])  1D space-time variation of, e.g. moving bar  2D space-time variation of, e.g. moving ball  3D space-time variation of, e.g. jumping ball

10. Ivan Laptev (2007) IRISA / INRIA Rennes France Space-Time interest points Motion event detection

11. Ivan Laptev (2007) IRISA / INRIA Rennes France Space-Time interest points Motion event detection: complex background

12. Ivan Laptev (2007) IRISA / INRIA Rennes France accelerations appearance/ disappearance Space-Time interest points split/merge

13. Ivan Laptev (2007) IRISA / INRIA Rennes France Relations to psychology ”... The world presents us with a continuous stream of activity which the mind parses into events. Like objects, they are bounded; they have beginnings, (middles,) and ends. Like objects, they are structured, composed of parts. However, in contrast to objects, events are structured in time...'' Tversky et.al.(2002), in ”The Imitative Mind” Events are well localized in time and are consistently identified by different people. The ability of memorizing activities has shown to be dependent on how fine we subdivide the motion into units.

14. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICPR’04)  How to describe the neighborhoods? (SCMVP’04)

15. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04) Scale and frequency transformations

16. Ivan Laptev (2007) IRISA / INRIA Rennes France Spatio-temporal scale selection Image sequence f can be influenced by changes in spatial and temporal resolution p P’ S point transformation covariance transformation

17. Ivan Laptev (2007) IRISA / INRIA Rennes France Spatio-temporal scale selection Want to estimate S from the data Estimate spatial and temporal extents of image structures  Scale selection Extension to space-time: Find normalization parameters a,b,c,d for Scale-selection in space [Lindeberg IJCV’98]

18. Ivan Laptev (2007) IRISA / INRIA Rennes France Spatio-temporal scale selection Analyze spatio-temporal blob Extrema constraints give parameter values a=1, b=1/4, c=1/2, d=3/4

19. Ivan Laptev (2007) IRISA / INRIA Rennes France  The normalized spatio-temporal Laplacian operator Assumes extrema values at positions and scales corresponding to the centers and the spatio- temporal extent of a Gaussian blob Spatio-temporal scale selection

20. Ivan Laptev (2007) IRISA / INRIA Rennes France Space-Time interest points H depends on  and, hence, on  and scale transformation S  adapt interest points by iteratively computing:  Scale estimation  Interest point detection (*)(*) ( ** ) 1.Fix 2.For each detected interest point ( ** ) 3.Estimate ( * ) 4.Update covariance 5.Re-detect using 6.Iterate 3-6 until convergence of and

21. Ivan Laptev (2007) IRISA / INRIA Rennes France Stability to size changes, e.g. camera zoom Spatio-temporal scale selection

22. Ivan Laptev (2007) IRISA / INRIA Rennes France Selection of temporal scales captures the frequency of events Spatio-temporal scale selection

23. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04) Scale and frequency transformations

24. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICPR’04)  How to describe the neighborhoods? (SCMVP’04) Transformations due to camera motion

25. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICPR’04)  How to describe the neighborhoods? (SCMVP’04) Transformations due to camera motion Stationary camera Moving camera time

26. Ivan Laptev (2007) IRISA / INRIA Rennes France Galilean transformation p P’ G point transformation covariance transformation G  ’’

27. Ivan Laptev (2007) IRISA / INRIA Rennes France Adapted interest points Velocity-adapted interest points Interest points Stabilized cameraStationary camera

28. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04)

29. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04)

30. Ivan Laptev (2007) IRISA / INRIA Rennes France Features from human actions

31. Ivan Laptev (2007) IRISA / INRIA Rennes France Space-time neighborhoods boxing walking hand waving

32. Ivan Laptev (2007) IRISA / INRIA Rennes France Local space-time descriptors A common choice for local descriptors is a local jet (Koenderink and van Doorn, 1987) computed from spatio- temporal Gaussian derivatives (here at interest points p i ) where Covariance-normalization to obtain transformation-invriant Descriptors:

33. Ivan Laptev (2007) IRISA / INRIA Rennes France Use of descriptors: Clustering c1 c2 c3 c4 Clustering Classification  Group similar points in the space of image descriptors using K-means clustering  Select significant clusters

34. Ivan Laptev (2007) IRISA / INRIA Rennes France Use of descriptors: Clustering c1 c2 c3 c4 Clustering Classification  Group similar points in the space of image descriptors using K-means clustering  Select significant clusters

35. Ivan Laptev (2007) IRISA / INRIA Rennes France Use of descriptors: Matching  Find similar events in pairs of video sequences

36. Ivan Laptev (2007) IRISA / INRIA Rennes France Other descriptors better?  Multi-scale spatio- temporal derivatives Consider the following choices: Spatio-temporal neighborhood  Projections to orthogonal bases obtained with PCA  Histogram-based descriptors

37. Ivan Laptev (2007) IRISA / INRIA Rennes France Multi-scale derivative filters Derivatives up to order 2 or 4; 3 spatial scales; 3 temporal scales: 9 x 3 x 3 = 81 or 34 x 3 x 3 = 306 dimensional descriptors

38. Ivan Laptev (2007) IRISA / INRIA Rennes France PCA descriptors  Compute normal flow or optic flow in locally adapted spatio- temporal neighborhoods of features  Subsample the flow fields to resolution 9x9x9 pixels  Learn PCA basis vectors (separately for each flow) from features in training sequences  Project flow fields of the new features onto the 100 most significant eigen-flow-vectors:

39. Ivan Laptev (2007) IRISA / INRIA Rennes France Position-dependent histograms...  Divide the neighborhood  i of each point p i into M^3 subneighborhoods, here M=1,2,3  Compute space-time gradients (L x, L y, L t ) T or optic flow (v x, v y ) T at combinations of 3 temporal and 3 spatial scales where are locally adapted detection scales  Compute separable histograms over all subneighborhoods, derivatives/velocities and scales

40. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04)

41. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04) Action recognition

42. Evaluation: Action Recognition walking running jogging handwaving handclapping boxing Database:  Represent sequences as ”Bags of Local Features”  Compute similarity of two sequences as  Use e.g. Nearest Neighbor Classifier (NNC) to classify test actions given a set of training actions

43. Results: Recognition rates Scale-adapted features Scale and velocity adapted features

44. Results: Comparison Global-STG-HIST: Zelnik-Manor and Irani CVPR’01 Spatial-4Jets: Spatial interest points (Harris and Stephens, 1988)

45. Confusion matrices Position-dependent histograms for space-time interest points Local jets at spatial interest points

46. Ivan Laptev (2007) IRISA / INRIA Rennes France

47. STG-PCA, EDSTG-PD2HIST, ED Confusion matrices

48. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04) Action recognition

49. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICCV’03)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04) Action recognition Sequence alignment

50. Ivan Laptev (2007) IRISA / INRIA Rennes France  Represent the gait pattern using classified spatio-temporal points corresponding the one gait cycle  Define the state of the model X for the moment t 0 by the position, the size, the phase and the velocity of a person:  Associate each phase  with a silhouette of a person extracted from the original sequence Sequence alignment

51. Ivan Laptev (2007) IRISA / INRIA Rennes France Sequence alignment  Given a data sequence with the current moment t 0, detect and classify interest points in the time window of length t w : (t 0, t 0 -t w )  Transform model features according to X and for each model feature f m,i =(x m,i, y m,i, t m,i,  m,i,  m,i, c m,i ) compute its distance d i to the most close data feature f d,j, c d,j =c m,i :  Define the ”fit function” D of model configuration X as a sum of distances of all features weighted w.r.t. their ”age” (t 0 -t m ) such that recent features get more influence on the matching

52. Ivan Laptev (2007) IRISA / INRIA Rennes France Sequence alignment data features model features At each moment t 0 minimize D with respect to X using standard Gauss-Newton minimization method

53. Ivan Laptev (2007) IRISA / INRIA Rennes France Experiments

54. Ivan Laptev (2007) IRISA / INRIA Rennes France Experiments

55. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICPR’04)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04) Action recognition Sequence alignment

56. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICCV’05)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04) Action recognition Sequence alignment Periodic motion detection

57. Periodic views can be approximately treated as stereopairs Fundamental matrix is generally time-dependent  Periodic motion estimation ~ sequence alignment Periodic motion detection

58. Ivan Laptev (2007) IRISA / INRIA Rennes France Periodic motion detection 1.Corresponding points have similar descriptors 2. Same period for all features 3. For constant gross motion of the object, spatial arrangement of features across periods satisfy epipolar constraint:  Use RANSAC to estimate F and p

59. Ivan Laptev (2007) IRISA / INRIA Rennes France Periodic motion detection Original space-time features RANSAC estimation of F,p

60. Ivan Laptev (2007) IRISA / INRIA Rennes France Periodic motion detection Original space-time features RANSAC estimation of F,p

61. Ivan Laptev (2007) IRISA / INRIA Rennes France Periodic motion detection Original space-time features RANSAC estimation of F,p

62. Assume periodic objects are planar  Periodic points can be related by a dynamic homography: linear in time Periodic motion segmentation

63. Assume periodic objects are planar  Periodic points can be related by a dynamic homography:  RANSAC estimation of H and p linear in time Periodic motion segmentation

64. Object-centered stabilization Periodic motion segmentation

65. Disparity estimation Graph-cut segmentation Periodic motion segmentation

67. Ivan Laptev (2007) IRISA / INRIA Rennes France Questions  How to find informative neighborhoods? (ICCV’03)  How to use obtained features for applications? (ICCV’05)  How to deal with transformations in the data? (ICCV’03)  How to describe the neighborhoods? (SCMVP’04) Action recognition Sequence alignment Periodic motion detection

68. Ivan Laptev (2007) IRISA / INRIA Rennes France Related work  Zelnik and Irani CVPR’01  Efros et.al. ICCV’03  Lowe ICCV’99  Mikolayczyk and Schmid CVPR’03, ECCV’02  Fablet, Bouthemy and Peréz PAMI’02  Harris and Stephens Alvey’88  Koenderink and Doorn PAMI 1992  Lindeberg IJCV 1998

69. Ivan Laptev (2007) IRISA / INRIA Rennes France Summary  Detection of local space-time interest points  Applications: action recognition, sequence alignment, periodic motion detection,... ?  Adaptation to scale and velocity transformations  Evaluation of local space-time descriptors