1. Analysis of Contour Motions Ce Liu William T. Freeman Edward H. Adelson Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Neural Information Processing Systems 2006

2. Visual Motion Analysis in Computer Vision Motion analysis is essential in –Video processing –Geometry reconstruction –Object tracking, segmentation and recognition –Graphics applications Is motion analysis solved? Do we have good representation for motion analysis? Is it computationally feasible to infer the representation from the raw video data? What is a good representation for motion?

3. Seemingly Simple Examples Kanizsa square From real video

4. Output from the State-of-the-Art Optical Flow Algorithm T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004 Optical flow field Kanizsa square

5. Output from the State-of-the-Art Optical Flow Algorithm T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004 Optical flow field Dancer

6. Optical flow representation: aperture problem Corners Lines Flat regions Spurious junctionsBoundary ownership Illusory boundaries

7. Optical Flow Representation Corners Lines Flat regions Spurious junctionsBoundary ownership Illusory boundaries We need motion representation beyond pixel level!

8. Layer Representation Video is a composite of layers Layer segmentation assumes sufficient textures for each layer to represent motion A true success? J. Wang & E. H. Adelson 1994 Y. Weiss & E. H. Adelson 1994 Achieved with the help of spatial segmentation

9. Layer Representation Video is a composite of layers Layer segmentation assumes sufficient textures for each layer to represent motion A true success? J. Wang & E. H. Adelson 1994 Y. Weiss & E. H. Adelson 1994 Achieved with the help of spatial segmentation Layer representation is good, but the existing layer segmentation algorithms cannot find the right layers for textureless objects

10. Challenge: Textureless Objects under Occlusion Corners are not always trustworthy (junctions) Flat regions do not always move smoothly (discontinuous at illusory boundaries) How about boundaries? –Easy to detect and track for textureless objects –Able to handle junctions with illusory boundaries

11. Analysis of Contour Motions Our approach: simultaneous grouping and motion analysis –Multi-level contour representation –Junctions are appropriated handled –Formulate graphical model that favors good contour and motion criteria –Inference using importance sampling Contribution –An important component in motion analysis toolbox for textureless objects under occlusion

12. Three Levels of Contour Representation –Edgelets : edge particles –Boundary fragments : a chain of edgelets with small curvatures –Contours : a chain of boundary fragments Forming boundary fragments: easy (for textureless objects) Forming contours: hard (the focus of our work)

13. Overview of our system 1. Extract boundary fragments 2. Edgelet tracking with uncertainty. 3. Boundary grouping and illusory boundary 4. Motion estimation based on the grouping

14. Forming Boundary Fragments Boundary fragments extraction in frame 1 –Steerable filters to obtain edge energy for each orientation band –Spatially trace boundary fragments –Boundary fragments: lines or curves with small curvature Temporal edgelet tracking with uncertainties (a)(b) (c)(d) –Frame 1: edgelet (x, y,  ) –Frame 2: orientation energy of  –A Gaussian pdf is fit with the weight of orientation energy –1D uncertainty of motion (even for T- junctions)

15. Forming Contours: Boundary Fragments Grouping Grouping representation: switch variables (attached to every end of the fragments) –Exclusive : one end connects to at most one other end –Reversible : if end (i,t i ) connects to (j,t j ), then (j,t j ) connects to (i,t i ) Arbitrarily possible connection Reversibility A legal contour grouping Another legal contour grouping 0 1 0 1 1 0

16. Local Spatial-Temporal Cues for Grouping Motion stimulus Illusory boundaries corresponding to the groupings (generated by spline interpolation)

17. Local spatial-temporal cues for grouping: (a) Motion similarity Motion stimulus Velocity space KL( ) < KL( ) The grouping with higher motion similarity is favored

18. Local spatial-temporal cues for grouping: (b) Curve smoothness Motion stimulus The grouping with smoother and shorter illusory boundary is favored

19. Local spatial-temporal cues for grouping: (c) Contrast consistency Motion stimulus The grouping with consistent local contrast is favored

20. The Graphical Model for Grouping Affinity metric terms –(a) Motion similarity –(b) Curve smoothness –(c) Contrast consistency The graphical model for grouping reversibilityaffinityno self-intersection

21. Motion estimation for grouped contours Gaussian MRF (GMRF) within a boundary fragment The motions of two end edgelets are similar if they are grouped together The graphical model of motion: joint Gaussian given the grouping This problem is solved in early work: Y. Weiss, Interpreting images by propagating Bayesian beliefs, NIPS, 1997.

22. Inference Two-step inference –Grouping (switch variables) –Motion based on grouping (easy, least square) Grouping: importance sampling to estimate the marginal of the switch variables –Bidirectional proposal density –Toss the sample if self-intersection is detected Obtain the optimal grouping from the marginal

23. Why bidirectional proposal in sampling?

24. b 1  b 2 : 0.39 b 1  b 3 : 0.01 b 1  b 4 : 0.60 Normalized affinity metrics b 4  b 1 : 0.20 b 4  b 2 : 0.05 b 4  b 3 : 0.85 b 2  b 1 : 0.50 b 2  b 3 : 0.45 b 2  b 4 : 0.05 b 3  b 1 : 0.01 b 3  b 2 : 0.45 b 3  b 4 : 0.54 b 1  b 2 : 0.1750 b 1  b 3 : 0.0001 b 1  b 4 : 0.1200 Affinity metric of the switch variable (darker, thicker means larger affinity) Bidirectional proposal

25. Why bidirectional proposal in sampling? b 1  b 2 : 0.39 b 1  b 3 : 0.01 b 1  b 4 : 0.60 Normalized affinity metrics Bidirectional proposal (Normalized) b 4  b 1 : 0.20 b 4  b 2 : 0.05 b 4  b 3 : 0.85 b 2  b 1 : 0.50 b 2  b 3 : 0.45 b 2  b 4 : 0.05 b 3  b 1 : 0.01 b 3  b 2 : 0.45 b 3  b 4 : 0.54 b 1  b 2 : 0.62 b 1  b 3 : 0.00 b 1  b 4 : 0.38 Bidirectional proposal of the switch variable (darker, thicker means larger affinity)

26. Example of Sampling Motion stimulus Self intersection

27. Example of Sampling Motion stimulus A valid grouping

28. Example of Sampling Motion stimulus More valid groupings

29. Example of Sampling Motion stimulus More valid groupings

30. From Affinity to Marginals Affinity metric of the switch variable (darker, thicker means larger affinity) Motion stimulus

31. From Affinity to Marginals Marginal distribution of the switch variable (darker, thicker means larger affinity) Motion stimulus Greedy algorithm to search for the best grouping based on the marginals

32. Experiments All the results are generated using the same parameter settings Running time depends on the number of boundary fragments, varying from ten seconds to a few minutes in MATLAB

33. Frame 1 Two Moving Bars

34. Frame 2 Two Moving Bars

35. Extracted boundary fragments. The green circles are the boundary fragment end points. Two Moving Bars

36. Optical flow from Lucas-Kanade algorithm. The flow vectors are only plotted at the edgelets Two Moving Bars

37. Estimated motion by our system after grouping Two Moving Bars

38. Boundary grouping and illusory boundaries (frame 1). The fragments belonging to the same contour are plotted in one color. Two Moving Bars

39. Boundary grouping and illusory boundaries (frame 2). The fragments belonging to the same contour are plotted in one color. Two Moving Bars

40. Kanizsa Square

41. Frame 1

42. Frame 2

43. Extracted boundary fragments

44. Optical flow from Lucas-Kanade algorithm

45. Estimated motion by our system, after grouping

46. Boundary grouping and illusory boundaries (frame 1)

47. Boundary grouping and illusory boundaries (frame 2)

48. Dancer

49. Frame 1

50. Frame 2

51. Extracted boundary fragments

52. Optical flow from Lucas-Kanade algorithm

53. Estimated motion by our system, after grouping

54. Lucas-Kanade flow field Estimated motion by our system, after grouping

55. Boundary grouping and illusory boundaries (frame 1)

56. Boundary grouping and illusory boundaries (frame 2)

57. Rotating Chair

58. Frame 1

59. Frame 2

60. Extracted boundary fragments

61. Estimated flow field from Brox et al.

62. Estimated motion by our system, after grouping

63. Boundary grouping and illusory boundaries (frame 1)

64. Boundary grouping and illusory boundaries (frame 2)

65. Conclusion A contour-based representation to estimate motion for textureless objects under occlusion Motion ambiguities are preserved and resolved through appropriate contour grouping An important component in motion analysis toolbox To be combined with the classical motion estimation techniques to analyze complex scenes

66. Thanks! Analysis of Contour Motions Ce Liu William T. Freeman Edward H. Adelson Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology http://people.csail.mit.edu/celiu/contourmotions/

67. Backup Slides

68. Why bidirectional proposal in sampling?

69. b 1  b 2 : 0.39 b 1  b 3 : 0.01 b 1  b 4 : 0.60 Normalized affinity metrics b 4  b 1 : 0.20 b 4  b 2 : 0.05 b 4  b 3 : 0.85 b 2  b 1 : 0.50 b 2  b 3 : 0.45 b 2  b 4 : 0.05 b 3  b 1 : 0.01 b 3  b 2 : 0.45 b 3  b 4 : 0.54 b 1  b 2 : 0.1750 b 1  b 3 : 0.0001 b 1  b 4 : 0.1200 Affinity metric of the switch variable (darker, thicker means larger affinity) Bidirectional proposal

70. Why bidirectional proposal in sampling? b 1  b 2 : 0.39 b 1  b 3 : 0.01 b 1  b 4 : 0.60 Normalized affinity metrics Bidirectional proposal (Normalized) b 4  b 1 : 0.20 b 4  b 2 : 0.05 b 4  b 3 : 0.85 b 2  b 1 : 0.50 b 2  b 3 : 0.45 b 2  b 4 : 0.05 b 3  b 1 : 0.01 b 3  b 2 : 0.45 b 3  b 4 : 0.54 b 1  b 2 : 0.62 b 1  b 3 : 0.00 b 1  b 4 : 0.38 Bidirectional proposal of the switch variable (darker, thicker means larger affinity)

71. Sampling Grouping (Switch Variables) Motion stimulus

72. Lucas-Kanade flow field Estimated motion by our system, after grouping