1. Analysis of Contour Motions
Neural Information Processing Systems 2006
Analysis of Contour Motions
Ce Liu William T. Freeman Edward H. Adelson
Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology

2. Visual Motion Analysis in Computer Vision
Motion analysis is essential in
Video processing
Geometry reconstruction
Object tracking, segmentation and recognition
Graphics applications
Is motion analysis solved?
Do we have good representation for motion analysis?
Is it computationally feasible to infer the representation from the raw video data?
What is a good representation for motion?
Motion analysis is the corner stone for many computer vision applications.

3. Seemingly Simple Examples
Let’s look at these two seemingly simple examples. We can clearly see a moving square occluding four circles from the Kanizsa square example, and one leg moving occluding the other from the real video. These two examples both contain textueless objects with occlusions. What do you expect the output from the state-of-the-art motion analysis algorithms for these two examples?
Kanizsa square
From real video

4. Output from the State-of-the-Art Optical Flow Algorithm
Kanizsa square
Optical flow field
T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004

5. Output from the State-of-the-Art Optical Flow Algorithm
Dancer
Optical flow field
T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004

6. Optical flow representation: aperture problem
How computer sees the motion in flow estimation?
Corners
Lines
Flat regions
Spurious junctions
Boundary ownership
Illusory boundaries

7. Optical Flow Representation
We need motion representation beyond pixel level!
Corners
Lines
Flat regions
Spurious junctions
Boundary ownership
Illusory boundaries

8. Layer Representation Video is a composite of layers
Layer segmentation assumes sufficient textures for each layer to represent motion
A true success?
J. Wang & E. H. Adelson 1994
Y. Weiss & E. H. Adelson 1994
Achieved with the help of spatial segmentation

9. Layer Representation Video is a composite of layers
Layer segmentation assumes sufficient textures for each layer to represent motion
A true success?
J. Wang & E. H. Adelson 1994
Y. Weiss & E. H. Adelson 1994
Layer representation is good, but the existing layer
segmentation algorithms cannot find the right layers
for textureless objects
Achieved with the help of spatial segmentation

10. Challenge: Textureless Objects under Occlusion
Corners are not always trustworthy (junctions)
Flat regions do not always move smoothly (discontinuous at illusory boundaries)
How about boundaries?
Easy to detect and track for textureless objects
Able to handle junctions with illusory boundaries

11. Analysis of Contour Motions
Our approach: simultaneous grouping and motion analysis
Multi-level contour representation
Junctions are appropriated handled
Formulate graphical model that favors good contour and motion criteria
Inference using importance sampling
Contribution
An important component in motion analysis toolbox for textureless objects under occlusion

12. Three Levels of Contour Representation
Edgelets: edge particles
Boundary fragments: a chain of edgelets with small curvatures
Contours: a chain of boundary fragments
Forming boundary fragments: easy (for textureless objects)
Forming contours: hard (the focus of our work)

13. Overview of our system 1. Extract boundary fragments
2. Edgelet tracking with uncertainty.
3. Boundary grouping and illusory boundary
4. Motion estimation based on the grouping

14. Forming Boundary Fragments
Boundary fragments extraction in frame 1
Steerable filters to obtain edge energy for each orientation band
Spatially trace boundary fragments
Boundary fragments: lines or curves with small curvature
Temporal edgelet tracking with uncertainties
Frame 1: edgelet (x, y, q)
Frame 2: orientation energy of q
A Gaussian pdf is fit with the weight of orientation energy
1D uncertainty of motion (even for T-junctions)
(a)
(b)
(c)
(d)

15. Forming Contours: Boundary Fragments Grouping
Grouping representation: switch variables (attached to every end of the fragments)
Exclusive: one end connects to at most one other end
Reversible: if end (i,ti) connects to (j,tj), then (j,tj) connects to (i,ti) 1
Arbitrarily possible connection
A legal contour grouping
Reversibility
Another legal contour grouping 1 1

16. Local Spatial-Temporal Cues for Grouping
Illusory boundaries corresponding to the groupings (generated by spline interpolation)
Motion stimulus

17. Local spatial-temporal cues for grouping: (a) Motion similarity
The grouping with higher motion similarity is favored
Velocity space
KL( ) < KL( )
Motion stimulus

18. Local spatial-temporal cues for grouping: (b) Curve smoothness
The grouping with smoother and shorter illusory boundary is favored
Motion stimulus

19. Local spatial-temporal cues for grouping: (c) Contrast consistency
The grouping with consistent local contrast is favored
Motion stimulus

20. The Graphical Model for Grouping
Affinity metric terms
(a) Motion similarity
(b) Curve smoothness
(c) Contrast consistency
The graphical model for grouping
affinity
reversibility
no self-intersection

21. Motion estimation for grouped contours
Gaussian MRF (GMRF) within a boundary fragment
The motions of two end edgelets are similar if they are grouped together
The graphical model of motion: joint Gaussian given the grouping
This problem is solved in early work: Y. Weiss, Interpreting images by propagating Bayesian beliefs, NIPS, 1997.

22. Inference Two-step inference
Grouping (switch variables)
Motion based on grouping (easy, least square)
Grouping: importance sampling to estimate the marginal of the switch variables
Bidirectional proposal density
Toss the sample if self-intersection is detected
Obtain the optimal grouping from the marginal

23. Why bidirectional proposal in sampling?

24. Why bidirectional proposal in sampling?
Affinity metric of the switch variable (darker, thicker means larger affinity)
b1b2: 0.39
b1b3: 0.01
b1b4: 0.60
b4b1: 0.20
b4b2: 0.05
b4b3: 0.85
b2b1: 0.50
b2b3: 0.45
b2b4: 0.05
b3b1: 0.01
b3b2: 0.45
b3b4: 0.54
b1b2:
b1b3:
b1b4:
Normalized affinity metrics
Bidirectional proposal

25. Why bidirectional proposal in sampling?
Bidirectional proposal of the switch variable (darker, thicker means larger affinity)
b1b2: 0.39
b1b3: 0.01
b1b4: 0.60
b4b1: 0.20
b4b2: 0.05
b4b3: 0.85
b2b1: 0.50
b2b3: 0.45
b2b4: 0.05
b3b1: 0.01
b3b2: 0.45
b3b4: 0.54
b1b2: 0.62
b1b3: 0.00
b1b4: 0.38
Normalized affinity metrics
Bidirectional proposal
(Normalized)

26. Example of Sampling
Motion stimulus
Self intersection

27. Example of Sampling
Motion stimulus
A valid grouping

28. Example of Sampling
Motion stimulus
More valid groupings

29. Example of Sampling
Motion stimulus
More valid groupings

30. From Affinity to Marginals
Affinity metric of the switch variable (darker, thicker means larger affinity)
Motion stimulus

31. From Affinity to Marginals
Marginal distribution of the switch variable (darker, thicker means larger affinity)
Greedy algorithm to search for the best grouping based on the marginals
Motion stimulus

32. Experiments
All the results are generated using the same parameter settings
Running time depends on the number of boundary fragments, varying from ten seconds to a few minutes in MATLAB

33. Two Moving Bars
Frame 1

34. Two Moving Bars
Frame 2

35. Two Moving Bars
Extracted boundary fragments. The green circles are the boundary fragment end points.

36. Two Moving Bars
Optical flow from Lucas-Kanade algorithm. The flow vectors are only plotted at the edgelets

37. Estimated motion by our system after grouping
Two Moving Bars
Estimated motion by our system after grouping

38. Two Moving Bars
Boundary grouping and illusory boundaries (frame 1). The fragments belonging to the same contour are plotted in one color.

39. Two Moving Bars
Boundary grouping and illusory boundaries (frame 2). The fragments belonging to the same contour are plotted in one color.

40. Kanizsa Square

41. Frame 1

42. Frame 2

43. Extracted boundary fragments

44. Optical flow from Lucas-Kanade algorithm

45. Estimated motion by our system, after grouping

46. Boundary grouping and illusory boundaries (frame 1)

47. Boundary grouping and illusory boundaries (frame 2)

48. Dancer

49. Frame 1

50. Frame 2

51. Extracted boundary fragments

52. Optical flow from Lucas-Kanade algorithm

53. Estimated motion by our system, after grouping

54. Lucas-Kanade flow field
Estimated motion by our system, after grouping

55. Boundary grouping and illusory boundaries
(frame 1)

56. Boundary grouping and illusory boundaries
(frame 2)

57. Rotating Chair

58. Frame 1

59. Frame 2

60. Extracted boundary fragments

61. Estimated flow field from Brox et al.

62. Estimated motion by our system, after grouping

63. Boundary grouping and illusory boundaries (frame 1)

64. Boundary grouping and illusory boundaries (frame 2)

65. Conclusion
A contour-based representation to estimate motion for textureless objects under occlusion
Motion ambiguities are preserved and resolved through appropriate contour grouping
An important component in motion analysis toolbox
To be combined with the classical motion estimation techniques to analyze complex scenes

66. Thanks! Analysis of Contour Motions
Ce Liu William T. Freeman Edward H. Adelson
Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology

67. Backup Slides

68. Why bidirectional proposal in sampling?

69. Why bidirectional proposal in sampling?
Affinity metric of the switch variable (darker, thicker means larger affinity)
b1b2: 0.39
b1b3: 0.01
b1b4: 0.60
b4b1: 0.20
b4b2: 0.05
b4b3: 0.85
b2b1: 0.50
b2b3: 0.45
b2b4: 0.05
b3b1: 0.01
b3b2: 0.45
b3b4: 0.54
b1b2:
b1b3:
b1b4:
Normalized affinity metrics
Bidirectional proposal

70. Why bidirectional proposal in sampling?
Bidirectional proposal of the switch variable (darker, thicker means larger affinity)
b1b2: 0.39
b1b3: 0.01
b1b4: 0.60
b4b1: 0.20
b4b2: 0.05
b4b3: 0.85
b2b1: 0.50
b2b3: 0.45
b2b4: 0.05
b3b1: 0.01
b3b2: 0.45
b3b4: 0.54
b1b2: 0.62
b1b3: 0.00
b1b4: 0.38
Normalized affinity metrics
Bidirectional proposal
(Normalized)

71. Sampling Grouping (Switch Variables)
Motion stimulus

72. Lucas-Kanade flow field
Estimated motion by our system, after grouping