1. Tzu ming Su Advisor ： S.J.Wang MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION 2013/1/28 L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving optical flow estimation. In CVPR, 2010. 1

2. OUTLINE 2013/1/28 Previous Work Optical flow Conventional optical flow estimation 2

3. MOTION FIELD 2013/1/28 Definition ： an ideal representation of 3D motion as it is projected onto a camera image. 3D motion vector 2D optical flow vector CCD 3

4. MOTION FIELD 2013/1/28 Applications ： Video enhancement ： stabilization, denoising, super resolution 3D reconstruction ： structure from motion (SFM) Video segmentation Tracking/recognition Advanced video editing (label propagation) 4

5. MOTION FIELD ESTIMATION 2013/1/28 Optical flow Recover image motion at each pixel from spatio-temporal image brightness variations Feature-tracking Extract visual features (corners, textured areas) and “track” them over multiple frames 5

6. OPTICAL FLOW 2013/1/28 Definition ： the apparent motion of brightness patterns in the images Map flow vector to color Magnitude: saturation Orientation: hue 6

7. OPTICAL FLOW 2013/1/28 Key assumptions Brightness constancy Small motion Spatial coherence Remark ： Brightness constancy is often violated  Use gradient constancy for addition, both of them are called data constraint 7

8. BRIGHTNESS CONSISTENCY 2013/1/28 1-D case IxIx vItIt 8

9. BRIGHTNESS CONSISTENCY I ( x, t ) p ? v 1-D case 2013/1/28 9

10. BRIGHTNESS CONSISTENCY I ( x, t ) p Spatial derivative Temporal derivative v 1-D case 2013/1/28 10

11. BRIGHTNESS CONSISTENCY 1-D case 2-D case One equation, two velocity (u,v) unknowns… u v 2013/1/28 11

12. APERTURE PROBLEM 2013/1/28 12

13. APERTURE PROBLEM 2013/1/28 13

14. APERTURE PROBLEM 2013/1/28 Time t ? Time t+dt We know the movement parallel to the direction of gradient, but not the movement orthogonal to the gradient We need additional constraints 14

15. CONVENTIONAL ESTIMATION 2013/1/28 Use data consistency & additional constraint to estimate optical flow Horn-Schunck Minimize energy function with smoothness term Lucas-Kanade Minimize least square error function with local region coherence 15

16. HORN-SCHUNCK ESTIMATION 2013/1/28 Imposing spatial smoothness to the flow field Adjacent pixels should move together as much as possible Horn & Schunck equation 16

17. HORN-SCHUNCK ESTIMATION 2013/1/28 Use 2D Euler Lagrange Can be iteratively solved 17

18. COARSE TO FINE ESTIMATION 2013/1/28 Optical flow is assumed to be small motions, but in fact most motions are not Solved by coarse to fine resolution 18

19. image I image I t-1 COARSE TO FINE ESTIMATION run iteratively...... 2013/1/28 19

20. OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 20

21. OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 21

22. MUTI-SCALE PROBLEM 2013/1/28 Conventional coarse to fine estimation can’t deal with large displacement. With different motion scales between foreground & background, even small motions can be miss detected. 22

23. MUTI-SCALE PROBLEM 2013/1/28 23

24. Ground truth … Estimate Ground truth 2013/1/28 24

25. MUTI-SCALE PROBLEM 2013/1/28 Large discrepancy between initial values and optimal motion vectors Solution ： Improve flow initialization to reduce the reliance on the initialization from coarser levels 25

26. 2013/1/28 Sparse feature matching Fusion Dense nearest-neighbor patch matching Selection 26

27. EXTENDED FLOW INITIALIZATION Sparse feature matching for each level

28. EXTENDED FLOW INITIALIZATION Identify missing motion vectors 2013/1/28 28

29. EXTENDED FLOW INITIALIZATION Identify missing motion vectors 2013/1/28 29

30. EXTENDED FLOW INITIALIZATION … 2013/1/28 30

31. EXTENDED FLOW INITIALIZATION Fuse 2013/1/28 31

32. 2013/1/28 Sparse feature matching Fusion Dense nearest-neighbor patch matching Selection 32

33. OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 33

34. CONSTRAINTS 2013/1/28 Brightness consistency Gradient consistency Average 34

35. Pixels moving out of shadow CONSTRAINTS Color constancy is violated Average: : ground truth motion of p 1 Gradient constancy holds 2013/1/28 35

36. Pixels undergoing rotational motion CONSTRAINTS Color constancy holds Gradient constancy is violated : ground truth motion of p 2 Average: 2013/1/28 36

37. SELECTIVE DATA TERM Selectively combine the constraints where 2013/1/28 37

38. SELECTIVE DATA TERM selective 2013/1/28 38

39. OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 39

40. DISCRETE-OPTIMIZATION 2013/1/28 40 Minimizing energy including discrete α & continuous u ： Try to separate α & u For α Probability of a particular state of MRF system

41. DISCRETE-OPTIMIZATION 2013/1/28 41 Partition function Sum over all possible values of α

42. Optimal condition (Euler-Lagrange equations) It decomposes to DISCRETE-OPTIMIZATION Minimization – Update α – Compute flow field

43. CONTINUOUS-OPTIMIZATION 2013/1/28 43 Energy function Variable splitting

44. CONTINUOUS-OPTIMIZATION 2013/1/28 44 Fix u, estimate w,p Fix w,p, estimate u The Euler-Lagrange equation Is linear.

45. OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 45

46. SELECTIVE DATA TERM Averaging Selective Difference 2013/1/28 46

47. EXPERIMENTAL RESULTS 2013/1/28 47

48. RESULTS FROM DIFFERENT STEPS Coarse-to-fine Extended coarse-to-fine 2013/1/28 48

49. 2013/1/28 49

50. LARGE DISPLACEMENT Overlaid Input 2013/1/28 50

51. LARGE DISPLACEMENT Motion Estimates Coarse-to-fineResult Warping Result 2013/1/28 51

52. LARGE DISPLACEMENT Motion Magnitude Maps LDOP [Brox et al. 09 ] [Steinbrucker et al. 09]] Result 2013/1/28 52

53. OVERLAID INPUT 2013/1/28 53

54. Conventional Coarse-to-fineResult 2013/1/28 54

55. EXPERIMENTAL RESULTS Overlaid Input 2013/1/28 55

56. Coarse-to-fine Result 2013/1/28 56

57. OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 57

58. CONCLUSION 2013/1/28 58 To solve the coarse-to-fine problem, it seems more easier to make a correctness in every level. Using optical flow for small motion & other tracking skill for large displacement seems reasonable. It takes 40s ~ 3mins to compute an optical flow field respect to the amount of missing parts. Tradeoff problem.

59. 2013/1/28 59 Thank you for your listening

60. FEATURE MATCHING 2013/1/28 Feature ： ” interesting “, ” unique” part of image Two components of feature ： Test image Detector : where are the local features? Descriptor : how to describe them? 60

61. FEATURE MATCHING 2013/1/28 Local measure of feature uniqueness Shifting the window in any direction causes a big change “flat” : no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions 61

62. SIFT FEATURE MATCHING 2013/1/28 SIFT ： Scale Invariant Feature Transform Problem ： non -invariant between image scales All points will be classified as edges Corner 62

63. SIFT FEATURE MATCHING 2013/1/28 Find scale that gives local maxima of some function f in both position and scale 63

64. SIFT FEATURE MATCHING 2013/1/28 Function f ： Laplacian-of-Gaussian 64

65. SIFT FEATURE MATCHING 2013/1/28 65 We define the characteristic scale as the scale that produces peak of Laplacian response

66. 66 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

67. 67 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

68. DETECTOR 2013/1/28 68

69. SCALE-SPACE EXTREMA DETECTION 2013/1/28 69 Use Difference of Gaussian instead of LOG More efficient DOG & LOG

70. KEYPOINT LOCALIZATION 2013/1/28 70 X is selected if it is larger or smaller than all 26 neighbors Eliminating edge responses

71. 71 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

72. ORIENTATION ASSIGNMENT 2013/1/28 72 Use orientation histogram in the window to vote for total orientation Rotation-Invariant

73. KEYPOINT DESCRIPTOR 2013/1/28 73 Describe the orientation histogram in 8x8 window near the pixel Illumination-robust Back

74. PATCH MATCHING 2013/1/28 SIFT still lose information about objects lacking features Using “patch” as a unit, minimizing Without smoothness term, it can detect large replacement, but also produce errors. Errors can be eliminate by fusion step. 74

75. PATCH MATCHING 2013/1/28 Randomized Correspondence Algorithm Idea ： Coherent matches with neighbors Algorithm Initialization Propagation Search 75

76. PATCH MATCHING 2013/1/28 76 Back

77. GRAPHIC CUT 2013/1/28 Regard every pixel of image as a random variable, then the image is a “ random field.” Every pixel is only related to its neighbors, the filed is a “ Markov random field. ”(MRF) MRF can be viewed as a graph. 77

78. GRAPHIC CUT 2013/1/28 Regard the optical field as a MRF. The value of a pixel is chosen within optical flow frames produced previously, it’s a “ labeling problem. ” The edges between pixels are smoothness relation. Cut the graph with minimum energy. 78

79. GRAPHIC CUT 2013/1/28 Minimize the energy function Multi-labeling problem expansion move algorithm ： expanse the label which can decrease the energy V. Lempitsky, S. Roth, and C. Rother, “Fusionflow: Discrete-Continuous Optimization for Optical Flow Estimation,”Proc. IEEE Conf. Computer Vision and Pattern Recognition,2008 79 Back

80. FEATURE MATCHING 2013/1/28 Find corners Change of intensity for the shift [u,v]: Intensity Shifted intensity Window function or Window function w(x,y) = Gaussian1 in window, 0 outside 80

81. FEATURE MATCHING 2013/1/28 For small shifts [ u,v ] we have a bilinear approximation: where M is a 2  2 matrix computed from image derivatives: 81

82. FEATURE MATCHING 2013/1/28 2 “Corner” 1 and 2 are large, 1 ~ 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Edge” 2 >> 1 “Flat” region Classification of image points using eigenvalues of M: 82