Tzu ming Su Advisor ： S.J.Wang MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION 2013/1/28 L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving optical flow estimation. In CVPR,

OUTLINE 2013/1/28 Previous Work Optical flow Conventional optical flow estimation 2

MOTION FIELD 2013/1/28 Definition ： an ideal representation of 3D motion as it is projected onto a camera image. 3D motion vector 2D optical flow vector CCD 3

MOTION FIELD 2013/1/28 Applications ： Video enhancement ： stabilization, denoising, super resolution 3D reconstruction ： structure from motion (SFM) Video segmentation Tracking/recognition Advanced video editing (label propagation) 4

MOTION FIELD ESTIMATION 2013/1/28 Optical flow Recover image motion at each pixel from spatio-temporal image brightness variations Feature-tracking Extract visual features (corners, textured areas) and “track” them over multiple frames 5

OPTICAL FLOW 2013/1/28 Definition ： the apparent motion of brightness patterns in the images Map flow vector to color Magnitude: saturation Orientation: hue 6

OPTICAL FLOW 2013/1/28 Key assumptions Brightness constancy Small motion Spatial coherence Remark ： Brightness constancy is often violated  Use gradient constancy for addition, both of them are called data constraint 7

BRIGHTNESS CONSISTENCY 2013/1/28 1-D case IxIx vItIt 8

BRIGHTNESS CONSISTENCY I ( x, t ) p ? v 1-D case 2013/1/28 9

BRIGHTNESS CONSISTENCY I ( x, t ) p Spatial derivative Temporal derivative v 1-D case 2013/1/28 10

BRIGHTNESS CONSISTENCY 1-D case 2-D case One equation, two velocity (u,v) unknowns… u v 2013/1/28 11

APERTURE PROBLEM 2013/1/28 12

APERTURE PROBLEM 2013/1/28 13

APERTURE PROBLEM 2013/1/28 Time t ? Time t+dt We know the movement parallel to the direction of gradient, but not the movement orthogonal to the gradient We need additional constraints 14

CONVENTIONAL ESTIMATION 2013/1/28 Use data consistency & additional constraint to estimate optical flow Horn-Schunck Minimize energy function with smoothness term Lucas-Kanade Minimize least square error function with local region coherence 15

HORN-SCHUNCK ESTIMATION 2013/1/28 Imposing spatial smoothness to the flow field Adjacent pixels should move together as much as possible Horn & Schunck equation 16

HORN-SCHUNCK ESTIMATION 2013/1/28 Use 2D Euler Lagrange Can be iteratively solved 17

COARSE TO FINE ESTIMATION 2013/1/28 Optical flow is assumed to be small motions, but in fact most motions are not Solved by coarse to fine resolution 18

image I image I t-1 COARSE TO FINE ESTIMATION run iteratively /1/28 19

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 20

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 21

MUTI-SCALE PROBLEM 2013/1/28 Conventional coarse to fine estimation can’t deal with large displacement. With different motion scales between foreground & background, even small motions can be miss detected. 22

MUTI-SCALE PROBLEM 2013/1/28 23

Ground truth … Estimate Ground truth 2013/1/28 24

MUTI-SCALE PROBLEM 2013/1/28 Large discrepancy between initial values and optimal motion vectors Solution ： Improve flow initialization to reduce the reliance on the initialization from coarser levels 25

2013/1/28 Sparse feature matching Fusion Dense nearest-neighbor patch matching Selection 26

EXTENDED FLOW INITIALIZATION Sparse feature matching for each level

EXTENDED FLOW INITIALIZATION Identify missing motion vectors 2013/1/28 28

EXTENDED FLOW INITIALIZATION Identify missing motion vectors 2013/1/28 29

EXTENDED FLOW INITIALIZATION … 2013/1/28 30

EXTENDED FLOW INITIALIZATION Fuse 2013/1/28 31

2013/1/28 Sparse feature matching Fusion Dense nearest-neighbor patch matching Selection 32

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 33

CONSTRAINTS 2013/1/28 Brightness consistency Gradient consistency Average 34

Pixels moving out of shadow CONSTRAINTS Color constancy is violated Average: : ground truth motion of p 1 Gradient constancy holds 2013/1/28 35

Pixels undergoing rotational motion CONSTRAINTS Color constancy holds Gradient constancy is violated : ground truth motion of p 2 Average: 2013/1/28 36

SELECTIVE DATA TERM Selectively combine the constraints where 2013/1/28 37

SELECTIVE DATA TERM selective 2013/1/28 38

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 39

DISCRETE-OPTIMIZATION 2013/1/28 40 Minimizing energy including discrete α & continuous u ： Try to separate α & u For α Probability of a particular state of MRF system

DISCRETE-OPTIMIZATION 2013/1/28 41 Partition function Sum over all possible values of α

Optimal condition (Euler-Lagrange equations) It decomposes to DISCRETE-OPTIMIZATION Minimization – Update α – Compute flow field

CONTINUOUS-OPTIMIZATION 2013/1/28 43 Energy function Variable splitting

CONTINUOUS-OPTIMIZATION 2013/1/28 44 Fix u, estimate w,p Fix w,p, estimate u The Euler-Lagrange equation Is linear.

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 45

SELECTIVE DATA TERM Averaging Selective Difference 2013/1/28 46

EXPERIMENTAL RESULTS 2013/1/28 47

RESULTS FROM DIFFERENT STEPS Coarse-to-fine Extended coarse-to-fine 2013/1/28 48

2013/1/28 49

LARGE DISPLACEMENT Overlaid Input 2013/1/28 50

LARGE DISPLACEMENT Motion Estimates Coarse-to-fineResult Warping Result 2013/1/28 51

LARGE DISPLACEMENT Motion Magnitude Maps LDOP [Brox et al. 09 ] [Steinbrucker et al. 09]] Result 2013/1/28 52

OVERLAID INPUT 2013/1/28 53

Conventional Coarse-to-fineResult 2013/1/28 54

EXPERIMENTAL RESULTS Overlaid Input 2013/1/28 55

Coarse-to-fine Result 2013/1/28 56

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 57

CONCLUSION 2013/1/28 58 To solve the coarse-to-fine problem, it seems more easier to make a correctness in every level. Using optical flow for small motion & other tracking skill for large displacement seems reasonable. It takes 40s ~ 3mins to compute an optical flow field respect to the amount of missing parts. Tradeoff problem.

2013/1/28 59 Thank you for your listening

FEATURE MATCHING 2013/1/28 Feature ： ” interesting “, ” unique” part of image Two components of feature ： Test image Detector : where are the local features? Descriptor : how to describe them? 60

FEATURE MATCHING 2013/1/28 Local measure of feature uniqueness Shifting the window in any direction causes a big change “flat” : no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions 61

SIFT FEATURE MATCHING 2013/1/28 SIFT ： Scale Invariant Feature Transform Problem ： non -invariant between image scales All points will be classified as edges Corner 62

SIFT FEATURE MATCHING 2013/1/28 Find scale that gives local maxima of some function f in both position and scale 63

SIFT FEATURE MATCHING 2013/1/28 Function f ： Laplacian-of-Gaussian 64

SIFT FEATURE MATCHING 2013/1/28 65 We define the characteristic scale as the scale that produces peak of Laplacian response

66 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

67 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

DETECTOR 2013/1/28 68

SCALE-SPACE EXTREMA DETECTION 2013/1/28 69 Use Difference of Gaussian instead of LOG More efficient DOG & LOG

KEYPOINT LOCALIZATION 2013/1/28 70 X is selected if it is larger or smaller than all 26 neighbors Eliminating edge responses

71 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

ORIENTATION ASSIGNMENT 2013/1/28 72 Use orientation histogram in the window to vote for total orientation Rotation-Invariant

KEYPOINT DESCRIPTOR 2013/1/28 73 Describe the orientation histogram in 8x8 window near the pixel Illumination-robust Back

PATCH MATCHING 2013/1/28 SIFT still lose information about objects lacking features Using “patch” as a unit, minimizing Without smoothness term, it can detect large replacement, but also produce errors. Errors can be eliminate by fusion step. 74

PATCH MATCHING 2013/1/28 Randomized Correspondence Algorithm Idea ： Coherent matches with neighbors Algorithm Initialization Propagation Search 75

PATCH MATCHING 2013/1/28 76 Back

GRAPHIC CUT 2013/1/28 Regard every pixel of image as a random variable, then the image is a “ random field.” Every pixel is only related to its neighbors, the filed is a “ Markov random field. ”(MRF) MRF can be viewed as a graph. 77

GRAPHIC CUT 2013/1/28 Regard the optical field as a MRF. The value of a pixel is chosen within optical flow frames produced previously, it’s a “ labeling problem. ” The edges between pixels are smoothness relation. Cut the graph with minimum energy. 78

GRAPHIC CUT 2013/1/28 Minimize the energy function Multi-labeling problem expansion move algorithm ： expanse the label which can decrease the energy V. Lempitsky, S. Roth, and C. Rother, “Fusionflow: Discrete-Continuous Optimization for Optical Flow Estimation,”Proc. IEEE Conf. Computer Vision and Pattern Recognition, Back

FEATURE MATCHING 2013/1/28 Find corners Change of intensity for the shift [u,v]: Intensity Shifted intensity Window function or Window function w(x,y) = Gaussian1 in window, 0 outside 80

FEATURE MATCHING 2013/1/28 For small shifts [ u,v ] we have a bilinear approximation: where M is a 2  2 matrix computed from image derivatives: 81

FEATURE MATCHING 2013/1/28 2 “Corner” 1 and 2 are large, 1 ~ 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Edge” 2 >> 1 “Flat” region Classification of image points using eigenvalues of M: 82