Tzu ming Su Advisor : S.J.Wang MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION 2013/1/28 L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving optical.

Slides:



Advertisements
Similar presentations
Feature Detection. Description Localization More Points Robust to occlusion Works with less texture More Repeatable Robust detection Precise localization.
Advertisements

The SIFT (Scale Invariant Feature Transform) Detector and Descriptor
Instructor: Mircea Nicolescu Lecture 13 CS 485 / 685 Computer Vision.
Computer Vision Optical Flow
A New Block Based Motion Estimation with True Region Motion Field Jozef Huska & Peter Kulla EUROCON 2007 The International Conference on “Computer as a.
Automatic Image Alignment (direct) : Computational Photography Alexei Efros, CMU, Fall 2006 with a lot of slides stolen from Steve Seitz and Rick.
Announcements Quiz Thursday Quiz Review Tomorrow: AV Williams 4424, 4pm. Practice Quiz handout.
Optical Flow Methods 2007/8/9.
Announcements Project1 artifact reminder counts towards your grade Demos this Thursday, 12-2:30 sign up! Extra office hours this week David (T 12-1, W/F.
Feature matching and tracking Class 5 Read Section 4.1 of course notes Read Shi and Tomasi’s paper on.
Motion Detail Preserving Optical Flow Estimation Li Xu 1, Jiaya Jia 1, Yasuyuki Matsushita 2 1 The Chinese University of Hong Kong 2 Microsoft Research.
Automatic Image Alignment (feature-based) : Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and.
Feature extraction: Corners and blobs
Announcements Project 1 test the turn-in procedure this week (make sure your folder’s there) grading session next Thursday 2:30-5pm –10 minute slot to.
Optical flow and Tracking CISC 649/849 Spring 2009 University of Delaware.
Scale Invariant Feature Transform (SIFT)
Motion Computing in Image Analysis
Optical Flow Estimation
Lecture 19: Optical flow CS6670: Computer Vision Noah Snavely
Visual motion Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Numerical Recipes (Newton-Raphson), 9.4 (first.
1 Stanford CS223B Computer Vision, Winter 2006 Lecture 7 Optical Flow Professor Sebastian Thrun CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado Slides.
COMP 290 Computer Vision - Spring Motion II - Estimation of Motion field / 3-D construction from motion Yongjik Kim.
3D Rigid/Nonrigid RegistrationRegistration 1)Known features, correspondences, transformation model – feature basedfeature based 2)Specific motion type,
Matching Compare region of image to region of image. –We talked about this for stereo. –Important for motion. Epipolar constraint unknown. But motion small.
Automatic Image Alignment (feature-based) : Computational Photography Alexei Efros, CMU, Fall 2006 with a lot of slides stolen from Steve Seitz and.
KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track
Optical Flow Digital Photography CSE558, Spring 2003 Richard Szeliski (notes cribbed from P. Anandan)
CSCE 641 Computer Graphics: Image Registration Jinxiang Chai.
Scale-Invariant Feature Transform (SIFT) Jinxiang Chai.
Computer vision.
Optical Flow Donald Tanguay June 12, Outline Description of optical flow General techniques Specific methods –Horn and Schunck (regularization)
Reporter: Fei-Fei Chen. Wide-baseline matching Object recognition Texture recognition Scene classification Robot wandering Motion tracking.
CSE 185 Introduction to Computer Vision Feature Tracking and Optical Flow.
Visual motion Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys.
December 9, 2014Computer Vision Lecture 23: Motion Analysis 1 Now we will talk about… Motion Analysis.
CSCE 643 Computer Vision: Extractions of Image Features Jinxiang Chai.
Computer Vision Spring ,-685 Instructor: S. Narasimhan Wean 5403 T-R 3:00pm – 4:20pm Lecture #16.
3D Imaging Motion.
Kylie Gorman WEEK 1-2 REVIEW. CONVERTING AN IMAGE FROM RGB TO HSV AND DISPLAY CHANNELS.
Joint Tracking of Features and Edges STAN BIRCHFIELD AND SHRINIVAS PUNDLIK CLEMSON UNIVERSITY ABSTRACT LUCAS-KANADE AND HORN-SCHUNCK JOINT TRACKING OF.
Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image.
Miguel Tavares Coimbra
Feature extraction: Corners and blobs. Why extract features? Motivation: panorama stitching We have two images – how do we combine them?
Features, Feature descriptors, Matching Jana Kosecka George Mason University.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Newton's method Wikpedia page
Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb.
Lecture 9 Feature Extraction and Motion Estimation Slides by: Michael Black Clark F. Olson Jean Ponce.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Newton's method Wikpedia page
Optical flow and keypoint tracking Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys.
Instructor: Mircea Nicolescu Lecture 10 CS 485 / 685 Computer Vision.
MASKS © 2004 Invitation to 3D vision Lecture 3 Image Primitives andCorrespondence.
MOTION Model. Road Map Motion Model Non Parametric Motion Field : Algorithms 1.Optical flow field estimation. 2.Block based motion estimation. 3.Pel –recursive.
Interest Points EE/CSE 576 Linda Shapiro.
Distinctive Image Features from Scale-Invariant Keypoints
Scale Invariant Feature Transform (SIFT)
3D Vision Interest Points.
Motion and Optical Flow
Nonparametric Semantic Segmentation
Motion Estimation Today’s Readings
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor
Announcements more panorama slots available now
MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION
Announcements Questions on the project? New turn-in info online
Announcements more panorama slots available now
Optical flow Computer Vision Spring 2019, Lecture 21
Filtering An image as a function Digital vs. continuous images
Optical flow and keypoint tracking
Presentation transcript:

Tzu ming Su Advisor : S.J.Wang MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION 2013/1/28 L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving optical flow estimation. In CVPR,

OUTLINE 2013/1/28 Previous Work Optical flow Conventional optical flow estimation 2

MOTION FIELD 2013/1/28 Definition : an ideal representation of 3D motion as it is projected onto a camera image. 3D motion vector 2D optical flow vector CCD 3

MOTION FIELD 2013/1/28 Applications : Video enhancement : stabilization, denoising, super resolution 3D reconstruction : structure from motion (SFM) Video segmentation Tracking/recognition Advanced video editing (label propagation) 4

MOTION FIELD ESTIMATION 2013/1/28 Optical flow Recover image motion at each pixel from spatio-temporal image brightness variations Feature-tracking Extract visual features (corners, textured areas) and “track” them over multiple frames 5

OPTICAL FLOW 2013/1/28 Definition : the apparent motion of brightness patterns in the images Map flow vector to color Magnitude: saturation Orientation: hue 6

OPTICAL FLOW 2013/1/28 Key assumptions Brightness constancy Small motion Spatial coherence Remark : Brightness constancy is often violated  Use gradient constancy for addition, both of them are called data constraint 7

BRIGHTNESS CONSISTENCY 2013/1/28 1-D case IxIx vItIt 8

BRIGHTNESS CONSISTENCY I ( x, t ) p ? v 1-D case 2013/1/28 9

BRIGHTNESS CONSISTENCY I ( x, t ) p Spatial derivative Temporal derivative v 1-D case 2013/1/28 10

BRIGHTNESS CONSISTENCY 1-D case 2-D case One equation, two velocity (u,v) unknowns… u v 2013/1/28 11

APERTURE PROBLEM 2013/1/28 12

APERTURE PROBLEM 2013/1/28 13

APERTURE PROBLEM 2013/1/28 Time t ? Time t+dt We know the movement parallel to the direction of gradient, but not the movement orthogonal to the gradient We need additional constraints 14

CONVENTIONAL ESTIMATION 2013/1/28 Use data consistency & additional constraint to estimate optical flow Horn-Schunck Minimize energy function with smoothness term Lucas-Kanade Minimize least square error function with local region coherence 15

HORN-SCHUNCK ESTIMATION 2013/1/28 Imposing spatial smoothness to the flow field Adjacent pixels should move together as much as possible Horn & Schunck equation 16

HORN-SCHUNCK ESTIMATION 2013/1/28 Use 2D Euler Lagrange Can be iteratively solved 17

COARSE TO FINE ESTIMATION 2013/1/28 Optical flow is assumed to be small motions, but in fact most motions are not Solved by coarse to fine resolution 18

image I image I t-1 COARSE TO FINE ESTIMATION run iteratively /1/28 19

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 20

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 21

MUTI-SCALE PROBLEM 2013/1/28 Conventional coarse to fine estimation can’t deal with large displacement. With different motion scales between foreground & background, even small motions can be miss detected. 22

MUTI-SCALE PROBLEM 2013/1/28 23

Ground truth … Estimate Ground truth 2013/1/28 24

MUTI-SCALE PROBLEM 2013/1/28 Large discrepancy between initial values and optimal motion vectors Solution : Improve flow initialization to reduce the reliance on the initialization from coarser levels 25

2013/1/28 Sparse feature matching Fusion Dense nearest-neighbor patch matching Selection 26

EXTENDED FLOW INITIALIZATION Sparse feature matching for each level

EXTENDED FLOW INITIALIZATION Identify missing motion vectors 2013/1/28 28

EXTENDED FLOW INITIALIZATION Identify missing motion vectors 2013/1/28 29

EXTENDED FLOW INITIALIZATION … 2013/1/28 30

EXTENDED FLOW INITIALIZATION Fuse 2013/1/28 31

2013/1/28 Sparse feature matching Fusion Dense nearest-neighbor patch matching Selection 32

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 33

CONSTRAINTS 2013/1/28 Brightness consistency Gradient consistency Average 34

Pixels moving out of shadow CONSTRAINTS Color constancy is violated Average: : ground truth motion of p 1 Gradient constancy holds 2013/1/28 35

Pixels undergoing rotational motion CONSTRAINTS Color constancy holds Gradient constancy is violated : ground truth motion of p 2 Average: 2013/1/28 36

SELECTIVE DATA TERM Selectively combine the constraints where 2013/1/28 37

SELECTIVE DATA TERM selective 2013/1/28 38

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 39

DISCRETE-OPTIMIZATION 2013/1/28 40 Minimizing energy including discrete α & continuous u : Try to separate α & u For α Probability of a particular state of MRF system

DISCRETE-OPTIMIZATION 2013/1/28 41 Partition function Sum over all possible values of α

Optimal condition (Euler-Lagrange equations) It decomposes to DISCRETE-OPTIMIZATION Minimization – Update α – Compute flow field

CONTINUOUS-OPTIMIZATION 2013/1/28 43 Energy function Variable splitting

CONTINUOUS-OPTIMIZATION 2013/1/28 44 Fix u, estimate w,p Fix w,p, estimate u The Euler-Lagrange equation Is linear.

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 45

SELECTIVE DATA TERM Averaging Selective Difference 2013/1/28 46

EXPERIMENTAL RESULTS 2013/1/28 47

RESULTS FROM DIFFERENT STEPS Coarse-to-fine Extended coarse-to-fine 2013/1/28 48

2013/1/28 49

LARGE DISPLACEMENT Overlaid Input 2013/1/28 50

LARGE DISPLACEMENT Motion Estimates Coarse-to-fineResult Warping Result 2013/1/28 51

LARGE DISPLACEMENT Motion Magnitude Maps LDOP [Brox et al. 09 ] [Steinbrucker et al. 09]] Result 2013/1/28 52

OVERLAID INPUT 2013/1/28 53

Conventional Coarse-to-fineResult 2013/1/28 54

EXPERIMENTAL RESULTS Overlaid Input 2013/1/28 55

Coarse-to-fine Result 2013/1/28 56

OUTLINE 2013/1/28 Previous Work Contributions Extended Flow Initialization Selective data term Efficient optimization solver Experimental result Conclusion 57

CONCLUSION 2013/1/28 58 To solve the coarse-to-fine problem, it seems more easier to make a correctness in every level. Using optical flow for small motion & other tracking skill for large displacement seems reasonable. It takes 40s ~ 3mins to compute an optical flow field respect to the amount of missing parts. Tradeoff problem.

2013/1/28 59 Thank you for your listening

FEATURE MATCHING 2013/1/28 Feature : ” interesting “, ” unique” part of image Two components of feature : Test image Detector : where are the local features? Descriptor : how to describe them? 60

FEATURE MATCHING 2013/1/28 Local measure of feature uniqueness Shifting the window in any direction causes a big change “flat” : no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions 61

SIFT FEATURE MATCHING 2013/1/28 SIFT : Scale Invariant Feature Transform Problem : non -invariant between image scales All points will be classified as edges Corner 62

SIFT FEATURE MATCHING 2013/1/28 Find scale that gives local maxima of some function f in both position and scale 63

SIFT FEATURE MATCHING 2013/1/28 Function f : Laplacian-of-Gaussian 64

SIFT FEATURE MATCHING 2013/1/28 65 We define the characteristic scale as the scale that produces peak of Laplacian response

66 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

67 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

DETECTOR 2013/1/28 68

SCALE-SPACE EXTREMA DETECTION 2013/1/28 69 Use Difference of Gaussian instead of LOG More efficient DOG & LOG

KEYPOINT LOCALIZATION 2013/1/28 70 X is selected if it is larger or smaller than all 26 neighbors Eliminating edge responses

71 ALGORITHM Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor ( ) local descriptor detector descriptor

ORIENTATION ASSIGNMENT 2013/1/28 72 Use orientation histogram in the window to vote for total orientation Rotation-Invariant

KEYPOINT DESCRIPTOR 2013/1/28 73 Describe the orientation histogram in 8x8 window near the pixel Illumination-robust Back

PATCH MATCHING 2013/1/28 SIFT still lose information about objects lacking features Using “patch” as a unit, minimizing Without smoothness term, it can detect large replacement, but also produce errors. Errors can be eliminate by fusion step. 74

PATCH MATCHING 2013/1/28 Randomized Correspondence Algorithm Idea : Coherent matches with neighbors Algorithm Initialization Propagation Search 75

PATCH MATCHING 2013/1/28 76 Back

GRAPHIC CUT 2013/1/28 Regard every pixel of image as a random variable, then the image is a “ random field.” Every pixel is only related to its neighbors, the filed is a “ Markov random field. ”(MRF) MRF can be viewed as a graph. 77

GRAPHIC CUT 2013/1/28 Regard the optical field as a MRF. The value of a pixel is chosen within optical flow frames produced previously, it’s a “ labeling problem. ” The edges between pixels are smoothness relation. Cut the graph with minimum energy. 78

GRAPHIC CUT 2013/1/28 Minimize the energy function Multi-labeling problem expansion move algorithm : expanse the label which can decrease the energy V. Lempitsky, S. Roth, and C. Rother, “Fusionflow: Discrete-Continuous Optimization for Optical Flow Estimation,”Proc. IEEE Conf. Computer Vision and Pattern Recognition, Back

FEATURE MATCHING 2013/1/28 Find corners Change of intensity for the shift [u,v]: Intensity Shifted intensity Window function or Window function w(x,y) = Gaussian1 in window, 0 outside 80

FEATURE MATCHING 2013/1/28 For small shifts [ u,v ] we have a bilinear approximation: where M is a 2  2 matrix computed from image derivatives: 81

FEATURE MATCHING 2013/1/28 2 “Corner” 1 and 2 are large, 1 ~ 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Edge” 2 >> 1 “Flat” region Classification of image points using eigenvalues of M: 82