Computer Vision Optical Flow Marc Pollefeys COMP 256 Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black, K. Toyama

Computer Vision 2 last week: polar rectification

Computer Vision 3 Last week: polar rectification

Computer Vision 4 Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data) Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Computer Vision 5 Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges ( Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Computer Vision 6 Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

Computer Vision 7 Example: reconstruct image from neighboring images

Computer Vision 8 Multi-view depth fusion Compute depth for every pixel of reference image –Triangulation –Use multiple views –Up- and down sequence –Use Kalman filter (Koch, Pollefeys and Van Gool. ECCV‘98) Allows to compute robust texture

Computer Vision 9 Real-time stereo on graphics hardware Computes Sum-of-Square-Differences Hardware mip-map generation used to aggregate results over support region Trade-off between small and large support window Yang and Pollefeys CVPR03 140M disparity hypothesis/sec on Radeon 9700pro e.g. 512x512x20disparities at 30Hz Shape of a kernel for summing up 6 levels

Computer Vision 10 Sample Re-Projections near far

Computer Vision 11 (1x1) (1x1+2x2) (1x1+2x2 +4x4+8x8) (1x1+2x2 +4x4+8x8 +16x16) Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass video

Computer Vision 12 Cool ideas Space-time stereo (varying illumination, not shape)

Computer Vision 13 More on stereo … The Middleburry Stereo Vision Research Page Recommended reading D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. IJCV 47(1/2/3):7-42, April-June PDF file (1.15 MB) - includes current evaluation. Microsoft Research Technical Report MSR-TR , November PDF file (1.27 MB).PDF file

Computer Vision 14 Jan 16/18-Introduction Jan 23/25CamerasRadiometry Jan 30/Feb1Sources & ShadowsColor Feb 6/8Linear filters & edgesTexture Feb 13/15Multi-View GeometryStereo Feb 20/22Optical flowProject proposals Feb27/Mar1Affine SfMProjective SfM Mar 6/8Camera CalibrationSilhouettes and Photoconsistency Mar 13/15Springbreak Mar 20/22SegmentationFitting Mar 27/29Prob. SegmentationProject Update Apr 3/5Tracking Apr 10/12Object Recognition Apr 17/19Range data Apr 24/26Final project Tentative class schedule

Computer Vision 15 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 16 Optical Flow: Where do pixels move to?

Computer Vision 17 Motion is a basic cue Motion can be the only cue for segmentation

Computer Vision 18 Motion is a basic cue Motion can be the only cue for segmentation

Computer Vision 19 Motion is a basic cue Even impoverished motion data can elicit a strong percept

Computer Vision 20 Applications tracking structure from motion motion segmentation stabilization compression Mosaicing …

Computer Vision 21 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 22 Definition of optical flow OPTICAL FLOW = apparent motion of brightness patterns Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image 

Computer Vision 23 Caution required ! Two examples : 1. Uniform, rotating sphere  O.F. = 0 2. No motion, but changing lighting  O.F.  0 

Computer Vision 24 Caution required !

Computer Vision 25 Mathematical formulation I (x,y,t) = brightness at (x,y) at time t Optical flow constraint equation :  Brightness constancy assumption:

Computer Vision 26 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 27 The aperture problem 1 equation in 2 unknowns 

Computer Vision 28 The aperture problem 0

Computer Vision 29

Computer Vision 30 The aperture problem

Computer Vision 31 Remarks

Computer Vision 32 Apparently an aperture problem 

Computer Vision 33 What is Optic Flow, anyway? Estimate of observed projected motion field Not always well defined! Compare: –Motion Field (or Scene Flow) projection of 3-D motion field –Normal Flow observed tangent motion –Optic Flow apparent motion of the brightness pattern (hopefully equal to motion field) Consider Barber pole illusion

Computer Vision 34

Computer Vision 35 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 36 Horn & Schunck algorithm Additional smoothness constraint : besides OF constraint equation term minimize e s + e c 

Computer Vision 37 Horn & Schunck The Euler-Lagrange equations : In our case, so the Euler-Lagrange equations are is the Laplacian operator 

Computer Vision 38 Horn & Schunck Remarks : 1. Coupled PDEs solved using iterative methods and finite differences 2. More than two frames allow a better estimation of I t 3. Information spreads from corner-type patterns 

Computer Vision 39 

Computer Vision 40 Horn & Schunck, remarks 1. Errors at boundaries 2. Example of regularisation (selection principle for the solution of illposed problems) 

Computer Vision 41 Results of an enhanced system

Computer Vision 42 Structure from motion with OF

Computer Vision 43 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 44

Computer Vision 45

Computer Vision 46

Computer Vision 47

Computer Vision 48

Computer Vision 49 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 50

Computer Vision 51

Computer Vision 52

Computer Vision 53 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 54

Computer Vision 55

Computer Vision 56

Computer Vision 57

Computer Vision 58

Computer Vision 59

Computer Vision 60

Computer Vision 61

Computer Vision 62

Computer Vision 63

Computer Vision 64

Computer Vision 65

Computer Vision 66

Computer Vision 67

Computer Vision 68

Computer Vision 69

Computer Vision 70

Computer Vision 71

Computer Vision 72

Computer Vision 73

Computer Vision 74

Computer Vision 75

Computer Vision 76

Computer Vision 77

Computer Vision 78

Computer Vision 79

Computer Vision 80

Computer Vision 81

Computer Vision 82

Computer Vision 83 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 84

Computer Vision 85

Computer Vision 86

Computer Vision 87 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 88

Computer Vision 89

Computer Vision 90

Computer Vision 91

Computer Vision 92

Computer Vision 93 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 94

Computer Vision 95

Computer Vision 96

Computer Vision 97

Computer Vision 98

Computer Vision 99

Computer Vision 100 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Computer Vision 101

Computer Vision 102 Rhombus Displays

Computer Vision 103

Computer Vision 104

Computer Vision 105

Computer Vision 106

Computer Vision 107

Computer Vision 108