O PTICAL F LOW & L AYERED M OTION Prepared by: Lidya Tonkonogi Mu’in Rizik 1

I NTRODUCTION – M OTION E STIMATION Motion estimation algorithms calculate the motion vectors that describe the transformation from one image to another 2

I NTRODUCTION – M OTION E STIMATION What if… there were no motion estimation algorithms… 3

I NTRODUCTION – M OTION E STIMATION Why Do we need Motion Estimation?  Video Compression  Video stabilization: 4

I NTRODUCTION – M OTION E STIMATION Old school Registration algorithms(Pre optical flow):  Slow and Costly  Failed in processing images with rotation and other distortions 5

O PTICAL FLOW 6

D EFINITION OPTICAL FLOW = apparent motion of brightness patterns Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image 7

P ROBLEM DEFINITION : OPTICAL FLOW How to estimate pixel motion from image H to image I? Solve pixel correspondence problem –given a pixel in H, look for nearby pixels of the same color in I 8

9 Key assumptions color constancy: a point in H looks the same in I –For grayscale images, this is brightness constancy small motion: points do not move very far This is called the optical flow problem

O PTICAL FLOW CONSTRAINTS ( GRAYSCALE IMAGES ) Let’s look at these constraints more closely brightness constancy: Q: what’s the equation? H(x,y)=I(x+u, y+v) 10

11 small motion: (u and v are less than 1 pixel) – suppose we take the Taylor series expansion of I:

OPTICAL FLOW EQUATION Combining these two equations In the limit as u and v go to zero, this becomes exact 12

O PTICAL FLOW EQUATION Q: how many unknowns and equations per pixel? 2 unknowns, one equation 13 Aperture Problem

W HAT ’ S IS THE PROBLEM T HE MOTION FROM LEFT TO RIGHT ??! 14

W HAT ’ S IS THE PROBLEM WRONG !!! 15

T HE A PERTURE P ROBLEM Different motions – classified as similar source: Ran Eshel 16

T HE A PERTURE P ROBLEM Similar motions – classified as different source: Ran Eshel 17

T HE A PERTURE P ROBLEM This problem results from the fact that we have only one equation and two unknowns. This equation : (the gradient) Define a line in (u,v) space 18

To solve this equation we will calculate the Normal flow We get at most “Normal Flow” – with one point we can only detect movement perpendicular to the brightness gradient. Solution is to take a patch of pixels Around the pixel of interest. T HE A PERTURE P ROBLEM 19

A PERTURE P ROBLEM AND N ORMAL F LOW 20

O PTICAL F LOW T ECHNIQUES Differential techniques: Local method: Lucas and Kanade Global method: Horn and Schunck 21

I NTRODUCTION – M OTION E STIMATION Lucas and Kanade 22

Lucas and Kanade proposed an algorithm to solve the aperture problem. We back to the function And they proposed another solution to find u and v. Question : How to get more equations for a pixel? Basic idea: impose additional constraints one method: pretend the pixel’s neighbors have the same (u,v) in other words If we use a kxk window, that gives us equations per pixel! L UCAS -K ANADE ALGORITHM 23

For example : Assume that we use a 5x5 window, that gives us 25 equations per pixel! L UCAS -K ANADE - EXAMPLE 24

We try to solve this problem by : In the previous example : Minimizing 25

L UKAS -K ANADE FLOW Prob:we have more equations than unknowns Solution : solve least squares problem minimum least squares solution given by solution of: 26 K x 22 x 1 K x 1 Minimize :

C ONDITIONS FOR SOLVABILITY Optimal (u, v) satisfies Lucas-Kanade equation When is This Solvable? A T A should be invertible A T A should not be too small due to noise –eigenvalues 1 and 2 of A T A should not be too small A T A is solvable when there is no aperture problem 27

Hessian Matrix I x = 0 I y = 0 M = 0 Non invertible I x = 0 I y = k M = 0 0 k 2 Non invertible I x = k I y = 0 M = k Non invertible I x = k 1 I y = k 2 RM = k Non invertible k 1, k 2 correlated (R = rotation) I x = k 1 I y = k 2 M = k k 2 2 invertible k 1 * k 2 = 0 28

E XAMPLE : 29

E DGE – large gradients, all the same – large  1, small 2 30

L OW TEXTURE REGION – gradients have small magnitude – small  1, small 2 31

H IGH TEXTURED REGION – gradients are different, large magnitudes – large  1, large 2 32

L UCAS -K ANADE Advantages easy and fast calculation accurate time derivatives Disadvantages errors on boundaries 33

E RRORS IN L UKAS -K ANADE What are the potential causes of errors in this procedure? Suppose A T A is easily invertible Suppose there is not much noise in the image When our assumptions are not valid Brightness constancy is not satisfied The motion is not small 34

H ORN & S CHUNCK Berthold Horn 35

Horn and Schunk proposed an algorithm to solve the aperture problem by using a Global method with smoothness constraint. Minimize a global energy functional with calculus of variations We back to the function And they proposed another solution to find u and v. H ORN -S CHUNCK ALGORITHM 36

H ORN -S CHUNCK ALGORITHM They assumed that in most of the pictures, the velocity of the objects do not change radically between adjacent pixels ! Here our new equation and we want to find the minimum : 37

H ORN -S CHUNCK ALGORITHM 38

H ORN -S CHUNCK ALGORITHM Iterative scheme The superscript k denotes the current iteration The solution depends on the neighboring values of the flow field, it must be repeated once the neighbors have been updated. 39

R EVISITING THE SMALL MOTION ASSUMPTION Is this motion small enough? Probably not—it’s much larger than one pixel (2 nd order terms dominate) How might we solve this problem? 40

A LIASING Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity. I.e., how do we know which ‘correspondence’ is correct? nearest match is correct (no aliasing) nearest match is incorrect (aliasing) coarse-to-fine estimation To overcome aliasing: coarse-to-fine estimation. actual shift estimated shift 41

R EDUCE THE RESOLUTION ! 42

image I image H Gaussian pyramid of image HGaussian pyramid of image I image I image H u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels C OARSE - TO - FINE OPTICAL FLOW ESTIMATION 43

image I image J Gaussian pyramid of image HGaussian pyramid of image I image I image H C OARSE - TO - FINE OPTICAL FLOW ESTIMATION run iterative L-K warp & upsample

I TERATIVE R EFINEMENT Iterative Lukas-Kanade Algorithm 1.Estimate velocity at each pixel by solving Lucas- Kanade equations 2.Warp I(t-1) towards I(t) using the estimated flow field - use image warping techniques 3.Repeat until convergence 45

O PTICAL F LOW R ESULTS 46

O PTICAL F LOW R ESULTS 47

L AYERED M OTION 48

W HAT IS L AYERED M OTION ? A system for representing moving images with sets of overlapping layers 49

W HY D O WE NEED IT ? Smooth Edges Reduces blurring – Each layer motion can be smooth 50

51

W HY D O WE NEED IT ? Smooth Edges Reduces blurring – Each layer motion can be smooth Used for video segmentation 52

V IDEO S EGMENTATION 53

H OW D O WE DO IT ? Edward Adelson and John Wang (1994) 54

A DELSON AND W ANG High Level – divide the image into three maps: Intensity Map Alpha Map (transparency) Velocity Map 55

A DELSON AND W ANG 56

ADELSON AND W ANG The garden scene: 57

ADELSON AND W ANG Divide the frame into non-overlapping patches 58

ADELSON AND W ANG Calculate the velocity with a coarse to fine optical flow algorithm 59

ADELSON AND W ANG Find the affine motion estimation, why? We assume that most of the motions in the world we can describe by affine transformation:  Translation  Rotation  Zoom  Shear 60

ADELSON AND W ANG 61

ADELSON AND W ANG One way to calculate them is by defining matching points between two images and calculating the affine transformation matrix 62

ADELSON AND W ANG 63

ADELSON AND W ANG 64

ADELSON AND W ANG 65

ADELSON AND W ANG 66

ADELSON AND W ANG 67

68

ADELSON AND W ANG The reliability of the hypothesis can be measured by it’s residual error (let’s skip it’s calculation) Patches with a Hypotheses with a residual error greater than a specified threshold are eliminated from the calculation (large residual error usually indicates that the patch contains object boundaries in it) 69

ADELSON AND W ANG Patches hypotheses with a residual error smaller than a defined threshold, and similar affine transformations we want to group together 70

ADELSON AND W ANG Divide the patches into clusters using k- means (remember the segmentation lecture) 71

ADELSON AND W ANG Now we have regions with similar affine transformation We want to verify our affine transformation vs. the actual movement 72

ADELSON AND W ANG 73

ADELSON AND W ANG 74

ADELSON AND W ANG 75

ADELSON AND W ANG We iterate to minimize the distortion Conversion is obtained when only a few pixels are reassigned or if we reached the max number of iterations. 76

ADELSON AND W ANG The Layers representation: 77

ADELSON AND W ANG In the last phase we warp the layers together, using a median filter to overcome slight changes in the intensity 78

ADELSON AND W ANG Final Result: 79

Good Job Adelson and Wang! But can we do better?? 80

L AYERED M OTION O PTIMIZATIONS Optimizations by Weiss and Adelson (1996) Automatically define the optimal number of layers Replace Affine motion model with smooth regularized motion models 81

L AYERED M OTION O PTIMIZATIONS Optimizations by Baker, Szeliski and Anandan (1998): Use homographies instead of affine motion estimation 82

83

L AYERED M OTION O PTIMIZATIONS Optimizations by Baker, Szeliski and Anandan (1998) – continue: Optimize The layers colors and opacity by minimizing the discrepancies between the re-synthesized and observed motion sequence 84

L AYERED M OTION O PTIMIZATIONS Baker, Szeliski and Anandan (1998) – Layers assignment: Adelson and Wang– Layers assignment: 85

D EMO TIME ! 86

S UMMARY Motions Estimation Algorithms Optical Flow The Aperture Problem Lucas and Kanade Algorithm Horn and Schunck Algorithm Coarse-to-fine Layered Motion Estimation (Adelson and Wang) 87

R EFERENCES Books: Computer Vision: Algorithms and Applications, Richard Szeliski Introductory Techniques for 3D Computer Vision, Emanuele Trucco, Alessandro Verri 88

R EFERENCES Papers: Layered Representation for Motion analysis, John Y.A. Wang & Edward H. Adelson hunck_method anade_method B%20L9%20Optical%20Flow.ppt 89

R EFERENCES Youtube: n=3&start=137&end=242&autoplay=0&hl=en_U S&rel=0 90