Optical flow , A tutorial of the paper: G. Farneback , “Two-frame Motion Estimation based on Polynomial Expansion”, 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29 – July 2, 2003. KH Wong Optical Flow v.5a (beta)

Introduction Optical flow : Based on two frames sampled at different slight different time, find the change of position of each pixel in the first image to the second image. Reference: G. Farneback , “Two-frame Motion Estimation based on Polynomial Expansion”, 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29 – July 2, 2003. Optical Flow v.5a

Problem definition and method Input: Two images Output : motion vectors (v) and xx( c )? of each pixel in the first image. The Farneback method [1] is fast and linear. Method: Step1: Approximate the image using a polynomial Step2: Use the model of polynomial to approximate the motion for each pixel. Optical Flow v.5a

Example Image1 Image2 Optical flow result , vectors showing the direction and intensity of flow of each pixel. Plotting all vectors is too messy, only plot one vector in a window of 10x10 pixels. Optical Flow v.5a

Ideal and method x=[x,y]’ x1 (just show one dimension of x) We use a polynomial (with parameters A,b,c) to model the intensity change of an image (small window). Assume x is dimension2 here, but it can be larger. In image f1 (x) is taken at t , and f2 (x) at t+dt f1 (x)=xTA1x+b1x+c1 x is 2x1, A1 is 2x2, b1 is 2x1, c1 is 1x1 Assume f2 is f1 with a global displacement by d=[d1,d2]’ (shape no change, just displacement) f2 (x)=xTA2x+b2x+c2 By comparing f1 and f2, we get A2=A1, b2=b1-2A1d C2=dTA1d-bT1d+c1 We observe that (when A1 is non-singular , i.e. invertible) 2A1d=-(b2-b1) d=-(1/2)(A1)-1(b2-b1) So if we can measure f1 and f2, can may find d=[d1,d2]’. d is the 2-D displacement and hence the flow of the image pixel. Intensity of image f1(x)) f2(x) d x Image 1, at t Image 2, at t=dt Use a polynomial (with parameters A,b,c) to model the intensity change of an image (assume a small window). Optical Flow v.5a

Step: 1a Model an image using a polynomial For a polynomial with parameters A,b,c : x= is 2x1 A is 2x2 b is 2x1 C is 1x1 d is 2x1 Optical Flow v.5a

Modeling images using polynomials Read the Ph.D thesis Orientation and Velocity, Estimation , Ph.D https://www.facebook.com/download/333523813505207/farneback99-phd.pdf (page22 the example) and also the code make_Abc_fast.m in Code, https://github.com/GunnarFarneback/spatial_domain_toolbox Optical Flow v.5a

step1b Assume all pixels in a small window I() behave the same. If we can measure A1, b1, c1 from f1 for each pixel in I() ; and A2, A2 c2 from f2 we can calculate d. Pointwise is too noisy Sum over an window of I() Optical Flow v.5a

Step2 : find displacement d=[dx,dy]’ (=optical flow) We assume d is not a constant over the window I(), but a function according to its position x,y based on affine transformation. Affine model of d S is 2x8 ST is 8x2 p is 8x1 d is 2x1 wi= is 1x1 0 A , p ,b can be measured. Put them in (19) and minimize (19), or set it to [0,0]’. We can solve for p using (20), if p is found d can be found by (15) Optical Flow v.5a

Some modification Optical Flow v.5a

Result Image: yos2 Image: yos8 Optical flow (averaged) using a sequence of 15 pictures. Plotting all vectors is too messy, only plot one vector in a window of 10x10 pixels. Using the code https://github.com/GunnarFarneback/spatial_domain_toolbox Optical Flow v.5a Image: yos16

Summary Studied a linear method to find the optical flow based on two frames of image. Optical Flow v.5a

References G. Farneback , “Two-frame Motion Estimation based on Polynomial Expansion”, 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29 – July 2, 2003. Publications of G. Farneback , http://lmi.bwh.harvard.edu/papers/gunnar.html G. Farneback , Spatial Domain Methods for Orientation and Velocity, Estimation , Ph.D https://www.facebook.com/download/333523813505207/farneback99-phd.pdf Code, https://github.com/GunnarFarneback/spatial_domain_toolbox Optical Flow v.5a