1. CSc83029 – 3-D Computer Vision/ Ioannis Stamos 3-D Computational Vision CSc 83029 Optical Flow & Motion The Factorization Method

2. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow & Motion  Finding the movement of scene objects from time-varying images.  Motion Field  Optical Flow  Computing Optical Flow

3. CSc83029 – 3-D Computer Vision/ Ioannis Stamos

4. Computing Time-to-Impact τ L l(t) v f D0D0 D 0 -vt l(t)=f L/D(t)

5. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Time-to-Impact τ L l(t) v f D0D0 D 0 -vt l(t)=f L/D(t) l(t) / l’(t) = τ Quantities measured from image sequence.

6. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Sudden change in viewing position/direction: Hard to compute motion field/optical flow.

7. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Views from a sequence of spatially close viewpoints: Motion Field/ Optical Flow

8. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Sub problems of Motion Analysis  Correspondence.  Reconstruction.  Segmentation.

9. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Field 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points. P V dt v dt p f Image plane Scene Point Velocity V=dr o /dt Image Velocity v =dr i /dt roro riri

10. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Field 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points. P V dt v dt p f Image plane roro riri Perspective projection Velocity of point P as a function of translation and rotation Motion field equations

11. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Field 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS:

12. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Field 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS: +

13. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Special Case 1: Pure Translation 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS: +

14. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Pure Translation: Radial Motion Field p0=(x0,y0)

15. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Pure Translation: Radial Motion Field 1.Tz < 0 : FOCUS OF EXPANSION. 2.Tz > 0: FOCUS OF CONTRACTION. 3.Tz = 0: PARALLEL MOTION FIELD. 4.Vanishing point (epipole) p0. p0=(x0,y0)

16. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Special Case 2: Moving Plane n P Plane moves: n,d are functions of time. Motion field=?

17. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Special Case 2: Moving Plane n P Plane moves: n,d are functions of time. Motion field=?

18. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Special Case 2: Moving Plane n P Plane moves: n,d are functions of time. Motion field=? Motion field: quadratic polynomial in (x,y,f) at any time t. The same motion field can be produced by 2 different planes undergoing 2 different 3-D motions.

19. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Parallax The relative motion field of two instantaneously coincident points does not depend on the rotational component of the motion… p0

20. CSc83029 – 3-D Computer Vision/ Ioannis Stamos The notion of Optical Flow Optical Flow: Estimation of the motion field from a sequence of images.

21. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

22. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow Image brightness constancy equation: E(x,y,t)=E(x+uδt, y+vδt, t+δt) or tt+δt (x,y) (x+uδt,y+vδt) u=δx/δt v=δy/δt

23. CSc83029 – 3-D Computer Vision/ Ioannis Stamos The Aperture Problem Image brightness constancy equation: E(x,y,t)=E(x+uδt, y+vδt, t+δt) Aperture problem 1 constraint 2 unknowns The component of the motion field in the direction orthogonal to the spatial image gradient is not constrained by the image brightness constancy equation.

24. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow At each point we know dE/dx dE/dy and dE/dt. How can we obtain dx/dt and dy/dt?

25. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Each provides one constraint Solution is

26. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Minimize Q

27. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Minimize Q =>Solve the linear system

28. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow global approach Minimize the error in the image brightness constancy constraint. [Schunck and Horn 81]

29. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow global approach Minimize the error in the image brightness constancy constraint. Minimize the deviation from smoothness of the motion vectors. [Schunck and Horn 81]

30. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow global approach Minimize the error in the image brightness constancy constraint. Minimize the deviation from smoothness of the motion vectors. Find solution by minimizing where lambda weights the smoothness term. [Schunck and Horn 81]

31. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

32. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

33. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

34. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

35. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow Solve for the 8 unknowns a, b, c, d, e, f, g and h. If the scene is planar the motion Is described by Using

36. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Tracking Rigid Bodies B B Random Sampling Algorithm Step 1: Find corners Step 2: Search for correspondence Step 3: Randomly choose small set of matches. Step 4: Estimate F matrix Step 5: Find total number of matches close to epipolar lines Step 6: Go to step 3 Step 7: Choose F with largest number of matches

37. CSc83029 – 3-D Computer Vision/ Ioannis Stamos

38. 1. Use multiple image stream to compute the information about camera motion and 3D structure of the scene 2. Tracking image features over time Tracked Features Structure and Motion Recovery from Video Original sequence From Jana Kosecka

39. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Structure and Motion Recovery from Video Computed model 3D coordinates of the feature points Original picture From Jana Kosecka

40. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Factorization Method FRAMES: i=1…N kiki jiji i

41. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Factorization Method kiki jiji i Pj FRAMES: i=1…N World Points: j=1…n (x ij,y ij )

42. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Factorization Method World Reference Frame kiki jiji i Pj FRAMES: i=1…N World Points: j=1…n (x ij,y ij ) Y X Z Ti

43. CSc83029 – 3-D Computer Vision/ Ioannis Stamos World Reference Frame kiki jiji i Pj FRAMES: i=1…N World Points: j=1…n (x ij,y ij ) Y X Z Ti ASSUMPTIONS: The camera model is orthographic! The positions of n image points have been tracked.

44. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Measurement Matrix 2*N (Frames) n points per frame 2*N (Frames) n points per frame Registered Measurement Matrix

45. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Factorization Method World Reference Frame kiki jiji i FRAMES: i=1…N World Points: j=1…n Y X Z Ti Pj 3D Centroid 2D Centroid

46. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Rank Theorem 2*N (Frames) n points per frame R: “Rotation Matrix” S: Shape Matrix

47. CSc83029 – 3-D Computer Vision/ Ioannis Stamos Rank Theorem 2*N (Frames) n points per frame R: “Rotation Matrix” S: Shape Matrix Rank of is 3.

48. CSc83029 – 3-D Computer Vision/ Ioannis Stamos The algorithm 2*N (Frames) n points per frame Decompose into R and S. Is the decomposition unique? Translation estimation?

49. CSc83029 – 3-D Computer Vision/ Ioannis Stamos