1. 3D Imaging
Motion

2. Motion Extract visual information from image sequences
Assume illumination conditions do not change
Differences in image
Camera moves
(Some) Objects in a scene move
Both camera and objects move

3. Difference from stereo
Stereo from multiple views
Images taken at the same time moment
Motion
Images taken in small time intervals
Differences
Camera position might not be known
Changes are smaller
Not necessary caused by a single 3D rigid transform
Rotation and Translation
Multiple objects

4. Why analyze motion Understand scene structure
Objects, Distance, Velocities
Understand scene properties without a priori knowledge on the world

5. Random Dot

6. Time to Collision v
An object of height L moves with constant velocity v:
At time t=0 the object is at:
D(0) = Do
At time t it is at
D(t) = Do – vt
It will crash with the camera at time:
D(t) = Do – vt = 0
t = Do/v L L
l(t) f t
t=0 Do
D(t)

7. Time to Collision The image of the object has size l(t): v L L
Taking derivative wrt time:
l(t) f t
t=0 Do
D(t)

8. Time to Collision v L L
l(t) f t
t=0 Do
And their ratio is:
D(t)

9. Time to Collision v Can be directly measured from image L L f t t=0 Do
l(t) f t
t=0 Do
And time to collision:
D(t)
Can be found, without knowing L or Do or v !!

10. Parallax
Parallax=apparent change in position when viewed from different lines of sights
Closer objects have larger parallax

11. Depth cues

12. Kinetic Depth Effect

13. Motion Field 3D velocity field 2D Motion field
Each point has a velocity vector
2D Motion field
Projection of these 3D velocity vector onto image plane

14. 2D Motion Field

15. 2D Optical Flow
Apparent motion of image brightness pattern

16. Assuming that illumination does not change:
Image changes are due to the RELATIVE MOTION between the scene and the camera.
There are 3 possibilities:
Camera still, moving scene
Moving camera, still scene
Moving camera, moving scene

17. Motion Analysis Problems
Correspondence Problem
Track corresponding elements across frames
Reconstruction Problem
Given a number of corresponding elements, and camera parameters, what can we say about the 3D motion and structure of the observed scene?
Segmentation Problem
What are the regions of the image plane which correspond to different moving objects?

18. Motion Field (MF)
The MF assigns a velocity vector to each pixel in the image.
These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene
The MF can be thought as the projection of the 3D velocities on the image plane.

19. 2D Motion Field and 2D Optical Flow
Motion field: projection of 3D motion vectors on image plane
Optical flow field: apparent motion of brightness patterns
We equate motion field with optical flow field

20. 2 Cases Where this Assumption Clearly is not Valid
(a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not.
(b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not.
(a) (b)

21. What is Meant by Apparent Motion of Brightness Pattern?
The apparent motion of brightness patterns is an awkward concept. It is not easy to decide which point P' on a contour C' of constant brightness in the second image corresponds to a particular point P on the corresponding contour C in the first image.

22. Aperture Problem (a) (b)
(a) Line feature observed through a small aperture at time t.
(b) At time t+t the feature has moved to a new position. It is not possible to determine exactly where each point has moved. From local image measurements only the flow component perpendicular to the line feature can be computed.
Normal flow: Component of flow perpendicular to line feature.

23. Brightness Constancy Equation
Let P be a moving point in 3D:
At time t, P has coords (X(t),Y(t),Z(t))
Let p=(x(t),y(t)) be the coords. of its image at time t.
Let E(x(t),y(t),t) be the brightness at p at time t.
Brightness Constancy Assumption:
As P moves over time, E(x(t),y(t),t) remains constant.

24. Brightness Constraint Equation
short:

25. Brightness Constancy Equation
Taking derivative wrt time:

26. Brightness Constancy Equation
Let
(Frame spatial gradient)
(optical flow)
(derivative across frames)
and

27. Brightness Constancy Equation
Becomes:
The OF is CONSTRAINED to be on a line !

28. Normal Motion/Aperture Problem

29. Barber Pole Illusion
A: u and v are unknown, 1 equation

30. Solving the aperture problem
How to get more equations for a pixel?
Basic idea: impose additional constraints
most common is to assume that the flow field is smooth locally
one method: pretend the pixel’s neighbors have the same (u,v)
If we use a 5x5 window, that gives us 25 equations per pixel!

31. Constant flow The summations are over all pixels in the K x K window
Prob: we have more equations than unknowns
Solution: solve least squares problem
minimum least squares solution given by solution (in d) of:
The summations are over all pixels in the K x K window

32. Taking a closer look at (ATA)
This is the same matrix we used for corner detection!

33. Taking a closer look at (ATA)
The matrix for corner detection:
is singular (not invertible) when det(ATA) = 0
But det(ATA) = Õ li = 0 -> one or both e.v. are 0
Aperture Problem !
One e.v. = 0 -> no corner, just an edge
Two e.v. = 0 -> no corner, homogeneous region

34. A Pattern of Hajime Ouchi

35. Copyright A.Kitaoka 2003

36. Reconstruction

37. Rotation
Represents a 3D rotation of the points in P.

38. First, look at 2D rotation (easier)
Matrix R acts on points by rotating them.
Also, RRT = Identity. RT is also a rotation matrix, in the opposite direction to R.

39. Simple 3D Rotation Rotation about z axis.
Rotates x,y coordinates. Leaves z coordinates fixed.

40. Full 3D Rotation
Any rotation can be expressed as combination of three rotations about three axes.
Rows (and columns) of R are orthonormal vectors.
R has determinant 1 (not -1).

41. 3D Motion Displacement model Velocity model
Alper Yilmaz, Fall 2004 UCF

42. Displacement Model and Its Projection onto Image Space
Perspective projection

43. Velocity Model in 3D Optical Flow
Alper Yilmaz, Fall 2004 UCF

44. Velocity Model in 3D Optical Flow
rotational velocities
translational velocities
3D optical flow
cross product
Alper Yilmaz, Fall 2004 UCF

45. Velocity Model in 2D
Perspective projection