3D Imaging Motion

Motion Extract visual information from image sequences Assume illumination conditions do not change Differences in image Camera moves (Some) Objects in a scene move Both camera and objects move

Difference from stereo Stereo from multiple views Images taken at the same time moment Motion Images taken in small time intervals Differences Camera position might not be known Changes are smaller Not necessary caused by a single 3D rigid transform Rotation and Translation Multiple objects

Why analyze motion Understand scene structure Objects, Distance, Velocities Understand scene properties without a priori knowledge on the world

Random Dot

Time to Collision v An object of height L moves with constant velocity v: At time t=0 the object is at: D(0) = Do At time t it is at D(t) = Do – vt It will crash with the camera at time: D(t) = Do – vt = 0 t = Do/v L L l(t) f t t=0 Do D(t)

Time to Collision The image of the object has size l(t): v L L Taking derivative wrt time: l(t) f t t=0 Do D(t)

Time to Collision v L L l(t) f t t=0 Do And their ratio is: D(t)

Time to Collision v Can be directly measured from image L L f t t=0 Do l(t) f t t=0 Do And time to collision: D(t) Can be found, without knowing L or Do or v !!

Parallax Parallax=apparent change in position when viewed from different lines of sights Closer objects have larger parallax

Depth cues

Kinetic Depth Effect

Motion Field 3D velocity field 2D Motion field Each point has a velocity vector 2D Motion field Projection of these 3D velocity vector onto image plane

2D Motion Field

2D Optical Flow Apparent motion of image brightness pattern

Assuming that illumination does not change: Image changes are due to the RELATIVE MOTION between the scene and the camera. There are 3 possibilities: Camera still, moving scene Moving camera, still scene Moving camera, moving scene

Motion Analysis Problems Correspondence Problem Track corresponding elements across frames Reconstruction Problem Given a number of corresponding elements, and camera parameters, what can we say about the 3D motion and structure of the observed scene? Segmentation Problem What are the regions of the image plane which correspond to different moving objects?

Motion Field (MF) The MF assigns a velocity vector to each pixel in the image. These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene The MF can be thought as the projection of the 3D velocities on the image plane.

2D Motion Field and 2D Optical Flow Motion field: projection of 3D motion vectors on image plane Optical flow field: apparent motion of brightness patterns We equate motion field with optical flow field

2 Cases Where this Assumption Clearly is not Valid (a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not. (b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not. (a) (b)

What is Meant by Apparent Motion of Brightness Pattern? The apparent motion of brightness patterns is an awkward concept. It is not easy to decide which point P' on a contour C' of constant brightness in the second image corresponds to a particular point P on the corresponding contour C in the first image.

Aperture Problem (a) (b) (a) Line feature observed through a small aperture at time t. (b) At time t+t the feature has moved to a new position. It is not possible to determine exactly where each point has moved. From local image measurements only the flow component perpendicular to the line feature can be computed. Normal flow: Component of flow perpendicular to line feature.

Brightness Constancy Equation Let P be a moving point in 3D: At time t, P has coords (X(t),Y(t),Z(t)) Let p=(x(t),y(t)) be the coords. of its image at time t. Let E(x(t),y(t),t) be the brightness at p at time t. Brightness Constancy Assumption: As P moves over time, E(x(t),y(t),t) remains constant.

Brightness Constraint Equation short:

Brightness Constancy Equation Taking derivative wrt time:

Brightness Constancy Equation Let (Frame spatial gradient) (optical flow) (derivative across frames) and

Brightness Constancy Equation Becomes: The OF is CONSTRAINED to be on a line !

Normal Motion/Aperture Problem

Barber Pole Illusion A: u and v are unknown, 1 equation

Solving the aperture problem How to get more equations for a pixel? Basic idea: impose additional constraints most common is to assume that the flow field is smooth locally one method: pretend the pixel’s neighbors have the same (u,v) If we use a 5x5 window, that gives us 25 equations per pixel!

Constant flow The summations are over all pixels in the K x K window Prob: we have more equations than unknowns Solution: solve least squares problem minimum least squares solution given by solution (in d) of: The summations are over all pixels in the K x K window

Taking a closer look at (ATA) This is the same matrix we used for corner detection!

Taking a closer look at (ATA) The matrix for corner detection: is singular (not invertible) when det(ATA) = 0 But det(ATA) = Õ li = 0 -> one or both e.v. are 0 Aperture Problem ! One e.v. = 0 -> no corner, just an edge Two e.v. = 0 -> no corner, homogeneous region

A Pattern of Hajime Ouchi

Copyright A.Kitaoka 2003

Reconstruction

Rotation Represents a 3D rotation of the points in P.

First, look at 2D rotation (easier) Matrix R acts on points by rotating them. Also, RRT = Identity. RT is also a rotation matrix, in the opposite direction to R.

Simple 3D Rotation Rotation about z axis. Rotates x,y coordinates. Leaves z coordinates fixed.

Full 3D Rotation Any rotation can be expressed as combination of three rotations about three axes. Rows (and columns) of R are orthonormal vectors. R has determinant 1 (not -1).

3D Motion Displacement model Velocity model Alper Yilmaz, Fall 2004 UCF

Displacement Model and Its Projection onto Image Space Perspective projection

Velocity Model in 3D Optical Flow Alper Yilmaz, Fall 2004 UCF

Velocity Model in 3D Optical Flow rotational velocities translational velocities 3D optical flow cross product Alper Yilmaz, Fall 2004 UCF

Velocity Model in 2D Perspective projection