3D Rigid/Nonrigid RegistrationRegistration 1)Known features, correspondences, transformation model – feature basedfeature based 2)Specific motion type, unknown correspondences – feature basedfeature based 3) Known transformation model, unknown correspondences – region basedregion based 4) Specific motion model – feature basedfeature based 5) Unknown motion model, unknown correspondences – region based

Visual Motion Jim Rehg (G.Tech)

Motion (Displacement) of Environment Image plane Scene Flow Motion Field Visual motion results from the displacement of the scene with respect to a fixed camera (or vice-versa). Motion field is the 2-D velocity field that results from a projection of the 3-D scene velocities

Examples of Visual Motion

Applications of Motion Analysis Visual tracking Structure recovery Robot (vehicle) navigation

Applications of Motion Analysis Visual tracking Structure recovery Robot (vehicle) navigation

Motion Segmentation Where are the independently moving objects (and how many are there)?

Optical Flow 2-D velocity field describing the apparent motion in an image sequence A vector at each pixel indicates its motion (between a pair of frames). Ground truthHorn and Schunk

Optical Flow and Motion Field In general the optical flow is an approximation to the motion field. When the scene can be segmented into rigidly moving objects (for example) the relationship between the two can be made precise. We can always think of the optical flow as summarizing the temporal change in an image sequence.

Computing Optical Flow Courtesy of Michael Black

Cost Function for Optical Flow Courtesy of Michael Black

Lucas-Kanade Method Brute-force minimization of SSD error can be inefficient and inaccurate Many redundant window evaluations Answer is limited to discrete u, v pairs

Lucas-Kanade Method Problems with brute-force minimization of SSD error Many redundant window evaluations Answer is limited to discrete u, v pairs Related to Horn-Schunk optical flow equations Several key innovations Early, successful use of patch-based model in low-level vision. Today these models are used everywhere. Formulation of vision problem as non-linear least squares optimization, a trend which continues to this day.

Optical Flow Estimation

Optical Flow Constraint I_t is one-to-one in the first Iteration and changes as u,v changes

Optimization

Quality of Image Patch Eigenvalues of the matrix contain information about local image structure Both eigenvalues (close to) zero: Uniform area One eigenvalue (close to) zero: Edge No eigenvalues (close to) zero: Corner

Contributions of Lucas-Kanade Basic idea of patch or template is very old (goes back at least to Widrow) But in practice patch models have worked much better than the alternatives: Point-wise differential equations with smoothness Edge-based descriptions Patches provide a simple compact enforcement of spatial continuity and support (robust) least-squares estimators.

Apparent Motion Apparent motion of objects on the image plane Caution required!! Consider a perfectly uniform sphere that is rotating but no change in the light direction Optic flow is zero Perfectly uniform sphere that is stationary but the light is changing Optic flow exists Hope – apparent motion is very close to the actual motion Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

Aperture Problem 1)The aperture problem arises due to uniformly colored surfaces in the scene. In the absence of strong lighting effects, a uniform surface in the scene appears nearly uniform in the projection. It is then impossible to determine correspondences within these regions. 2) We are able to only measure the component Of the optic flow that is in the direction of the Intensity gradient. (Unable to measure component In the tangential direction, edge).

Aperture Problem We can measure Terms that can be measured Terms to be computed Number of equations - 1 The component of the motion field that is orthogonal to the spatial image gradient is not constrained by the image brightness constancy assumption Intuitively The component of the flow in the gradient direction is determined The component of the flow parallel to an edge is unknown Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

Different physical motion but same measurable motion within a fixed window