Optic Flow and Motion Detection Cmput 498/613 Martin Jagersand Readings: Papers and chapters on 613 web page

Image motion Somehow quantify the frame-to-frame differences in image sequences.Somehow quantify the frame-to-frame differences in image sequences. 1.Image intensity difference. 2.Optic flow dim image motion computation

Motion is used to: Attention: Detect and direct using eye and head motionsAttention: Detect and direct using eye and head motions Control: Locomotion, manipulation, toolsControl: Locomotion, manipulation, tools Vision: Segment, depth, trajectoryVision: Segment, depth, trajectory

Small camera re-orientation Note: Almost all pixels change!

MOVING CAMERAS ARE LIKE STEREO Locations of points on the object (the “structure”) The change in spatial location between the two cameras (the “motion”)

Classes of motion Still camera, single moving objectStill camera, single moving object Still camera, several moving objectsStill camera, several moving objects Moving camera, still backgroundMoving camera, still background Moving camera, moving objectsMoving camera, moving objects

The optic flow field Vector field over the image:Vector field over the image: [u,v] = f(x,y), u,v = Vel vector, x,y = Im pos [u,v] = f(x,y), u,v = Vel vector, x,y = Im pos FOE, FOC Focus of Expansion, ContractionFOE, FOC Focus of Expansion, Contraction

Optic/image flow Assumption: Image intensities from object points remain constant over time.Assumption: Image intensities from object points remain constant over time.

Taylor expansion of intensity variation Keep linear terms Keep linear terms Use constancy assumption and rewrite:Use constancy assumption and rewrite: Notice: Linear constraint, but no unique solutionNotice: Linear constraint, but no unique solution

Aperture problem Rewrite as dot productRewrite as dot product Each pixel gives one equation in two unknowns:Each pixel gives one equation in two unknowns: f. n = k Notice: Can only detect vectors normal to gradient directionNotice: Can only detect vectors normal to gradient direction The motion of a line cannot be recovered using only local informationThe motion of a line cannot be recovered using only local ðñ á îx îy òó = r Im á îx îy òó f n f ît

Aperture problem 2

The flow continuity constraint Flows of nearby patches are nearly equalFlows of nearby patches are nearly equal Two equations, two unknowns:Two equations, two unknowns: f. n 1 = k 1 f. n 2 = k 2 Unique solution exists, provided n 1 and n 2 not parallelUnique solution exists, provided n 1 and n 2 not parallel f n f f n f

Sensitivity to error n 1 and n 2 might be almost paralleln 1 and n 2 might be almost parallel Tiny errors in estimates of k’s or n’s can lead to huge errors in the estimate of fTiny errors in estimates of k’s or n’s can lead to huge errors in the estimate of f f n f f n f

Sometimes the continuity constraint is false

Using several points Typically solve for motion in 2x2, 4x4 or larger patches.Typically solve for motion in 2x2, 4x4 or larger patches. Over determined equation system:Over determined equation system: dI = Mu dI = Mu Can be solved in e.g. least squares sense using matlab u = M\dICan be solved in e.g. least squares sense using matlab u = M\dI

3-6D Optic flow Generalize to many freedooms (DOFs)Generalize to many freedooms (DOFs)

All 6 freedoms X Y Rotation Scale Aspect Shear

Flow vectors Norbert’s trick: Use an mpeg-card to speed up motion computationNorbert’s trick: Use an mpeg-card to speed up motion computation

Application: mpeg compression

Other applications: Recursive depth recovery: Kostas Danilidis and Jane MulliganRecursive depth recovery: Kostas Danilidis and Jane Mulligan Motion control (we will cover)Motion control (we will cover) SegmentationSegmentation TrackingTracking

Lab: Purpose:Purpose: –Hands on optic flow experience –Simple translation tracking Posted on www page.Posted on www page. Prepare through readings: (on-line papers.Prepare through readings: (on-line papers. Lab tutorial: How to capture.Lab tutorial: How to capture. Questions: or course newsgroupQuestions: or course newsgroup Due 1 week.Due 1 week.

Organizing different kinds of motion Two ideas: 1.Encode different motion structure in the M- matrix. Examples in 4,6,8 DOF image plane tracking. 3D a bit harder. 2.Attempt to find a low dimensional subspace for the set of motion vectors. Can use e.g. PCA

Remember: The optic flow field Vector field over the image:Vector field over the image: [u,v] = f(x,y), u,v = Vel vector, x,y = Im pos [u,v] = f(x,y), u,v = Vel vector, x,y = Im pos FOE, FOC Focus of Expansion, ContractionFOE, FOC Focus of Expansion, Contraction

Remember last lecture: Solving for the motion of a patchSolving for the motion of a patch Over determined equation system: Im t = Mu Im t = Mu Can be solved in e.g. least squares sense using matlab u = M\Im tCan be solved in e.g. least squares sense using matlab u = M\Im t t t+1

3-6D Optic flow Generalize to many freedooms (DOFs)Generalize to many freedooms (DOFs) Im = Mu

Know what type of motion (Greg Hager, Peter Belhumeur) u’ i = A u i + d E.g. Planar Object => Affine motion model: I t = g(p t, I 0 )

Mathematical Formulation Define a “warped image” gDefine a “warped image” g –f(p,x) = x’ (warping function), p warp parameters –I(x,t) (image a location x at time t) –g(p,I t ) = (I(f(p,x 1 ),t), I(f(p,x 2 ),t), … I(f(p,x N ),t))’ Define the Jacobian of warping functionDefine the Jacobian of warping function –M(p,t) = Model Model – I 0 = g(p t, I t ) (image I, variation model g, parameters p) –  I = M(p t, I t )  p  (local linearization M) Compute motion parametersCompute motion parameters –  p = (M T M) -1 M T  I where M = M(p t,I t )

Planar 3D motion From geometry we know that the correct plane- to-plane transform isFrom geometry we know that the correct plane- to-plane transform is 1.for a perspective camera the projective homography 2.for a linear camera (orthographic, weak-, para- perspective) the affine warp

Planar Texture Variability 1 Affine Variability Affine warp functionAffine warp function Corresponding image variabilityCorresponding image variability Discretized for imagesDiscretized for images

On The Structure of M u’ i = A u i + d Planar Object + linear (infinite) camera -> Affine motion model X Y Rotation Scale Aspect Shear

Planar Texture Variability 2 Projective Variability Homography warpHomography warp Projective variability:Projective variability: Where,Where, and and

Planar motion under perspective projection Perspective plane-plane transforms defined by homographiesPerspective plane-plane transforms defined by homographies

Planar-perspective motion 3 In practice hard to compute 8 parameter model stably from one image, and impossible to find out-of plane variationIn practice hard to compute 8 parameter model stably from one image, and impossible to find out-of plane variation Estimate variability basis from several images:Estimate variability basis from several images: Computed Estimated Computed Estimated

Another idea Black, Fleet) Organizing flow fields Express flow field f in subspace basis mExpress flow field f in subspace basis m Different “mixing” coefficients a correspond to different motionsDifferent “mixing” coefficients a correspond to different motions

Example: Image discontinuities

Mathematical formulation Let: Mimimize objective function: =Where Motion basis Robust error norm

Experiment Moving camera 4x4 pixel patches4x4 pixel patches Tree in foreground separates wellTree in foreground separates well

Experiment: Characterizing lip motion Very non-rigid!Very non-rigid!

Summary Three types of visual motion extractionThree types of visual motion extraction 1.Optic (image) flow: Find x,y – image velocities 2.3-6D motion: Find object pose change in image coordinates based more spatial derivatives (top down) 3.Group flow vectors into global motion patterns (bottom up) Visual motion still not satisfactorily solved problemVisual motion still not satisfactorily solved problem

Sensing and Perceiving Motion in Humans and Animals Martin Jagersand

How come perceived as motion? Im = sin(t)U5+cos(t)U6Im = f1(t)U1+…+f6(t)U6

Counterphase sin grating Spatio-temporal patternSpatio-temporal pattern –Time t, Spatial x,y

Counterphase sin grating Spatio-temporal patternSpatio-temporal pattern –Time t, Spatial x,y Rewrite as dot product: = + Result: Standing wave is superposition of two moving waves

Analysis: Only one term: Motion left or rightOnly one term: Motion left or right Mixture of both: Standing waveMixture of both: Standing wave Direction can flip between left and rightDirection can flip between left and right

Reichardt detector QT movieQT movie

Several motion models Gradient: in Computer VisionGradient: in Computer Vision Correlation: In bio visionCorrelation: In bio vision Spatiotemporal filters: Unifying modelSpatiotemporal filters: Unifying model

Spatial response: Gabor function Definition:Definition:

Temporal response: Adelson, Bergen ’85 Note: Terms from taylor of sin(t) Spatio-temporal D=D s D t

Receptor response to Counterphase grating Separable convolutionSeparable convolution

Simplified: For our grating: (Theta=0)For our grating: (Theta=0) Write as sum of components:Write as sum of components: = exp(…)*(acos… + bsin…) = exp(…)*(acos… + bsin…)

Space-time receptive field

Combined cells Spat: Temp:Spat: Temp: Both:Both: Comb:Comb:

Result: More directionally specific responseMore directionally specific response

Energy model: Sum odd and even phase componentsSum odd and even phase components Quadrature rectifierQuadrature rectifier

Adaption: Motion aftereffect

Where is motion processed?

Higher effects:

Equivalence: Reich and Spat

Conclusion Evolutionary motion detection is importantEvolutionary motion detection is important Early processing modeled by Reichardt detector or spatio-temporal filters.Early processing modeled by Reichardt detector or spatio-temporal filters. Higher processing poorly understoodHigher processing poorly understood