Affine structure from motion

Slides:



Advertisements
Similar presentations
Structure from motion.
Advertisements

The fundamental matrix F
3D reconstruction.
MASKS © 2004 Invitation to 3D vision Lecture 7 Step-by-Step Model Buidling.
N-view factorization and bundle adjustment CMPUT 613.
Self-calibration.
Lecture 11: Structure from Motion
Camera calibration and epipolar geometry
Structure from motion.
Algorithms & Applications in Computer Vision
Projective structure from motion
Stanford CS223B Computer Vision, Winter 2005 Lecture 11: Structure From Motion 2 Sebastian Thrun, Stanford Rick Szeliski, Microsoft Hendrik Dahlkamp and.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Today Feature Tracking Structure from Motion on Monday (1/29)
Stanford CS223B Computer Vision, Winter 2007 Lecture 8 Structure From Motion Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.
Computer Vision Projective structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Structure from motion. Multiple-view geometry questions Scene geometry (structure): Given 2D point matches in two or more images, where are the corresponding.
Uncalibrated Geometry & Stratification Sastry and Yang
Stanford CS223B Computer Vision, Winter 2006 Lecture 8 Structure From Motion Professor Sebastian Thrun CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado.
Multiple View Geometry
Many slides and illustrations from J. Ponce
Self-calibration Class 21 Multiple View Geometry Comp Marc Pollefeys.
CAU Kiel DAGM 2001-Tutorial on Visual-Geometric 3-D Scene Reconstruction 1 The plan for today Leftovers and from last time Camera matrix Part A) Notation,
Computer Vision Optical Flow Marc Pollefeys COMP 256 Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black,
© 2003 by Davi GeigerComputer Vision October 2003 L1.1 Structure-from-EgoMotion (based on notes from David Jacobs, CS-Maryland) Determining the 3-D structure.
Previously Two view geometry: epipolar geometry Stereo vision: 3D reconstruction epipolar lines Baseline O O’ epipolar plane.
Computer Vision Structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …
Projected image of a cube. Classical Calibration.
CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.
CSE473/573 – Stereo Correspondence
Advanced Computer Vision Structure from Motion. Geometric structure-from-motion problem: using image matches to estimate: The 3D positions of the corresponding.
Camera parameters Extrinisic parameters define location and orientation of camera reference frame with respect to world frame Intrinsic parameters define.
Computer Vision Calibration Marc Pollefeys COMP 256 Read F&P Chapter 2 Some slides/illustrations from Ponce, Hartley & Zisserman.
CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.
Computer Vision Optical Flow Marc Pollefeys COMP 256 Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black,
3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
Euclidean cameras and strong (Euclidean) calibration Intrinsic and extrinsic parameters Linear least-squares methods Linear calibration Degenerate point.
Brief Introduction to Geometry and Vision
1 Preview At least two views are required to access the depth of a scene point and in turn to reconstruct scene structure Multiple views can be obtained.
Projective cameras Motivation Elements of Projective Geometry Projective structure from motion Planches : –
Structure from Motion Computer Vision CS 143, Brown James Hays 11/18/11 Many slides adapted from Derek Hoiem, Lana Lazebnik, Silvio Saverese, Steve Seitz,
Projective cameras Trifocal tensors Euclidean/projective SFM Self calibration Line geometry Purely projective cameras Je ne suis pas la la semaine prochaine.
CSCE 643 Computer Vision: Structure from Motion
Multiview Geometry and Stereopsis. Inputs: two images of a scene (taken from 2 viewpoints). Output: Depth map. Inputs: multiple images of a scene. Output:
Geometric Camera Models
Geometry of Multiple Views
Affine Structure from Motion
EECS 274 Computer Vision Affine Structure from Motion.
1 Chapter 2: Geometric Camera Models Objective: Formulate the geometrical relationships between image and scene measurements Scene: a 3-D function, g(x,y,z)
776 Computer Vision Jan-Michael Frahm & Enrique Dunn Spring 2013.
Structure from Motion ECE 847: Digital Image Processing
MASKS © 2004 Invitation to 3D vision Uncalibrated Camera Chapter 6 Reconstruction from Two Uncalibrated Views Modified by L A Rønningen Oct 2008.
Uncalibrated reconstruction Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration.
Structure from Motion Paul Heckbert, Nov , Image-Based Modeling and Rendering.
Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches : –
Projective cameras The end of projective SFM Euclidean upgrades Line geometry Purely projective cameras Présentations mercredi 26 mai de 9h30 à midi, salle.
EECS 274 Computer Vision Projective Structure from Motion.
11/25/03 3D Model Acquisition by Tracking 2D Wireframes Presenter: Jing Han Shiau M. Brown, T. Drummond and R. Cipolla Department of Engineering University.
Reconstruction of a Scene with Multiple Linearly Moving Objects Mei Han and Takeo Kanade CISC 849.
Motion Segmentation with Missing Data using PowerFactorization & GPCA
René Vidal and Xiaodong Fan Center for Imaging Science
Parameter estimation class 5
Epipolar geometry.
Structure from motion Input: Output: (Tomasi and Kanade)
3D reconstruction class 11
Advanced Computer Vision
Uncalibrated Geometry & Stratification
Structure from motion Input: Output: (Tomasi and Kanade)
Presentation transcript:

Affine structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …

Last time: Optical Flow Ixu Ix u Ixu=- It It Aperture problem: two solutions: - regularize (smoothness prior) constant over window (i.e. Lucas-Kanade) Coarse-to-fine, parametric models, etc…

Tentative class schedule Jan 16/18 - Introduction Jan 23/25 Cameras Radiometry Jan 30/Feb1 Sources & Shadows Color Feb 6/8 Linear filters & edges Texture Feb 13/15 Multi-View Geometry Stereo Feb 20/22 Optical flow Project proposals Feb27/Mar1 Affine SfM Projective SfM Mar 6/8 Camera Calibration Silhouettes and Photoconsistency Mar 13/15 Springbreak Mar 20/22 Segmentation Fitting Mar 27/29 Prob. Segmentation Project Update Apr 3/5 Tracking Apr 10/12 Object Recognition Apr 17/19 Range data Apr 24/26 Final project

AFFINE STRUCTURE FROM MOTION The Affine Structure from Motion Problem Elements of Affine Geometry Affine Structure from Motion from two Views A Geometric Approach Affine Epipolar Geometry An Algebraic Approach Affine Structure from Motion from Multiple Views From Affine to Euclidean Images Structure from motion of multiple and deforming object Reading: Chapter 12.

Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990).  1990 Optical Society of America. Given m pictures of n points, can we recover the three-dimensional configuration of these points? the camera configurations? (structure) (motion)

Orthographic Projection Parallel Projection

Weak-Perspective Projection Paraperspective Projection

The Affine Structure-from-Motion Problem Given m images of n fixed points P we can write j Problem: estimate the m 2x4 matrices M and the n positions P from the mn correspondences p . i j ij 2mn equations in 8m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares!

The Affine Ambiguity of Affine SFM When the intrinsic and extrinsic parameters are unknown If M and P are solutions, i j So are M’ and P’ where i j and Q is an affine transformation.

Affine Spaces: (Semi-Formal) Definition

2 Example: R as an Affine Space

In General The notation is justified by the fact that choosing some origin O in X allows us to identify the point P with the vector OP. Warning: P+u and Q-P are defined independently of O!!

Barycentric Combinations Can we add points? R=P+Q NO! But, when we can define Note:

Affine Subspaces

Affine Coordinates Coordinate system for U: Coordinate system for Y=O+U: Affine coordinates: Coordinate system for Y: Barycentric coordinates:

When do m+1 points define a p-dimensional subspace Y of an n-dimensional affine space X equipped with some coordinate frame basis? Rank ( D ) = p+1, where Writing that all minors of size (p+2)x(p+2) of D are equal to zero gives the equations of Y.

Affine Transformations Bijections from X to Y that: map m-dimensional subspaces of X onto m-dimensional subspaces of Y; map parallel subspaces onto parallel subspaces; and preserve affine (or barycentric) coordinates. Bijections from X to Y that: map lines of X onto lines of Y; and preserve the ratios of signed lengths of line segments. 3 In E they are combinations of rigid transformations, non-uniform scalings and shears.

Affine Transformations II Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B). Given an affine transformation from X to Y, one can always write: When coordinate frames have been chosen for X and Y, this translates into:

Affine projections induce affine transformations from planes onto their images.

Affine Shape Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation y: X X such that X’ = y ( X ). Affine structure from motion = affine shape recovery. = recovery of the corresponding motion equivalence classes.

Geometric affine scene reconstruction from two images (Koenderink and Van Doorn, 1991).

Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990).  1990 Optical Society of America. (Koenderink and Van Doorn, 1991)

The Affine Epipolar Constraint Note: the epipolar lines are parallel.

Affine Epipolar Geometry

The Affine Fundamental Matrix where

An Affine Trick.. Algebraic Scene Reconstruction

The Affine Structure of Affine Images Suppose we observe a scene with m fixed cameras.. The set of all images of a fixed scene is a 3D affine space!

has rank 4!

From Affine to Vectorial Structure Idea: pick one of the points (or their center of mass) as the origin.

What if we could factorize D? (Tomasi and Kanade, 1992) Affine SFM is solved! Singular Value Decomposition We can take

From uncalibrated to calibrated cameras Weak-perspective camera: Calibrated camera: Problem: what is Q ? Note: Absolute scale cannot be recovered. The Euclidean shape (defined up to an arbitrary similitude) is recovered.

Reconstruction Results (Tomasi and Kanade, 1992) Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi and T. Kanade, Proc. IEEE Workshop on Visual Motion (1991).  1991 IEEE.

More examples Tomasi Kanade’92, Poelman & Kanade’94

More examples Tomasi Kanade’92, Poelman & Kanade’94

More examples Tomasi Kanade’92, Poelman & Kanade’94

Further Factorization work Factorization with uncertainty Factorization for indep. moving objects (now) Factorization for articulated objects (now) Factorization for dynamic objects (now) Perspective factorization (next week) Factorization with outliers and missing pts. (Irani & Anandan, IJCV’02) (Costeira and Kanade ‘94) (Yan and Pollefeys ‘05) (Bregler et al. 2000, Brand 2001) (Sturm & Triggs 1996, …) (Jacobs ‘97 (affine), Martinek & Pajdla‘01 Aanaes’02 (perspective))

Structure from motion of multiple moving objects

Structure from motion of multiple moving objects

Shape interaction matrix Shape interaction matrix for articulated objects looses block diagonal structure Costeira and Kanade’s approach is not usable for articulated bodies (assumes independent motions)

Articulated motion subspaces Motion subspaces for articulated bodies intersect (Yan and Pollefeys, CVPR’05) (Tresadern and Reid, CVPR’05) Joint (1D intersection) (joint=origin) (rank=8-1) Hinge (2D intersection) (hinge=z-axis) (rank=8-2) Exploit rank constraint to obtain better estimate Also for non-rigid parts if (Yan & Pollefeys, 06?)

Results Toy truck Student Segmentation Segmentation Intersection

Articulated shape and motion factorization (Yan and Pollefeys, 2006?) Automated kinematic chain building for articulated & non-rigid obj. Estimate principal angles between subspaces Compute affinities based on principal angles Compute minimum spanning tree

Structure from motion of deforming objects (Bregler et al ’00; Brand ‘01) Extend factorization approaches to deal with dynamic shapes

Representing dynamic shapes (fig. M.Brand) represent dynamic shape as varying linear combination of basis shapes

Projecting dynamic shapes (figs. M.Brand) Rewrite:

Dynamic image sequences One image: (figs. M.Brand) Multiple images

Dynamic SfM factorization? Problem: find J so that M has proper structure

Dynamic SfM factorization (Bregler et al ’00) Assumption: SVD preserves order and orientation of basis shape components

Results (Bregler et al ’00)

Dynamic SfM factorization (Brand ’01) constraints to be satisfied for M constraints to be satisfied for M, use to compute J hard! (different methods are possible, not so simple and also not optimal)

Non-rigid 3D subspace flow (Brand ’01) Same is also possible using optical flow in stead of features, also takes uncertainty into account

Results (Brand ’01)

Results (Brand ’01)

Results (Bregler et al ’01)

Next class: Projective structure from motion Reading: Chapter 13