Camera calibration Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz,, Fred Pighin and Marc Pollefyes.

Slides:



Advertisements
Similar presentations
Single-view geometry Odilon Redon, Cyclops, 1914.
Advertisements

Last 4 lectures Camera Structure HDR Image Filtering Image Transform.
RANSAC and mosaic wrap-up cs129: Computational Photography James Hays, Brown, Fall 2012 © Jeffrey Martin (jeffrey-martin.com) most slides from Steve Seitz,
CS 691 Computational Photography Instructor: Gianfranco Doretto 3D to 2D Projections.
More Mosaic Madness : Computational Photography Alexei Efros, CMU, Fall 2011 © Jeffrey Martin (jeffrey-martin.com) with a lot of slides stolen from.
RANSAC and mosaic wrap-up cs195g: Computational Photography James Hays, Brown, Spring 2010 © Jeffrey Martin (jeffrey-martin.com) most slides from Steve.
Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C.
Computer vision: models, learning and inference
Chapter 6 Feature-based alignment Advanced Computer Vision.
CSc D Computer Vision – Ioannis Stamos 3-D Computer Vision CSc Camera Calibration.
Camera calibration and epipolar geometry
CS6670: Computer Vision Noah Snavely Lecture 17: Stereo
Announcements. Projection Today’s Readings Nalwa 2.1.
Lecture 5: Projection CS6670: Computer Vision Noah Snavely.
Structure from motion. Multiple-view geometry questions Scene geometry (structure): Given 2D point matches in two or more images, where are the corresponding.
Announcements Project 2 questions Midterm out on Thursday
Uncalibrated Epipolar - Calibration
Lecture 11: Structure from motion CS6670: Computer Vision Noah Snavely.
Announcements Mailing list Project 1 test the turnin procedure *this week* (make sure it works) vote on best artifacts in next week’s class Project 2 groups.
Lecture 20: Light, color, and reflectance CS6670: Computer Vision Noah Snavely.
Lecture 13: Projection, Part 2
Panorama signups Panorama project issues? Nalwa handout Announcements.
Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely.
Lecture 16: Single-view modeling, Part 2 CS6670: Computer Vision Noah Snavely.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Single-view geometry Odilon Redon, Cyclops, 1914.
The Pinhole Camera Model
More Mosaic Madness : Computational Photography Alexei Efros, CMU, Fall 2005 © Jeffrey Martin (jeffrey-martin.com) with a lot of slides stolen from.
Panoramas and Calibration : Rendering and Image Processing Alexei Efros …with a lot of slides stolen from Steve Seitz and Rick Szeliski.
CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.
More Mosaic Madness : Computational Photography Alexei Efros, CMU, Fall 2006 © Jeffrey Martin (jeffrey-martin.com) with a lot of slides stolen from.
CSE 576, Spring 2008Projective Geometry1 Lecture slides by Steve Seitz (mostly) Lecture presented by Rick Szeliski.
Camera parameters Extrinisic parameters define location and orientation of camera reference frame with respect to world frame Intrinsic parameters define.
Today: Calibration What are the camera parameters?
Sebastian Thrun CS223B Computer Vision, Winter Stanford CS223B Computer Vision, Winter 2006 Lecture 4 Camera Calibration Professor Sebastian Thrun.
Projective Geometry and Single View Modeling CSE 455, Winter 2010 January 29, 2010 Ames Room.
Single-view modeling, Part 2 CS4670/5670: Computer Vision Noah Snavely.
776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2013.
Image Stitching Ali Farhadi CSE 455
CSC 589 Lecture 22 Image Alignment and least square methods Bei Xiao American University April 13.
Camera Geometry and Calibration Thanks to Martial Hebert.
Epipolar geometry The fundamental matrix and the tensor
Last Lecture (optical center) origin principal point P (X,Y,Z) p (x,y) x y.
Course 12 Calibration. 1.Introduction In theoretic discussions, we have assumed: Camera is located at the origin of coordinate system of scene.
Vision Review: Image Formation Course web page: September 10, 2002.
Project 2 –contact your partner to coordinate ASAP –more work than the last project (> 1 week) –you should have signed up for panorama kitssigned up –sign.
Peripheral drift illusion. Multiple views Hartley and Zisserman Lowe stereo vision structure from motion optical flow.
Single-view geometry Odilon Redon, Cyclops, 1914.
Ch. 3: Geometric Camera Calibration
EECS 274 Computer Vision Geometric Camera Calibration.
Single-view geometry Odilon Redon, Cyclops, 1914.
Project 1 Due NOW Project 2 out today –Help session at end of class Announcements.
Example: warping triangles Given two triangles: ABC and A’B’C’ in 2D (12 numbers) Need to find transform T to transfer all pixels from one to the other.
Project 2 went out on Tuesday You should have a partner You should have signed up for a panorama kit Announcements.
Lecture 14: Projection CS4670 / 5670: Computer Vision Noah Snavely “The School of Athens,” Raphael.
Camera Calibration Course web page: vision.cis.udel.edu/cv March 24, 2003  Lecture 17.
Calibrating a single camera
Geometric Camera Calibration
More Mosaic Madness © Jeffrey Martin (jeffrey-martin.com)
Depth from disparity (x´,y´)=(x+D(x,y), y)
Digital Visual Effects, Spring 2007 Yung-Yu Chuang 2007/4/17
Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/4/19
Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/15
More Mosaic Madness : Computational Photography
More Mosaic Madness © Jeffrey Martin (jeffrey-martin.com)
Digital Visual Effects Yung-Yu Chuang
Projective geometry Readings
More Mosaic Madness © Jeffrey Martin (jeffrey-martin.com)
Announcements Project 2 extension: Friday, Feb 8
Presentation transcript:

Camera calibration Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz,, Fred Pighin and Marc Pollefyes

Outline Camera projection models Camera calibration Nonlinear least square methods A camera calibration tool Applications

Camera projection models

Pinhole camera

Pinhole camera model (optical center) origin principal point P (X,Y,Z) p (x,y)

Pinhole camera model principal point

Pinhole camera model principal point

Principal point offset intrinsic matrix principal point only related to camera projection

Intrinsic matrix non-square pixels (digital video) skew radial distortion Is this form of K good enough?

Distortion Radial distortion of the image –Caused by imperfect lenses –Deviations are most noticeable for rays that pass through the edge of the lens No distortionPin cushionBarrel

Camera rotation and translation extrinsic matrix

Two kinds of parameters internal or intrinsic parameters such as focal length, optical center, aspect ratio: what kind of camera? external or extrinsic (pose) parameters including rotation and translation: where is the camera?

Other projection models

Orthographic projection Special case of perspective projection –Distance from the COP to the PP is infinite –Also called “ parallel projection ” : (x, y, z) → (x, y) Image World

Other types of projections Scaled orthographic –Also called “ weak perspective ” Affine projection –Also called “ paraperspective ”

Illusion

Fun with perspective

Perspective cues

Fun with perspective Ames room Ames videoBBC story

Forced perspective in LOTR

Camera calibration

Estimate both intrinsic and extrinsic parameters. Two main categories: 1.Photometric calibration: uses reference objects with known geometry 2.Self calibration: only assumes static scene, e.g. structure from motion

Camera calibration approaches 1.linear regression (least squares) 2.nonlinear optimization

Chromaglyphs (HP research)

Camera calibration

Linear regression

Directly estimate 11 unknowns in the M matrix using known 3D points (X i,Y i,Z i ) and measured feature positions (u i,v i )

Linear regression

Solve for Projection Matrix M using least-square techniques

Normal equation Given an overdetermined system the normal equation is that which minimizes the sum of the square differences between left and right sides

Linear regression Advantages: –All specifics of the camera summarized in one matrix –Can predict where any world point will map to in the image Disadvantages: –Doesn’t tell us about particular parameters –Mixes up internal and external parameters pose specific: move the camera and everything breaks –More unknowns than true degrees of freedom

Nonlinear optimization A probabilistic view of least square Feature measurement equations Probability of M given {( u i,v i )} P

Optimal estimation Likelihood of M given {( u i,v i )} It is a least square problem (but not necessarily linear least square) How do we minimize L ? PL

Optimal estimation Non-linear regression (least squares), because the relations between û i and u i are non-linear functions of M We can use Levenberg-Marquardt method to minimize it known constant We could have terms like in this unknown parameters

Nonlinear least square methods

Least square fitting number of data points number of parameters

Linear least square fitting y t

y t modelparameters

Linear least square fitting y t modelparameters

Linear least square fitting y t modelparameters residual prediction

Linear least square fitting y t is linear, too. modelparameters residual prediction

Nonlinear least square fitting model parameters residuals

Function minimization It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum. Least square is related to function minimization.

Function minimization

Quadratic functions Approximate the function with a quadratic function within a small neighborhood

Quadratic functions A is positive definite. All eigenvalues are positive. For all x, x T Ax>0. negative definite A is indefinite A is singular

Function minimization Why? By definition, if is a local minimizer, is small enough

Function minimization

Descent methods

Descent direction

Steepest descent method the decrease of F(x) per unit along h direction → h sd is a descent direction because h T sd F’(x) = -F’(x) 2 <0

Line search

Steepest descent method isocontourgradient

Steepest descent method It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.

Newton’s method → → → →

Another view Minimizer satisfies

Newton’s method It requires solving a linear system and H is not always positive definite. It has good performance in the final stage of the iterative process, where x is close to x*.

Gauss-Newton method Use the approximate Hessian No need for second derivative H is positive semi-definite

Hybrid method This needs to calculate second-order derivative which might not be available.

Levenberg-Marquardt method LM can be thought of as a combination of steepest descent and the Newton method. When the current solution is far from the correct one, the algorithm behaves like a steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Newton’s method.

Nonlinear least square

Levenberg-Marquardt method

μ=0 → Newton’s method μ → ∞ → steepest descent method Strategy for choosing μ –Start with some small μ –If F is not reduced, keep trying larger μ until it does –If F is reduced, accept it and reduce μ for the next iteration

Recap (the Rosenbrock function) Global minimum at (1,1)

Steepest descent

In the plane of the steepest descent direction

Regularized Least- Squares Steepest descent (1000 iterations)

Gauss-Newton method With the approximate Hessian No need for second derivative H is positive semi-definite

Regularized Least- Squares Newton’s method (48 evaluations)

Levenberg-Marquardt Blends steepest descent and Gauss-Newton At each step, solve for the descent direction h If μ large,, steepest descent If μ small,, Gauss-Newton

Regularized Least- Squares Levenberg-Marquardt (90 evaluations)

A popular calibration tool

Multi-plane calibration Images courtesy Jean-Yves Bouguet, Intel Corp. Advantage Only requires a plane Don’t have to know positions/orientations Good code available online! –Intel’s OpenCV library: –Matlab version by Jean-Yves Bouget: –Zhengyou Zhang’s web site:

Step 1: data acquisition

Step 2: specify corner order

Step 3: corner extraction

Step 4: minimize projection error

Step 4: camera calibration

Step 5: refinement

Optimized parameters

Applications

How is calibration used? Good for recovering intrinsic parameters; It is thus useful for many vision applications Since it requires a calibration pattern, it is often necessary to remove or replace the pattern from the footage or utilize it in some ways…

Example of calibration

Videos from GaTech DasTatoo, MakeOfDasTatooMakeOf P!NG, MakeOfP!NGMakeOf Work, MakeOfWorkMakeOf LifeInPaints, MakeOfLifeInPaintsMakeOf

PhotoBook MakeOf PhotoBook