 Marc Levoy Using Plane + Parallax to Calibrate Dense Camera Arrays Vaibhav Vaish, Bennett Wilburn, Neel Joshi, Marc Levoy Computer Science Department.

Slides:



Advertisements
Similar presentations
Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.
Advertisements

A Robust Super Resolution Method for Images of 3D Scenes Pablo L. Sala Department of Computer Science University of Toronto.
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 2 Calibration Axel Pinz.
Fourier Slice Photography
Light Fields PROPERTIES AND APPLICATIONS. Outline  What are light fields  Acquisition of light fields  from a 3D scene  from a real world scene 
Two-View Geometry CS Sastry and Yang
Light field photography and videography Marc Levoy Computer Science Department Stanford University.
Light Field Rendering Shijin Kong Lijie Heng.
Geometry of Images Pinhole camera, projection A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix.
Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Uncalibrated Geometry & Stratification Sastry and Yang
Computational Photography
3D reconstruction of cameras and structure x i = PX i x’ i = P’X i.
CAU Kiel DAGM 2001-Tutorial on Visual-Geometric 3-D Scene Reconstruction 1 The plan for today Leftovers and from last time Camera matrix Part A) Notation,
Computer Science Department
Image Stitching and Panoramas
Previously Two view geometry: epipolar geometry Stereo vision: 3D reconstruction epipolar lines Baseline O O’ epipolar plane.
Linear View Synthesis Using a Dimensionality Gap Light Field Prior
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.
 Marc Levoy Synthetic aperture confocal imaging Marc Levoy Billy Chen Vaibhav Vaish Mark Horowitz Ian McDowall Mark Bolas.
 Marc Levoy IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry.”
 Marc Levoy IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry.”
Camera parameters Extrinisic parameters define location and orientation of camera reference frame with respect to world frame Intrinsic parameters define.
CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.
Light field photography and microscopy Marc Levoy Computer Science Department Stanford University.
3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.
Light Field. Modeling a desktop Image Based Rendering  Fast Realistic Rendering without 3D models.
Light field photography and videography Marc Levoy Computer Science Department Stanford University.
CSE 6367 Computer Vision Stereo Reconstruction Camera Coordinate Transformations “Everything should be made as simple as possible, but not simpler.” Albert.
Automatic Camera Calibration
What Does the Scene Look Like From a Scene Point? Donald Tanguay August 7, 2002 M. Irani, T. Hassner, and P. Anandan ECCV 2002.
Structure from images. Calibration Review: Pinhole Camera.
Geometry and Algebra of Multiple Views
Epipolar geometry The fundamental matrix and the tensor
Dynamically Reparameterized Light Fields Aaron Isaksen, Leonard McMillan (MIT), Steven Gortler (Harvard) Siggraph 2000 Presented by Orion Sky Lawlor cs497yzy.
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
The Brightness Constraint
ABSTRACT A single camera can be a useful surveillance tool, but video recorded from a single point of reference becomes ineffective when objects of interest.
Image-based Rendering. © 2002 James K. Hahn2 Image-based Rendering Usually based on 2-D imagesUsually based on 2-D images Pre-calculationPre-calculation.
1 Finding depth. 2 Overview Depth from stereo Depth from structured light Depth from focus / defocus Laser rangefinders.
Multiview Geometry and Stereopsis. Inputs: two images of a scene (taken from 2 viewpoints). Output: Depth map. Inputs: multiple images of a scene. Output:
Vision Review: Image Formation Course web page: September 10, 2002.
Interreflections : The Inverse Problem Lecture #12 Thanks to Shree Nayar, Seitz et al, Levoy et al, David Kriegman.
A Flexible New Technique for Camera Calibration Zhengyou Zhang Sung Huh CSPS 643 Individual Presentation 1 February 25,
Plane-based external camera calibration with accuracy measured by relative deflection angle Chunhui Cui , KingNgiNgan Journal Image Communication Volume.
Feature Matching. Feature Space Outlier Rejection.
 Vaibhav Vaish Synthetic Aperture Focusing Using Dense Camera Arrays Vaibhav Vaish Computer Graphics Laboratory Stanford University.
1Ellen L. Walker 3D Vision Why? The world is 3D Not all useful information is readily available in 2D Why so hard? “Inverse problem”: one image = many.
3D Reconstruction Using Image Sequence
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
MASKS © 2004 Invitation to 3D vision Uncalibrated Camera Chapter 6 Reconstruction from Two Uncalibrated Views Modified by L A Rønningen Oct 2008.
Chapter 2: The Lens. Focal Length is the distance between the center of a lens and the film plane when focused at infinity.
Uncalibrated reconstruction Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration.
Digital Image Processing Additional Material : Imaging Geometry 11 September 2006 Digital Image Processing Additional Material : Imaging Geometry 11 September.
Image-Based Rendering Geometry and light interaction may be difficult and expensive to model –Think of how hard radiosity is –Imagine the complexity of.
EECS 274 Computer Vision Projective Structure from Motion.
Automatic 3D modelling of Architecture Anthony Dick 1 Phil Torr 2 Roberto Cipolla 1 1 Department of Engineering 2 Microsoft Research, University of Cambridge.
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
Lec 26: Fundamental Matrix CS4670 / 5670: Computer Vision Kavita Bala.
CSE 185 Introduction to Computer Vision Stereo 2.
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry
Depth of Field Objective: to photograph a subject with a foreground and background, giving a better understanding of aperture and its relation to depth.
The Brightness Constraint
Epipolar geometry.
The Brightness Constraint
The Brightness Constraint
Multiple View Geometry for Robotics
Uncalibrated Geometry & Stratification
Presentation transcript:

 Marc Levoy Using Plane + Parallax to Calibrate Dense Camera Arrays Vaibhav Vaish, Bennett Wilburn, Neel Joshi, Marc Levoy Computer Science Department Stanford University

 Marc Levoy The Stanford Multi-Camera Array

 Marc Levoy Synthetic Aperture Photography: Seeing through Foliage

 Marc Levoy Synthetic Aperture Photography: Seeing through Foliage

 Marc Levoy Outline Problem Statement –Synthetic aperture photography using an array of cameras –Calibration required Calibration Pipeline Results Future Work

 Marc Levoy Synthetic aperture photography 

 Marc Levoy SAP: Prior Work Synthetic Aperture Radar Light Field Rendering [Levoy 96] Dynamically Reparametrized Light Fields [Isaksen 00] Single lens SAP [Favaro 03]

 Marc Levoy Outdoor SAP: Array layout width of aperture2m number of cameras45 spacing between cameras13cm camera’s field of view4.5°

 Marc Levoy Outdoor SAP: The scene distance to occluder33m distance to targets45m field of view at target3m

 Marc Levoy Outdoor SAP: Calibration Narrow field of view and long-range imaging makes accurate pose estimation difficult Cannot take calibration measurements at the desired focal depth (behind occluding bushes) Calibration Volume Focal Depth 28 m 5m a = 2m 12m

 Marc Levoy Calibration Goal Given a focal plane, compute the projective transform (homography) to project each camera image onto the plane. Focal Plane

 Marc Levoy Approaches to Calibration Metric Calibration –Computes camera intrinsics, position, orientation (10 parameters/camera) –Nonlinear optimization, requires initial guess –Not stable for narrow angle lenses and long range imaging Non-metric Calibration –Plane + Parallax methods [Irani 96, Triggs 00] –Homography Spaces [Zelnik-Manor 99]

 Marc Levoy Calibration Pipeline Problem Statement Calibration Pipeline –Focus cameras on one plane (using homographies) –Compute relative camera positions from parallax measurements –Use camera positions to vary focal plane over a range of depths Results Future Work

 Marc Levoy Focusing on one plane + Add camera images so that points on one plane are in good focus

 Marc Levoy Focusing at different depths +

 Marc Levoy Focusing at different depths + To focus at a different depth, we have to shift the images by an amount equal to the parallax

 Marc Levoy Parallax and Camera Geometry  p 1 =  X 1.  z/(Z 0 +  z) =  X 1. d P Parallax = Camera shift * Relative Depth Reference Plane Camera Plane P P p1p1 X1X1 zz Z0Z0

 Marc Levoy Parallax and Camera Geometry Measure parallax of P in all cameras (wrt reference camera) [  p 1  p 2... ] T Reference Plane Camera Plane p1p1 X1X1 P p2p2 X2X2

 Marc Levoy Recovering Camera Positions Parallax of point P: For multiple points P 1, … P n : Relative camera positions  X i can be recovered robustly (up to scale) using SVD.

 Marc Levoy Computing SAP Images at different focal depths To change the focal depth, images have to be shifted by the amount of parallax. In camera C 1, the parallax for a parallel focal plane is f.  X 1,where f is a constant that depends only on the depth of the plane. f is analogous to the focus distance of the synthetic lens: varying f changes the depth of the focal plane. p1p1 X1X1

 Marc Levoy Algorithm for SAP 1.Focus cameras onto a frontoparallel plane 2.Compute parallax for one (or more) scene points 3.Recover relative camera positions  X i (up to an unknown scale) 4.For a range of values of f : Shift image from camera C i by f.  X i and average shifted images. Varying f corresponds to changing the focus

 Marc Levoy Results Synthetic Aperture Sequence

 Marc Levoy Parallax v/s Metric Calibration Parallax-based calibration Metric calibration

 Marc Levoy Summary Calibration of camera arrays for synthetic aperture photography –Decompose warps into reference homography and shifts –Use parallax measurements to compute camera positions –Avoids computing camera intrinsics and orientation explicitly –Robust, linear solution Metric information not available Algorithm requires planar camera array and frontoparallel reference plane

 Marc Levoy Extension :Tilted Focal Planes Reference Plane Focal Plane e (epipole) L (line of intersection) - Parallax is described by a projective warp (not a shift) - Rank-1 factorization is still possible

 Marc Levoy Future Work Real-time applications –Warp images in hardware –Track moving objects by moving focal plane 3D Reconstruction from synthetic focus –Is this more robust to occlusions ? Quantitative analysis of synthetic aperture photography –Effect of occluder density, number of cameras, aperture shape

 Marc Levoy Acknowledgements Sponsors –NSF IIS –DARPA NBCH Assistance in acquisition – Gaurav Garg, Augusto Roman, Billy Chen, Pradeep Sen, Doantam Phan, Guillaume Poncin, Jeff Klingner High Speed Videography Using a Dense Camera Array B Wilburn, N Joshi, V Vaish, M Levoy, M Horowitz Session 5A, 4:20pm