 Marc Levoy Using Plane + Parallax to Calibrate Dense Camera Arrays Vaibhav Vaish, Bennett Wilburn, Neel Joshi, Marc Levoy Computer Science Department Stanford University

 Marc Levoy The Stanford Multi-Camera Array

 Marc Levoy Synthetic Aperture Photography: Seeing through Foliage

 Marc Levoy Synthetic Aperture Photography: Seeing through Foliage

 Marc Levoy Outline Problem Statement –Synthetic aperture photography using an array of cameras –Calibration required Calibration Pipeline Results Future Work

 Marc Levoy Synthetic aperture photography 

 Marc Levoy SAP: Prior Work Synthetic Aperture Radar Light Field Rendering [Levoy 96] Dynamically Reparametrized Light Fields [Isaksen 00] Single lens SAP [Favaro 03]

 Marc Levoy Outdoor SAP: Array layout width of aperture2m number of cameras45 spacing between cameras13cm camera’s field of view4.5°

 Marc Levoy Outdoor SAP: The scene distance to occluder33m distance to targets45m field of view at target3m

 Marc Levoy Outdoor SAP: Calibration Narrow field of view and long-range imaging makes accurate pose estimation difficult Cannot take calibration measurements at the desired focal depth (behind occluding bushes) Calibration Volume Focal Depth 28 m 5m a = 2m 12m

 Marc Levoy Calibration Goal Given a focal plane, compute the projective transform (homography) to project each camera image onto the plane. Focal Plane

 Marc Levoy Approaches to Calibration Metric Calibration –Computes camera intrinsics, position, orientation (10 parameters/camera) –Nonlinear optimization, requires initial guess –Not stable for narrow angle lenses and long range imaging Non-metric Calibration –Plane + Parallax methods [Irani 96, Triggs 00] –Homography Spaces [Zelnik-Manor 99]

 Marc Levoy Calibration Pipeline Problem Statement Calibration Pipeline –Focus cameras on one plane (using homographies) –Compute relative camera positions from parallax measurements –Use camera positions to vary focal plane over a range of depths Results Future Work

 Marc Levoy Focusing on one plane + Add camera images so that points on one plane are in good focus

 Marc Levoy Focusing at different depths +

 Marc Levoy Focusing at different depths + To focus at a different depth, we have to shift the images by an amount equal to the parallax

 Marc Levoy Parallax and Camera Geometry  p 1 =  X 1.  z/(Z 0 +  z) =  X 1. d P Parallax = Camera shift * Relative Depth Reference Plane Camera Plane P P p1p1 X1X1 zz Z0Z0

 Marc Levoy Parallax and Camera Geometry Measure parallax of P in all cameras (wrt reference camera) [  p 1  p 2... ] T Reference Plane Camera Plane p1p1 X1X1 P p2p2 X2X2

 Marc Levoy Recovering Camera Positions Parallax of point P: For multiple points P 1, … P n : Relative camera positions  X i can be recovered robustly (up to scale) using SVD.

 Marc Levoy Computing SAP Images at different focal depths To change the focal depth, images have to be shifted by the amount of parallax. In camera C 1, the parallax for a parallel focal plane is f.  X 1,where f is a constant that depends only on the depth of the plane. f is analogous to the focus distance of the synthetic lens: varying f changes the depth of the focal plane. p1p1 X1X1

 Marc Levoy Algorithm for SAP 1.Focus cameras onto a frontoparallel plane 2.Compute parallax for one (or more) scene points 3.Recover relative camera positions  X i (up to an unknown scale) 4.For a range of values of f : Shift image from camera C i by f.  X i and average shifted images. Varying f corresponds to changing the focus

 Marc Levoy Results Synthetic Aperture Sequence

 Marc Levoy Parallax v/s Metric Calibration Parallax-based calibration Metric calibration

 Marc Levoy Summary Calibration of camera arrays for synthetic aperture photography –Decompose warps into reference homography and shifts –Use parallax measurements to compute camera positions –Avoids computing camera intrinsics and orientation explicitly –Robust, linear solution Metric information not available Algorithm requires planar camera array and frontoparallel reference plane

 Marc Levoy Extension :Tilted Focal Planes Reference Plane Focal Plane e (epipole) L (line of intersection) - Parallax is described by a projective warp (not a shift) - Rank-1 factorization is still possible

 Marc Levoy Future Work Real-time applications –Warp images in hardware –Track moving objects by moving focal plane 3D Reconstruction from synthetic focus –Is this more robust to occlusions ? Quantitative analysis of synthetic aperture photography –Effect of occluder density, number of cameras, aperture shape

 Marc Levoy Acknowledgements Sponsors –NSF IIS –DARPA NBCH Assistance in acquisition – Gaurav Garg, Augusto Roman, Billy Chen, Pradeep Sen, Doantam Phan, Guillaume Poncin, Jeff Klingner High Speed Videography Using a Dense Camera Array B Wilburn, N Joshi, V Vaish, M Levoy, M Horowitz Session 5A, 4:20pm