Marc Levoy Using Plane + Parallax to Calibrate Dense Camera Arrays Vaibhav Vaish, Bennett Wilburn, Neel Joshi, Marc Levoy Computer Science Department Stanford University
Marc Levoy The Stanford Multi-Camera Array
Marc Levoy Synthetic Aperture Photography: Seeing through Foliage
Marc Levoy Synthetic Aperture Photography: Seeing through Foliage
Marc Levoy Outline Problem Statement –Synthetic aperture photography using an array of cameras –Calibration required Calibration Pipeline Results Future Work
Marc Levoy Synthetic aperture photography
Marc Levoy SAP: Prior Work Synthetic Aperture Radar Light Field Rendering [Levoy 96] Dynamically Reparametrized Light Fields [Isaksen 00] Single lens SAP [Favaro 03]
Marc Levoy Outdoor SAP: Array layout width of aperture2m number of cameras45 spacing between cameras13cm camera’s field of view4.5°
Marc Levoy Outdoor SAP: The scene distance to occluder33m distance to targets45m field of view at target3m
Marc Levoy Outdoor SAP: Calibration Narrow field of view and long-range imaging makes accurate pose estimation difficult Cannot take calibration measurements at the desired focal depth (behind occluding bushes) Calibration Volume Focal Depth 28 m 5m a = 2m 12m
Marc Levoy Calibration Goal Given a focal plane, compute the projective transform (homography) to project each camera image onto the plane. Focal Plane
Marc Levoy Approaches to Calibration Metric Calibration –Computes camera intrinsics, position, orientation (10 parameters/camera) –Nonlinear optimization, requires initial guess –Not stable for narrow angle lenses and long range imaging Non-metric Calibration –Plane + Parallax methods [Irani 96, Triggs 00] –Homography Spaces [Zelnik-Manor 99]
Marc Levoy Calibration Pipeline Problem Statement Calibration Pipeline –Focus cameras on one plane (using homographies) –Compute relative camera positions from parallax measurements –Use camera positions to vary focal plane over a range of depths Results Future Work
Marc Levoy Focusing on one plane + Add camera images so that points on one plane are in good focus
Marc Levoy Focusing at different depths +
Marc Levoy Focusing at different depths + To focus at a different depth, we have to shift the images by an amount equal to the parallax
Marc Levoy Parallax and Camera Geometry p 1 = X 1. z/(Z 0 + z) = X 1. d P Parallax = Camera shift * Relative Depth Reference Plane Camera Plane P P p1p1 X1X1 zz Z0Z0
Marc Levoy Parallax and Camera Geometry Measure parallax of P in all cameras (wrt reference camera) [ p 1 p 2... ] T Reference Plane Camera Plane p1p1 X1X1 P p2p2 X2X2
Marc Levoy Recovering Camera Positions Parallax of point P: For multiple points P 1, … P n : Relative camera positions X i can be recovered robustly (up to scale) using SVD.
Marc Levoy Computing SAP Images at different focal depths To change the focal depth, images have to be shifted by the amount of parallax. In camera C 1, the parallax for a parallel focal plane is f. X 1,where f is a constant that depends only on the depth of the plane. f is analogous to the focus distance of the synthetic lens: varying f changes the depth of the focal plane. p1p1 X1X1
Marc Levoy Algorithm for SAP 1.Focus cameras onto a frontoparallel plane 2.Compute parallax for one (or more) scene points 3.Recover relative camera positions X i (up to an unknown scale) 4.For a range of values of f : Shift image from camera C i by f. X i and average shifted images. Varying f corresponds to changing the focus
Marc Levoy Results Synthetic Aperture Sequence
Marc Levoy Parallax v/s Metric Calibration Parallax-based calibration Metric calibration
Marc Levoy Summary Calibration of camera arrays for synthetic aperture photography –Decompose warps into reference homography and shifts –Use parallax measurements to compute camera positions –Avoids computing camera intrinsics and orientation explicitly –Robust, linear solution Metric information not available Algorithm requires planar camera array and frontoparallel reference plane
Marc Levoy Extension :Tilted Focal Planes Reference Plane Focal Plane e (epipole) L (line of intersection) - Parallax is described by a projective warp (not a shift) - Rank-1 factorization is still possible
Marc Levoy Future Work Real-time applications –Warp images in hardware –Track moving objects by moving focal plane 3D Reconstruction from synthetic focus –Is this more robust to occlusions ? Quantitative analysis of synthetic aperture photography –Effect of occluder density, number of cameras, aperture shape
Marc Levoy Acknowledgements Sponsors –NSF IIS –DARPA NBCH Assistance in acquisition – Gaurav Garg, Augusto Roman, Billy Chen, Pradeep Sen, Doantam Phan, Guillaume Poncin, Jeff Klingner High Speed Videography Using a Dense Camera Array B Wilburn, N Joshi, V Vaish, M Levoy, M Horowitz Session 5A, 4:20pm