 Vaibhav Vaish Synthetic Aperture Focusing Using Dense Camera Arrays Vaibhav Vaish Computer Graphics Laboratory Stanford University

 Vaibhav Vaish Cameras are Getting Cheaper … (Photo: AP)

 Vaibhav Vaish Why Camera Arrays ? High performance imaging Virtual reality Matrix: Bullet scene (Manex) Eyevision, SuperBowl 2001

 Vaibhav Vaish Stanford Multi-camera Array 100 cameras X 640x480 pixels X 30 frames/sec = 1 GB/sec Scalable Flexible Per-camera processing

 Vaibhav Vaish Stanford Multi-camera Array

 Vaibhav Vaish Demo #1: High Speed Video N cameras, each running at 30Hz Stagger the frames of cameras by 1/N th of a frame Align images to single perspective Video: 52 cameras, 1560 Hz

 Vaibhav Vaish Demo #2: High Resolution Video 12 × 8 array of VGA cameras total field of view = 29° wide seamless stitching cameras individually metered Tiled Video: 7 Megapixels

 Vaibhav Vaish Camera Array: Portable Version 48 cameras in 16 x 3 layout 2m wide baseline

 Vaibhav Vaish Synthetic Aperture Focusing: Scene distance to occluder 33m distance to targets 40m field of view at target 3m

 Vaibhav Vaish Synthetic Aperture Focusing: Results Synthetic aperture sequence

 Vaibhav Vaish Synthetic Aperture Focusing: Results Synthetic aperture sequence

 Vaibhav Vaish Outline Synthetic aperture focusing: basics Technical challenges –Determining the image transformations –How to refocus efficiently Real-time system Future Work

 Vaibhav Vaish Synthetic Aperture Focusing

 Vaibhav Vaish Synthetic Aperture Focusing

 Vaibhav Vaish Synthetic Aperture Focusing

 Vaibhav Vaish Synthetic Aperture Focusing 

 Vaibhav Vaish Synthetic Aperture Focusing 

 Vaibhav Vaish Synthetic Aperture Focusing 

 Vaibhav Vaish Synthetic Aperture Focusing: Properties Focusing is a computational process (as opposed to optical) –can vary focal plane after capturing images Can use arbitrary apertures Averaging multiple images improves signal-to-noise ratio

 Vaibhav Vaish Focusing on one Plane Backproject each camera image on to focal plane This is a 2D image warp called a homography (3x3 matrix) Focal Plane

 Vaibhav Vaish Focusing on one Plane The final image is the average of all the backprojected camera images Focal Plane

 Vaibhav Vaish Example: Focusing on one plane + Add camera images so that points on one plane are in good focus

 Vaibhav Vaish Technical Challenges How do we determine the projections for focusing digitally ? –what camera parameters do we need to calibrate ? –what are the image warps required ? Are there efficient algorithms for varying the focal plane ? –a homography requires 3 adds + 1 divide/pixel –100 video cameras = 90 million pixels/sec –computationally intensive!

 Vaibhav Vaish Varying the Focal Plane Plane 1 Plane 2 Camera Image Plane 1 Plane 2 Camera Image Plane 2Plane 1

 Vaibhav Vaish Varying the Focal Plane Reference Plane Focal Plane Camera Image Reference Plane Focal Plane Camera Image Focal Plane Reference Plane

 Vaibhav Vaish Planar Homologies Refocusing requires projecting image from reference plane on to new focal plane This reprojection is called a planar homology A homology is described by a matrix of the form H i = I +  i e i l T

 Vaibhav Vaish Focusing Algorithm 1.Calibration (Pre-process) Homographies for projection on to the reference plane Epipoles [Vaish 2004, Hartley 2000] 2.Project images on reference plane 3.Vary the focal plane by applying the homologies given by H i = I +  i e i l T

 Vaibhav Vaish Example: varying focal planes Rotating focal plane

 Vaibhav Vaish Case 1: Parallel Reference Plane When the reference plane is parallel to the camera plane, the epipoles e i = [ x i y i 0 ] T are points at infinity Homologies H i = I +  i e i l T reduce to affine transforms Reference Plane Camera Plane Focal Planes

 Vaibhav Vaish Case 2: Parallel Focal Planes When focal planes are parallel to reference plane, the line l = [ ] T is at infinity Homologies H i = I +  i e i l T reduce to a scale and shift Parallel Focal Planes Reference Plane

 Vaibhav Vaish Case 3: Frontoparallel Planes When the reference plane, camera plane and focal planes are parallel, the epipoles e i and line l are both at infinity Homologies H i = I +  i e i l T reduce to shifts [Vaish 2004] Reference Plane Camera Plane Focal Planes

 Vaibhav Vaish Case 4: Scheimpflug Configuration When the camera plane, reference plane and focal planes intersect in the same line, l and e i can both be mapped to infinity Homologies H i = I +  i e i l T again reduce to shifts Reference Plane Camera Plane Focal Planes

 Vaibhav Vaish Taxonomy of Homologies 1. Affine 4. Shift + post warp of final image 3. Shift 2. Scale + shift

 Vaibhav Vaish Real-time Implementation 1.Projection on to fixed reference plane (Look-up table) 2.Shift image for desired focal plane (FPGA) 3.Send MPEG stream to client PC (Firewire) 4.Decompress and add streams on client PC 5.Server adds streams from clients and displays live synthetic aperture video

 Vaibhav Vaish Real-time system: Demo

 Vaibhav Vaish Real-time System: Discussion Reference plane is fixed –Initial projection (homographies) can be implemented via look-up table which is computed beforehand Varying focal plane requires shifting images –Easy to realize in FPGAs (2 adds/camera) –Keeps per-camera cost low Extensions –Implement affine warps (2 adds/pixel) –Computer assisted focusing –Study other architectures for digital focusing (GPU)

 Vaibhav Vaish Outline Synthetic aperture focusing: basics Technical challenges –Determining the image transforms –How to refocus efficiently Real-time system Future Work –Matted synthetic apertures –General focal surfaces

 Vaibhav Vaish Matted Synthetic Apertures

 Vaibhav Vaish Crowd scene

 Vaibhav Vaish Crowd scene

 Vaibhav Vaish Curved focal surfaces planar focal surface cylindrical focal surface unoccluded

 Vaibhav Vaish General Focal Surfaces Can we reconstruct the correct focal depth for every pixel ? Single camera view Shape from focus “Unoccluded” view

 Vaibhav Vaish General Focal Surfaces Can we reconstruct the correct focal depth for every pixel ? Single camera view Shape from stereo “Unoccluded” view

 Vaibhav Vaish Summary Using a camera array for synthetic aperture focusing –large synthetic aperture allows seeing through partial occluders Geometry of digital focusing Real-time system Future work –explore general apertures –reconstruct correct focal depths for each pixel –study design space of synthetic aperture camera arrays

 Vaibhav Vaish Acknowledgements Sponsors –Bosch Research –NSF IIS –DARPA NBCH Acquisition assistance –Augusto Roman, Billy Chen, Abhishek Bapna, Mike Cammarano Listeners –Gaurav Garg, Ren Ng, Jeff Klingner, Doantam Phan, Niloy Mitra, Sriram Sankaranarayanan

 Vaibhav Vaish The Camera Array Team staff –Mark Horowitz –Marc Levoy –Bennett Wilburn students –Vaibhav Vaish –Gaurav Garg –Eino-Ville Talvala –Emilio Antunez –Andrew Adams –Neel Joshi –Georg Petschnigg –Guillaume Poncin –Monica Goyal collaborators –Mark Bolas –Ian McDowall –Microsoft Research funding –Bosch Research –Intel –Sony –Interval Research –NSF –DARPA

 Vaibhav Vaish Effect of feature size see-around ability increases with aperture width (a) and separation (Δz) relative to distance (d) from the cameras and to feature size (s) can see around 2” leaves at 125’ using a 16” aperture (ours was 6’) independent of number of cameras d = 125’ Δz =15’ a = 6’ s = 2” see-around ability ~ a Δz / d s

 Vaibhav Vaish Effect of occluder density see-through ability increases with number of cameras (n) relative to occluder opacity (α) independent of aperture size our bushes averaged 97% opaque (needs better measurement) qualitative figure of merit depends on human perception