 Marc Levoy Synthetic aperture confocal imaging Marc Levoy Billy Chen Vaibhav Vaish Mark Horowitz Ian McDowall Mark Bolas

 Marc Levoy Outline technologies –large camera arrays –large projector arrays –camera–projector arrays optical effects synthetic aperture photography synthetic aperture illumination synthetic confocal imaging applications –partially occluding environments –weakly scattering media examples foliage murky water

 Marc Levoy Stanford Multi-Camera Array [Wilburn 2002] 640 × 480 pixels × 30fps × 128 cameras synchronized timing continuous video streaming flexible physical arrangement

 Marc Levoy Ways to use large camera arrays widely spaced light field capture tightly packed high-performance imaging intermediate spacing synthetic aperture photography

 Marc Levoy Synthetic aperture photography

 Marc Levoy Synthetic aperture photography

 Marc Levoy Synthetic aperture photography

 Marc Levoy Synthetic aperture photography 

 Marc Levoy Synthetic aperture photography 

 Marc Levoy Synthetic aperture photography 

 Marc Levoy Related work not like synthetic aperture radar (SAR) more like X-ray tomosynthesis [Levoy and Hanrahan, 1996] [Isaksen, McMillan, Gortler, 2000]

 Marc Levoy Example using 45 cameras

 Marc Levoy Synthetic pull-focus

 Marc Levoy Crowd scene

 Marc Levoy Crowd scene

 Marc Levoy Synthetic aperture photography using an array of mirrors 11-megapixel camera 22 planar mirrors ?

 Marc Levoy

Synthetic aperture illumation

 Marc Levoy Synthetic aperture illumation technologies –array of projectors –array of microprojectors –single projector + array of mirrors applications –bright display –autostereoscopic display [Matusik 2004] –confocal imaging [this paper]

 Marc Levoy Confocal scanning microscopy pinhole light source

 Marc Levoy Confocal scanning microscopy pinhole light source photocell pinhole r

 Marc Levoy Confocal scanning microscopy pinhole light source photocell pinhole

 Marc Levoy Confocal scanning microscopy pinhole light source photocell pinhole

[UMIC SUNY/Stonybrook]

 Marc Levoy Synthetic confocal scanning light source → 5 beams → 0 or 1 beam

 Marc Levoy Synthetic confocal scanning light source → 5 beams → 0 or 1 beam

 Marc Levoy Synthetic confocal scanning → 5 beams → 0 or 1 beam works with any number of projectors ≥ 2 discrimination degrades if point to left of no discrimination for points to left of slow! poor light efficiency d.o.f.

 Marc Levoy Synthetic coded-aperture confocal imaging different from coded aperture imaging in astronomy [Wilson, Confocal Microscopy by Aperture Correlation, 1996]

 Marc Levoy Synthetic coded-aperture confocal imaging

 Marc Levoy Synthetic coded-aperture confocal imaging

 Marc Levoy Synthetic coded-aperture confocal imaging

 Marc Levoy Synthetic coded-aperture confocal imaging 100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5

 Marc Levoy Synthetic coded-aperture confocal imaging 100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5 floodlit → 2 beams trials – ¼ × floodlit → 1 – ¼ ( 2 ) ≈ 0.5 → 0.5 – ¼ ( 2 ) ≈ 0

 Marc Levoy Synthetic coded-aperture confocal imaging 50% light efficiency any number of projectors ≥ 2 no discrimination to left of works with relatively few trials (~16) 100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5 floodlit → 2 beams trials – ¼ × floodlit → 1 – ¼ ( 2 ) ≈ 0.5 → 0.5 – ¼ ( 2 ) ≈ 0

 Marc Levoy Synthetic coded-aperture confocal imaging 50% light efficiency any number of projectors ≥ 2 no discrimination to left of works with relatively few trials (~16) needs patterns in which illumination of tiles are uncorrelated 100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5 floodlit → 2 beams trials – ¼ × floodlit → 1 – ¼ ( 2 ) ≈ 0.5 → 0.5 – ¼ ( 2 ) ≈ 0

 Marc Levoy Example pattern

 Marc Levoy Patterns with less aliasing

 Marc Levoy Implementation using an array of mirrors

 Marc Levoy Confocal imaging in scattering media small tank –too short for attenuation –lit by internal reflections

 Marc Levoy Experiments in a large water tank 50-foot flume at Wood’s Hole Oceanographic Institution (WHOI)

 Marc Levoy Experiments in a large water tank 4-foot viewing distance to target surfaces blackened to kill reflections titanium dioxide in filtered water transmissometer to measure turbidity

 Marc Levoy Experiments in a large water tank stray light limits performance one projector suffices if no occluders

 Marc Levoy Seeing through turbid water floodlit scanned tile

 Marc Levoy Application to underwater exploration [Ballard/IFE 2004]

 Marc Levoy Research challenges in SAP and SAI theory –aperture shapes and sampling patterns –illumination patterns for confocal imaging optical design –How to arrange cameras, projectors, lenses, mirrors, and other optical elements? –How to compare the performance of different arrangements (in foliage, underwater,...)?

 Marc Levoy Challenges (continued) systems design –multi-camera or multi-projector chips –communication in camera-projector networks –calibration in long-range or mobile settings algorithms –tracking and stabilization of moving objects –compression of dense multi-view imagery –shape from light fields

 Marc Levoy Challenges (continued) applications of confocal imaging –remote sensing and surveillance –shape measurement –scientific imaging applications of shaped illumination –shaped searchlights for surveillance –shaped headlamps for driving in bad weather –selective lighting of characters for stage and screen

 Marc Levoy Computational imaging in other fields medical imaging –rebinning –tomography airborne sensing –multi-perspective panoramas –synthetic aperture radar astronomy –coded-aperture imaging –multi-telescope imaging

 Marc Levoy Computational imaging in other fields geophysics –accoustic array imaging –borehole tomography biology –confocal microscopy –deconvolution microscopy physics –optical tomography –inverse scattering

 Marc Levoy The team staff –Mark Horowitz –Marc Levoy –Bennett Wilburn students –Billy Chen –Vaibhav Vaish –Katherine Chou –Monica Goyal –Neel Joshi –Hsiao-Heng Kelin Lee –Georg Petschnigg –Guillaume Poncin –Michael Smulski –Augusto Roman collaborators –Mark Bolas –Ian McDowall –Guillermo Sapiro funding –Intel –Sony –Interval Research –NSF –DARPA

 Marc Levoy Related papers The Light Field Video Camera Bennett Wilburn, Michael Smulski, Hsiao-Heng Kelin Lee, and Mark Horowitz Proc. Media Processors 2002, SPIE Electronic Imaging 2002 Using Plane + Parallax for Calibrating Dense Camera Arrays Vaibhav Vaish, Bennett Wilburn, Neel Joshi,, Marc Levoy Proc. CVPR 2004 High Speed Video Using a Dense Camera Array Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc Levoy, Mark Horowitz Proc. CVPR 2004 Spatiotemporal Sampling and Interpolation for Dense Camera Arrays Bennett Wilburn, Neel Joshi, Katherine Chou, Marc Levoy, Mark Horowitz ACM Transactions on Graphics (conditionally accepted)

 Marc Levoy