1. Synthetic aperture confocal imaging
21:00 stopping video after scattering, ~22:00 in actual delivery
Synthetic aperture confocal imaging
Marc Levoy
Billy Chen
Vaibhav Vaish
Mark Horowitz
Ian McDowall
Mark Bolas
34:15 total + 30% = ~45 minutes
Time =

2. Outline technologies optical effects applications examples
0:30, 1:15 for entire zoomin
technologies
large camera arrays
large projector arrays
camera–projector arrays
optical effects
synthetic aperture photography
synthetic aperture illumination
synthetic confocal imaging
applications
partially occluding environments
weakly scattering media
examples
foliage
murky water
Time =

3. Stanford Multi-Camera Array [Wilburn 2002]
0:15
flexible physical arrangement
gives us a lot of mileage
640 × 480 pixels × 30fps × 128 cameras
synchronized timing
continuous video streaming
flexible physical arrangement
Time =

4. Ways to use large camera arrays
0:30
if the cameras are tightly packed
relative to the distance to the scene
then we can think of the array as a single virtual camera of high performance
along one or more imaging dimensions, such as
resolution, speed, dynamic range, and so on
papers at CVPR
what I’ll be talking about today...intermediate spacing
where the cameras essentially sample a synthetic aperture
widely spaced light field capture
tightly packed high-performance imaging
intermediate spacing synthetic aperture photography
Time =

5. Synthetic aperture photography
2:00, 3:45 for SAP
Time =

6. Synthetic aperture photography
Time =

7. Synthetic aperture photography
Taken to the extreme
objects off the focal plane become so blurry that they effectively disappear
at least if they are smaller than the aperture
Leonardo noticed 500 years ago
a needle placed in front of his eye
because it was smaller than his pupil
did not occlude his vision
Time =

8. Synthetic aperture photography
Time =

9. Synthetic aperture photography
Time =

10. Synthetic aperture photography
Another way to think about synthetic aperture photography
take the images from all the cameras,
rectify them to a common plane
shift them by a certain amount,
and add them together
Objects that become aligned by the shifting process
will be sharply focused
objects in front of that plane...
objects in back of that plane... 
Time =

11. Related work not like synthetic aperture radar (SAR)
0:30
not like synthetic aperture radar (SAR)
more like X-ray tomosynthesis
[Levoy and Hanrahan, 1996]
[Isaksen, McMillan, Gortler, 2000]
Time =

12. Example using 45 cameras
0:30
Time =

13. Synthetic pull-focus
Time =

14. Crowd scene
0:15
Time =

15. Crowd scene
Time =

16. Synthetic aperture photography using an array of mirrors
0:30 ?
11-megapixel camera
22 planar mirrors
Time =

17. Time =

18. Time =

19. Synthetic aperture illumation
0:45, 1:15 for SAI
if you replace cameras with projectors
many of the principles I’ve talked about still apply
in particular,
an array of projectors, suitably aligned, produces a real image in space
with such a shallow depth of field that
if you stick your hand astride the image, or a white angled screen in this case,
the image is focused only at one place on your hand,
it’s blurry in front of and behind that point
in this case
I’m using only a few projectors, so there’s some aliasing off the focal plane
if I had enough projectors, it would be a smooth blur
Time =

20. Synthetic aperture illumation
0:30
single projector + array of mirrors
as we do in this paper
bright display
we wear sunglasses in our laboratory when looking at these images
autostereoscopic display
i.e. a light field display
technologies
array of projectors
array of microprojectors
single projector + array of mirrors
applications
bright display
autostereoscopic display [Matusik 2004]
confocal imaging [this paper]
Time =

21. Confocal scanning microscopy
1:00, 5:15 up to and including patterns
at this point
I need to back up and tell you a bit about confocal microscopy
if you introduce a pinhole
only one point on the focal plane will be illuminated
light source
pinhole
Time =

22. Confocal scanning microscopy
...and a matching optical system,
hence the word confocal
this green dot
will be both strongly illuminated and sharply imaged
while this red dot
will have less light falling on it by the square of distance r,
because the light is spread over a disk
and it will also be more weakly imaged by the square of distance r,
because its image is blurred out over an disk on the pinhole mask, and only a little bit is permitted through
so the extent to which the red dot contributes to the final image
falls off as the fourth power of r, the distance from the focal plane
pinhole
light source r
photocell
pinhole
Time =

23. Confocal scanning microscopy
of course, you’ve only imaged one point
so you need to move the pinholes
and scan across the focal plane
light source
pinhole
pinhole
photocell
Time =

24. Confocal scanning microscopy
light source
pinhole
pinhole
photocell
Time =

25. [UMIC SUNY/Stonybrook]
0:15
the object in the lower-right image is actually spherical,
but portions of it that are off the focal plane are both blurry and dark,
effectively disappearing
[UMIC SUNY/Stonybrook]
Time =

26. Synthetic confocal scanning
1:30
our goal
is to approximate this effect at the large scale
we can understand the photometry of this setup
using a simplified counting argument
if we had 5 cameras
then the ratio would be 25:1 instead of 5:1
light source
→ 5 beams
→ 0 or 1 beam
Time =

27. Synthetic confocal scanning
5:0 or 5:1
if we had 5 cameras as well as 5 projectors, then the ratio would be 25:0 or 25:1
light source
→ 5 beams
→ 0 or 1 beam
Time =

28. Synthetic confocal scanning
depth of field
a microscoper would call it the axial resolution
to make the depth of field shallower, spread out the projectors, i.e. a larger synthetic aperture
→ 5 beams
→ 0 or 1 beam
d.o.f.
works with any number of projectors ≥ 2
discrimination degrades if point to left of
no discrimination for points to left of
slow!
poor light efficiency
Time =

29. Synthetic coded-aperture confocal imaging
2:15
different from coded aperture imaging in astronomy
[Wilson, Confocal Microscopy by Aperture Correlation, 1996]
Time =

30. Synthetic coded-aperture confocal imaging
Time =

31. Synthetic coded-aperture confocal imaging
Time =

32. Synthetic coded-aperture confocal imaging
Time =

33. Synthetic coded-aperture confocal imaging
100 trials
→ 2 beams × ~50/100 trials ≈ 1
→ ~1 beam × ~50/100 trials ≈ 0.5
Time =

34. Synthetic coded-aperture confocal imaging
100 trials
→ 2 beams × ~50/100 trials ≈ 1
→ ~1 beam × ~50/100 trials ≈ 0.5
floodlit
→ 2 beams
trials – ¼ × floodlit
→ 1 – ¼ ( 2 ) ≈ 0.5
→ – ¼ ( 2 ) ≈ 0
Time =

35. Synthetic coded-aperture confocal imaging
note all the tildas in the formulas
this algorithm is statistical in nature
for example, if we flip a coin to decide whether to illumninate a particular tile on a particular trial
the binomial theorem tells us how much variability we’ll get over a given number of trials
the effect of this variability
the image of our focal plane will be slightly non-uniform, and
objects off the focal plane won’t be entirely dark after the confocal subtraction
but for visual purposes
our technique works well with a modest number of trials, like 16
far fewer than would be required to scan out the focal plane, as in the usual confocal scanning algorithm
something else about this variability...
100 trials
→ 2 beams × ~50/100 trials ≈ 1
→ ~1 beam × ~50/100 trials ≈ 0.5
floodlit
→ 2 beams
trials – ¼ × floodlit
→ 1 – ¼ ( 2 ) ≈ 0.5
→ – ¼ ( 2 ) ≈ 0
50% light efficiency
any number of projectors ≥ 2
no discrimination to left of
works with relatively few trials (~16)
Time =

36. Synthetic coded-aperture confocal imaging
let’s examine the red dot in isolation
if due to this variability it is not entirely dark
then we at least want the beams affecting its color, shown here with red dashed lines
to be uncorrelated with the beams affecting the color of adjacent points, like this orange one
so that objects like this off the focal plane don’t develop objectionable spatial patterns
for this to happen
we need patterns in which the illumination of different tiles are spatially uncorrelated
100 trials
→ 2 beams × ~50/100 trials ≈ 1
→ ~1 beam × ~50/100 trials ≈ 0.5
floodlit
→ 2 beams
trials – ¼ × floodlit
→ 1 – ¼ ( 2 ) ≈ 0.5
→ – ¼ ( 2 ) ≈ 0
50% light efficiency
any number of projectors ≥ 2
no discrimination to left of
works with relatively few trials (~16)
needs patterns in which illumination of tiles are uncorrelated
Time =

37. Example pattern
0:15
Time =

38. Patterns with less aliasing
Time =

39. Implementation using an array of mirrors
4:00 up to and including scattering media
Time =

40. 4:00 up to and including scattering media
Time =

41. Confocal imaging in scattering media
0:30, 1:30 for WHOI setup
I observed a confocal effect
but it was modest
theory said the effect should be stronger
indeed,
it is stronger
as I confirmed this summer...
small tank
too short for attenuation
lit by internal reflections
Time =

42. Experiments in a large water tank
1:00
50-foot flume at Wood’s Hole Oceanographic Institution (WHOI)
Time =

43. Experiments in a large water tank
titanium dioxide
the stuff in white paint
4-foot viewing distance to target
surfaces blackened to kill reflections
titanium dioxide in filtered water
transmissometer to measure turbidity
Time =

44. Experiments in a large water tank
stray light limits performance
one projector suffices if no occluders
Time =

45. Seeing through turbid water
0:45, 1:45 for WHOI results
this is very turbid water
the “attenuation length” (a technical term that roughly translates to “how far you can see clearly”) is about 8 inches
and I’m trying to see through 4 feet
if you contrast enhance these images,
you can see the improvement in signal-to-noise ratio
floodlit
scanned tile
Time =

46. Application to underwater exploration
0:30
so I think we can expect
video projectors being mounted on future underwater vehicles
this is the Hercules remotely operated vehicle
exploring the wreck of the Titanic two months ago in the North Atlantic
the question is
can you produce an overhead view like this, of the Titanic
in a single shot taken from far away using shaped illumination
rather than by mowing the lawn with the underwater vehicle
which is difficult, dangerous, time consuming, and produces a mosaic with parallax errors
[Ballard/IFE 2004]
[Ballard/IFE 2004]
Time =

47. Research challenges in SAP and SAI
1:45, 2:45 until end
Where do we go from here?
sampling patterns
meaning – arrangements of cameras or projectors
There are many optical design problems here
theory
aperture shapes and sampling patterns
illumination patterns for confocal imaging
optical design
How to arrange cameras, projectors, lenses, mirrors, and other optical elements?
How to compare the performance of different arrangements (in foliage, underwater,...)?
Time =

48. Challenges (continued)
if you agree that
arrays of cameras and projectors are a good idea, then
How can we build them compactly and inexpensively?
systems design
multi-camera or multi-projector chips
communication in camera-projector networks
calibration in long-range or mobile settings
algorithms
tracking and stabilization of moving objects
compression of dense multi-view imagery
shape from light fields
Time =

49. Challenges (continued)
one of my favorites is
in the theater
perform confocal imaging of an actor on a stage, in real-time, using imperceptible structured light generated by an ultra-high-speed projector (such as my co-authors are exhibiting in etech)
to produce an moving matte for the actor, then
use that matte to throw a spotlight only on the actor,
without casting a shadow on the stage, and
without creating a pool of light on the ground!
applications of confocal imaging
remote sensing and surveillance
shape measurement
scientific imaging
applications of shaped illumination
shaped searchlights for surveillance
shaped headlamps for driving in bad weather
selective lighting of characters for stage and screen
Time =

50. Computational imaging in other fields
0:30
let me end with this thought...
the techniques I’ve presented today
arose from a study of computational imaging in other scientific disciplines
I think we’ve only scratched the surface here
for example,
the original idea for light field rendering 10 years ago,
creating new perspective images by rebinning other images,
was inspired by a technique called rebinning in medical imaging
many of the ideas in the present paper
came out of studying synthetic aperture radar and coded-aperture imaging in astronomy
medical imaging
rebinning
tomography
airborne sensing
multi-perspective panoramas
synthetic aperture radar
astronomy
coded-aperture imaging
multi-telescope imaging
Time =

51. Computational imaging in other fields
there are many other fields
with interesting computational techniques
that might have application in computer graphics or computer vision
geophysics
accoustic array imaging
borehole tomography
biology
confocal microscopy
deconvolution microscopy
physics
optical tomography
inverse scattering
Time =

52. The team staff students collaborators funding Mark Horowitz Marc Levoy
0:30
staff
Mark Horowitz
Marc Levoy
Bennett Wilburn
students
Billy Chen
Vaibhav Vaish
Katherine Chou
Monica Goyal
Neel Joshi
Hsiao-Heng Kelin Lee
Georg Petschnigg
Guillaume Poncin
Michael Smulski
Augusto Roman
collaborators
Mark Bolas
Ian McDowall
Guillermo Sapiro
funding
Intel
Sony
Interval Research
NSF
DARPA
Time =

53. Related papers The Light Field Video Camera
Bennett Wilburn, Michael Smulski, Hsiao-Heng Kelin Lee, and Mark Horowitz
Proc. Media Processors 2002, SPIE Electronic Imaging 2002
Using Plane + Parallax for Calibrating Dense Camera Arrays
Vaibhav Vaish, Bennett Wilburn, Neel Joshi,, Marc Levoy
Proc. CVPR 2004
High Speed Video Using a Dense Camera Array
Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc Levoy, Mark Horowitz
Spatiotemporal Sampling and Interpolation for Dense Camera Arrays
Bennett Wilburn, Neel Joshi, Katherine Chou, Marc Levoy, Mark Horowitz
ACM Transactions on Graphics (conditionally accepted)
Time =

54.
Time =