1. Computer Science Department
Synthetic aperture photography and illumination using arrays of cameras and projectors
Marc Levoy
Computer Science Department
Stanford University
34:15 total + 30% = ~45 minutes
Time =

2. Outline technologies optical effects applications examples
large camera arrays
large projector arrays
camera–projector arrays
optical effects
synthetic aperture photography
synthetic aperture illumination
synthetic confocal imaging
applications
partially occluding environments
weakly scattering media
examples
foliage
murky water
Time =

3. Multi-camera systems multi-camera vision systems
Kang’s multi- baseline stereo
multi-camera
as opposed to a single moving camera, allows a snapshot of a dynamic scene
single viewpoint versus multiple viewpoint
narrow FOV or panoramic FOV
camera arrays
1D versus 2D
dense versus sparse
other issues
single snapshot versus video
multi-camera vision systems
omni-directional vision
1D camera arrays
2D camera arrays
Nayar’s
Omnicam
Kanade’s 3D room
Immersive Media’s dodeca camera
Manex’s bullet time array
Time =

4. Stanford multi-camera array
snapshot or video
at different quality levels
multiple viewpoint, usually dense
but if sufficiently dense relative to the distance to the scene, it is “nearly” a single viewpoint
we’ll come back to this point shortly...
usually 2D array
although 1D arrangements are possible
inward-looking
“inverse panoramic”
640 × 480 pixels × 30 fps × 128 cameras ÷ 18:1 MPEG = 512 Mbs
snapshot or video
synchronized timing
continuous streaming
cheap sensors & optics
flexible arrangement
Time =

5. Applications for the array
tightly packed
scene is far away, or
range of depths is small, or
cameras are very close together, so that
all cameras see about the same view
can be described quantitatively in terms of disparity or parallax, but
that’s beyond the scope of this talk
How are the cameras arranged?
tightly packed high-performance imaging
widely spaced light fields
intermediate spacing synthetic aperture photography
Time =

6. Cameras tightly packed: high-performance imaging
16 x 8 array of VGA cameras = 10,240 x 3,840 pixel camera 
high-resolution
by abutting the cameras’ fields of view
high speed
by staggering their triggering times
high dynamic range
mosaic of shutter speeds, apertures, density filters
high precision
averaging multiple images improves contrast
high depth of field
mosaic of differently focused lenses
Time =

7. Abutting fields of view
16 x 8 array of VGA cameras = 10,240 x 3,840 pixel camera
Q. Can we align images this well?
A. Yes.
Time =

8. Cameras tightly packed: high-performance imaging
high-resolution
by abutting the cameras’ fields of view
high speed
by staggering their triggering times
high dynamic range
mosaic of shutter speeds, apertures, density filters
high precision
averaging multiple images improves contrast
high depth of field
mosaic of differently focused lenses 
Time =

9. High-speed photography
Harold Edgerton, Stopping Time, 1964
Time =

10. A virtual high-speed video camera [Wilburn, 2004 (submitted) ]
52 cameras, each 30 fps
packed as closely as possible
staggered firing, short exposure
result is 1560 fps camera
continuous streaming
no triggering needed
Time =

11. Example
Time =

12. A virtual high-speed video camera [Wilburn, 2004 (submitted) ]
52 cameras, 30 fps, 640 × 480
packed as closely as possible
short exposure, staggered firing
result is 1536 fps camera
continuous streaming
no triggering needed
scalable to more cameras
limited by available photons
overlap exposure times?
100 cameras
3072 fps
Time =

13. Cameras tightly packed: high-X imaging
in a digital camera
move lens over a sequence of frames
requires image stabilization
high-resolution
by abutting the cameras’ fields of view
high speed
by staggering their triggering times
high dynamic range
mosaic of shutter speeds, apertures, density filters
high precision
averaging multiple images improves contrast
high depth of field
mosaic of differently focused lenses 
Time =

14. High dynamic range (HDR)
negative film
actually 1000:1, but top and bottom are non-linear, so usually ignored
increased dynamic range
detail in shadows without blowing out highlights
or equivalently, low-noise in shadows
overcomes one of photography’s key limitations
negative film = 250:1 (8 stops)
paper prints = 50:1
[Debevec97] = 250,000:1 (18 stops)
hot topic at recent SIGGRAPHs 
Time =

15. Cameras tightly packed: high-X imaging
in a digital camera
move lens over a sequence of frames
requires image stabilization
high-resolution
by abutting the cameras’ fields of view
high speed
by staggering their triggering times
high dynamic range
mosaic of shutter speeds, apertures, density filters
high precision
averaging multiple images improves contrast
high depth of field
mosaic of differently focused lenses 
Time =

16. Seeing through murky water
scattering decreases contrast
noise limits perception in low contrast images
averaging multiple images decreases noise
Time =

17. Seeing through murky water
scattering decreases contrast, but does not blur
noise limits perception in low contrast images
averaging multiple images decreases noise
Time =

18. Seeing through murky water
16 images
1 image
Time =

19. Cameras tightly packed: high-X imaging
in a digital camera
move lens over a sequence of frames
requires image stabilization
high-resolution
by abutting the cameras’ fields of view
high speed
by staggering their triggering times
high dynamic range
mosaic of shutter speeds, apertures, density filters
high precision
averaging multiple images improves contrast
high depth of field
mosaic of differently focused lenses 
Time =

20. High depth-of-field adjacent views use different focus settings
in a digital camera
move lens over a sequence of frames
requires image stabilization
adjacent views use different focus settings
for each pixel, select sharpest view
[Haeberli90]
close focus
distant focus
composite
Time =

21. Synthetic aperture photography
Time =

22. Synthetic aperture photography
Time =

23. Synthetic aperture photography
Taken to the extreme
objects off the focal plane become so blurry that they effectively disappear
at least if they are smaller than the aperture
Leonardo noticed 500 years ago
a needle placed in front of his eye
because it was smaller than his pupil
did not occlude his vision
Time =

24. Synthetic aperture photography
Time =

25. Synthetic aperture photography
Time =

26. Synthetic aperture photography
Another way to think about synthetic aperture photography
take the images from all the cameras,
rectify them to a common plane
shift them by a certain amount,
and add them together
Objects that become aligned by the shifting process
will be sharply focused
objects in front of that plane...
objects in back of that plane... 
Time =

27. Long-range synthetic aperture photography
Time =

28. Synthetic pull-focus
Time =

29. Crowd scene hemi-circular motion lego sheriff behind most soldiers
but one is just off-frame
other people in same plane wouldn’t occlude sheriff anyway
Time =

30. Crowd scene hemi-circular motion lego sheriff behind most soldiers
but one is just off-frame
other people in same plane wouldn’t occlude sheriff anyway
Time =

31. Synthetic aperture photography using an array of mirrors?
11-megapixel camera
22 planar mirrors
Time =

32. Time =

33. Time =

34. Synthetic aperture illumation
if you replace cameras with projectors
many of the principles I’ve talked about still apply
in particular,
an array of projectors, suitably aligned, produces a real image in space
with such a shallow depth of field that
if you stick your hand astride the image, or a white angled screen in this case,
the image is focused only at one place on your hand,
it’s blurry in front of and behind that point
in this case
I’m using only a few projectors, so there’s some aliasing off the focal plane
if I had enough projectors, it would be a smooth blur
Time =

35. Synthetic aperture illumation
single projector + array of mirrors
as we do in this paper
bright display
we wear sunglasses in our laboratory when looking at these images
autostereoscopic display
i.e. a light field display
technologies
array of projectors
array of microprojectors
single projector + array of mirrors
applications
bright display
autostereoscopic display [Matusik 2004]
confocal imaging [this paper]
Time =

36. Confocal scanning microscopy
at this point
I need to back up and tell you a bit about confocal microscopy
if you introduce a pinhole
only one point on the focal plane will be illuminated
light source
pinhole
Time =

37. Confocal scanning microscopy
...and a matching optical system,
hence the word confocal
this green dot
will be both strongly illuminated and sharply imaged
while this red dot
will have less light falling on it by the square of distance r,
because the light is spread over a disk
and it will also be more weakly imaged by the square of distance r,
because its image is blurred out over an disk on the pinhole mask, and only a little bit is permitted through
so the extent to which the red dot contributes to the final image
falls off as the fourth power of r, the distance from the focal plane
pinhole
light source r
photocell
pinhole
Time =

38. Confocal scanning microscopy
of course, you’ve only imaged one point
so you need to move the pinholes
and scan across the focal plane
light source
pinhole
pinhole
photocell
Time =

39. Confocal scanning microscopy
light source
pinhole
pinhole
photocell
Time =

40. [UMIC SUNY/Stonybrook]
the object in the lower-right image is actually spherical,
but portions of it that are off the focal plane are both blurry and dark,
effectively disappearing
[UMIC SUNY/Stonybrook]
Time =

41. Synthetic confocal scanning
our goal
is to approximate this effect at the large scale
we can understand the photometry of this setup
using a simplified counting argument
if we had 5 cameras
then the ratio would be 25:1 instead of 5:1
light source
→ 5 beams
→ 0 or 1 beam
Time =

42. Synthetic confocal scanning
5:0 or 5:1
if we had 5 cameras as well as 5 projectors, then the ratio would be 25:0 or 25:1
light source
→ 5 beams
→ 0 or 1 beam
Time =

43. Synthetic confocal scanning
depth of field
a microscoper would call it the axial resolution
to make the depth of field shallower, spread out the projectors, i.e. a larger synthetic aperture
→ 5 beams
→ 0 or 1 beam
d.o.f.
works with any number of projectors ≥ 2
discrimination degrades if point to left of
no discrimination for points to left of
slow!
poor light efficiency
Time =

44. Synthetic coded-aperture confocal imaging
different from coded aperture imaging in astronomy
[Wilson, Confocal Microscopy by Aperture Correlation, 1996]
Time =

45. Synthetic coded-aperture confocal imaging
Time =

46. Synthetic coded-aperture confocal imaging
Time =

47. Synthetic coded-aperture confocal imaging
Time =

48. Example pattern
0:15
Time =

49. Patterns with less aliasing
Time =

50. Implementation using an array of mirrors
Time =

51. (video available at http://graphics.stanford.edu/papers/confocal/)
Time =

52. Synthetic aperture confocal imaging
single viewpoint
synthetic aperture image
confocal image
combined
Time =

53. Selective illumination using object-aligned mattes
Time =

54. Confocal imaging in scattering media
I observed a confocal effect
but it was modest
theory said the effect should be stronger
indeed,
it is stronger
as I confirmed this summer...
small tank
too short for attenuation
lit by internal reflections
Time =

55. Experiments in a large water tank
50-foot flume at Wood’s Hole Oceanographic Institution (WHOI)
Time =

56. Experiments in a large water tank
titanium dioxide
the stuff in white paint
4-foot viewing distance to target
surfaces blackened to kill reflections
titanium dioxide in filtered water
transmissometer to measure turbidity
Time =

57. Experiments in a large water tank
stray light limits performance
one projector suffices if no occluders
Time =

58. Seeing through turbid water
this is very turbid water
the “attenuation length” (a technical term that roughly translates to “how far you can see clearly”) is about 8 inches
and I’m trying to see through 4 feet
if you contrast enhance these images,
you can see the improvement in signal-to-noise ratio
floodlit
scanned tile
Time =

59. Other patterns sparse grid staggered grid swept stripe
to speed things up,
one can use patterns that illuminate several tiles at once
similar strategies have been used in confocal microscopy
the problem with this one
the illumination beams (coming in from the right side) intersect the lines of sight to other tiles, degrading contrast
here’s a pattern in which they don’t
a staggered grid
here’s another pattern in which they don’t
a simple swept stripe
with the light coming in from the right side
staggered grid
sparse grid
swept stripe
Time =

60. Other patterns floodlit swept stripe scanned tile
here’s how a swept stripe stacks up against the other patterns
somewhere in the middle, in terms of quality
a swept stripe has a number of advantages, though...
floodlit
swept stripe
scanned tile
Time =

61. Stripe-based illumination
no sweeping is needed
the forward motion of the vehicle will sweep out the stripe!
in addition, since the stripe is coming from the side...
essentially constitutes a structured-light rangefinder
this is not a new idea
Jules Jaffe proposed it 14 years ago (in a paper in Oceanic Engineering)
but he had no technology to implement it
compact video projectors provide that technology
so we can easily envision
video projectors being mounted on future underwater vehicles
this is the Hercules remotely operated vehicle
exploring the wreck of the Titanic two months ago in the North Atlantic
if vehicle is moving, no sweeping is needed!
can triangulate from leading (or trailing) edge of stripe, getting range (depth) for free
[Jaffe90]
Time =

62. sum of floodlit
floodlit
swept line
scanned tile
Time =

63. Strawman conclusions on imaging through a scattering medium
2-3x further
precise results will be published in a forthcoming paper
regardless of pattern
that’s why individual tiles are better than a stripe
shaping the illumination lets you see 2-3x further, but requires scanning or sweeping
use a pattern that avoids illuminating the media along the line of sight
contrast degrades with increasing total illumination, regardless of pattern
Time =

64. Application to underwater exploration
so I think we can expect
video projectors being mounted on future underwater vehicles
this is the Hercules remotely operated vehicle
exploring the wreck of the Titanic two months ago in the North Atlantic
the question is
can you produce an overhead view like this, of the Titanic
in a single shot taken from far away using shaped illumination
rather than by mowing the lawn with the underwater vehicle
which is difficult, dangerous, time consuming, and produces a mosaic with parallax errors
[Ballard/IFE 2004]
[Ballard/IFE 2004]
Time =

65. The team staff students collaborators funding Mark Horowitz Marc Levoy
Bennett Wilburn
students
Billy Chen
Vaibhav Vaish
Katherine Chou
Monica Goyal
Neel Joshi
Hsiao-Heng Kelin Lee
Georg Petschnigg
Guillaume Poncin
Michael Smulski
Augusto Roman
collaborators
Mark Bolas
Ian McDowall
Guillermo Sapiro
funding
Intel
Sony
Interval Research
NSF
DARPA
Time =

66. Relevant publications
(in reverse chronological order)
Spatiotemporal Sampling and Interpolation for Dense Camera Arrays
Bennett Wilburn, Neel Joshi, Katherine Chou, Marc Levoy, Mark Horowitz
ACM Transactions on Graphics (conditionally accepted)
Interactive Design of Multi-Perspective Images for Visualizing Urban Landscapes
Augusto Román, Gaurav Garg, Marc Levoy
Proc. IEEE Visualization 2004
Synthetic aperture confocal imaging
Marc Levoy, Billy Chen, Vaibhav Vaish, Mark Horowitz, Ian McDowall, Mark Bolas
Proc. SIGGRAPH 2004
High Speed Video Using a Dense Camera Array
Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc Levoy, Mark Horowitz
Proc. CVPR 2004
The Light Field Video Camera
Bennett Wilburn, Michael Smulski, Hsiao-Heng Kelin Lee, and Mark Horowitz
Proc. Media Processors 2002, SPIE Electronic Imaging 2002
Time =

67.
Time =