1. Multiplexed Illumination
Yoav Y. Schechner
Technion, Israel
Shree Nayar, Peter Belhumeur
Columbia University
Acknowledgments: Taub Foundation, BSF, NSF, Harish Peri

2. Georghiades, Belhumeur & Kriegman
Yale Face Database B
Shape Recovery
BRDF
Material Recognition
Human Vision
Rendering
Object / Face Recognition

3. Image-Based Rendering
Synthesized illumination
Raw images: individual sources
based on
Schechner et. al. Multiplexed Illumination

4. Schechner, Nayar, Belhumeur Multiplexed Illumination
Our Method
High quality
New capabilities
Very simple, yet flexible setup
Schechner, Nayar, Belhumeur Multiplexed Illumination

5. Schechner et. al. Multiplexed Illumination
Image-Based Rendering / Recognition + +
Schechner et. al. Multiplexed Illumination

6. Illumination Direction Resolution
Low res.
High res.
Schechner et. al. Multiplexed Illumination

7. Trade-Off resolution vs. Illumination direction Image intensity
Schechner et. al. Multiplexed Illumination

8. Trade-Off ~ N resolution vs. signal = SNR N elements noise
Image intensity 1 N ~
= SNR
signal
noise
Exposure time
N elements
Illumination direction
resolution
N independent images
Schechner, Nayar, Belhumeur Multiplexed Illumination

9. ~ N resolution vs. signal = SNR N elements noise N independent images
Image intensity 1 N ~
= SNR
signal
noise
Exposure time
N elements
Illumination direction
resolution
N independent images
Schechner, Nayar, Belhumeur Multiplexed Illumination

10. Standard Illumination
Multiplexed Illumination

11. Multiplexed Illumination
i = intensity under source k
a = acquired measurement 1 2 3 k a 1 1 2 3 i = 1 2 3 a 2
i a
i = a
i a 1 2 3
Estimating i 1 2 3 a 3

12. Standard Illumination
i = intensity under source k
i = estimate of i
a = acquired measurement k 1 2 3
a = i s
a i
a i - 1 2 3 1 2 3
i = a
i a
i a 1 2 3
+ s 1 2 3

13. Multiplexed Illumination
i = intensity under source k
i = estimate of i
a = acquired measurement k 1 2 3
a = i
a i
a i 1 2 3 1 2 3
+ s
i a
i = a
i a 1 2 3 1 2 3 s 3 4

14. Multiplexed Illumination
Demultiplexed Images

15. W W Theory of Multiplexed Illumination W W s i
i = intensities under individual sources
a = acquired measurements
For each pixel W
a = i
+ s
i = a W -1
Variance ( ) = i 2 s N
Trace [ ] -1 t
W W
Minimize

16. Optimal Multiplexing Codes
Based on Hadamard Codes
Spectroscopy, X-ray astronomy: Harwit & Sloane 1979
Trace [ ] -1 t
W W
Minimize
Schechner, Nayar, Belhumeur Multiplexed Illumination

17. Optimal Multiplexing Codes
Based on Hadamard Codes
Spectroscopy, X-ray astronomy: Harwit & Sloane 1979
Solution W
w = or 0
m,s
source On/Off 2
N+1
1’s
N-1
0’s
~ half the sources are On
Schechner, Nayar, Belhumeur Multiplexed Illumination

18. Schechner, Nayar, Belhumeur Multiplexed Illumination
Easy to Invert 1 4 W -1 =
SNR
multiplex
single = 2
+1/ N ~
Schechner, Nayar, Belhumeur Multiplexed Illumination

19. Schechner, Nayar, Belhumeur Multiplexed Illumination
demultiplexed
standard
Fixed acquisition time T
SNR 2 N =
SNR
multiplex
single
Schechner, Nayar, Belhumeur Multiplexed Illumination

20. Schechner, Nayar, Belhumeur Multiplexed Illumination
demultiplexed
standard
Fixed acquisition time T
SNR 2 N =
SNR
multiplex
single
Fixed high quality
SNR
multiplex
demultiplexed
standard T T = 2 N
multiplex
single
standard
Schechner, Nayar, Belhumeur Multiplexed Illumination

21. SNR N = SNR SNR T T = N T N = SNR 2 2 0.6 multiplex single multiplex
demultiplexed
standard
Fixed acquisition time T
SNR 2 N =
SNR
multiplex
single
Fixed high quality
SNR
multiplex
demultiplexed
standard T T = 2 N
multiplex
single
standard
Fixed acquisition time T
single
demultiplexed
Fixed high quality
0.6 N
multiplex
single = 3
SNR
multiplex

22. Schechner, Nayar, Belhumeur Multiplexed Illumination
Setup Scalable, Simple, Flexible
projector
Schechner, Nayar, Belhumeur Multiplexed Illumination

23. Illumination Patterns
Multiplexed Illumination
Single-Source Illumination
Schechner, Nayar, Belhumeur Multiplexed Illumination

24. Schechner, Nayar, Belhumeur Multiplexed Illumination
Setup
Schechner, Nayar, Belhumeur Multiplexed Illumination

25. Schechner, Nayar, Belhumeur Multiplexed Illumination
Anistropic illumination 1 2 -1 -2 -3 -4 -5
Schechner, Nayar, Belhumeur Multiplexed Illumination

26. Adaptive directional resolution
Schechner, Nayar, Belhumeur Multiplexed Illumination

27. Schechner, Nayar, Belhumeur Multiplexed Illumination
The Quadtree algorithm
bright
dark
dark
bright
dark
bright
bright
bright
Schechner, Nayar, Belhumeur Multiplexed Illumination

28. Schechner, Nayar, Belhumeur Multiplexed Illumination
The Quadtree algorithm
bright
bright
bright
Schechner, Nayar, Belhumeur Multiplexed Illumination

29. Schechner, Nayar, Belhumeur Multiplexed Illumination
Raw Images (Diffuse Objects)
Multiplexed images
Single-source images
Schechner, Nayar, Belhumeur Multiplexed Illumination

30. Schechner, Nayar, Belhumeur Multiplexed Illumination
Single-Source Images
Schechner, Nayar, Belhumeur Multiplexed Illumination

31. Schechner, Nayar, Belhumeur Multiplexed Illumination
Demultiplexed (Decoded) Images
Schechner, Nayar, Belhumeur Multiplexed Illumination

32. Schechner, Nayar, Belhumeur Multiplexed Illumination
Specular Objects
Schechner, Nayar, Belhumeur Multiplexed Illumination

33. Schechner, Nayar, Belhumeur Multiplexed Illumination
Specular Objects
Schechner, Nayar, Belhumeur Multiplexed Illumination

34. Schechner, Nayar, Belhumeur Multiplexed Illumination

35. noise std [gray-levels]
Quantitative verification
Single-source
noise std [gray-levels]
Demultiplexed
measurement samples
Average ratio = 7.97
= Expected ratio = 8.02 2
+1/ N

36. High Definition Specularities
Image-Based Rendering
Raw Images
Schechner, Nayar, Belhumeur Multiplexed Illumination

37. Schechner, Nayar, Belhumeur Multiplexed Illumination
Color l
intensity
blue
green
red
cyan
yellow
magenta
Schechner, Nayar, Belhumeur Multiplexed Illumination

38. Schechner, Nayar, Belhumeur Multiplexed Illumination
Limit
Saturation
Saturation
Benefit
Highlights
Diffuse Object
Schechner, Nayar, Belhumeur Multiplexed Illumination

39. Camera Noise Model - Revisited
Detector (pixel)
50% efficiency
Light energy E
light
Electric energy
elect E
light 2 =
Schechner, Nayar, Belhumeur Multiplexed Illumination

40. Schechner, Nayar, Belhumeur Multiplexed Illumination
Photon (shot) Noise
Detector (pixel)
Electrons or e {
nothing
Photon
either
50% quantum efficiency
Schechner, Nayar, Belhumeur Multiplexed Illumination

41. Schechner, Nayar, Belhumeur Multiplexed Illumination
Photon (shot) Noise
Electrons e {
nothing
Photons
either
50% quantum efficiency
Schechner, Nayar, Belhumeur Multiplexed Illumination

42. Camera Noise Model - Revisited50
100
150 I
1.5 2
0.5 1
250
200 s
readout noise variance
shot noise variance
* Point-Grey Dragonfly, Gain 3.5 dB, 30fps
Schechner, Nayar, Belhumeur Multiplexed Illumination

43. Schechner, Nayar, Belhumeur Multiplexed Illumination
PtGrey Dragonfly 9 8 4 2 1
20%
40%
60%
80%
saturation limit
N=255
SNR Gain = 8.02
SNR Gain
Dalsa 1M75
Pixelink
Redlake MotionPro
PCO Sensican
Schechner, Nayar, Belhumeur Multiplexed Illumination

44. Yoav Schechner Shree Nayar Peter Belhumeur
Conclusions
Multiplexing: Higher quality
Very simple, yet flexible setup
Yoav Schechner Shree Nayar Peter Belhumeur