1. Dept. Elect. Eng. Technion – Israel Institute of Technology Radiometric Nonidealities: A Unified Framework Anatoly Litvinov, Yoav Y. Schechner Support: Ollendorff Foundation (BMBF), GIF 1

2. Spatial Non-Uniformity camera Iris A. Litvinov & Y. Schechner, radiometric nonidealities 2

3. Spatial Non-Uniformity iris Automatic Gain Control (AGC) time gain Temporal Non-Uniformity: Litvinov, Schechner, radiometric nonidealities

4. Automatic Gain Control (AGC) A. Litvinov & Y. Schechner, radiometric nonidealities AGC G

5. Image Mosaicing Standard seam removal: “feathering” Levin; Zomet; Peleg; Weiss (04) Duplaquet (98) Burt; Adelson (83) Not addressing the root of inconsistencies Soderblum et. al (78) Shum; Szeliski (00) Jia; Tang (03) A. Litvinov & Y. Schechner, radiometric nonidealities 3

6. electronic readout light intensity Fill factory CMOS chip Nonlinear Radiometric Response 4

7. Image Intensifier Photo cathode Fiber optic coupler CCD Accelerating potential A. Litvinov & Y. Schechner, radiometric nonidealities Phosphor screen: nonlinear Nonlinear Radiometric Response 5

8. The Eye A. Litvinov & Y. Schechner, radiometric nonidealities Spatial non-uniformity Auto gain Radiometric response 42

9. Prior Techniques Pre-calibrations Standard target Integrating sphere Exposure variations Debevec; Malik (98) Mitsunaga; Nayar (99) Mann; Picard (95) Based on edge detection Lin et al. (04) AND OTHERS … Spatial non-uniformity estimation Schechner; Nayar (01) Only radiometric response Only spatial non-uniformity Only radiometric response Only radiometric response Only spatial non-uniformity Combined AGC and radiometric response Mann; Mann (01)Kim; Pollefeys (04) not including spatial non-uniformity Candocia; Mandarino (05) 6

10. Integrating Sphere A. Litvinov & Y. Schechner, radiometric nonidealities Spatial non-uniformity pre-calibration visible mismatch 97 % - not accurate enough

11. We Achieve : Blind estimation of: spatial non-uniformity pixelelectronic readout light intensity A. Litvinov & Y. Schechner, radiometric nonidealities non-uniformity AGC nonlinearity 7

12. Without “seam-removal” steps A. Litvinov & Y. Schechner, radiometric nonidealities Blind estimation of: pixelelectronic readout light intensity non-uniformity AGC nonlinearity 8

13. Camera Model G M(x)M(x) E r(I)r(I) A. Litvinov & Y. Schechner, radiometric nonidealities 10

14. irradiance of pixel x spatial non-uniformity of x radiometric response grayscale value of x r[ ] I(x)I(x) M(x)M(x) v(x) = log{r [v(x)]} = log(G) + log[M(x)] + log[I(x)] ρ(v)ρ(v) m(x)m(x) G g gain ρ(v) - m(x) – g = i i A. Litvinov & Y. Schechner, radiometric nonidealities r log

15. A. Litvinov & Y. Schechner, radiometric nonidealities ρ (255) m (x=1) m (x=N) 1 -1 -1 = g 1 g F ρ(v) - m(x) – g = i 1 -1 -1 i 1 i N ρ (0) 1 -1 -1 i s B = 12

16. Bs = i B γ s = γ i exp ρ(v) = γ ρ (v) ^ m(x) = γ m(x) ^ g = γ g ^ ff r (v) = [r (v)] ^ γ M(x) = M(x) γ ^ G = G ^ f f γ A. Litvinov & Y. Schechner, radiometric nonidealities 13

17. An Image Invariance log I = γ log I ^ A. Litvinov & Y. Schechner, radiometric nonidealities 14

18. Nevertheless … Mutually Consistent A. Litvinov & Y. Schechner, radiometric nonidealities 15

19. ρ(v ) – m(x ) - g = log[I(x)] k k pp Frame: k Frame: p A. Litvinov & Y. Schechner, radiometric nonidealities k p ρ (v ) - ρ(v ) - m(x ) + m(x ) - g + g = 0 p pkk k p m(x) : log of mask at pixel x  (v) : log of inverse-radiometric response g : log of gain at frame f : f 16

20. A. Litvinov & Y. Schechner, radiometric nonidealities ρ (0) ρ (255) m (x=1) m (x=N) 1 -1 -1 1 -1 1 -1 1 -1 1 -1 1 = 0 g 1 g F 1 -1 -1 1 -1 1 ρ (v ) - ρ(v ) - m(x ) + m(x ) - g + g = 0 p pkk k p Rs = 0

21. Experiment: AGC, Vignetting, Gamma pixel 0 460 1 transmittance (vignetting) frame (time) 1 4 15 0 amplifier gain 1 0 255 graylevel irradiance radiometric response 17

22. SVD solution Rs = 0 Null space of R s = 0 s = [ ρ = const, m = const, g = const ] T A. Litvinov & Y. Schechner, radiometric nonidealities 123

23. Trivial Solutions ρ(v) = const light intensity electronic readout exp r r light intensity electronic readout A. Litvinov & Y. Schechner, radiometric nonidealities

24. Entropy Maximization A. Litvinov & Y. Schechner, radiometric nonidealities hist

25. Anatoly Litvinov Yoav Schechner A Unified Framework 1 4 15 0 1 0 255 0 460 1 log I = γ log I ^ Image invariance 18