Dept. Elect. Eng. Technion – Israel Institute of Technology Radiometric Nonidealities: A Unified Framework Anatoly Litvinov, Yoav Y. Schechner Support: Ollendorff Foundation (BMBF), GIF 1

Spatial Non-Uniformity camera Iris A. Litvinov & Y. Schechner, radiometric nonidealities 2

Spatial Non-Uniformity iris Automatic Gain Control (AGC) time gain Temporal Non-Uniformity: Litvinov, Schechner, radiometric nonidealities

Automatic Gain Control (AGC) A. Litvinov & Y. Schechner, radiometric nonidealities AGC G

Image Mosaicing Standard seam removal: “feathering” Levin; Zomet; Peleg; Weiss (04) Duplaquet (98) Burt; Adelson (83) Not addressing the root of inconsistencies Soderblum et. al (78) Shum; Szeliski (00) Jia; Tang (03) A. Litvinov & Y. Schechner, radiometric nonidealities 3

electronic readout light intensity Fill factory CMOS chip Nonlinear Radiometric Response 4

Image Intensifier Photo cathode Fiber optic coupler CCD Accelerating potential A. Litvinov & Y. Schechner, radiometric nonidealities Phosphor screen: nonlinear Nonlinear Radiometric Response 5

The Eye A. Litvinov & Y. Schechner, radiometric nonidealities Spatial non-uniformity Auto gain Radiometric response 42

Prior Techniques Pre-calibrations Standard target Integrating sphere Exposure variations Debevec; Malik (98) Mitsunaga; Nayar (99) Mann; Picard (95) Based on edge detection Lin et al. (04) AND OTHERS … Spatial non-uniformity estimation Schechner; Nayar (01) Only radiometric response Only spatial non-uniformity Only radiometric response Only radiometric response Only spatial non-uniformity Combined AGC and radiometric response Mann; Mann (01)Kim; Pollefeys (04) not including spatial non-uniformity Candocia; Mandarino (05) 6

Integrating Sphere A. Litvinov & Y. Schechner, radiometric nonidealities Spatial non-uniformity pre-calibration visible mismatch 97 % - not accurate enough

We Achieve : Blind estimation of: spatial non-uniformity pixelelectronic readout light intensity A. Litvinov & Y. Schechner, radiometric nonidealities non-uniformity AGC nonlinearity 7

Without “seam-removal” steps A. Litvinov & Y. Schechner, radiometric nonidealities Blind estimation of: pixelelectronic readout light intensity non-uniformity AGC nonlinearity 8

Camera Model G M(x)M(x) E r(I)r(I) A. Litvinov & Y. Schechner, radiometric nonidealities 10

irradiance of pixel x spatial non-uniformity of x radiometric response grayscale value of x r[ ] I(x)I(x) M(x)M(x) v(x) = log{r [v(x)]} = log(G) + log[M(x)] + log[I(x)] ρ(v)ρ(v) m(x)m(x) G g gain ρ(v) - m(x) – g = i i A. Litvinov & Y. Schechner, radiometric nonidealities r log

A. Litvinov & Y. Schechner, radiometric nonidealities ρ (255) m (x=1) m (x=N) = g 1 g F ρ(v) - m(x) – g = i i 1 i N ρ (0) i s B = 12

Bs = i B γ s = γ i exp ρ(v) = γ ρ (v) ^ m(x) = γ m(x) ^ g = γ g ^ ff r (v) = [r (v)] ^ γ M(x) = M(x) γ ^ G = G ^ f f γ A. Litvinov & Y. Schechner, radiometric nonidealities 13

An Image Invariance log I = γ log I ^ A. Litvinov & Y. Schechner, radiometric nonidealities 14

Nevertheless … Mutually Consistent A. Litvinov & Y. Schechner, radiometric nonidealities 15

ρ(v ) – m(x ) - g = log[I(x)] k k pp Frame: k Frame: p A. Litvinov & Y. Schechner, radiometric nonidealities k p ρ (v ) - ρ(v ) - m(x ) + m(x ) - g + g = 0 p pkk k p m(x) : log of mask at pixel x  (v) : log of inverse-radiometric response g : log of gain at frame f : f 16

A. Litvinov & Y. Schechner, radiometric nonidealities ρ (0) ρ (255) m (x=1) m (x=N) = 0 g 1 g F ρ (v ) - ρ(v ) - m(x ) + m(x ) - g + g = 0 p pkk k p Rs = 0

Experiment: AGC, Vignetting, Gamma pixel transmittance (vignetting) frame (time) amplifier gain graylevel irradiance radiometric response 17

SVD solution Rs = 0 Null space of R s = 0 s = [ ρ = const, m = const, g = const ] T A. Litvinov & Y. Schechner, radiometric nonidealities 123

Trivial Solutions ρ(v) = const light intensity electronic readout exp r r light intensity electronic readout A. Litvinov & Y. Schechner, radiometric nonidealities

Entropy Maximization A. Litvinov & Y. Schechner, radiometric nonidealities hist

Anatoly Litvinov Yoav Schechner A Unified Framework log I = γ log I ^ Image invariance 18