Michael Grossberg and Shree Nayar CAVE Lab, Columbia University Partially funded by NSF ITR Award What can be Known about the Radiometric Response from Images? ECCV Conference May, 2002, Copenhagen, Denmark

Radiometric Response Function Response: u Inverse response function: g g(u)=I Response function: f(I)=u Response = Gray-level Irradiance I u Image Plane Irradiance: I 0255 Scene Radiance: R

Radiometric Calibration Critical for Photometric Applications Example: Photometric Stereo 3D Structure Changes Scene Radiance Lighting Changes + 3D structure Changes Image Irradiance Changes Image Brightness Reveals Radiometric Response f Cannot recover 3D structure without g Inverse Response g

Response Recovery from Images What is measured?What is needed?What is recovered? Images at different exposures Correspondence of gray- levels between images Inverse Radiometric Response, g Exposure Ratios k 3 k 1 k 2 k 1 k 3 k 2 Response Irradiance u I Exposure Ratios Gray-levels: Image A Gray-levels: Image B Gray-levels: Image C Gray-levels: Image D uAuA uBuB Recovery Algorithms: S. Mann and R. Picard, 1995, P. E. Debevec, and J. Malik, 1997, T. Mitsunaga S. K. Nayar 1999, S. Mann 2001, Y. Tsin, V. Ramesh and T. Kanade 2001

How is Radiometric Calibration Done? Recovery Algorithms Response Irradiance u I k 1 k 3 k 2 Images at Different ExposuresCorresponding Gray-levelsInverse Response g, Exposure Ratio k Geometric Correspondences We eliminate the need for geometric correspondences: Static Scenes Dynamic Scenes We find: All ambiguities in recovery Assumptions that break them

Constraint Equations Constraint on irradiance I:  I B = kI A Constraint on g:  g(u B )=kg(u A ) T IBIB IAIA Filter Brighter image Darker image g(T(u A ))=kg(u A ) Brightness Transfer Function T:  u B =T(u A ) Constraint on g in terms of T

How Does the Constraint Apply? Exposure ratio k known Constraint makes curve self-similar

Self-Similar Ambiguity: Can We Recover g? Conclusions:  Constraint gives no information in [T -1 (1),1]  Regularity assumptions break ambiguity  Known k: only Self-similar ambiguity Gray-levels Irradiance 01T -1 (1) Choose anything here 1 and copy 1/k 1/k 2 1/k 3 u I

Exponential Ambiguity: Can We Recover g and k ? Exposure ratio Inverse Response Function g γ Brightness Transfer Function T Response Irradiance Gray-level Image A Gray-level Image B γ =1/3 γ =1/2 γ =1 γ =2 γ =3 I k=2 1/2 k=2 1/3 k=2 k=2 2 k=2 3 T(M)=2M T(u) = g -1 (kg(u)) = g - γ (k - γ g γ (u)) = T(u) We cannot disambiguate (g γ, k γ ) from (g, k) using T! U

Ambiguity with Multiple Images Inverse Response: gInverse Response: g γ k1γk1γ k2k2 k 1 γ k 2 γ =(k 1 k 2 ) γ k2γk2γ k1k1 k1k2k1k2 Conclusions Cannot recover both k and g, without making assumptions on g Exponential and self-similar ambiguities complete ambiguities of recovery

Direct Recovery of Exposure Ratios Brightness in Image A Brightness in Image B T 2 (u)=g -1 (4g(u)) T 3 (u)=g -1 (8g(u)) T 1 (u)=g -1 (2g(u)) g(u)=2u/(1+u) k 3 =8=T 3 ′(0) k 1 =2=T 1 ′(0) k 2 =4=T 2 ′(0) k = T ′(0) when g′(0)≠0

Obtaining the Brightness Transfer Function (S. Mann, 2001) Registered Static Images at Different Exposures 2D-Gray-level Histogram Brightness Transfer Function Regression Scenes must be static. Gray-level Image A Gray-level Image B Gray-level Image A Gray-level Image B

Brightness Transfer Function Histogram Specification Brightness HistogramsUnregistered Images at Different Exposures Scenes may have motion. Brightness Transfer Function without Registration Gray-level Image A Gray-level Image B Gray-level Image A Gray-level Image B

How does Histogram Specification Work? Gray-levels in Image A Cumulative Area (Fake Irradiance) Histogram Equalization Gray-levels in Image B Histogram Specification = Brightness Transfer Function Histogram Specification

Why Does Histogram Specification = Brightness Transfer Function? Image A Image B Area of intensities ≤ u A in image A = Area of intensities ≤ u B in image B  u B =T(u A )

Results: Object Motion Recovered Inverse Radiometric Response Curves Red Response Irradiance Green Response Recovered Response Macbeth Chart Data Irradiance Blue Response Irradiance

Results: Camera Motion Green Irradiance Blue Irradiance Red Irradiance Red Response Blue Response Recovered Inverse Radiometric Response Curves Green Response Recovered Response Macbeth Chart Data Irradiance

Results: Object and Camera Motion Green Irradiance Blue Irradiance Red Irradiance Red Response Blue Response Green Response Recovered Inverse Radiometric Response Curves Recovered Response Macbeth Chart Data Irradiance

Conclusions: What can be Known about Inverse Response g from Images? Recovery of g from T Self-similar Ambiguity + Exponential Ambiguity Need assumptions on g and k to recover g Exposure ratio k known Exposure ratio k unknown A2: In theory, we can recover exposure ratio directly from Brightness Transfer Function T A3: Geometric correspondence step eliminated allowing recovery in dynamic scenes: A1: