EE 7700 High Dynamic Range Imaging

Bahadir K. Gunturk2 References Slides and papers by Debevec, Ward, Pattaniak, Nayar, Durand, et al…

Bahadir K. Gunturk3 High Dynamic Range (HDR) Imaging star light moon light office light day light search light The range of luminances is more than 10^14 candela/m2 Range of human eye at an instant is around 10^4:1 (4log units) Human eye can adapt to see much wider range. Candela (cd) is the unit of luminous intensity (power emitted by a light source in a particular direction, with wavelengths weighted by the sensitivity of the human eye.) A common candle emits roughly 1 cd.candle A 100 W incandescent lightbulb emits about 120 cd.lightbulb

Bahadir K. Gunturk4 Spectral Sensitivity of Human Visual System: Luminosity Function Photopic (black) and scotopic [1] (green) luminosity functions. The photopic includes the CIE 1931 standard [2] (solid), the Judd-Vos 1978 modified data [3] (dashed), and the Sharpe, Stockman, Jagla & Jägle 2005 data [4] (dotted). The horizontal axis is wavelength in nm. (from Wikipedia) One candela is defined as the luminous intensity of a monochromatic 540 THz light source that has a radiant intensity of 1/683 watts per steradian, or about mW/sr. The 540 THz frequency corresponds to a wavelength of about 555 nm, which is green light near the peak of the eye's response. A typical candle produces very roughly one candela of luminous intensity. Quantity Derived SI Unit Symbol Luminance candela per square meter cd/m2 Luminous flux lumen cd * sr = lm Illuminance lux lm/m2 = lx

Bahadir K. Gunturk5 HDR star light moon light office light day light search light The range of radiances is more than 10^14 candela/m Range of Typical Displays: from ~1 to ~100 cd/m 2

Bahadir K. Gunturk cd/m^2 Cone dominated log L a Gain rod cone log Gain Sensitivity of Eye

Bahadir K. Gunturk cd/m^2 Rod dominated log L a Gain rod cone log Gain Sensitivity of Eye

Bahadir K. Gunturk8 Sensitivity of Eye

Bahadir K. Gunturk9 HDR The range of image capture devices is also low

Bahadir K. Gunturk10 HDR The range of image capture devices is also low

Bahadir K. Gunturk11 HDR HDR image rendered to be displayed on a LDR display.

Bahadir K. Gunturk12 HDR Problems: How to capture an HDR image with LDR cameras? How to display an HDR image on LDR displays?

Bahadir K. Gunturk13 Capture multiple images with varying exposure. Combine them to produce an HDR image.

Bahadir K. Gunturk14 Creating HDR from Multiple Pictures Measured intensity, z t1 t2 t1 t2 Irradiance, E (=total power per unit area)

Bahadir K. Gunturk15 Creating HDR from Multiple Pictures Measured intensity, z t1 t2 t1 t2 Irradiance, E z1 z2 E z1 = t1 * E z2 = t2 * E E1=z1/t1 E2=z2/t2 Estimates: Take a weighted sum of E1 and E2: w1 w2 E=( w1*E1 + w2*E2 ) / (w1+w2) E

Bahadir K. Gunturk16 Creating HDR from Multiple Pictures Measured intensity, z t1 t2 t1 t2 Irradiance, E z1 z2 E z1 = t1 * E z2 = t2 * E E1=z1/t1 E2=z2/t2 Estimates: Take a weighted sum of E1 and E2: w E=( w(z1)*E1 + w(z2)*E2 ) / (w(z1)+w(z2)) z 255

Bahadir K. Gunturk17 Creating HDR from Multiple Pictures Measured intensity, z t1 t2 t1 t2 Irradiance, E z1 z2 E z1 = t1 * E z2 = t2 * E E1=z1/t1 E2=z2/t2 Estimates: Take a weighted sum of E1 and E2: w E=( w(z1)*E1 + w(z2)*E2 ) / (w(z1)+w(z2)) z 255 Question: If t1 and t2 are not given, how can we estimate them?

Bahadir K. Gunturk18 Creating HDR from Multiple Pictures In general, the camera response is not linear. t1 t2 z1 = f ( t1 * E ) z2 = f ( t2 * E ) E1= g (z1) / t1 E2= g (z2) / t2 E=( w(z1)*E1 + w(z2)*E2 ) / (w(z1)+w(z2)) f g w z z Questions: How to estimate g and t?  One approach is based on polynomial model (Nayar). w is sometimes chosen as the derivative of f. (Mann)

Bahadir K. Gunturk19 Radiometric Self Calibration Polynomial model Exposure ratios: Cost function Solve using If exposure ratios are not known, solve iteratively Intensity Irradiance Pixel Image number

Bahadir K. Gunturk20 Tone Mapping Given an HDR image, how are we going to display it in an LDR display?

Bahadir K. Gunturk21 Tone Mapping Given an HDR image, how are we going to display it in an LDR display? Linear Nonlinear

Bahadir K. Gunturk22 Durand & Dorsey

Bahadir K. Gunturk23 Durand & Dorsey

Bahadir K. Gunturk24 Durand & Dorsey

Bahadir K. Gunturk25 Durand & Dorsey

Bahadir K. Gunturk26 Durand & Dorsey

Bahadir K. Gunturk27 Durand & Dorsey  Bilateral filter

Bahadir K. Gunturk28 Durand & Dorsey

Bahadir K. Gunturk29 Fattal et al in 1D 2500:1 log derivative 7.5:1exp integrate attenuate

Bahadir K. Gunturk30 Reinhard et al. L_white is the smallest luminance that will be mapped to pure white (1). Set L_white = L_max to have no “burn-out”.

Bahadir K. Gunturk31 Durand & Dorsey

Bahadir K. Gunturk32 Durand & Dorsey

Bahadir K. Gunturk33 Informal comparison Bilateral [Durand et al.] Photographic [Reinhard et al.] Gradient domain [Fattal et al.] Bilateral [Durand et al.] Photographic [Reinhard et al.] Gradient domain [Fattal et al.]

Bahadir K. Gunturk34 Spatially Varying Exposures Instead of capturing multiple pictures, allow different amounts of light pass for different pixel positions. Estimate the missing pixels. Combine to obtain an HDR image. 100%75% 50%25% Nayar

Bahadir K. Gunturk35 Image Reconstruction: Interpolation

Bahadir K. Gunturk36 Image Reconstruction: Aggregation

Bahadir K. Gunturk37 HDR image examples

Bahadir K. Gunturk38 HDR image examples

Bahadir K. Gunturk39 HDR image examples

Bahadir K. Gunturk40 The Bilateral Filter (BF) The SUSAN filter, which is essentially the bilateral filter, was used for corner/edge detection and denoising in [Smith & Brady 97]. The BF was presented in [Tomasi & Manduchi 98]. [Elad 02] and [Barash 02] show that the BF is related to the weighted least squares estimation and anisotropic diffusion. Fast implementations/approximations have been proposed, e.g., in [Paris & Durand 06]. In addition to image denoising, the BF is used in tone mapping of HDR images, contrast enhancement, 3D mesh smoothing, blocking artifact reduction, etc.

Bahadir K. Gunturk41 Bilateral Filtering Intensity (range) proximity Spatial (domain) proximity

Bahadir K. Gunturk42 Bilateral Filtering Input Gaussian Bilateral

Bahadir K. Gunturk43 What are the optimal values of the parameters of the Bilateral Filter? MSE=49.8MSE=50.9 MSE=30.3MSE=43.4 MSE=42.5MSE=71.5 MSE=100.0