1. INFORMATIK Design of a Tone Mapping Operator for High Dynamic Range Images based upon Psychophysical Evaluation and Preference Mapping Design of a Tone Mapping Operator for High Dynamic Range Images based upon Psychophysical Evaluation and Preference Mapping F. Drago 1, W. Martens 2, K. Myszkowski 3, and N. Chiba 1 1 Iwate University and 2 Aizu University, Japan 3 Max-Planck-Institut für Informatik, Germany F. Drago 1, W. Martens 2, K. Myszkowski 3, and N. Chiba 1 1 Iwate University and 2 Aizu University, Japan 3 Max-Planck-Institut für Informatik, Germany

2. INFORMATIK Overview MotivationMotivation Previous workPrevious work Psychophysical experimentPsychophysical experiment Enhancements of Retinex for HDR imagesEnhancements of Retinex for HDR images ConclusionsConclusions MotivationMotivation Previous workPrevious work Psychophysical experimentPsychophysical experiment Enhancements of Retinex for HDR imagesEnhancements of Retinex for HDR images ConclusionsConclusions

3. INFORMATIK Motivation Many applications Lighting simulation and realistic renderingLighting simulation and realistic rendering High Dynamic Range photographyHigh Dynamic Range photography Multimedia: distributing HDR video streamsMultimedia: distributing HDR video streams Many applications Lighting simulation and realistic renderingLighting simulation and realistic rendering High Dynamic Range photographyHigh Dynamic Range photography Multimedia: distributing HDR video streamsMultimedia: distributing HDR video streams

4. INFORMATIK HDR Photographs + Rendering: Real World Lighting 1) Photographs of mirror sphere at varying exposure times 2) High-dynamic range environment map 3) Use as light source in Monte Carlo radiosity algorithm Philippe Bekaert

5. INFORMATIK Goals Technical requirementTechnical requirement –Match the dynamic range of image to the range available on a given display device Various objectivesVarious objectives –Get good perceptual match between the real-world and corresponding images –Reproducing details –Maximize reproducible contrast –Just to get “nice-looking” images Technical requirementTechnical requirement –Match the dynamic range of image to the range available on a given display device Various objectivesVarious objectives –Get good perceptual match between the real-world and corresponding images –Reproducing details –Maximize reproducible contrast –Just to get “nice-looking” images

6. INFORMATIK Various Classifications Theoretical foundationsTheoretical foundations –Perception-based –Pure image processing techniques Mapping functionMapping function –Global – the same for all pixels –Local – depends on local image contents Temporal processingTemporal processing –Static –Dynamic Theoretical foundationsTheoretical foundations –Perception-based –Pure image processing techniques Mapping functionMapping function –Global – the same for all pixels –Local – depends on local image contents Temporal processingTemporal processing –Static –Dynamic

7. INFORMATIK Previous Work: Global Methods Perception-based Tumblin and Rushmeier (1993,1999)Tumblin and Rushmeier (1993,1999) –Brightness matching Ward (1994), Ferwerda et al. (1996)Ward (1994), Ferwerda et al. (1996) –Contrast matching (a linear function is used) Ward et al. (1997)Ward et al. (1997) –Adjusting image histogram to avoid exceeding display contrast in respect to the real-world scene Efficiency-driven Schlick (1994)Schlick (1994) –Rational functions Perception-based Tumblin and Rushmeier (1993,1999)Tumblin and Rushmeier (1993,1999) –Brightness matching Ward (1994), Ferwerda et al. (1996)Ward (1994), Ferwerda et al. (1996) –Contrast matching (a linear function is used) Ward et al. (1997)Ward et al. (1997) –Adjusting image histogram to avoid exceeding display contrast in respect to the real-world scene Efficiency-driven Schlick (1994)Schlick (1994) –Rational functions

8. INFORMATIK Examples Ferwerda et al. Tumblin (1999) Ward et al. Schlick

9. INFORMATIK Previous Work: Local Methods Early methods – prone to halo artifactsEarly methods – prone to halo artifacts –Chiu et al. (1993), Schlick (1994), –Land (1971), Jobson et al. (1997): Retinex –Pattanaik et al. (1998): The most comprehensive model of HVS used in CG LCIS: Tumblin and Turk (1999)LCIS: Tumblin and Turk (1999) –Based on an anisotropic diffusion procedure –Emphasize on details but compress excessively contrast New wave: Fattal et al., Reinhard et al., Durand and Dorsey, Ashikhmin (2002)New wave: Fattal et al., Reinhard et al., Durand and Dorsey, Ashikhmin (2002) Early methods – prone to halo artifactsEarly methods – prone to halo artifacts –Chiu et al. (1993), Schlick (1994), –Land (1971), Jobson et al. (1997): Retinex –Pattanaik et al. (1998): The most comprehensive model of HVS used in CG LCIS: Tumblin and Turk (1999)LCIS: Tumblin and Turk (1999) –Based on an anisotropic diffusion procedure –Emphasize on details but compress excessively contrast New wave: Fattal et al., Reinhard et al., Durand and Dorsey, Ashikhmin (2002)New wave: Fattal et al., Reinhard et al., Durand and Dorsey, Ashikhmin (2002)

10. INFORMATIK Tumblin and TurkRetinexAshikhmin Examples

11. INFORMATIK Durand and DorseyReinhard et al.Fattal et al. Examples

12. INFORMATIK Durand and Dorsey Reinhard et al.Fattal et al. Ashikhmin

13. INFORMATIK Psychophysical Experiment Perceptual evaluation of subject preference by pairwise comparison of tone mapped imagesPerceptual evaluation of subject preference by pairwise comparison of tone mapped images Seven tone mapping algorithms examined:Seven tone mapping algorithms examined: –Tumblin and Rushmeier (1993), –Ferwerda et al. (1996), –Ward et al. (1997), –Schlick (1994), –Retinex - based on Funt and Ciurea (2001) implementation but with our extensions toward suppressing halo –Reinhard et al. (2002) – photographic method –Tumblin and Turk (1999) - LCIS Four scenes consideredFour scenes considered Perceptual evaluation of subject preference by pairwise comparison of tone mapped imagesPerceptual evaluation of subject preference by pairwise comparison of tone mapped images Seven tone mapping algorithms examined:Seven tone mapping algorithms examined: –Tumblin and Rushmeier (1993), –Ferwerda et al. (1996), –Ward et al. (1997), –Schlick (1994), –Retinex - based on Funt and Ciurea (2001) implementation but with our extensions toward suppressing halo –Reinhard et al. (2002) – photographic method –Tumblin and Turk (1999) - LCIS Four scenes consideredFour scenes considered

14. INFORMATIK

15. INFORMATIK Statistical Data Processing 11 subjects participated11 subjects participated Dissimilarity ratings for pairwise comparisons of images submitted to Individual Differences Scaling (INDSCAL) analysisDissimilarity ratings for pairwise comparisons of images submitted to Individual Differences Scaling (INDSCAL) analysis Stimulus Space configures the stimuli such that Euclidian distances between the stimuli match the obtained dissimilarity judgmentsStimulus Space configures the stimuli such that Euclidian distances between the stimuli match the obtained dissimilarity judgments Axes labeled based upon correlation of the dimensional coordinates with independently generated attribute ratings (naturalness, detail and contrast reproduction)Axes labeled based upon correlation of the dimensional coordinates with independently generated attribute ratings (naturalness, detail and contrast reproduction) “Ideal” preference point obtained through PREFMAP analysis“Ideal” preference point obtained through PREFMAP analysis 11 subjects participated11 subjects participated Dissimilarity ratings for pairwise comparisons of images submitted to Individual Differences Scaling (INDSCAL) analysisDissimilarity ratings for pairwise comparisons of images submitted to Individual Differences Scaling (INDSCAL) analysis Stimulus Space configures the stimuli such that Euclidian distances between the stimuli match the obtained dissimilarity judgmentsStimulus Space configures the stimuli such that Euclidian distances between the stimuli match the obtained dissimilarity judgments Axes labeled based upon correlation of the dimensional coordinates with independently generated attribute ratings (naturalness, detail and contrast reproduction)Axes labeled based upon correlation of the dimensional coordinates with independently generated attribute ratings (naturalness, detail and contrast reproduction) “Ideal” preference point obtained through PREFMAP analysis“Ideal” preference point obtained through PREFMAP analysis

16. INFORMATIK Subject Preferences –T: Tumblin & R. –V: Ferwerda et al. –H: Ward et al. –Q: Schlick –X: Retinex –P: Reinhard et al. –T: Tumblin & R. –V: Ferwerda et al. –H: Ward et al. –Q: Schlick –X: Retinex –P: Reinhard et al.

17. INFORMATIK Retinex We use the “Frankle-McCann Retinex” algorithm ratio-product-reset-averageratio-product-reset-average –NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows: –Reset test In each iteration (the number of iterations predefined by the user)In each iteration (the number of iterations predefined by the user) –the distance D between pixels (x,y) and (xs,ys) is halved –the direction for pixel comparison is rotated 90 o clockwise Main problem: Suppressing halo effectsMain problem: Suppressing halo effects We use the “Frankle-McCann Retinex” algorithm ratio-product-reset-averageratio-product-reset-average –NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows: –Reset test In each iteration (the number of iterations predefined by the user)In each iteration (the number of iterations predefined by the user) –the distance D between pixels (x,y) and (xs,ys) is halved –the direction for pixel comparison is rotated 90 o clockwise Main problem: Suppressing halo effectsMain problem: Suppressing halo effects

18. INFORMATIK Retinex Extensions: for HDR Main problem: Suppressing halo effectsMain problem: Suppressing halo effects –Adding counterclockwise rotation of the path  suggested by Coopers –Spatially varying levels of pixel interaction based contrast information  Suggested by Sobol, but we use a smooth function for clipping –Adjusting a reset ratio to the maximum luminance of the display device instead of the maximum luminance of the scene Main problem: Suppressing halo effectsMain problem: Suppressing halo effects –Adding counterclockwise rotation of the path  suggested by Coopers –Spatially varying levels of pixel interaction based contrast information  Suggested by Sobol, but we use a smooth function for clipping –Adjusting a reset ratio to the maximum luminance of the display device instead of the maximum luminance of the scene

19. INFORMATIK Halo Reduction: Retinex Rotation CounterClockwiseClockwiseBoth Ways All images for 40 iterations

20. INFORMATIK Halo Reduction: Retinex Contrast Crop with Bias

21. INFORMATIK Standard Retinex 33 iterations cw and ccw The same settings but crop with bias added

22. INFORMATIK Halo Reduction: Retinex Contrast Crop with Bias 33 Retinex iterations

23. INFORMATIK Halo Reduction: Retinex Contrast Crop with Bias 4 Retinex iterations 30 Retinex iterations

24. INFORMATIK Retinex Maximum Reset Maximum = 226.5 cd/m^2Maximum = 100 cd/m^2

25. INFORMATIK LinearmappingRetinex 4 iterations ExtendedRetinex ExtendedRetinex

26. INFORMATIK Retinex + Tone Mapping Op. Ferwerda et al. (1996) Logmap - new

27. INFORMATIK Logmap Equation

28. INFORMATIK Performance: SoftwareSoftware –30 fps on PentiumIV, 2.2GHz HardwareHardware –? Performance: SoftwareSoftware –30 fps on PentiumIV, 2.2GHz HardwareHardware –? Adaptive Logarithmic Mapping

29. INFORMATIK Conclusions We performed psychophysical of seven existing tone mapping operators. More details in our TechRep:We performed psychophysical of seven existing tone mapping operators. More details in our TechRep: http://data.mpi-sb.mpg.de/internet/reports.nsf/AG4NumberView?OpenView Good performance of Retinex in the experiment encouraged us extend it toward reducing hallo artifactsGood performance of Retinex in the experiment encouraged us extend it toward reducing hallo artifacts Addind a regular tone mapping processing atop of Retinex results make the resulting images more independent on the number of Retinex iterations and improve the image naturalnessAddind a regular tone mapping processing atop of Retinex results make the resulting images more independent on the number of Retinex iterations and improve the image naturalness Future work: repeating psychophysical with all recent local tone mapping operators and our extended RetinexFuture work: repeating psychophysical with all recent local tone mapping operators and our extended Retinex We performed psychophysical of seven existing tone mapping operators. More details in our TechRep:We performed psychophysical of seven existing tone mapping operators. More details in our TechRep: http://data.mpi-sb.mpg.de/internet/reports.nsf/AG4NumberView?OpenView Good performance of Retinex in the experiment encouraged us extend it toward reducing hallo artifactsGood performance of Retinex in the experiment encouraged us extend it toward reducing hallo artifacts Addind a regular tone mapping processing atop of Retinex results make the resulting images more independent on the number of Retinex iterations and improve the image naturalnessAddind a regular tone mapping processing atop of Retinex results make the resulting images more independent on the number of Retinex iterations and improve the image naturalness Future work: repeating psychophysical with all recent local tone mapping operators and our extended RetinexFuture work: repeating psychophysical with all recent local tone mapping operators and our extended Retinex

30. INFORMATIK Color Balance Correction Retinex Applied to All Channels in LMS Color Space

31. INFORMATIK Stanford Memorial Church Photograph

32. INFORMATIK

33. INFORMATIK Acknowledgments We would like to thank Michael Ashikhmin, Paul Debevec, Fredo Durand, Dani Lischinski, Eric Reinhard, and Greg Ward for providing us with some images used in this presentation. We would like also to thank Greg Ward for his precious comments concerning our work. We would like to thank Michael Ashikhmin, Paul Debevec, Fredo Durand, Dani Lischinski, Eric Reinhard, and Greg Ward for providing us with some images used in this presentation. We would like also to thank Greg Ward for his precious comments concerning our work.