1. P ERCEPTUAL E VALUATION OF M ULTI -E XPOSURE I MAGE F USION A LGORITHMS Kai Zeng, Kede Ma, Rania Hassen and Zhou Wang Department of Electrical and Computer Engineering University of Waterloo September 2014

2. Introduction Subjective Experiment Results Conclusion Multi-Exposure Fusion (MEF)... 2 / 18

3. Introduction Subjective Experiment Results Conclusion MEF vs Tone Mapping HDR reconstruction Tone mapping MEF Alternative Methods High dynamic range (HDR) image reconstruction followed by tone mapping MEF – bypassing the intermediate HDR stage 3 / 18

4. Introduction Subjective Experiment Results Conclusion Existing MEF algorithms Cues used in existing MEF methods Local energy and correlation (in Laplacian pyramid) [Burt1993] Entropy [Goshtasby2005] Contrast, color saturation and luminance exposure [Mertens2007] Edge information (using bilateral filter) [Raman2009] Visual contrast and scene gradient [Song2012] Visibility and consistency (from gradient information) [Zhang2012] Local contrast, brightness and color dissimilarity [Li2012] Pixel saliency and spatial consistency [Li2013]...... Question: Which method produces the best quality image? 4 / 18

5. Introduction Subjective Experiment Results Conclusions Existing objective image quality assessment (IQA) models for image fusion Existing IQA models for image fusion Information theoretical models [Hossny, 2008, Cvejic, 2006, Wang, 2008] Image feature based models [Xydeas, 2000, WangPW, 2008, Zheng, 2007] Image structural similarity based models [Piella, 2003] Human perception based models [Chen, 2007, Chen, 2009] Question: Which model better predicts perceptual quality? 5 / 20

6. Image Database 17 natural scenes of images with multi-exposure levels 8 image fusion algorithms Simple methods: local and global energy weighted linear combination Advanced models: [Raman09, Gu12, ShutaoLi12, Shutaoli13, Li12, Mertens07] Subjective Test Absolute category rating from 1 to 10 No reference images are shown 25 naïve observers (15 male and 10 female, between 22 and 30) Each session limited to within 30 mins 6 / 18 Introduction Subjective Experiment Results Conclusion Subjective Experiment Mapping

7. Introduction Subjective Experiment Results Conclusion Reference Images 7 / 18

8. Introduction Subjective Experiment Results Conclusion Images Examples (a1) Under Exposure (a2) Normal Exposure(a3) Over Exposure (b) Global Energy Weighted (c) Gu12 (d) Local Energy Weighted 8 / 18

9. Introduction Subjective Experiment Results Conclusion Images Examples (e) Li12 (f) Li13 (g) Mertens07 (h) Raman09 (i) ShutaoLi12 9 / 18

10. Introduction Subjective Experiment Results Conclusion Subjective Data Analysis Table: Correlation between individual and average subject scores 10 / 18 SubjectPLCCSRCCSubjectPLCCSRCC 10.87430.8631130.84110.7989 20.82450.7984140.87810.8743 30.71020.6735150.89880.8924 40.80930.8182160.74130.7313 50.67850.6649170.73470.6488 60.65440.6567180.77970.7486 70.81980.8030190.67320.6814 80.89510.8849200.78540.7643 90.79610.7835210.60450.5638 100.69240.6826220.62130.6121 110.82980.8275230.79760.7558 120.61450.5795Average 0.76330.7438

11. Introduction Subjective Experiment Results Conclusion Subjective Data Analysis: Pearson Linear CC Figure: Mean and standard deviation of PLCC between individual subject and average subject for each image set. The rightmost column gives the average performance of all subjects. 11 / 18

12. Introduction Subjective Experiment Results Conclusion Subjective Data Analysis: Spearman ROCC Figure: Mean and standard deviation of SRCC between individual subject and average subject for each image set. The rightmost column gives the average performance of all subjects. 12 / 18

13. Considerable consistency between subjects Behavior of individual subjects may vary “Average Subject”: a useful concept Baseline to observe behaviors of individual subject Baseline to evaluate behaviors of objective models Introduction Subjective Experiment Results Conclusion Subjective Data Analysis: Observations

14. Introduction Subjective Experiment Results Conclusion Performance of MEF Algorithms Figure: Mean and standard deviation of subjective rankings of individual fuser across all image sets. 13 / 18

15. Considerable subject agreement on individual MEF but performance difference between MEFs could be small No single method performs best on all image sets Best performance [Mertens07] (on average) Second best [Li12]: a detail-enhanced version of [Mertens07] Global energy weighting significantly better than local energy weighting Some “advanced” MEFs show no advantage over simple methods Introduction Subjective Experiment Results Conclusion Performance of MEF Algorithms: Observations

16. Table: Performance evaluation of objective IQA models 16 / 18 IQA modelPLCCSRCC µσµσ Hossny08-0.29390.2054-0.27840.2803 Cvejic060.06040.43110.05900.4968 Q. Wang08-0.29920.2008-0.25240.2976 Xydeas000.69490.16550.61980.2452 P.W. Wang080.63560.16340.57710.1761 Zheng070.43320.23170.46140.1820 Piella030.37980.24090.41310.1725 H. Chen07-0.55440.4089-0.56110.4640 Y. Chen090.26670.48300.32740.4628 Introduction Subjective Experiment Results Conclusion Performance of Objective IQA Models

17. Introduction Subjective Experiment Results Conclusion Performance of Objective IQA Models 17 / 18

18. State-of-the-art IQA models do not provide adequate predictions Entropy-based models: very poor performance entropy too limited at capturing image characteristics Models based on structure preservation: most promising but they are limited in capturing spatial consistency Introduction Subjective Experiment Results Conclusion Performance of Objective IQA Models: Observations

19. Introduction Subjective Experiment Results Conclusion Conclusions A new subject-rated MEF image database Evaluation of MEF algorithms Evaluation of objective IQA model for image fusion Summary 19 / 20 Future Work: Design of MEF and objective IQA algorithms No existing IQA model works adequately Need balance between structure-preservation and spatial consistency

20. Introduction Subjective Experiment Results Conclusion Thank you 20 / 18