P ERCEPTUAL E VALUATION OF M ULTI -E XPOSURE I MAGE F USION A LGORITHMS Kai Zeng, Kede Ma, Rania Hassen and Zhou Wang Department of Electrical and Computer Engineering University of Waterloo September 2014

Introduction Subjective Experiment Results Conclusion Multi-Exposure Fusion (MEF)... 2 / 18

Introduction Subjective Experiment Results Conclusion MEF vs Tone Mapping HDR reconstruction Tone mapping MEF Alternative Methods High dynamic range (HDR) image reconstruction followed by tone mapping MEF – bypassing the intermediate HDR stage 3 / 18

Introduction Subjective Experiment Results Conclusion Existing MEF algorithms Cues used in existing MEF methods Local energy and correlation (in Laplacian pyramid) [Burt1993] Entropy [Goshtasby2005] Contrast, color saturation and luminance exposure [Mertens2007] Edge information (using bilateral filter) [Raman2009] Visual contrast and scene gradient [Song2012] Visibility and consistency (from gradient information) [Zhang2012] Local contrast, brightness and color dissimilarity [Li2012] Pixel saliency and spatial consistency [Li2013] Question: Which method produces the best quality image? 4 / 18

Introduction Subjective Experiment Results Conclusions Existing objective image quality assessment (IQA) models for image fusion Existing IQA models for image fusion Information theoretical models [Hossny, 2008, Cvejic, 2006, Wang, 2008] Image feature based models [Xydeas, 2000, WangPW, 2008, Zheng, 2007] Image structural similarity based models [Piella, 2003] Human perception based models [Chen, 2007, Chen, 2009] Question: Which model better predicts perceptual quality? 5 / 20

Image Database 17 natural scenes of images with multi-exposure levels 8 image fusion algorithms Simple methods: local and global energy weighted linear combination Advanced models: [Raman09, Gu12, ShutaoLi12, Shutaoli13, Li12, Mertens07] Subjective Test Absolute category rating from 1 to 10 No reference images are shown 25 naïve observers (15 male and 10 female, between 22 and 30) Each session limited to within 30 mins 6 / 18 Introduction Subjective Experiment Results Conclusion Subjective Experiment Mapping

Introduction Subjective Experiment Results Conclusion Reference Images 7 / 18

Introduction Subjective Experiment Results Conclusion Images Examples (a1) Under Exposure (a2) Normal Exposure(a3) Over Exposure (b) Global Energy Weighted (c) Gu12 (d) Local Energy Weighted 8 / 18

Introduction Subjective Experiment Results Conclusion Images Examples (e) Li12 (f) Li13 (g) Mertens07 (h) Raman09 (i) ShutaoLi12 9 / 18

Introduction Subjective Experiment Results Conclusion Subjective Data Analysis Table: Correlation between individual and average subject scores 10 / 18 SubjectPLCCSRCCSubjectPLCCSRCC Average

Introduction Subjective Experiment Results Conclusion Subjective Data Analysis: Pearson Linear CC Figure: Mean and standard deviation of PLCC between individual subject and average subject for each image set. The rightmost column gives the average performance of all subjects. 11 / 18

Introduction Subjective Experiment Results Conclusion Subjective Data Analysis: Spearman ROCC Figure: Mean and standard deviation of SRCC between individual subject and average subject for each image set. The rightmost column gives the average performance of all subjects. 12 / 18

Considerable consistency between subjects Behavior of individual subjects may vary “Average Subject”: a useful concept Baseline to observe behaviors of individual subject Baseline to evaluate behaviors of objective models Introduction Subjective Experiment Results Conclusion Subjective Data Analysis: Observations

Introduction Subjective Experiment Results Conclusion Performance of MEF Algorithms Figure: Mean and standard deviation of subjective rankings of individual fuser across all image sets. 13 / 18

Considerable subject agreement on individual MEF but performance difference between MEFs could be small No single method performs best on all image sets Best performance [Mertens07] (on average) Second best [Li12]: a detail-enhanced version of [Mertens07] Global energy weighting significantly better than local energy weighting Some “advanced” MEFs show no advantage over simple methods Introduction Subjective Experiment Results Conclusion Performance of MEF Algorithms: Observations

Table: Performance evaluation of objective IQA models 16 / 18 IQA modelPLCCSRCC µσµσ Hossny Cvejic Q. Wang Xydeas P.W. Wang Zheng Piella H. Chen Y. Chen Introduction Subjective Experiment Results Conclusion Performance of Objective IQA Models

Introduction Subjective Experiment Results Conclusion Performance of Objective IQA Models 17 / 18

State-of-the-art IQA models do not provide adequate predictions Entropy-based models: very poor performance entropy too limited at capturing image characteristics Models based on structure preservation: most promising but they are limited in capturing spatial consistency Introduction Subjective Experiment Results Conclusion Performance of Objective IQA Models: Observations

Introduction Subjective Experiment Results Conclusion Conclusions A new subject-rated MEF image database Evaluation of MEF algorithms Evaluation of objective IQA model for image fusion Summary 19 / 20 Future Work: Design of MEF and objective IQA algorithms No existing IQA model works adequately Need balance between structure-preservation and spatial consistency

Introduction Subjective Experiment Results Conclusion Thank you 20 / 18