1. No-reference Image Quality Assessment for High Dynamic Range Images
November 9, 2016
No-reference Image Quality Assessment for High Dynamic Range Images
Debarati Kundu, Deepti Ghadiyaram,
Alan C. Bovik and Brian L. Evans
The University of Texas at Austin

2. Introduction Scene luminance 10-4 to 106 cd/m2 [Narwaria2013]
Standard dynamic range (SDR) High dynamic range (HDR)
HDR picture capture (e.g. smart phones and DSLR cameras)
HDR video displays for home (e.g. Samsung)
HDR streaming content (e.g. Amazon Video and Netflix)
HDR graphics rendering (e.g. Unreal and CryEngine)
DSLR

3. Tonemapping HDR to SDR format
Uniformly spaced quantization of luminance overexposes view through window
World luminance floating-point values for window office in cd/m2 [Larson1997]
Global nonuniform quantization of luminance preserves visibility of indoor & outdoor features

4. Tonemapping operators
Estimate radiance map by merging pixels from different exposures
Tonemap floating point irradiance map to SDR
HDR but in SDR format
Registered exposure stack of SDR images
Requires camera calibration and motion comp.
Distort gradients
Spatially-varying transfer function [Szeliski2010] [Larson1997]
Shrink gradients to fit within available dynamic range [Szeliski2010]

5. Multi-exposure fusion
Merge exposure stack directly to get fused image
ith pixel index
kth exposure image
Xk(i) luminance
Wk(i) weight for perceptual importance of exposure level k
HDR but in SDR format
Registered SDR stack
Distorts gradients
Reversals in image gradients [Ma2015] and ghosting artifacts [Tursun2016]
Deghosting methods: banding, blending, structural distortion [Tursun2016]

6. Image Quality Assessment (IQA)
Full reference IQA approaches
Tonemapping synthesizes an irradiance map [Yeganeh2013]
Multi-exposure fusion does not have single reference image
No-reference IQA using scene statistics
Statistics of pristine pictures occur irrespective of content
Statistics of distorted pictures deviate from these
Previously used in predicting quality in SDR pictures
Contributions
Propose two new no-reference HDR IQA algorithms
Evaluate new algorithms based on crowdsourced HDR scores
Evaluate new algorithms on well-established SDR databases

7. ESPL-LIVE HDR database [Debarati2016]
ESPL-LIVE HDR database [Debarati2016]
1,811 HDR images from 605 source scenes
960x540 landscape & 540x304 portrait orientation
Single stimulus continuous quality scale (0-100)
327,720 raw quality scores from 5,462 subjects
Images annotated with mean opinion scores (MOS)
“Surreal” Effect
Distribution of HDR processing methods
Grunge and Surreal effects applied to an HDR image in SDR format using Photomatix software.
ESPL-LIVE Database Web page
Sample images
Number of images in database

8. Mean subtracted contrast normalization
MSCN pixels for image
Weighted local mean
Weighted local standard deviation
Models divisive normalization in retina
[Ruderman1993]
Original image
Use 11 x 11 windows with uninform Gaussian blur (sigma = 1.17)
MSCN image

9. MSCN and local σ-map distributions
(a) MOS = 40.47
(b) MOS = 49.23
(c) MOS = 52.80
Explain intuition behind gradient

10. Distribution of gradient domain features

11. Rank correlation between each feature
Contribution #1
Rank correlation between each feature
Domain
Feature Description
Corr.
Spatial
[f1 − f2]
Shape and Scale parameters of a generalized Gaussian distribution (GGD) to MSCN coefficients
0.238
[f3 − f16]
Shape and Scale parameters of a GGD fitted to log-
derivative of the seven types of neighbors
0.439
[f17 − f18]
Mean and standard deviation based features extracted from the σ-field
0.369
Gradient
[f19 − f20]
Shape and Scale parameters of a GGD fitted to the MSCN coefficients of gradient magnitude field
0.250
[f21 − f34]
derivative of seven types of neighbors of gradient magnitude field
0.386
[f35 − f36]
Mean and standard deviation based features extracted from the σ-field of gradient magnitude field
0.388
Correlation is Spearman’s Rank Ordered Correlation Coefficient (SROCC) .
G-IQA: [f1 − f36]
Features computed across 2 levels in LAB color space

12. Evaluate NR-IQA algorithms
Contribution #2
Evaluate NR-IQA algorithms
Correlate predicted ratings with crowdsourced ratings from 5,462 subjects for ESPL-LIVE HDR image quality database
G-IQA: 80% training, 20% testing, 100 random train-test splits
No content overlap to prevent artificial inflation of correlations
Algorithms
Tone Mapping Operators
Multi Exposure Fusion
Effects
Overall
G-IQA
0.692
0.691
0.582
0.716
G-IQA (L)
0.651
0.623
0.489
0.662
DESIQUE
0.503
0.550
0.476
0.565
GM-LOG
0.521
0.527
0.562
CurveletIQA
0.542
0.512
0.435
0.546
DIIVINE
0.485
0.456
0.335
0.480
BLIINDS-II
0.385
0.421
0.448
BRISQUE
0.267
0.431
0.402
Correlation is Spearman’s Rank Ordered Correlation Coefficient (SROCC) .

13. Box plots for the 100 trials
Contribution #2
Box plots for the 100 trials
Box: Line is median and edges 25th & 75th percentiles
Whiskers span extreme non-outlier points & outliers are +

14. No-reference IQAs on LIVE Database
Contribution #3
No-reference IQAs on LIVE Database
Algorithms
JP2K
JPEG GN
Blur FF
Overall
GM-LOG
0.882
0.878
0.978
0.915
0.899
0.914
G-IQA
0.905
0.883
0.983
0.917
0.836
0.906
BRISQUE
0.852
0.962
0.941
0.863
0.902
BLIINDS-II
0.907
0.846
0.939
0.884
0.897
DESIQUE
0.875
0.824
0.975
0.908
0.829
CurveletQA
0.816
0.827
0.969
0.896
0.826
DIIVINE
0.759
0.937
0.854
MS-SSIM
0.963
0.979
0.977
0.954
SSIM
0.947
0.964
0.913
PSNR
0.865
0.752
0.874
0.864
LIVE IQA Database
H. R. Sheikh, M. F. Sabir, and A. C. Bovik, “A statistical evaluation of recent full reference image quality assessment algorithms,” IEEE Trans. Image Process., vol. 15, no. 11, pp. 3440–3451, Nov 2006.
Correlation results with subjective scores given above
Italics indicate full-reference algorithms

15. Conclusion Future work Proposed two no-reference IQA methods
HDR image formation causes gradient distortion
Use statistics in pixel and gradient domains
Proposed IQA methods correlate well with human scores
ESPL-LIVE HDR subjective image quality database (2016)
LIVE image quality assessment database (2006)
Improve performance of NR-IQA HDR algorithms
Use the methods to improve HDR processing algorithms such as tone-mapping or multi-exposure fusion
Future work

16. References
[Debarati2016] [Larson1997] G. W. Larson, H. Rushmeier, C. Piatko. “A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes”, IEEE Trans. on Visualization and Computer Graphics 3, 4, Oct. 1997, pp [Ma2015] K. Ma, K. Zeng, Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Trans. Image Processing, vol. 24, no. 11, pp. 3345–3356, Nov [Nafchi2014] H. Ziaei Nafchi, A. Shahkolaei, R. Farrahi Moghaddam, M. Cheriet, “Fsitm: A feature similarity index for tone-mapped images,” IEEE Signal Processing Letters, [Narwaria2013] M. Narwaria, M. Perreira Da Silva, P. Le Callet, and R. Pepion, “Tone mapping-based high-dynamic-range image compression: study of optimization criterion and perceptual quality,” Optical Engineering, vol. 52, no. 10, Oct [Nasrinpour2015] 1 H. R. Nasrinpour and N. D. Bruce, “Saliency weighted quality assessment of tone-mapped images,” Proc. IEEE Int. Conf. Image Proc., Sep [Ruderman1993] D. L. Ruderman and W. Bialek, “Statistics of natural images: Scaling in the woods,” Proc. Neural Info. Processing Sys. Conf. and Workshops, [Szeliski2010] R. Szeliski, Computer Vision: Algorithms and Applications, 1st ed., [Tursun2016] O. T. Tursun, A. O. Akyuz, A. Erdem, E. Erdem, “An Objective Deghosting Quality Metric for HDR Images”, Eurographics, vol. 35, [Yeganeh2013] H. Yeganeh and Z. Wang, “Objective quality assessment of tone-mapped images,” IEEE Trans. on Image Processing , vol. 22, no. 2, pp , Feb 2013.

17. Questions?

18. Multi-exposure fusion distortion [Tursun 2016]
Exposure stack for scene with person walking
MEF HDR Image + Deghosting
Gradient mag & visual difference artifacts

19. HDR-IQA: Subjective Testing Methodology
Contribution #4
HDR-IQA: Subjective Testing Methodology
12 subjects evaluated 27 images in laboratory setting
5 ‘Gold Standard’ images
Amazon Mechanical Turk used for crowdsourcing
Training images: 11
Test images: 49
‘Gold Standard’ images: 5 (Viewed by every subject)
Randomly repeated images : 5
At the end subjects answered questions on demographics, display parameters, and familiarity with HDR imaging
Source: Amazon.com
Introduction | Synthetic Image Quality| HDR Image Quality| Conclusions

20. HDR-IQA: Processing of the raw scores
Contribution #4
HDR-IQA: Processing of the raw scores
Subject rejection strategies:
Only subjects with AMT confidence values greater than 0.75 participated
If scores assigned to multiple copies of the same image differed by more than 25.5 for three images, scores from that user was rejected
388 subjects among 5,462 removed as outliers
Processing of remaining scores:
On an average, every image evaluated by 110 subjects
Mean Opinion Scores: Mean of Z-score for every image
Spans
Raw MOS scores span –
Consistency with laboratory setting:
Median PLCC between individual scores and MOS values for ‘Gold Standard’ images in laboratory setting was
Details
Introduction | Synthetic Image Quality| HDR Image Quality| Conclusions

21. Distorted Image Statistics
Different distortions affect scene statistics characteristically
Used for distortion classification and blind quality prediction
Replot
MSCN Coefficients
Steerable Pyramid Wavelet
Coefficients
Curvelet Coefficients
Back

22. Tone Mapped Quality Index [Yeganeh2013]
Tonemapping meant to change local intensity & contrast
Structural fidelity modifies Structural Similarity (SSIM)
Penalizes large change in strength in HDR vs. SDR image patch
Local standard deviations nonlinearly mapped via Gaussian CDF
Significant signal strength mapped to 1
Insignificant signal strength mapped to 0
Structural fidelity computation over five scales
Naturalness measure of tonemapped SDR image
Distribution of global means in 3000 natural images
Distribution of global standard deviations in 3000 natural images
Back

23. Itti and Koch’s Saliency
Back
Different scales
Implemented as Gaussian Pyramid
Center Surround mechanism
Implemented with DoG
LPF repeated over multiple scales
3 scales, 4 orientations used

24. Generalized Gaussian Density
PDF for shape parameter a and scale parameter g
Includes Laplacian (a=1), Gaussian (a=2) and uniform (a=oo)
GGD behavior of bandpass image signals
Wavelet coefficients
DCT coefficients
Usually reported that a » 1 but varies (0.8 < a < 1.4)
[A. C. Bovik, EE381V Digital Video, UT Austin, Spring 2015]

25. Calculating Correlations
Spearman’s Rank-Order Correlation Coefficient (SRCC)
di is difference between ith image’s ranks is subjective and objective evaluations
N is number of rankings
Kendall’s correlation coefficient (KCC)
Nc and Nd are the number of concordant (of consistent rank order) and discordant (of inconsistent rank order) pairs in the data set respectively
Pearson’s Linear Correlation Coefficient (PLCC)
Back

26. Scatter plot of MOS vs IQA scores
G-IQA
DESIQUE
GM-LOG