Just Noticeable Difference Estimation For Images with Structural Uncertainty WU Jinjian Xidian University

Outline Concept of JND Existing Works Inspiration JND Estimation Experimental Results Conclusion

Original image JND noise (PSNR=26.4) contaminated image

Concept of Just Noticeable Difference Original image JND noise (PSNR=26.4) contaminated image +

Concept of Just Noticeable Difference Original image JND noise (PSNR=26.4) contaminated image +

Concept of Just Noticeable Difference Original image JND noise (PSNR=26.4) contaminated image  Just-noticeable difference (JND): below which the change cannot be detected by the majority (e.g., 75%) of viewers;  JND is useful in perceptual based signal processing, e.g., image/video coding, watermarking, quality assessment, etc.

Factors determine JND threshold ：  Luminance adaptation: the HVS presents different resolutions for different background illuminations ； Existing works

Factors determine JND threshold ：  Luminance adaptation: the HVS presents different resolutions for different background illuminations ； Existing works

Factors determine JND threshold ：  Luminance adaptation: the HVS presents different resolutions for different background illuminations ； Existing works

Factors determine JND threshold ：  Luminance adaptation: the HVS presents different resolutions for different background illuminations ； Existing works

Factors determine JND threshold ：  Luminance adaptation: the HVS presents different resolutions for different background illuminations ； Existing works

Factors determine JND threshold ：  Contrast masking: the spatial non-uniformity of the background luminance causes masking effect;

Existing works Factors determine JND threshold ：  Contrast masking: the spatial non-uniformity of the background luminance causes masking effect;

Existing works Factors determine JND threshold ：  Contrast masking: the spatial non-uniformity of the background luminance causes masking effect;

contaminated Existing works Shortcoming of the existing works ：  Overestimate the orderly regions (edge);  Underestimate the disorderly regions (texture); original

contaminated Existing works Shortcoming of the existing works ：  Overestimate the orderly regions (edge);  Underestimate the disorderly regions (texture); Too much noise original

contaminated Existing works Shortcoming of the existing works ：  Overestimate the orderly regions (edge);  Underestimate the disorderly regions (texture); original

contaminated Existing works Shortcoming of the existing works ：  Overestimate the orderly regions (edge);  Underestimate the disorderly regions (texture); Too little noise original

Luminance Adaptation Luminance Adaptation JND Threshold JND Threshold Factors determine JND threshold ： Existing works Contrast Masking Contrast Masking

Luminance Adaptation Luminance Adaptation JND Threshold JND Threshold Factors determine JND threshold ： Existing works Contrast Masking Contrast Masking ? ? There must be another factor, which also determines JND

Inspiration Intuitive perception ：  The HVS is sensitive to image regions with simple/regular structures(with small JND); Regular structures

Inspiration Intuitive perception ：  The HVS is sensitive to image regions with simple/regular structures(with small JND);  The HVS is insensitive to image regions with complex/irregular structures (with large JND); Regular structuresirregular structures

Theoretical Support: (Free-Energy Principle)  Recent perceptual science findings indicate that the HVS possesses an internal generative mechanism (IGM); Inspiration

Inspiration

Inspiration

By adjusting the internal structure of neurons By adjusting the internal structure of neurons Actively predicts the visual signal, while ignores the residual Actively predicts the visual signal, while ignores the residual When Perceiving an input scene Inspiration

Inspiration Theoretical Support: (Free-Energy Principle)  According to the free-energy principle, orderly structures are easily to be predicted, and the HVS is sensitive to them; Orderly structure

Inspiration Theoretical Support: (Free-Energy Principle)  According to the free-energy principle, orderly structures are easily to be predicted, and the HVS is sensitive to them;  Disorderly structures possess abundant uncertainty, and the HVS is insensitive to them. Orderly structure Disorderly structures

Inspiration Theoretical Support: (Free-Energy Principle)  According to the free-energy principle, orderly structures are easily to be predicted, and the HVS is sensitive to them;  Disorderly structures possess abundant uncertainty, and the HVS is insensitive to them.  Structural uncertainty determines the HVS sensitivity. Orderly structure Disorderly structures

Luminance Adaptation Luminance Adaptation JND Threshold JND Threshold Factors determine JND threshold ： Contrast Masking Contrast Masking Structural Uncertainty Structural Uncertainty Inspiration

Luminance Adaptation Luminance Adaptation JND Threshold JND Threshold Factors determine JND threshold ： Contrast Masking Contrast Masking Structural Uncertainty Structural Uncertainty How to calculate structural uncertainty? Inspiration

JND threshold estimation Structural Uncertainty:  Represent the disorderly degree of the visual structure;

JND threshold estimation Structural Uncertainty:  Represent the disorderly degree of the visual structure;  Decompose disorderly content from orderly content; Disorderly content Acquisition Disorderly content Acquisition AR prediction Based decomposition AR prediction Based decomposition TheoryTechnology

JND threshold estimation Structural Uncertainty:  Represent the disorderly degree of the visual structure;  Decompose disorderly content from orderly content;  Analyze the visual structure of the disorderly content; Disorderly content Acquisition Disorderly content Acquisition AR prediction Based decomposition AR prediction Based decomposition TheoryTechnology Visual Structure Characteristic Visual Structure Characteristic Local Binary Pattern (LBP) Local Binary Pattern (LBP)

JND threshold estimation Structural Uncertainty:  Represent the disorderly degree of the visual structure;  Decompose disorderly content from orderly content;  Analyze the visual structure of the disorderly content;  Calculate the structural uncertainty. Disorderly content Acquisition Disorderly content Acquisition AR prediction Based decomposition AR prediction Based decomposition TheoryTechnology Visual Structure Characteristic Visual Structure Characteristic Local Binary Pattern (LBP) Local Binary Pattern (LBP) Structural Uncertainty Structural Uncertainty Shannon Entropy Shannon Entropy

JND threshold estimation AR based disorderly content decomposition:  AR based prediction:

JND threshold estimation AR based disorderly content decomposition:  AR based prediction: The correlation between the central pixel and its i-th neighbor The correlation between the central pixel and its i-th neighbor

JND threshold estimation AR based disorderly content decomposition:  AR based prediction:  Disorderly content decomposition: The correlation between the central pixel and its i-th neighbor The correlation between the central pixel and its i-th neighbor

Input AR Pred. AR based disorderly content decomposition: JND threshold estimation

Input Orderly content Disorderly content AR Pred. AR based disorderly content decomposition: JND threshold estimation

Structure analysis on the disorderly content:  Local binary pattern (LBP) is successful in structure description;  The structure of a pixel is described as the correlations with its surround neighbors:  By assigning the binomial factors, LBP form is acquired: JND threshold estimation

Structural uncertainty computation: JND threshold estimation  Shannon Entropy is adopted;  Structural uncertainty is computed as the entropy of LBP on the disorderly content; The LBP distribution on a local region centered at The LBP distribution on a local region centered at

JND threshold estimation Structural Uncertainty: Original Image Structural Uncertainty

Pattern masking: JND threshold estimation Contrast Masking Affection of Structural Uncertainty

Pattern masking: JND threshold estimation Contrast Masking Affection of Structural Uncertainty

JND threshold estimation Pattern masking : parameter setting  for contrast masking function Linear Fitting The ground truth (the circle points) from subjective contrast masking testing

Pattern masking : parameter setting  for structural uncertainty function The ground truth comes from subjective viewing testing JND threshold estimation

JND computation model JND threshold estimation Luminance Adaptation Luminance Adaptation JND Threshold JND Threshold Pattern Masking Pattern Masking Luminance Contrast Luminance Contrast Structural Uncertainty Structural Uncertainty

JND computation model JND threshold estimation Luminance Adaptation Luminance Adaptation JND Threshold JND Threshold Pattern Masking Pattern Masking Luminance Contrast Luminance Contrast Structural Uncertainty Structural Uncertainty

Experimental Results Pattern Masking VS. Contrast Masking Contrast MaskingPattern Masking Original

Experimental Results Pattern Masking VS. Contrast Masking Contrast MaskingPattern Masking Original

Experimental Results Pattern Masking VS. Contrast Masking Contrast MaskingPattern Masking Original

Comparison among JND models ： Wu2013Proposed Yang2005 Liu2011 Original Experimental Results

Comparison among JND models ： Wu2013Proposed Yang2005 Liu2011 Original Experimental Results

Comparing with the existing JND models (Positive values means the proposed JND outperforms the compared model) Experimental Results

Comparing with the existing JND models (Positive values means the proposed JND outperforms the compared model) Experimental Results Subjective quality comparison (38 subjects are invited)

Conclusion The shortcoming of the existing JND model is analyzed; We suggest structural uncertainty is another factor which determines JND; Structural uncertainty is defined and calculated; Novel JND model is proposed; Experimental results demonstrate the effectiveness;

Related Works J. Wu, W. Lin, G. Shi, etc., “Pattern Masking Estimation in Image with Structural Uncertainty,” IEEE TIP. J. Wu, G. Shi, W. Lin, etc., “Just Noticeable Difference Estimation For Images with Free-Energy Principle,” IEEE TMM. J. Wu, F. Qi, and G. Shi. “Self-Similarity Based Structural Regularity for Just Noticeable Difference Estimation,” JVCI. J. Wu, W. Lin, and G. Shi. “Visual Masking Estimation Based On Structural Uncertainty,” ISCAS2013.(Best student paper).