1. Subjectif tests requirements for HDR
Jean-Charles Gicquel, Jérôme Fournier, Orange Labs
VQEG, July 2014
presentation title

2. Introduction : Context
High sensitivity of CMOS sensors
Up to 16 f-stops (Sony F65)
More and more video cameras
Availability of displays with high dynamic range of luminance
Up to 5000 Cd/m2
Just 2 references at the moment: Dolby and Sim2
Several demonstrations of such technologies
Philips, Technicolor, BBC, Dolby,…
Workshop DVB-EBU in London 2013 and Munich 2014
Subjectif tests requirements for HDR

3. Introduction : Context
HDR and EDR under consideration for upcoming video standards
UHD-phase2
DVB, ITU, SMPTE
MPEG and EBU: need to have a common subjective method to evaluate fairly HDR technology
Subjectif tests requirements for HDR

4. Introduction : Orange position
Acquisition of a SIM2 display 2 months ago
Characterization a the display
Preparation of HDR, EDR contents and their SDR version in a format adapted to the HDR display
Visualization of these HDR and EDR contents and their SDR versions on it (viewing session with experts and non-experts)
First conclusions
Various opinions
Experts have mixed feelings regarding the type of contents: sceptical especially for movies contents
Globally non-experts appreciate the improvements
Subjectif tests requirements for HDR

5. Introduction : Orange position
To evaluate the added value of HDR technology in terms of QoE
To compare HDR and SDR references
New perceptual dimensions to be assessed?
realness, presence, visual discomfort/annoyance…
To have a fair comparison of existing and ongoing technologies
Impact on current end-to-end video architectures
HDR video formats
Components
Bandwidth and associated video interfaces
Transfer function and tone mapping issues
Compression schemes
SDR backward compatibility
 Development of a new subjective test method required
Subjectif tests requirements for HDR

6. Introduction : HDR assessment main issues
Classification of test sources
Mean, peak and minimum luminance of test material …
Test bed definition for subjective assessment of HDR contents
To display sources test material reducing potential degradations
Format conversion: bit depth, color components, EOTF, TMO
Video player
To propose a method to adjust HDR screen features
Specific to each HDR display?
Test charts dedicated to HDR rendering?
New subjective test method
Viewing conditions  use case relationship (cinema, TV, mobile)
Test protocol, subjective scales…
Subjectif tests requirements for HDR

7. Technical architecture new requirements
How to present HDR contents on HDR displays with different properties regarding the dynamic of luminance?
Exchange file format: OpenEXR?
Input file format (components, bit depth): display dependant?
Max and min luminance (contrast)
Psychophysical senstivity function in accordance with the use case (cinema, TV)? Weber, Barten, Moon & Spencer, …?
Transfer function (luminance and color quantization)
From 16 bits to 8, 10, 12 bits?
What is the position of VQEG considering these issues?
File format, conversion tools, selection of psychophysical function…
Subjectif tests requirements for HDR

8. Technical architecture new requirements
Additional keys issues
How to present SDR content on HDR displays?
SDR is the reference at the moment. It is needed to evaluate the added value of HDR compared to SDR
How to present REC. 709 contents on HDR display?
How to present HDR content on SDR displays?
Evaluation of backward compatibility
Tone Mapping Operator (Direct and inverse) for content generation
Subjectif tests requirements for HDR

9. Test method (1/3) Feature Factors Related issues HDR Impact
General Viewing Conditions
Luminance and contrast ratio
Peak luminance of display (Visual discomfort or annoyance => Sun glasses, visual fatigue for long duration visualization).
Required contrast ratio adapted to eye performances.
Yes
Background and room illumination
Define background and room illumination to avoid visual discomfort and annoyance during visualization: Still 10-15%?
Do we have to consider different use cases (Indoor/outdoor with cinema, TV and mobile applications)?
Display
Physical interfaces, Input video formats, Transfer functions (luminance…)
Calibration (luminance and colors)
Viewing distance
According to the fact that HDR increases the perceived resolution (higher contrast), do we need to reconsider PVD or DVD recommendations? ?
Viewing position
The peak luminance seems to be dependant of the viewing position in front of the display. Could we have more than one viewer?
Subjectif tests requirements for HDR

10. Test method (2/3) Feature Factors Related issues HDR Impact
Source signals
Video format
Which HDR exchange format? OpenEXR?
Yes
Video format conversion
Components-based conversion (RGB to Log(Y),u,v
Selection of test materials
Video content complexity
2D video complexity is depending on texture and motion levels.
What about HDR complexity? (min luminance, peak luminance, colors,…)
Content acquisition and calibration
Dynamic Range conversion (Tone mapping issue)
Observers
Number
Stability and reliability of results ?
Viewer’s screening
Measurement of visual system performances? HDR impact on the HVS behavior?
Subjectif tests requirements for HDR

11. Test method (3/3) Feature Factors Related issues HDR Impact
Presentation
What about the temporal presentation of SDR and HDR?
Yes
Test session duration
Re-evaluation of duration for presentation, voting, session length
Subjective scales
Realness, Visual comfort, Visual annoyance, Video quality,...
Comparison, Preference…
Visual fatigue
Objective and subjective measurement of visual fatigue induced by HDR
Test Results analysis
Viewer factors
Rejection criteria, others? ?
Multidimensional indicators analysis
Statistical methods for analysis, e.g. relation, interaction and combination of subjectively measured QoE indicators
Subjectif tests requirements for HDR

12. Orange proposal Test method
To design a subjective test method to answer the following question: What is the added value of HDR in comparison with SDR for TV services in terms of QoE?
Technical architecture based on a Sim2 display
Test method
Based on the Philips test method
Different room illuminations
EOTF adapted to HDR Contents
Preference scale
Tests material with various luminance levels (mean values?)
Additional tests using others perceptual scales
Questionnaires to understand viewers criteria to assess HDR contents
Realness, visual comfort/annoyance, perceived video quality…
Subjectif tests requirements for HDR

13. Thank you!
presentation title