Comparison of subjective test methodologies VQEG Berlin meeting June 2009 P. Le Callet, R. Pépion.

Slides:

Advertisements

Similar presentations

Fountain Coding-based Video Transmission System over Heterogeneous Wireless Networks Presented by Hyunchul Joo POSTECH

Advertisements

Proposal for 3D evaluation test plan in VQEG Dec Jun Okamoto (NTT)

VQEG Boston meeting April 2006

International Telecommunication Union Committed to connecting the world Trondheim, 21 June ITU (International Telecommunication Union) ITU-T (Telecommunication.

Standards, process, requirements 4K PLAYBACK EXPLAINED.

Basics of MPEG Picture sizes: up to 4095 x 4095 Most algorithms are for the CCIR 601 format for video frames Y-Cb-Cr color space NTSC: 525 lines per frame.

(JEG) HDR Project Boulder meeting January 2014 Phil Corriveau-Patrick Le Callet- Manish Narwaria.

報告人：張景舜作者： Gardlo, B. ； Ries, M. ; Rupp, M. ; Jarina, R. ； Dept. of Telecommun. & Multimedia, Univ. of Zilina, Zilina, Slovakia QoE Evaluation Methodology.

QoE Assessment in Olfactory and Haptic Media Transmission: Influence of Inter-Stream Synchronization Error Sosuke Hoshino, Yutaka Ishibashi, Norishige.

SCHOOL OF COMPUTING SCIENCE SIMON FRASER UNIVERSITY CMPT 820 : Error Mitigation Schaar and Chou, Multimedia over IP and Wireless Networks: Compression,

An Error-Resilient GOP Structure for Robust Video Transmission Tao Fang, Lap-Pui Chau Electrical and Electronic Engineering, Nanyan Techonological University.

Perceptual Quality Assessment of P2P Assisted Streaming Video for Chunk-level Playback Controller Design Tom Z.J. Fu, CUHK W. T. Leung, CUHK P. Y. Lam,

Designing QoE experiments to evaluate Peer-to-Peer streaming applications Tom Z.J. Fu, CUHK Dah Ming Chiu, CUHK Zhibin Lei, ASTRI VCIP 2010, Huang Shan,

8th and 9th June 2004 Mainz, Germany Workshop on Wideband Speech Quality in Terminals and Networks: Assessment and Prediction 1 Vincent Barriac, Jean-Yves.

Copyright © Magnum Semiconductor, Unpublished Introduction to Deinterlacing by Mark Korhonen.

Proposal to study objective quality assessment of 4K resolution video in the future VQEG tests Osamu Sugimoto KDDI R&D Laboratories, Inc.

Over-the-Air (OTA) Bit Management David Felland Milwaukee Public Television.

© Tanner, KCL 2007 How do I decide if JPEG 2000 is for me? Choosing standards when there are so many… Simon Tanner Director.

Results of the ATIS/T1A1.1 Ad Hoc Group on Full-Reference Video Quality Metrics (FR-VQM) VSF Meeting October 3, 2001 John Pearson Sarnoff Corporation

Slide title In CAPITALS 50 pt Slide subtitle 32 pt Bitstream and Hybrid Model VQEG Meeting, Kyoto, March 2008 Jörgen Gustafsson and Martin Pettersson.

Maria Grazia Albanesi, Riccardo Amadeo University of Pavia, Faculty of Engineering, Computer Department Impact of Fixation Time on Subjective Video Quality.

NTT Confidential Test plan for evaluation of video quality models for use with stereoscopic three-dimensional television content Taichi Kawano (NTT)

Windows Media Video 9 Tarun Bhatia Multimedia Processing Lab University Of Texas at Arlington 11/05/04.

Impact of Lab Effects and Environment on Audiovisual Quality Margaret H. Pinson.

VQEG MM Phase 2 Working document towards a project plan.

(JEG) HDR Project: update from IRCCyN July 2014 Patrick Le Callet-Manish Narwaria.

1 Hybrid Bit-stream Models. 2 Hybrid bit-stream model: Type 1  Pros: Simple. All we need are open-source codecs.  Cons: May lose some available information.

1 Requirements for the Transmission of Streaming Video in Mobile Wireless Networks Vasos Vassiliou, Pavlos Antoniou, Iraklis Giannakou, and Andreas Pitsillides.

1 Hybrid Bit-Stream Models (January 24-26, 2009 San Jose)

Content Clustering Based Video Quality Prediction Model for MPEG4 Video Streaming over Wireless Networks Asiya Khan, Lingfen Sun & Emmanuel Ifeachor 16.

V-Factor QoE Platform Q-1000 Solution Overview. PAGE 2© COPYRIGHT SYMMETRICOM ( ) V-Factor Components Headend AnalyzerNetwork Probes Software.

November 1, 2005IEEE MMSP 2005, Shanghai, China1 Adaptive Multi-Frame-Rate Scheme for Distributed Speech Recognition Based on a Half Frame-Rate Front-End.

Adaptive Multi-path Prediction for Error Resilient H.264 Coding Xiaosong Zhou, C.-C. Jay Kuo University of Southern California Multimedia Signal Processing.

1 Hybrid Bit-Stream Models (January 24-26, 2009 San Jose)

1 Hybrid Bit-Stream Models (September 25, 2008 Ghent) FINAL.

DT_200906_VQEGMM2_AVtest.ppt1 Audio-visual quality tests. VQEG meeting, June 2009, Berlin Marie-Neige Garcia, Alexander Raake.

Image Compression Fasih ur Rehman. Goal of Compression Reduce the amount of data required to represent a given quantity of information Reduce relative.

AGH and Lancaster University. Assess based on visibility of individual packet loss –Frame level: Frame dependency, GoP –MB level: Number of affected MBs/slices.

AGH, NTIA, Intel, IRCCyN, Opticom, Technicolor.  Wide Range of Quality ◦ Original—Heavily Compressed  VGA  60 clips  10 SRC ◦ 50% English Language.

Rate-distortion Optimized Mode Selection Based on Multi-channel Realizations Markus Gärtner Davide Bertozzi Classroom Presentation 13 th March 2001.

1 JEG hybrid model Iñigo Sedano June, Three years working at Tecnalia Technology Corporation, Telecom Unit, Broadband networks group, Spain (

Towards a unique subjective experiment dataset for 3DTV – « 3DTV phase 1 » Marcus Barkowsky.

VQEG Joint Effort Group Berlin meeting A. Bourret- K. Brunnström P. Le Callet.

JEG-Hybrid. JEG Full database AGH database with additional packet loss HRCs AGH database of coding conditions JEG264HMIX1.

Nippon Telegraph and Telephone Corporation (NTT), Japan Taichi Kawano.

CS654: Digital Image Analysis

1. hybrid - bitstream models VQEG Berlin Meeting June 2009 for Discussion Jens Berger, Silvio Borer.

1 Video Quality for Public Safety Applications Margaret Pinson Public Safety Communications Research U.S. Dept. of Commerce June 6, 2011.

Discussion for JEG in Singapore Marcus Barkowsky.

Biostatistics Case Studies 2006 Peter D. Christenson Biostatistician Session 2: Correlation of Time Courses of Simultaneous.

T tests comparing two means t tests comparing two means.

Joint Effort Group Ghent meeting A. Bourret P. Le Callet.

Proposal for 3D evaluation test plan in VQEG June Jun Okamoto (NTT)

Transcoding based optimum quality video streaming under limited bandwidth *Michael Medagama, **Dileeka Dias, ***Shantha Fernando *Dialog-University of.

Block-based coding Multimedia Systems and Standards S2 IF Telkom University.

YONSEI Univ. High Dimensional Signal Processing Lab. 1 Comparison of DSCQS and SSCQE Chulhee Lee Yonsei University.

NTT Confidential Test plan for evaluation of video quality models for use with stereoscopic three-dimensional television content Taichi Kawano (NTT)

Moving from HDTV-1 to HDTV-2 VQEG agreed to use the HDTV Phase 1 test plan for the first draft of HDTV Phase 2 test plan The question for this meeting.

(JEG) HDR-WCG Project:

V ENUS INTERNATIONAL COLLEGE OF TECHNOLOGY Guided by : Rinkal mam.

(JEG) HDR & WCG? Project: September 2015 Patrick Le Callet.

2015 DOLBY LABORATORIES, INC. Overview of EBU and EPFL subjective tests on High Dynamic Range video 1 Ludovic Malfait Senior Engineer – Communication Group.

ELIS – Multimedia Lab Marcus Barkowsky Lucjan Janowski Glenn Van Wallendael VQEG JEG-Hybrid.

4K Resolution The future of resolutions. Table of Contents 1. What is resolution? 2. Common Terms 3. Background & Current Technologies 4. 4K resolution.

1 Hybrid Bit-stream Models. 2 Hybrid bit-stream model: Type 1  Pros: Simple. All we need are open-source codecs.  Cons: May lose some available information.

1 Quality of experience in High Definition Television: subjective assessments and objective metrics Stéphane Péchard October, 2 nd 2008 IV C.

MDC METHOD FOR HDTV TRANSMISSION OVER EXISTING IP NETWORK

Quantifying uncertainty using the bootstrap

Standards Presentation ECE 8873 – Data Compression and Modeling

The first test of validity

Presentation transcript:

Comparison of subjective test methodologies VQEG Berlin meeting June 2009 P. Le Callet, R. Pépion

Context, methodologies and issues Context: HRCs (coder, processing, transmission …) Resolutions Applications and services ACR (5, 11 categories …) Pair Comparison SAMVIQ DSCQS The value (e.g. accuracy, stability) of protocols might depend on the context … and the targeted goals

Outline Study 1: ACR 5 vs SAMVIQ HD H264 Study 2: Preference Tests vs SAMVIQ «processing» Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing Study 4: ACR5 for encoded + transmission error

Study 1: ACR 5 vs SAMVIQ HD Motivations: HDTV high quality in a short range => quality measure should be precise and discriminative Absolute Category Rating (ACR) - random order - only one viewing - category scale - no explicit reference... Good Subjective Assessment Methodology for Video Quality (SAMVIQ) - user-driven order - multiple viewing (natural?) - continuous scale - explicit reference

5 Previous and new studies [Brotherton, 2006] correlation on CIF (352x288): CC(MOS ACR, MOS SAMVIQ ) = 0.94 New studies: - Resolutions: QVGA, VGA and HD 1080i50 (viewing distance according to the resolution) - HRC: coding artefacts only (H264 AVC and SVC) CC(MOS ACR, MOS SAMVIQ ) = HDTV VGA QVGA 13° 19° 33° visual field RMSDiff= ACR and seems to provide “equivalent” results up to a certain resolution

6 Accuracy vs Number of observers confidence interval number of observers 24 « Suitable methodology in subjective video quality assessment: a resolution dependent paradigm » Stéphane Péchard, Romuald Pépion and Patrick Le Callet Proceedings of the Third International Workshop on Image Media Quality and its Applications, IMQA2008, Chiba, Japan, September 2008

Outline Study 1: ACR 5 vs SAMVIQ HD H264 Study 2: Preference Tests vs SAMVIQ «processing» Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing Study 4: ACR5 for encoded + transmission error

Study 2: Preference Test vs SAMVIQ « processing » Motivations: HDTV pre post processing, comparison between format on a 1080p display = > No other impairments 1080p SRC Pre Processing (interleaced and down Scaling) 1080i, 720p Pre Processing (deinterleaced + down Scaling) 720p Post Processing deinterleaced + up Scaling) Post Processing (up Scaling) 1080p PVS

Study 2: some results SAMVIQ Preference Test 1080p SRC compared to other PVS 7 categories preference test Generally good agreement but …further analysis is required (Thurstone Mosteller, CI …)

Outline Study 1: ACR 5 vs SAMVIQ HD H264 Study 2: Preference Tests vs SAMVIQ «processing» Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing Study 4: ACR5 for encoded + transmission error

Study 3: ACR5 vs ACR 11vs SAMVIQ encoded + processing Motivations: Comparison of 1080p50 with other HD and SD formats on a 1080p display => compression + processing Compression: H264 coder All formats (e.g. 1080p or i, 720p …) are coded at 3,6 and 9Mb/s and decoded before post processing. Processing: All formats are displayed in 1080p50 after decoding 1 deinterlacer : Smooth (VirtualDub/Avisynth), 2 Upscalers : Bilinear and Lanczos (VirtualDub/Avisynth).

Study 3: PVS generation 1080i50 720p x 1080i x 1080p50 SD Deint Upscale 1 Upscale 2 Deint Upscale 1 Upscale 2 Deint Upscale 2 Upscale 1 Upscale 2 29 HRC (8x3 HD +2x2 SD +1Ref) x 3 SRC = 87 PVS 3Mb/s 6Mb/s 9Mb/s Not for SD Upscale 1

ACR5 vs ACR11: correlation correlation between ACR 5 and 11: 0.98

Study 3: SAMVIQ vs ACR11, PVS generation X 1080i50 720p x 1080i x 1080p50 SD Deint Upscale 1 Upscale 2 Deint Upscale 1 Upscale 2 Deint Upscale 1 Upscale 2 10 HRC (8HD +1SD +1Ref) x 2 SRC = 20 PVS

Study 3: ACR11 vs SAMVIQ (on 20 PVS) Good correlation between ACR and SAMVIQ (0.97) => may be questionnable for high quality score

Study 3: score distribution ACR5 ACR11 SAMVIQ

Study 3: CI distribution

Outline Study 1: ACR 5 vs SAMVIQ HD H264 Study 2: Preference Tests vs SAMVIQ «processing» Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing Study 4: ACR5 for encoded + transmission error

Study 4: ACR5 encoded + transmission error The goal : analyse the relation between the position of the transmission error and the MOS on SD sequences. Each content is coded at 4 or 6Mb/s and some simulation of transmission errors are tested. Advanced FEC and Error concealment technique (ROI based)

Study 4: ACR5 encoded + transmission error X 14 HRC (Trans- Mission Errors) = 84 PVS

Study 4: ACR5 encoded + transmission error Reminder: coding only (study 3)