Overview of the Scalable Video Coding Dan Grois Ben-Gurion University of the Negev Based on the presentation of Heiko Schwarz, Detlev Marpe, and Thomas Wiegand, and the presentation of Thomas Wiegand: http://iphome.hhi.de/wiegand/assets/pdfs/DIC_SVC_07.pdf Fraunhofer Institute, Nchrichtentechnik, Heinrich-Hertz Institute. 9/4/2018 Dan Grois, Ph.D. Candidate

Outline Introduction Problems Definition Functionality Goal Competition Applications Targets History of SVC Structure of SVC Temporal Scalability Spatial Scalability Quality Scalability Combined Scalability Profiles of SVC Conclusions 9/4/2018 Dan Grois, Ph.D. Candidate

Introduction - problem Non-Scalable Video Streaming Multiple video streams are needed for heterogeneous clients 8Mb/s 512Kb/s 1Mb/s 6Mb/s 4Mb/s 9/4/2018 Dan Grois, Ph.D. Candidate

Introduction - definition Scalable video stream Scalability Removal of parts of the video bit-stream to adapt to the various needs of end users and to varying terminal capabilities or network conditions Sub-stream n Sub-stream ki High quality … … reconstruction Sub-stream 2 Sub-stream k2 Sub-stream 1 Sub-stream k1 Low quality 9/4/2018 Dan Grois, Ph.D. Candidate

Scalability of Video - Modalities

Scalability of Video – Modalities (Cont.) Dan Grois, Ph.D. Candidate 9/4/2018

Introduction – functionality

Introduction – functionality (Cont.) Functionality of SVC Bit-rate adaptation to match the channel throughput Format adaptation for backwards compatible extension Power adaptation for trade-off between runtime and quality Graceful degradation when “right” parts of the bit-stream are lost 9/4/2018 Dan Grois, Ph.D. Candidate

Introduction - goal Goal of SVC Scalability mode Fidelity reduction (SNR scalability) Picture size reduction (spatial scalability) Frame rate reduction (temporal scalability) Selection of content (ROI or object-based scalability) Sub-stream ki H.264/AVC bit-stream … = (Quality) Sub-stream k2 Sub-stream k1 9/4/2018 Dan Grois, Ph.D. Candidate

Introduction - competition SVC is an old research topic (> 20 years) and has been included in H.262/MPEG-2, H.263, and MPEG-4 Visual. Rarely used because The characteristics of traditional video transmission systems Significant loss of coding efficiency and large increase in decoder complexity Competition Simulcast Transcoding 9/4/2018 Dan Grois, Ph.D. Candidate

Introduction - applications Heterogeneous clients Unequal protection Surveillance Problems of SVC Increased decoder complexity Decreased coding efficiency Temporal scalability is more often supported than spatial and quality scalability. 9/4/2018 Dan Grois, Ph.D. Candidate

Introduction - targets Little decrease in coding efficiency Little increase in decoding complexity Support of temporal, spatial, and quality scalability A backward compatible base layer Simple bit-stream adaptations after encoding 9/4/2018 Dan Grois, Ph.D. Candidate

Outline Introduction History of SVC Structure of SVC Temporal Scalability Hierarchical prediction structure Spatial Scalability Quality Scalability Combined Scalability Profiles of SVC Conclusions 9/4/2018 Dan Grois, Ph.D. Candidate

Temporal Scalability Hierarchical prediction structures 9/4/2018 Dan Grois, Ph.D. Candidate

Temporal Scalability (Cont.) 9/4/2018 Dan Grois, Ph.D. Candidate

Temporal Scalability Combination with multiple reference picture Arbitrary modification of the prediction structure Issue of quantization Lower layers with higher fidelity  Smaller QPs are used in lower layers Propagation of quantization error  smaller QPs are used in higher layers 9/4/2018 Dan Grois, Ph.D. Candidate

Temporal Scalability Video Coding Experiment with H.264/MPEG4-AVC Foreman, CIF 30Hz @ 1320kbps Performance as a function of N Cascaded QP assignment QP(P)  QP(B0)-3  QP(B1)-4  QP(B2)-5 9/4/2018 Dan Grois, Ph.D. Candidate This slide is copied from JVT-W132-Talk

Temporal Scalability Coding efficiency of hierarchical prediction 9/4/2018 Dan Grois, Ph.D. Candidate

Outline Introduction History of SVC Structure of SVC Temporal Scalability Spatial Scalability Inter layer prediction Quality Scalability Combined Scalability Profiles of SVC Conclusions 9/4/2018 Dan Grois, Ph.D. Candidate

Spatial Scalability texture Hierarchical MCP & Intra-prediction Base layer coding motion Inter-layer prediction Intra Motion Residual Spatial decimation Hierarchical MCP & Intra-prediction texture Base layer coding Multiplex Scalable bit-stream motion Inter-layer prediction Intra Motion Residual Spatial decimation H.264/AVC compatible base layer bit-stream H.264/AVC MCP & Intra-prediction texture Base layer coding motion H.264/AVC compatible coder 9/4/2018 Dan Grois, Ph.D. Candidate

Spatial Scalability Similar to MPEG-2, H.263, and MPEG-4 Arbitrary resolution ratio The same coding order in all spatial layers Combination with temporal scalability Inter-layer prediction Spatial 1 Temporal 2 Intra Spatial 0 Temporal 0 Temporal 1 Intra 9/4/2018 Dan Grois, Ph.D. Candidate

Spatial Scalability The prediction signals are formed by MCP inside the enhancement layer (Temporal) Up-sampling from the lower layer (Spatial) Average of the above two predictions (Temporal + Spatial) Inter-layer prediction Three kinds of inter-layer prediction Inter-layer motion prediction Inter-layer residual prediction Inter-layer intra prediction Base mode MB Only residual are transmitted, but no additional side info. 9/4/2018 Dan Grois, Ph.D. Candidate

Spatial Scalability Past spatial scalable video: Single-loop decoding Inter-layer intra prediction requires completely decoding of base layer. Multiple motion compensation and deblocking filter are needed. Full decoding + inter-layer prediction: complexity > simulcast. Single-loop decoding Inter-layer intra prediction is restricted to MBs for which the co-located base layer is intra-coded 9/4/2018 Dan Grois, Ph.D. Candidate

Spatial Scalability Generalized spatial scalability in SVC Arbitrary ratio Only restriction: Neither the horizontal nor the vertical resolution can decrease from one layer to the next. Cropping Containing new regions Higher quality of interesting regions 9/4/2018 Dan Grois, Ph.D. Candidate

9/4/2018 Dan Grois, Ph.D. Candidate

Outline Introduction History of SVC Structure of SVC Temporal Scalability Spatial Scalability Quality Scalability CGS MGS Drift control Combined Scalability Profiles of SVC Conclusions 9/4/2018 Dan Grois, Ph.D. Candidate

Coarse-grain quality scalability (CGS) 9/4/2018 Dan Grois, Ph.D. Candidate

Coarse-grain quality scalability (Cont.) Smaller quantization step sizes for higher enhancement residual layers Designed for only several selected bit-rate points Supported bit-rate points = Number of layers 9/4/2018 Dan Grois, Ph.D. Candidate

Quality Scalability Medium-grain quality scalability (MGS) More enhancement layers are supported Refinement quality layers of residual Key pictures Drift control Switch can occur at any access units CGS + key pictures + refinement quality layers 9/4/2018 Dan Grois, Ph.D. Candidate

DRIFT IN PAST SNR SCALABLE CODING

SNR SCALABILITY RESULTS: PAST CODECS 9/4/2018 Dan Grois, Ph.D. Candidate

SNR SCALABILITY RESULTS: H.264/SVC 9/4/2018 Dan Grois, Ph.D. Candidate

SNR SCALABILITY RESULTS: H.264/SVC 9/4/2018 Dan Grois, Ph.D. Candidate

CODER CONTROL 9/4/2018 Dan Grois, Ph.D. Candidate

ENCODER OPTIMIZATION: JSVM ENCODER CONTROL 9/4/2018 Dan Grois, Ph.D. Candidate

ENCODER OPTIMIZATION: JOINT CONTROL 9/4/2018 Dan Grois, Ph.D. Candidate

RESULTS FOR SNR SCALABILITY 9/4/2018 Dan Grois, Ph.D. Candidate

Outline Introduction History of SVC Structure of SVC Temporal Scalability Spatial Scalability Quality Scalability Combined Scalability SVC encoder structure Dependence and Quality refinement layers Bit-stream format Bit-stream switching Profiles of SVC Conclusions 9/4/2018 Dan Grois, Ph.D. Candidate

Combined Scalability SVC encoder structure Dependency layer The same motion/prediction information Dependency layer Temporal Decomposition The same motion/prediction information 9/4/2018 Dan Grois, Ph.D. Candidate

Combined Scalability Dependency and Quality refinement layers Q = 2 Scalable bit-stream D = 1 Q = 1 Q = 0 Q = 2 D = 0 Q = 1 Q = 0 9/4/2018 Dan Grois, Ph.D. Candidate

NAL unit header extension Combined Scalability Bit-stream format NAL unit header NAL unit header extension NAL unit payload 2 6 3 3 2 1 1 1 1 1 3 P T D Q P (priority_id): indicates the importance of a NAL unit T (temporal_id): indicates temporal level D (dependency_id): indicates spatial/CGS layer Q (quality_id): indicates MGS/FGS layer 9/4/2018 Dan Grois, Ph.D. Candidate

Outline Introduction History of SVC Structure of SVC Temporal Scalability Spatial Scalability Quality Scalability Combined Scalability Profiles of SVC Scalable Baseline Scalable High Scalable High Intra Conclusions 9/4/2018 Dan Grois, Ph.D. Candidate

Profiles of h.264/aVC 9/4/2018 Dan Grois, Ph.D. Candidate

Profiles of h.264/aVC (CONT.) 9/4/2018 Dan Grois, Ph.D. Candidate

Profiles of h.264/aVC (CONT.) 9/4/2018 Dan Grois, Ph.D. Candidate

SVC PROFILES 9/4/2018 Dan Grois, Ph.D. Candidate

Conclusions Dan Grois, Ph.D. Candidate 9/4/2018

Conclusions (Cont.) 9/4/2018 Dan Grois, Ph.D. Candidate

References 9/4/2018 Dan Grois, Ph.D. Candidate