Future Video Coding Prof. Dr.-Ing. Jörn Ostermann

Technology Improvements Video Coding Technology Improvements Infrastructure Device Constant need for improved compression More devices, more pixels, more frames, more user-generated content 50% rate reduction in 10 years Constant need for additional functionality High dynamic range, wide color gamut, head mounted displays, augmented reality, virtual reality Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de

New usages: Panorama mono Video Coding New usages: Panorama mono Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de

New usages: Panorama stereo Video Coding New usages: Panorama stereo Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de

Use Cases for Future Video Codec Existing Markets Terrestrial, and Satellite broadcasting, cable services, managed IPTV services, surveillance New Markets Over-the-top (OTT) services, User generated content, video conferencing, gaming Emerging Markets Augmented and Virtual reality, multicast streaming/broadcast; media centric internet of things Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de

Requirements on Future Video Codec Source material 8k, 0Hz, 120 Hz, WCG, HDR, 10bit Rectangular, panorama, projected / viewports Natural, synthetic Monoscopic, multiview Network interface Streaming, error resilience, scalability, low delay Emerging Markets Augmented and Virtual reality, multicast streaming/broadcast; media centric internet of things Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de

Relevant Bodies ISO/IEC JTC1/SC29/WG11 Motion Picture Experts Group (MPEG) ITU-T Study Group 16 with Q6/16 (VCEG) Common work between MPEG and VCEG MPEG-4 AVC (ITU-T H.264, ISO/IEC 14496-10) HEVC (ITU-T H.265, ISO/IEC 23008-2) MV-HEVC, 3D-HEVC as extensions of HEVC Joint Video Experts Team (JVET) Future Video Coding Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de

Experimental open source software “Joint Exploration Model“ (JEM) JEM – where we are Experimental open source software “Joint Exploration Model“ (JEM) Intended to investigate potential for better compression beyond HEVC Open source code available for download https://jvet.hhi.fraunhofer.de/ Documentation openly and freely available for download: http://phenix.it-sudparis.eu/jvet/ HEVC software plus additional or improved compression tools General architecture similar to HEVC Hybrid closed-loop, high level syntax, reference picture management, etc. Improved/new compression tools Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de

Overall: Hybrid Coding / Closed Loop Decoded Picture Buffer Filter Control Data Header Formatting & CABAC Scaling & Inverse Transform Motion Compensation General Control Quantized Transform Coefficients Intra Prediction Intra/Inter Selection Estimation Transform, Scaling & Quantization - Input Video Signal Split into CTUs Intra-Picture Prediction Deblocking & SAO Filters Output Coded Bitstream Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de Source: Prof. Dr.-Ing. Jens Ohm

JEM Results compared to HEVC (HM) Measured average rate reduction for same PSNR (Bjontegaard delta method) Average encoder and decoder run time Random Access Over HM-16.16 (sequential, gcc6.2) Y U V EncT DecT Class A1 -27.2% -33.1% -35.2% 10.7x 5.6x Class A2 -36.3% -45.2% -38.2% 7.8x 6x Class B -27.0% -36.9% -31.7% 8.8x 6.3x Class C -26.3% -31.2% -33.6% 11.3x Class D -26.0% -30.0% -31.5% 11.5x 11.7x Overall -28.5% -35.4% -33.9% 9.9x 7.2x Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de Source: Prof. Dr-Ing. Jens Ohm

JEM – HEVC – AV1 Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de

Call for Evidence results Joint Call for Evidence on Video Compression with Capability beyond HEVC (April 2017) Test cases with eight 4K/SDR, five HD/SDR, five HD/HDR, and five 360-degree video sequences 4 rate points each Results of Joint Call for Evidence on Video Compression with Capability beyond HEVC (July 2017) Evaluates JEM anchors and other proposals against HEVC Main 10 Expert viewing tests with double-stimulus approach, allows grading on an MOS (mean opinion score) scale Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de Source: Prof. Dr.-Ing. Jens Ohm

Call for Evidence results Test cases SDR / UHD 4K, MOS over rate [kbps "Daylight Road" "Campfire Party" Source: JVET-G1004 Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de Source: Prof. Dr.-Ing. Jens Ohm

Call for Evidence results Test cases HDR / HD, MOS over rate [kbps] "Hurdles" "Cosmos" Source: JVET-G1004 Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de Source: Prof. Dr.-Ing. Jens Ohm

Call for Evidence results Test cases 360° Video, dynamic viewports, MOS over rate [kbps] Source: JVET-G1004 "Kite Flite" "Harbour" Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de Source: Prof. Dr.-Ing. Jens Ohm

Next Steps Joint Call for Proposals on Video Compression with Capability beyond HEVC Test cases with 4K/SDR, HD/SDR, HD/HDR, 360-degree video 4 rate points each HM anchors and JEM anchors Formal subjective assessment in labs Timeline: CfP October 2017 Submissions by February 2018, subjective testing March/April Evaluation of proposals: April 2018 Next generation standard by end of 2020 Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de Source: Prof. Dr.-Ing. Jens Ohm

Future Video Coding will focus on: Conclusions Future Video Coding will focus on: Rectangular 10 bit WCG HDR UHD video Future Video Coding will enable: Support for panoramic video Edge-based rendering for interactivity, games Future Video coding will deliver in 2020: Coding efficiency for UHD at 50% of the rate of HEVC for the same quality UHD at typically less than 4MBit/s MPEG continues to deliver compression technology beyond the state of the art Prof. Dr-Ing- Jörn Ostermann ostermann@tnt.uni-hannover.de