Video Coding --- State of the Art and Beyond Dr. Mona Mathur Advanced Systems Technology, STMicroelectronics, GNoida, India mona.mathur@st.com August 8, 2011

Agenda Overview of concepts in Video Coding State of the art Compression standards Evolving Video Coding standards The Road ahead ---- Challenges

Digital Video characteristics Frame : is a 2D rectilinear array of pixels Frame rate number of frames displayed per second Frame dimensions width and height of the frame in pixels Pixel depth Number of bits per pixel Each pixel has two values associated with it Luminance – represents the intensity (Y) Chrominance – represents the colour of the pixel (Cr, Cb) OBESE

Very high secondary storage and transmission requirements. Why compression ???? Images require considerably more storage space than text Audio and Video demand even more!! Text (VGA screen @ 2 bytes/char) ~ 640 * 480 * 2 / (8 * 8) ~ 9.4 KB Uncompressed stereo audio ~ 2 * 44100 * 2 (44.1 KHz, 16 bits per sample) ~ 172 KB/s PAL (720 pixels, 576 lines, 25 frames) ~ 249 Mbits/s !! A 2 hour movie ~ 1669GB more than 2000 CDs Very high secondary storage and transmission requirements.

History of Video Standards ITU --- International Telecommunications Union --- is focused on telecom applications ---- has created H.26x standards for low bit rate telephony, x---1to4 ISO --- International Standards Organization ---- is focused on consumer applications in entertainment & computers ----- defined MPEG standards for videos, MPEG-1, MPEG-2 and MPEG-4 H.263 baseline profile has commonalities with MPEG-4 Part2 Simple profile

Use and Spread of Standards Critical for widespread adoption of codec technology The three players Customers Economies of scale --- so products are affordable Industry Interoperability between vendors Content providers Long life and broad demand of the content Ensures widespread adoption Proprietary standards AVS (China), WMV9 (Microsoft), On2, VP8, Nancy, RV10 (Real Networks Real Video) Standards are often Targeted for specific applications Proprietary standards available basically for media streaming applications WMV9 – standardized in the SMPTE(Society of Motion Pictures and Television Engineers) as VC-1 On2 --- formerly “Duck Technologies” developed a codec called “True Motion”--- acquired by Google in Feb 2010 ---

Compression rates for diff technologies Intra-frame Motion Compensation Integer-pel Half-pel Variable Block size Quarter-pel Multi-frame

Evolution of MPEG4

Compression ??? : How Irrelevancy Reduction Statistical Redundancy in the signal Spatial: Correlation between adjacent pixels Temporal: Correlation between pixels in adjacent pictures Irrelevancy Reduction Omits parts of the signal not perceived by the Human Visual System (HVS) No Change in pixels

Coding Tools in state of the art codecs Spatial redundancy Intra prediction Temporal redundancy Inter prediction Transform -- DCT Entropy coding VLC, CABAC Deblocking Filter

Frame – from Macroblock view

Spatial Coding

Temporal Coding

H.264 Encoder

Is Need for higher Compression Relevant Today? Compression Today?? Is Need for higher Compression Relevant Today? YES !!! Why ???? When a scene is captured by multiple cameras, e.g. for the purpose of user-controlled navigation through a video scene, or for 3D display with view adaptation, the video data rate may in the worst case linearly increase by the number of cameras that are used for simultaneous capturing. It will therefore be necessary to further compress the overall rate by proper exploitation of coherences between the different available views (inter-view redundancy) instead of independently compressing the different views. Ref -- -Jens papers on AdvancesInVideoCoding

Evolving Products Smart phones Tablets Ipads Net enabled STBs ….

Emerging Video Content &Type Video as a medium is changing. Spatial resolution QCIF -- 176x144 --- pixels HD --- 1920x1080 ---- 2,073,600 pixels UHD --- 4096x3,072 ---- 12,582,912 pixels Temporal resolution (25- 50 -200 fps) Increasing dynamic range of each component From 8 bit per component to 16 bit per component New visual modalities stereoscopic, Multi-view, 3D, holography, the myriad of consumer devices playing stored

Demands of Consumers… Imagine… …being able to see high quality, high definition video, on demand. …renting a DVD with choices of standard or high definition video, with multiple camera views Applications like Facebook, twitter, Google+ Mobile Video Services like : football highlights, personal video conferencing, or downloads of movie trailers such services will increase What about spectrum availability

Video Coding today …. New extensions within existing framework SVC, MVC, Free View point TV Next evolution of the video coding standards KTA, H.265 New Technologies Parametric Video Coding Texture Analysis and Synthesis

Video Coding today …. New extensions within existing framework SVC, MVC, Free View point TV Next evolution of the video coding standards KTA, H.265 New Technologies Parametric Video Coding Texture Analysis and Synthesis

HEVC ---High Efficiency Video Coding Proposed Video compression standard Successor to H.264/MPEG-4 AVC A joint team by ISO- MPEG and ITU-T --VCEG has been established JCT-VC --- Joint Collaborative Team on Video Coding in 2010 Nick named --- H.265 Goals: Increase compression by 50% over H.264 at comparable image/video quality At the expense of increased computational complexity Targeted platforms: Next generation HDTV displays and content capture systems Display resolutions from QVGA (320x240) upto 1080p and Ultra HDTV (7680x4320) http://en.wikipedia.org/wiki/High_Efficiency_Video_Coding

Testing Results --- BDrate with complexity Source --- JCTVC-D181

Benefits of arbitrary unit representation Codec optimized for content, applications and devices. Examples are: low resolution video services Videos with large homogenous regions A simple & elegant representation LCTB size, max hierarchical depth, & a series of split flags. Size-independent syntax representation Syntactical representation of remaining coding tools is simplified Simplifies the standard and the parsing process