CMPT365 Multimedia Systems 1 Media Compression - Video Coding Standards Spring 2015 CMPT 365 Multimedia Systems

CMPT365 Multimedia Systems 2 Video Coding Standards H.264/AVC

CMPT365 Multimedia Systems 3 Coding Rate and Standards kbit/sMbit/s Very low bitrate Low bitrateMedium bitrateHigh bitrate Mobile videophone Videophone over PSTN ISDN videophone Digital TVHDTVVideo CD MPEG-4MPEG-1MPEG-2H.261H.263

CMPT365 Multimedia Systems 4 Standardization Organizations r ITU-T VCEG (Video Coding Experts Group) m standards for advanced moving image coding methods appropriate for conversational and non-conversational audio/visual applications. r ISO/IEC MPEG (Moving Picture Experts Group) m standards for compression and coding, decompression, processing, and coded representation of moving pictures, audio, and their combination r Relation m ITU-T H.262~ISO/IEC (mpeg2) Generic Coding of Moving Pictures and Associated Audio. m ITU-T H.263~ISO/IEC (mpeg4) WG - work group SG – sub group ISO/IEC JTC 1/SC 29/WG 1 Coding of Still Pictures ISO/IEC JTC 1/SC 29/WG 11

CMPT365 Multimedia Systems 5 Introduction r H.261 r MPEG-1 r MPEG-2 r H.263 r MPEG-4 r H.264

CMPT365 Multimedia Systems 6 H.261 r Earliest DCT-based video standard: 1990 r ITU Recommendation for videoconferencing and videophones over ISDN  Targeted bit rate: p x 64 kbps (p=1, …, 30) m Videophone: low rate, e.g., 64kbps m Videoconferencing: high rate, e.g., 384kbps (p=6) m Max: 1.92Mbps (p=30) r Picture format: m CIF (Common Intermediate Format, 352 x 288) m QCIF (Quarter CIF): 176 x 144. r Max delay: 150 ms (for bidirectional interactivity) r Sequential search r Amenable to low-cost VLSI implementation r No B mode

CMPT365 Multimedia Systems 7 MPEG-1 r Committee formed in 1988 r Finalized in 1991 r Used for VCD r Random access, fast forward/reverse search r Delay: 1 sec (for unidirectional video access) r 1/2-pixel ME/MC r No deblocking filter r B frames r Software-only decoding is possible r MPEG-1 Audio coding: m 3 layers of encoding: Layer 1: 4 : 1 compression ratio with CD quality Layer 2: 6 : 1 to 8 : 1 Layer 3 (MP3): 10 : 1 to 12 : 1

CMPT365 Multimedia Systems 8 MPEG-2 r Completed in 1994 r Extension of MPEG-1 r Standard for DVD, SDTV, HDTV r Support interlaced inputs r Support scalable coding r Flexible frame size r Low delay r Support a wide range of applications r Source format: 4:4:4:, 4:2:2, 4:2:0 r 1/2-pixel ME/MC (bilinear interpolation) r B frames r MPEG-2 Audio: m Support 5.1 channels m AAC: 30% fewer bits than MP3

CMPT365 Multimedia Systems 9 Profiles and Levels r Defined to manage the large number of coding tools and the broad range of formats and bit rates supported r Profiles and levels define a set of conformance points, each targeting a class applications r Maximize interoperability and limiting the complexity r Profile: a subset of the entire bit stream syntax r Levels: a specified set of constraints imposed on values of the syntax elements in the bit stream (maximum bit rate, buffer size, pic. resolution)

CMPT365 Multimedia Systems 10 MPEG-2 Levels LevelMax PixelsMax LinesMax Frame/s Low Main High High

CMPT365 Multimedia Systems 11 H.263 r Derived from H.261 r Intended for very low bit-rate application m Better quality at 18-24kbps than H.261 at 64 kbps  Used in MS NetMeeting, Messenger … r Can handle high resolution (up to 16CIF: 1408 x 1152) r No loop filter r 1/2-pixel ME/MC r Optional coding modes (defined in 8 Annexes): m Unrestricted motion vector (Annex D): MV can point outside of picture boundary by extrapolating the boundary pixels (repeat padding is usually used) MV range: [-31.5, 31.5] m Arithmetic coding m Advanced prediction (Annex F): Overlapped block motion compensation 4MV: 1 for each 8x8 block

CMPT365 Multimedia Systems 12 H.263+ and H r H.263+: Second version of H.263 r Some further optional features: Annex I to T. r Annex J: in-loop deblocking filter r H.263++: three more optional modes (2000) m Annex V: Data partitioned slice mode For enhanced resilience to transmission error

CMPT365 Multimedia Systems 13 MPEG-4 r Based on H.263 r A new concept rather than an improved algorithm r Deal with a variety of multimedia contents: audio, visual, image, graphic. r Part 2: Visual m Based on H.263 m Object-based coding m Coding of animated objects m Scalability: Fine Granular Scalability (FGS) m Texture coding: wavelet-based r Part 10: Advanced Video Coding m H.264

CMPT365 Multimedia Systems 14 Video Objects (VO) r MPEG-4 treats a video sequence as a collection of video objects r Each scene is decomposed into multiple objects m The segmentation method is not part of the standard r Each object is specified by shape, motion, and texture. r Natural visual Objects: m Image, video, sprite (background) r Synthetic visual object: m Face and body m 2-D mesh m 3-D mesh r The decoder can compose different scenes by using different number of decoded objects

CMPT365 Multimedia Systems 15 Scene Composition r The decoder can compose different scenes by using different number of decoded objects

CMPT365 Multimedia Systems 16 MPEG-4 Structure A/V object Decoder MUX Compositor BitstreamAudio/Video scene A/V object Decoder A/V object Decoder

CMPT365 Multimedia Systems 17 A video frame Background VOP VOP More MPEG-4 Example r Instead of ”frames”: Video Object Planes r Shape Adaptive DCT Alpha mapSA DCT

CMPT365 Multimedia Systems 18 Object 2 Object 1Object 3 Object 4 Example Problems, comments?

CMPT365 Multimedia Systems 19 Example

CMPT365 Multimedia Systems 20 Status r Microsoft, RealVideo, QuickTime,... m But only recentagular frame based r H.264 = MPEG-4 part 10 (2003)

CMPT365 Multimedia Systems 21 H.264 r 1998: Call for proposal for H.26L issued by ITU-T VCEG (Video Coding Expert Group) r Objective: m 50% bit rate savings compared to MPEG-2 m High quality video at both low and high bit rates m More error resilience tools r Oct. 1999: First draft design r Dec. 2001: VCEG and MPEG formed the Joint Video Team (JVT) r Approved in 2003: m ITU-T H.264 and ISO/IEC MPEG-4 Part 10 Advanced Video Coding (AVC)

CMPT365 Multimedia Systems 22 H. 264 Applications r Bit rate: 64kbps to 240Mbps  Broadcast over cable, satellite, DSL …  Interactive/serial storage on optical/magnetic devices, DVD … r Conversational services over network r Video on demand, streaming media over network r Multimedia messaging service over network r Three Profiles: Baseline, Main, and Extended r 15 levels r Four new profiles in Fidelity Range Extenstions (FRExt): m High, High 10, High 4:2:2, High 4:4:4

CMPT365 Multimedia Systems 23 H. 264 Slice coding Slice 1 Slice 2 Slice 3 r Slices can have different shapes and sizes r Each slice is self-contained m Can be decoded without knowing data other slices r Useful for: m Error resilience and concealment m Parallel processing

CMPT365 Multimedia Systems 24 Summary of Standards StandardDigitisation format Compressed rateExample applications H. 261CIF/ QCIFX 64 kbpsVideo conferencing over LANs H. 263S-QCIF/ QCIF<64kbpsVideo conferencing over low bits rate channels MPEG 1SIF<1.5MbpsVHS quality video storage MPEG 2 Low Main High 1440 High SIF 4:2:0 4:2:2 4:2:0 4:2:2 4:2:0 <4Mbps <15Mbps <20Mbps <60Mbps <80Mbps <100Mbps VHS quality video recording Digital video broadcasting High definition TV (4/3) High definition TV (16/9) MPEG 4Various5kbps – tens MbpsVersatile multimedia coding standard H.264Various SIF: Standard Interchange Format, 352x240 pixels at 30 Hz.

CMPT365 Multimedia Systems 25 Beyond H.264 r H.265 m 50% goal (bitrate reduction) m Start from 2010 m February 2012: Committee Draft (complete draft of standard) m July 2012: Draft International Standard m January 2013: Final Draft International Standard (ready to be ratified as a Standard) m July 7: Formal release r Scalable video coding r Multiview video/3D video r 4K UHD