Source Coding for Video Application Video coding is the compression or decompression of digital video. The compression is usually lossy. Historically, video was stored as an analog signal on magnetic tape. After the compact disc appeared and entered the market as a digital format. It replaced the analog audio, it became possible to also begin storing the video in digital from.

Most Common video codecs H.261 MPEG-1 Part 2 H.262/MPEG-2 Part 2 H.263/MPEG-4 Part 2 H.264/MPEG-4 Part 10/AVC H.265/MPEG-H HEVC WMV (Windows Media Video) AVS (Standard in China, similar to H.264) ITU Telecommunication Standardization Sector Moving Picture Experts Group

Most Common video codecs H.261 MPEG-1 Part 2 H.262/MPEG-2 Part 2 H.263/MPEG-4 Part 2 H.264/MPEG-4 Part 10/AVC H.265/MPEG-H HEVC WMV (Windows Media Video) AVS (Standard in China, similar to H.264)

H.261 First ratified in 1988 First digital video coding standard that was truly useful in practical All subsequent international video coding standards have been based closely on the H.261 design, MPEG-1 Part 2 H.262/MPEG-2 Part 2 H.263 MPEG-4 Part 2 H.264/MPEG-4 Part 10

H.261 Structure

CIF and QCIF Frame Formats

GOB and Resynchronization Purpose of Group of Blocks is resynchronization GOB starts with a sync code (binary: 00000000 00000001) Within a GOB, encoded MBs don’t even start on byte boundaries If there’s a bit error and you lose sync, or you join in the middle, you can’t decode the next bits (you don’t know where you are in the bit stream) Scan for the next GOB sync code, and then you can start decoding.

Macroblocks Macroblock is basic unit for compression Each macroblock is 16x16 pixels Represent as YUV 4:2:0 data 16x16 Luminance (Y) and subsampled 8x8 Cr, 8x8 Cb Represent this as 6 Blocks of 8x8 pixels:

Color Representation Requires three colors Red (R), Green (G), and Blue (B) Nominally requires three times B&W bandwidth Can compressed into 1.25 – 2 times B&W bandwidth by matrix: Luminance component (Y) Two chrominance components (I&Q or U&V) Luminance represent the color in black and white TV compatible

RGB-YIQ conversion RGB-YUV conversion 𝑌=0.299𝑅+0.587𝐺+0.114𝐵 𝑈= 𝐵−𝑌 2.03 𝑉= 𝑅−𝑌 1.14

Macroblock Coding Three Rules: Don’t send it if is hasn’t changed since last frame. Intra-frame compression Do DCT, Quantize, Zig-zag, Run-length encoding, and Huffman coding. Just like JPEG Inter-frame compression Calculate difference from previous version of same block Can use motion estimation to indicate block being differenced can from a slightly different place in previous frame Same DCT/quant/Huffman coding as Intra, but data is differences rather than absolute values

Intra-frame compression Intra-frame coding of Microblock is very similar to JPEG 8x8 DCT (Discrete Cosine Transformation) Quantization Uniform quantizer (∆=8) for intra-mode DC coefficients Uniform threshold quantizer (∆=2,4,…,62) for AC coefficients in intra-mode and all coefficients in inter-mode Zigzag Scan Order coefficients in zig-zag order Run-length encode Huffman code The discrete cosine transform (DCT) helps separate the image into parts (or spectral sub-bands) of differing importance (with respect to the image's visual quality). The DCT is similar to the discrete Fourier transform: it transforms a signal or image from the spatial domain to the frequency domain.

Inter-frame compression Basic compression process is the same as intra-frame compression, but the data is the differences from the immediately preceding frame rather that the raw samples themselves. Frame differencing Often the amount of information in the difference between two frames is a lot less than in the second frame itself. Motion estimation Motion in the scene will increase the differences If you can figure out the motion (where each block came from in the previous frame): Encode the motion as a motion vector (two small integers indicating motion in x and y directions) Encode the differences from the moved block using DCT + quantization + RLE + Huffman encoding Basic compression process is the same as intra-frame compression, but the data is the differences from the immediately preceding frame rather that the raw samples themselves. Often the amount of information in the difference between two frames is a lot less than in the second frame itself.

General Description of H.261 H.261 achieves compression by removing redundancy Temporal redundancy: reduced by motion compensation Spatial redundancy: reduced by DCT Statistical redundancy: reduced by entropy coding (e.g Huffman / arithmetic coding) Perceptual irrelevancy: reduced by quantization

General Description of H.261 Two type of coding applied to the frames in H.261 Intra-frame coding: (I frames) Removes spatial redundancy by JPEG-like coding Predictive coding: (P frames) Reduce spatial redundancy by JPEG-like coding Reduce temporal redundancy by motion compensation Achieve higher compression ratio than intra-frame coding Normally, every I frame is followed by many P frames. A new I frame would be used when there is a scene change When there is no scene change, I frames are inserted periodically to stop any error from propagating

General Description of H.261 Two type of coding applied to the frames in H.261 Intra-frame coding: (I frames) Removes spatial redundancy by JPEG-like coding Predictive coding: (P frames) Reduce spatial redundancy by JPEG-like coding Reduce temporal redundancy by motion compensation Achieve higher compression ratio than intra-frame coding

Intra-Block Encoding

Inter-Block Encoding

Bitstream Structure

H.263 Son of H.261 Standardized in 1996 Replcaing H.261 in many applications Basic design is very similar to H.261 (DCT/Quantization based, using intra or inter framecoding) Numerous optional improvements to improve compression, robustness, and flexibility of use

H.263 Improvements SQCIF: 128x96 4CIF: 704x576 QCIF: 176x144 Half-pixel precision in motion vectors (vs full-pixel precision for H.261) New options: Unrestricted Motion Vectos, Syntax-based arithmetic coding (replace RLE/Huffman) Advance preodiction (Uses 4 8x8 blocks instead of 1 16x16 which gives better detail) Forward and Backward frame prediction similar to MPEG Five resolutions (H.261 only have QCIF and CIF): SQCIF: 128x96 4CIF: 704x576 QCIF: 176x144 16CIF: 1408x1152 CIF: 352x288

History of video coding standards H.261/263/264, MPEG-1/2/4

Several MPEG standards MPEG-1: video compression at 1.5Mbit/s, CD-ROM MPEG-2: video compression at 2-20Mbit/s, CCIR601, HDTV MPEG-3: dissolved (combined with MPEG2) MPEG-4: video compression at less than 28.8kbit/s, Videophone

Major Applications of Video Compression Digital television broadcasting 2 – 5 Mbps (10 – 20 Mpbs for HD) MPEG-2 H.264/AVC DVD video HD-DVD Blu-ray Disk 4 – 8 Mbps H.264/AVC, VC-1 Internet video streaming 20 – 600 kbps Proprietary, similar to H.263, MPEG-4, or H.264/AVC, VC-1 Videoconferencing, videotelephony 20 – 320 kbps H.261, H.263, H.264/AVC Video over 3G wireless 20 – 200 kbps H.263, MPEG-4, H.264/AVC, VC-1

References H.261:  ITU-T (1988). "H.261 : Video codec for audiovisual services at p x 384 kbit/s - Recommendation H.261 (11/88)". Retrieved 2010-10-21. JPEG compression: http://www.cs.cf.ac.uk/Dave/Multimedia/node234.html H.263: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=556485 MPEG: http://www.google.com.hk/books?hl=zh-TW&lr=&id=GtCCIlpecBkC&oi=fnd&pg=PR5&dq=MPEG- 2&ots=Lxyhnl7BVG&sig=4G7kQp7Kj4Qmx5OuyhgBL1dAv1k&redir_esc=y#v=onepage&q=MPEG- 2&f=false

Q & A

Arrangement of Group of Pictures (GOPs) in a Video Sequence I = Intra-frame coding P = Predictive coding B = Bidirectional coding

Coding Order of I,B,P-frame