Image and Video Compression Fundamentals Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2013 Last updated

Heejune AHN: Image and Video Compressionp Driving Force of Video Compr. Uncompressed Video Bandwidth Ver. Resolution x Hor. Resolution x Time Resolution x Colors Eg. CCIR 601 (TV Quality) 720x480x30x24 = 248,832,000 bps Typical Storage and Network DVD 4.7 GB (about 80 sec for CCIR) ADSL 100Mbps < CCIR BW

Heejune AHN: Image and Video Compressionp. 3 Typical values Typical Video Bandwidth ITU CCIR 601 L(858x525) C(429 x525) 30fps => 216.0Mbps CIF L (352x288) C(176x144) 30fps => 36.5Mbps QCIF L (176x144) C(88x72) 15fps => 4.6Mbps Typical Storage /Transmission Capacity Terrestrial TV broadcasting channel~20 Mbps CD/DVD-5 640MB/4.7GB Ethernet/Fast Ethernet <10/100 Mbps ADSL/VDSL downlink 2048 kbps/100Mbps Wireless cellular (2G/3G/3G+) 9.6/384/2000kbps

Heejune AHN: Image and Video Compressionp Image and Video Compression Information Theory 1950’s Claude Shannon (Bell Lab) pioneered. Providing Mathematical Limits for Information Processing/Communications Coding Source Coding How to Reduce the data for information representation Channel coding How to Transmit Data though Noise/Distored Channels Note : TDMA, FDMA, CDMA, OFDMA, and MIMO are all for the channelization methods Claude Elwood Shannon (April 30, 1916 – February 24, 2001)

Heejune AHN: Image and Video Compressionp. 5 Typical Visual Comm. System Typical path Info source Source coder channel coder modulator demodulator channel decoder Source decoder Info output Channel (wired/wirless/ storage)

Heejune AHN: Image and Video Compressionp. 6 Codec Codec = enCOder&DECoder Codec Types Lossless compression X == X’ Used for document file (ZIP), Medical Images (JPEG lossless) Entropy coding (Arithmetic coding, Huffman coding), Predictive coding Lossy compression X ~ X’ Used for Entertainment, Communication Multimedia (DCT), Quantization Encoder Decoder X Y X’

Heejune AHN: Image and Video Compressionp. 7 Uncompressed, Zipped, H264-encoded of same video Video Compression System Feature Source model Note: zip is source-independent encoding Human Visual System HVS does not notice many distortions

Heejune AHN: Image and Video Compressionp Predictive Coding DPCM (Differential Pulse Coded Modulation) Highly Correlated pixel values in Spatial Domain Code current (S 0 ) using previously coded ones (S 1, S 2, S 3 etc) Coder Block Diagram line of pixels above current line of pixels Predictor Entropy Coder Entropy Decoder Predictor Encoder Decoder

Heejune AHN: Image and Video Compressionp. 9 DPCM example original

Heejune AHN: Image and Video Compressionp. 10 Motion Compensation Prediction Temporal domain prediction How to use the temporal correlation? Model and representation methods Two successive video frames Change detection mask

Heejune AHN: Image and Video Compressionp. 11 Model based MC 2D/3D Model dx, dy, dz and rotations Estimate (ie. Calculate) the parameters in encoder and use for decoder Difficulties Too high Shape encoding, Estimation Complexity for now In MPEG-4 Object Oriented coding Moving area picked up by change detector Moving areas missed by change detector

Heejune AHN: Image and Video Compressionp. 12 Block Based MC Segment Fixed Size Block and find best matching displacement Easier Implementation in HW and SW Real Motion MV X(t) X(t+1)

Heejune AHN: Image and Video Compressionp Transform coding Transform Spatial Domain to Frequency Domain Easy for quantization Energy Compaction Properties and HVS properties No Compression itself

Heejune AHN: Image and Video Compressionp. 14 Block transform (fixed-size) Block Transform Easy for implementation Normally 2-D separable Transform

Heejune AHN: Image and Video Compressionp. 15 Transform types KL Transform is proved optimal DCT is fixed and similar to KL for image signals Wavelet and Fractal Transform etc (1) Karhunen Loève transform [1948/1960] (2) Haar transform [1910] (3) Walsh-Hadamard transform [1923] (4) Slant transform [Enomoto, Shibata, 1971] (5) Discrete CosineTransform (DCT) [Ahmet, Natarajan, Rao, 1974] (1) (2) (3) (4) (5)

Heejune AHN: Image and Video Compressionp. 16 Transform size The Larger Block, The more efficient, but The more Computationally complex 8x8 or 4x4 are used for Standards

Heejune AHN: Image and Video Compressionp Quantization Approximation of Values Lossy Coding (key data reduction) Applied to 2D transform Coefficient

Heejune AHN: Image and Video Compressionp. 18 Qstep (or qscale) Distortion Range The Larger/Coarse Q step The More Compression The Larger Distortion Rate Distortion Theory In Video Coding Applied to 2D transform Coefficients HVS Smaller in low freq Larger in high frequency Quantizer input Quantizer output   

Heejune AHN: Image and Video Compressionp Entropy Coding Statistical redundancy in video coding Many zeroes in quantized transform coefficients Unequal histogram of control info, like motion vectors and coding type Entropy coding Principle “Shorter Code words for More Frequency events” Variable Length Coding (VLC) Huffman coding Integer VLC: each code words are integer length Used for most Standards Arithmetic Coding Fractional Length Coding Started from H.263+ but used in H.264 practically

Heejune AHN: Image and Video Compressionp. 20 VLC coding in Image Coding Zigzag scan used for more statistical correlation 2-D Run-Length Code (num of zeros, no zero value) Q (8) Run-level coding Zig-zag scan Transformed 8x8 block

Heejune AHN: Image and Video Compressionp Codec Design Hybrid Codec Most Standards Codec MC => DCT => Quant => Entropy Coding Intra-frame Decoder Motion- Compensated Predictor Control Data DCT Coefficients Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Intra-frame DCT Coder - Entropy coder Quant DeQ

Heejune AHN: Image and Video Compressionp. 22 Complexity Consideration Asymmetric Complexity Encoders are more complex for most standards Non-real time Encoding but Real time Encoding (e.g. Broadcasting, Storage) One time encoding many time decoding Encoder and decoder Cost Parallel Processing and HW/SW implementation (in MPEG-2) Motion Compensation (~ 55%) DCT/DCT (~15%) VLC encoding/Decoding (~15%) Other (post processing) (15%)