1. Entropy Coding Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2013 Last updated 2013. 10. 28

2. Heejune AHN: Image and Video Compressionp. 2 Agenda [Midterm exam solution] Entropy Coding in Video Codec Basic Information Theory on Entropy coding Huffman Coding and Implementation Arithmetic Coding and Implementation

3. Heejune AHN: Image and Video Compressionp. 3 1. Entropy Coding in Video Coding Video Encoder Components A source model + an Entropy Coder Design Goals of entropy coder (lossless) Compression efficiency : smaller output bit count Computational efficiency: Implementation suitability Error robustness and access efficiency: easy to synchronize Entropy coder Source Model Texture Info Block Motion Transfor m Quantiz er Symbol mapper Entropy coders Motion Info Control/Mode info

4. Heejune AHN: Image and Video Compressionp. 4 Data symbol Mapping Symbols Input to (General) Entropy Coder Well-defined Information units from real-raw data (numbers) e.g.) Coefficients in blocks, Motion Vectors for MBs, Coding Modes for Video Coding (DCT) Coefficient symbol mapping Properties Sparse non-zeros Near DC (0, 0) vertical and horizontal : even or not Solution Zig-zag scanning (Run, Length) symbol mapping 102 -33 - 4 - 2 - 1 - 3 21 -2 - 1 - 3 - 1 2 1 1 - 2

5. Heejune AHN: Image and Video Compressionp. 5 Zigzag Scanning Frame picture Field Picture (horizontal-first scan) Run-Length (run, length) : 2-D symbol mapping in H.261 and MPEG-1/2 EOB (end of block) : extra symbol for all remaining zeros (run, length, last) : 3-D in H.263 and MPEG-4 Intra DC is differential coded form previous block Do the text example (p 166)

6. Heejune AHN: Image and Video Compressionp. 6 Motion Vector Symbols Differential coded Other H.261 Quaitzation Parameters Differential coded : DQ = Qcurrent -Qprev No abrupt quant step change (-> abrupt quality change) p i = Median (mv1 i, mv2 i, mv3 i ) mvd i = mv i -p i

7. Heejune AHN: Image and Video Compressionp. 7 Coded Mode Coding Mode Motion compensated or not, Coded or not etc block has data or not No texture info Synchronization Headers High-Level Header Global parameters (coding options): resolution, pixel aspect ratio, quantization options, etc

8. Heejune AHN: Image and Video Compressionp. 8 2. Entropy Coding Background Definition of Entropy (by. C. Shannon) Input symbols space & occurrence prob. : Information Uncertainty = information (when revealed) length, amount of info for a symbol Rare Events (p = 0.0000001, e.g...) => big news ! => more info Interpretation (the implication) The amount of Information of S (Qualitatively) The less predictable, the more information Shannon, Information theory (Quantitatively) H(S) is the theoretical limit (upper-bound) bits for representing S.

9. Heejune AHN: Image and Video Compressionp. 9 Binary Signal Example (0,1) Entropy Entropy is maximum at p = ½, ie. equi-probable.

10. Heejune AHN: Image and Video Compressionp. 10 Entropy coding Concept Assign shorter length code for more frequent symbol, Average code- length is reduced So, VLC (variable-length-coding) Lossless coding algorithm: used in File compression An example Let code a file with “squeeze” {s, q, u, e, z} with prob. { 1/7, 1/7, 1/7, 3/7, 1/7} Fixed length coding 3 bits (5 symbol)* 7 symbol = 21 bits VLC coding {u, s, u, q, z} => {‘1’, ‘000’, ‘001’, ‘010’, ‘011’} 1bit*3symbols + 3bits*4symbols = 3 + 12 = 15 bits ( << 21 bits)

11. Heejune AHN: Image and Video Compressionp. 11 Coding algorithm How to code? Still “Open” Problem Welcome you if you dare Huffman Coding Integer length Code Book Used in H.261, H.263, MPEG-2 Pioneered David Huffman (in 1952) Arithmetic Coding Factional length Coding file compression (zip) Used in H.263, MPEG-4, H.264 Pioneered by Jorma Rissanen David Huffman Jorma Rissanen

12. Heejune AHN: Image and Video Compressionp. 12 Prefix-free coding A code is not same as the prefix part of any other code So, uniquely decodable by scanning from the start Huffman Code is a kind of prefix code Compare two codes symbolcodeword a1 a2 a3 a4 0 10 101 0101 Symbolcodeword a1 a2 a3 a4 0 10 110 1110

13. Heejune AHN: Image and Video Compressionp. 13 Huffman Coding Mapping a symbol to a variable length code variable & integer length code (an optimal integer length code) uniquely decodable (prefix-free) Huffman Code Generation Step 1 : Order the symbols in increasing order of probability Step 2 : Take two smallest, assign ‘1’ to the larger probability symbol and ‘0’ to the smaller. Step 3 : Combine the two smallest-probability symbols into a node (group), and assign the combined probability to the node Step 4 : IF only one node remains, STOP, ELSE reorder the resultant symbols (considering the combined node as one symbol)

14. Heejune AHN: Image and Video Compressionp. 14 Huffman Encoding & Decoding Encoding Table lookup Symbol => code (code value, length) Decoding Comparing codes bit by bit Bit stream => boundary (code value and length) => symbol 00001 111 210 3000001 400001 …01 N-3001 N-1000000 01… 111 102 001N-3 00010 000014 0000013 000000N-1 Encode Table Decode Table

15. Heejune AHN: Image and Video Compressionp. 15 An Example Motion Vector Code book Generation (See Fig 8.5) Encoding example 3 motion values, 1, 0, 0.5 => 01011/1/011 Decoding example 01011/1/011 => 1, 0, 0.5 mvxProb.log 2 (1/P) -1.50.0146.16 0.0245.38 -0.50.1173.10 00.6460.63 0.50.1013.31 10.0275.21 1.50.0165.97 mvxCodebit length (Huffman) Bit length (Ideal) 0110.63 -0.50023.10 +0.501133.31 -1.50100055.21 1.50100155.38 0101055.97 10101156.16

16. Heejune AHN: Image and Video Compressionp. 16 Code Table Design Ideal Huffman Code Input statistics adaptive; performance Disadvantages Statistics calculation burden (big data) Transmission overhead from an encoder to decoders Implementation difficulty Generic Huffman code based on large set data (various video sequences) Disadvantages Lower performance possible Video coding standards Provides generic code with optional extension

17. Heejune AHN: Image and Video Compressionp. 17 Code table design example H.263/MPEG-4 TCOEF 3 Dimensional (last, run, level) Ideal table size = 2 x 63 x 128 x 2 (sign). Huge!! Only 102 most frequent values has specific code (upto 13 bits) Others : ESCAPE Code (0000011) + 13 bit (last, run, level) Note : Fixed-length code is better for very small probability codes No successive 8 0’s allowed Used for synchronization start code

18. Heejune AHN: Image and Video Compressionp. 18 26 partial list ‘s’ : sign bit LastRunLEVELCode 00110s 011110s 0211110s 0021111s 1010111s 03101101s 04101100s 05101011s 003010101s 012010100s 061010011s 071010010s 081010001s 091010000s LastRunLEVELCode 111001111s 121001110s 131001101s 141001100s 0040010111s 01010010110s 01110010101s 01210010100s 1510010011s 1610010010s 1710010001s 1810010000s ESCAPE0000011s.etc

19. Heejune AHN: Image and Video Compressionp. 19 H.263/MPEG-4 MVD (motion vector difference) dmvx and dmvy See table 8.6 and Figure 8.8 MVD = 0 is 1 bit (‘1’) H.264 Universal VLC (UVLC) Systemic (Universal) table generation (just need ordering) 1 1-bit code, 2 3-bit codes, 4 5-bits codes…. Easy to implementation and statistics adaptive 1 => most frequent 1 symbol 0 x 0 1 => the next 2 symbols 0 x 1 0 x 0 1 => the next 4 symbols 0 x 2 0 x 1 0 x 0 1 => … 0 X 3 0 x 2 0 x 0 1 …

20. Heejune AHN: Image and Video Compressionp. 20 Example Textbook Coeff matrix (0,0,4), (0,0,-1), (0,1,2), (0,0,-3), (0,5, -1), (1,3,1) Final bit stream 00101110/101/0101000/010111/0011010 41 -3 2

21. Heejune AHN: Image and Video Compressionp. 21 Implementation Issues Encoder Length required (because of Variable length) Decoder Packing Register (Shifter) Lookup Or Calculate VLC Code Length Symbol Table select Output bitstream Packing Register (Shifter) Lookup Or Calculate VLC bitstream Length Symbol Error Input bitstream

22. Heejune AHN: Image and Video Compressionp. 22 Difficulties Table Size is huge num of last X num of run X num level H.263/MPEG-4 TCOEF => 16002 entries over 64 MB TLB for Encoder Over 16K Comparator for Decoder Complexity Reduction in Encoder If(|level| < 13 && run < 39){ output TLB[last][run][level]; }else{ output ESCAPE CODE; output last; output run; output level; } Switch(last){ case.. switch(run){ case.. switch(level){ } … …. Table Based Method Code Based Method If((last, run, level) in most frequent range){ output TLB1[last][run][level]; }else if((last, run, level) in 2nd frequent range){ } else ….

23. Heejune AHN: Image and Video Compressionp. 23 Complexity reduction in decoder do{ Bi = take first Li bits from stream; symbol = TLB1[Bi]; }while( symbol != continue); 01… 111 102 001NA 001N-3 00010 000014 0000013 000000N-1 01… 111 102 001N-3 00010 000014 0000013 000000N-1

24. Heejune AHN: Image and Video Compressionp. 24 Error Handling Cause of error Transmission error, storage error, system errors packet loss, bit-reversal, bit-erosion Error detection Undefined code (in syntax), visual corruption (in semantics) Error recovery Even after Forward error correction/Feedback Recovery Synchronization Code (slice or picture level) Reversible VLC code (in MPEG-4) Real Error Error detection Sync. code

25. Heejune AHN: Image and Video Compressionp. 25 Arithmetic Coding Motivating example Table 8.1: 0.5 (p = 0.101)  ideally 3.31 bits but 5 bits assigned Why ideal and why 5 bit assigned ? Arithmetic coding Assign code a range not a value (fractional bit code) Better compression than Integer Coding Especially one big symbol Patent IBM: Q-Coder (-2) (-1) (0) (+1) (+2) 0.0 0.1 0.3 0.7 0.9 1.0

26. Heejune AHN: Image and Video Compressionp. 26 Arithmetic Encoding Algorithm Step 1: Step 2: Calculate Range for n-th symbol Repeat remaining symbols or Stop Send any value in the range Normally the shortest representation (e.g. binary)

27. Heejune AHN: Image and Video Compressionp. 27 Example: Encoding and Decoding Input symbols 0, -1, 0, 2 (0) 0.0 0.1 0.3 0.7 0.9 1.0 (-1) 0.3 0.34 0.42 0.58 0.66 0.7 (0) 0.34 0.348 0.364 0.396 0.412 0.42 (2) 0.364 0.3672 0.3736 0.3864 0.3928 0.396 0.394531 => check !!

28. Heejune AHN: Image and Video Compressionp. 28 More Issues on Arithmetic Coding Performance 0.394531 =0.011001010b => 9 bits Theoretic min = 0.828 bits Termination method Why I stopped decoding with 4 symbols? I knew that 4 symbols required (predefined length) Extra “End of Sequence” code Fixed point arithmetic no symbolic computing CPU yet. All floating point is not exact. H.263/4 14 bits accuracy, i.e. smallest probability is 1/16384 Shift the determined MSB bits and contains only LSB bits in buffer 01011111010100101010010101 Unchangeable after new symbol range calculation

29. Heejune AHN: Image and Video Compressionp. 29 Conclusion Entropy coding Symbol to bit stream mapping and reverse Data compression using statistical redundancy approaching to the theoretical limit (entropy) Integer length code like Huffman code Fractional length codes like Arithmetic code Entropy coding in video coding Texture (DCT coefficients) Motion Vectors Mode and side information Huffman code is traditional tool for entropy coding Arithmetic code is new stronger tool but not widely used yet.