1. MPEG2 FGS Implementation ECE 738 Advanced Digital Image Processing Author: Deshan Yang 05/01/2003

2. Introduction of FGS FGS = fine granularity scalability For MPEG2 / MPEG4 and others Comparing to SNR, temporal, spatial scalability, FGS enhances video quality continuous instead of stepwise. Using bitplane coding for enhancement layer, base layer is still using regular MPEGx coding Enhancement layer bitstream can be truncated. The quality of video stream is proportional to the amount of enhancement bitstream data used.

3. Project outline Implement MPEG2 FGS enhancement into the current MPEG2 codec project from MPEG MSSG (MPEG Software Simulation Group). Implementation includes: Updating the current MPEG2 codec source code to support FGS Implementing bitplane VL Codec and bit streaming Implementing a new FGS enabled MPEG2 player Challenges Understanding and using the MSSG MPEG2 codec source code (much more difficult than it seems to be) Bitplane coding / decoding MPEG2 player on Windows platform Media data to play with

4. Project outline (cont.)

5. Source code MPEG2 codec source code is from: MPEG Software Simulation Group (MSSG) at http://www.mpeg.org/MPEG/MSSG/ http://www.mpeg.org/MPEG/MSSG/ MPEG2 player source code: searching google.com for mpg2w11b.zip. My MPEG2 FGS player is combined from the the MPEG2 player with my MPEG2 FGS decoder MPEG2 player source.

6. MPEG2 Encoder More configurable options to control FGS encoding. Integrating FGS enhancement layer bitplane encoder into the original MPEG2 encoder Encoding base layer in very low bit rate Calculate the DCT coefficients residues before and after the quantization step. Encoding the DCT coefficients residues with the bitplane VLC Encoded bitplane VLC output data are stored (as binary files) as the enhancement layer bitstream Supporting I-frame and P-frame, not B-frame

7. MPEG2 FGS Encoder Architecture

8. MPEG2 Decoder More configurable options to control FGS decoding Integrating the FGS bitplane VLD into the original MPEG2 decoder Accessing the enhancement layer bit stream (from binary files on disk) Decoding DCT residues from the enhancement bit stream Control the degree of enhancement Generating the enhanced DCT coefficients Decoding the enhanced DCT coefficients to get better video over base layer Supporting I-frame and P-frame, not B-frame

9. MPEG2 FGS Decoder Architecture

10. MPEG2 Player Combination of original MPEG2 player with the new MPEG2 FGS decoder New features Supporting FGS Replay Stop Stretching to window Supporting wide screen format FGS scale change on the fly Turning FGS on/off Player window resizing Supporting new command line options

11. DCT Coefficients Before quantization 88,-14,0,0,-3,0,0,0,-40,-2,-2,0,0,0,0,0,-7,8,3,0,0,0,0,0,- 5,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,2 After base layer quantization 88,0,0,0,0,0,0,0,-32,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,- 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1 Difference 0,-14,0,0,-3,0,0,0,-8,-2,-2,0,0,0,0,0,-7,8,3,0,0,0,0,0,- 5,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,1 Sign bits x,1,x,x,1,x,x,x,1,1,1,x,x,x,x,x,1,0,0,x,x,x,x,x,1,x,x,x,x,x,x,x,x,0,x,x,x,x,x,x,x,0,x,x,x,x,x,x,x,x,x, x,x,x,x,x,x,x,0,x,x,x,x,0 Compressed sign bits 1,1,1,1,1,1,0,0,1,0,0,0,0

12. Bitplane coding DCT coefficient differences: 0, -14, 0, 0, -3, 0, 0, 0, -8, -2, -2, 0, 0, 0, 0, 0, -7, 8, 3, 0, 0, 0, 0, 0, -5, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0 Bit planes: 0:0000000000000000000000000000000000000000000000000000000000000000 1:0000000000000000000000000000000000000000000000000000000000000000 2:0000000000000000000000000000000000000000000000000000000000000000 3:0000000000000000000000000000000000000000000000000000000000000000 4:0100000010000000010000000000000000000000000000000000000000000000 5:0100000000000000100000001000000000000000000000000000000000100000 6:0100100001100000101000000000000001000000010000000000000000000000 7:0000100000000000101000001000000001000000010000000000000000000000 8:0200200022200000211000002000000001000000010000000000000000100000 (sign bits) Compressed sign bits: 1,1,1,1,1,1,0,0,1,0,0,0,0 Bitplane VLC encoding Planes 0,1,2,3 are all zero Plane 4: (1, 0, 0), (6, 0, 0), (8, 1, 0) Plane 5: (1, 0, 0), (14, 0, 0), (7, 0, 0), (33, 1, 0) Plane 6: (1, 0, 0), (2, 0, 0), (4, 0, 0), (0, 0, 0), (5, 0, 0), (1, 0, 0), (14, 0, 0), (7, 1, 0) Plane 7: (4, 0, 0), (11, 0, 0), (1, 0, 0), (5, 0, 0), (8, 0, 0), (7, 1, 1)

13. Bitplane coding (cont.) Bit planeVLC codingBinary code 0 All zero 3F 1 All zero 3F 2 All zero 3F 3 All zero 3F 4 (1, 0, 0), (6, 0, 0), (8, 1, 0) 01, 06, 48 5 (1, 0, 0), (14, 0, 0), (7, 0, 0), (33, 1, 0) 01, 0E, 07, 61 6 (1, 0, 0), (2, 0, 0), (4, 0, 0), (0, 0, 0), (5, 0, 0), (1, 0, 0), (14, 0, 0), (7, 1, 0) 01, 02, 04, 00, 05, 01, 0E, 47 7 (4, 0, 0), (11, 0, 0), (1, 0, 0), (5, 0, 0), (8, 0, 0), (7, 1, 1) 04, 0B, 01, 05, 08, C7 Code format: BPnnnnnn B: End of block flag, P: End of plane flag, nnnnnn: number of 0 before 1 3F=00111111=“All zero plane”, BF=0x10111111=“All planes are zero)

14. Bitplane VL Codec VLC Splitting DCT coefficients to bitplanes Converting each bitplane to VL codes Converting VL codes to binary codes (Huffman codes) Storing binary codes as enhancement layer bitstream (binary file) VLD Loading binary codes from enhancement layer bitstream (binary file) Converting binary codes (Huffman codes) to VL codes Converting VL codes to bitplanes Merging bitplanes to DCT coefficients

15. Enhancement Layer Bitstream

16. No FGS15% FGS 30% FGS 65% FGS

17. Demo Run fgs.bat

18. Issues Enhancement layer bitstream size is much bigger than the base layer bitstream size. Motion prediction can not be applied to enhancement layer Current implementation only supports I-frame and P- frame, not B-frame yet Current implementation does not support Huffman VLC coding in the bitplane VLC. I don’t have a good Huffman VLC table, and Huffman VLC will only help to reduce the enhancement bitstream size ¼ anyway.