1. Image/Video Coding Techniques for IPTV Applications Wen-Jyi Hwang ( 黃文吉 ) Department of Computer Science and Information Engineering, National Taiwan Normal University, Taipei, 117, Taiwan

2. Goal of This Talk  Overview of Image/Video Coding Techniques for IP TV Applications Image Coding: JPEG, JPEG2000 Video Coding: H.261, MPEG1, MPEG2, H.263, MPEG4, H.264

3. WHY Compression ? DSL ~200 kbps ~ 1,000 : 1 ~ 10,000 : 1 ITU-R 601 166 Mbps SDTV broadcasting ~2 Mbps ~ 100 : 1 Dial-up modem, wireless link ~ 20 kbps

4. Image Coding Techniques  JPEG (Joint Photographic Experts Group) DCT-Based Technique Published in 1992  JPEG2000 Wavelet-Based Technique Published in 2000

5. Principles of Image Coding Technique Color Transform DCT/WaveletQuantization Bitstream Formation

6. Color Transform (JPEG) Process the data in blocks of 8×8 samples Convert RGB into Luminance (Y) and Chrominance (Cr and Cb). Use half resolution for Chrominance (because eye is more sensitive to Luminance)

7. DCT (JPEG) Transform each block of 8×8 samples into 64 DCT coefficients –energy tends to be concentrated into a few significant coefficients

8. Quantization (JPEG) Divide each DCT coefficient by an integer, discard remainder Typically, a few non-zero coefficients are left.

9. Quantization (JPEG)

10. Wavelet Transform (JPEG2000)

12. Quantization(JPEG2000)

16. Quantization for Resolution- Scalable Transmission

17. Quantization (JPEG2000)

18. Layers 1+2Layers 1+2+3Layers 1+2+3+4

19. Quantization for SNR-Scalable Transmission

20. JPEG2000-Based Layered Transmission

21. JPEG2000-Based Region of Interest (ROI) Encoding

23. Comparisons (JPEG and JPEG2000) JPEG (64:1) JPEG2000 (64:1)

24. Video Compression-History “It has been customary in the past to transmit successive complete images of the transmitted picture.” [...] “In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object.”

25. Features of Moving Pictures  Moving images contain significant temporal redundancy successive frames are very similar

26. Motion Estimation and Compensation  The amount of data to be coded can be reduced significantly if the previous frame is subtracted from the current frame.

27. Block-Matching Results of block- matching The predicted current frame The reconstructed previous frameThe current frame

28. Predicted Frame and Prediction Error Predicted Current Frame Displaced Frame Difference (DFD, 估測誤差 )

29. I, B and P Frames  A video sequences may contain three different types of frames: (a)Intra frames, (I-frames) (b)Predicted frames (P-frames), (c)Bi-directional frames (B-frames)

30. Basic MPEG Standard Encoding order: I 0, P 3, B 1, B 2, P 6, B 4, B 5, I 9, B 7, B 8. Playback order: I 0, B 1, B 2, P 3, B 4, B 5, P 6, B 7, B 8, I 9.

31. Basic Video Encoder Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Video in DFD Motion Vector Encoded DFD Reconstructed DFD Reconstructed current frame Predicted current frame

32. Basic Video Decoder Encoded bitstream in Motion vector Encoded DFD Predicted current frame Reconstructed DFD Reconstructed current frame

33. Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in

34. Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in ¼-pixel accuracy

35. Fractional Motion Estimation

36. Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Adaptive block sizes...

37. Adaptive Block Size

38. Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Multiple Past Reference Frames

39. Advantages of Multiple Reference Frames

40. Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Generalized B-Frames

41. Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in De-Blocking Filtering

42. without deblock filtering with deblock filtering

43. Comparisons (H.264 and JPEG2000) JPEG2000 H.264

44. Comparisons (MPEG-4 and H.264)

45. MPEG-4 H.264 1 Mbps

46. Concluding Remarks  Image/Video Coding Techniques such as JPEG2000 and H.264 are able to provide high fidelity while significantly reduce bandwidth for image/video transmission.  These coding techniques therefore are effective for IPTV and/or DVB applications with limited bandwidth resources.