1. 1 Introduction to Video Compression תשס"ט – 2009 דחיסת מולטימדיה – הנדסת מערכות תקשורת Part A Introduction to Video Compression תשס"ט – 2009 דחיסת מולטימדיה – הנדסת מערכות תקשורת Part A Dr. Ofer Hadar Communication Systems Engineering Dept., BGU URL: http://www.cse.bgu.ac.il/hadarhttp://www.cse.bgu.ac.il/hadar Based on: 1 Lecture of John G. Apostolopoulos, Streaming Media System Group, HP Labs, Palo Alto, CA, japos@hpl.hp.com japos@hpl.hp.com 2. Lecture of Didier Le Gall, MPEG: A Video Compression Standard for Multimedia Applications Copyright @2006, O. Hadar

2. 2 Introduction to Video Coding

3. 3 Digital Data  Lots of Bytes (1) (40% of original size) 1200 lines x 1600 pixels per line RGB, 24 bit (3 bytes) per color pixel Total uncompressed (raw) size is 5.8 Mbyte 36 photo’s film: 200 Mbyte (about 1/3 of a CD-ROM) Download time: 1.5 hours (GSM); 12 minutes (ISDN/56k); 46 second (ADSL); 5 seconds (slow IntraNet); <1 second (fast IntraNet) Digital camera technology 2000

4. 4 Digital Data  Lots of Bytes (2) 576 lines x 720 pixels per line YUV, 16 bit (2 bytes) per color pixel Total uncompressed (raw) size per frame is 830 kByte 1 hour of video 75 GByte Download time: 16 hours (slow IntraNet); 2 hours (fast IntraNet) Digital (MiniDV) camcorder technology

5. 5 Need for Video Compression

6. 6

7. 7 Need for Image & Video Compression

8. 8 Video Data Size size of uncompressed video in gigabytes image size of video 1280x720 (1.77) 640x480 (1.33) 320x240 160x120

9. 9 Video Bit Rate Calculation width  pixels(160, 320, 640, 720, 1280, 1920, …) height  pixels(120, 240, 480, 485, 720, 1080, …) depth  bits(1, 4, 8, 15, 16, 24, …) fps  frames per second (5, 15, 20, 24, 30, …) compression factor(1, 6, 24, …) width * height * depth * fps compression factor = bits/sec

10. 10 Effects of Compression storage for 1 hour of compressed video in megabytes 3 bytes/pixel, 30 frames/sec

11. 11 Be Careful... mpeg 200:1, jpeg 24:1 analog source digital representation compressed representation vs

12. 12 Types of Compression (1) n Lossless compression u Reversible compression 001010010110 001011010101 Compressed photo Identical CompressDecompress

13. 13 Types of Compression (2) n Lossy compression u Irreversible compression 000011110110 101011110100 Compressed photo No longer Identical Compress (Coder) Decompress (Decoder)

14. 14 Compression System CoderDecoder signal insignal out bit rate compressed signal Quantify the numerical/perceptual difference

15. 15 Compression to VBR or CBR n CBR (Constant Bit Rate) vs VBR (Variable Bit Rate) Scene more complex  Higher bit rate for same quality CBR  variable quality (example : Video CD artefact) Constant quality  VBR necessary (e.g.: DVD-Video)

16. 16 Compression Technology Change the “Format”

17. 17 50 %, 1.4 MByte 25%, 360 kByte 12.5%, 90 kByte 100%, 5.8 Mbyte Reduce the Resolution (1)

18. 18 Reduce the Resolution (2) n Compression Factor 1.0 4.0 16.0 64.0

19. 19 24 bit (16777200 different colors) Number of bits to represent one image (color) pixel Reduce the Number of “Colors” (1)

20. 20 8 bit (256 different colors) Compression factor 3 Reduce the Number of “Colors” (2)

21. 21 6 bit (64 different colors) Compression factor 4 Reduce the Number of “Colors” (3)

22. 22 4 bit (16 different colors) Compression factor 6 Reduce the Number of “Colors” (4)

23. 23 1 bit (2 different colors) Compression factor 24 Reduce the Number of “Colors” (5)

24. 24 8 bit (256 different gray values) Compression factor 3 Reduce the Number of “Colors” (6)

25. 25 4 bit (16 different gray values) Compression factor 6 Reduce the Number of “Colors” (7)

26. 26 3 bit (8 different gray values) Compression factor 8 Reduce the Number of “Colors” (8)

27. 27 2 bit (4 different gray values) Compression factor 12 Pulse Code Modulation (PCM) Compression Reduce the Number of “Colors” (9)

28. 28 Compression Technology Transform Coding

29. 29 Images and Video Frame of Picture Sequence Line time Pixel or Pel

30. 30 Spatial and Temporal Redundancy

31. 31 Neighboring Pixels are Correlated Locally there is little variation in color. If you know one pixel intensity, you can make a fair prediction about its neighbors

32. 32 Spatial Redundancy n Take advantage of similarity among most neighboring pixels

33. 33 Temporal Redundancy n Take advantage of similarity between successive frames 950951952 (Simpsons News Clip Here)

34. 34 “Talking Head” Temporal Activity

35. 35 Temporal Redundancy Reduction

36. 36 Temporal Redundancy Reduction

37. 37 Temporal Redundancy Reduction

38. 38 Temporal Redundancy Reduction (Cont.)

39. 39 Group of Pictures (GOP) n Starts with an I-frame n Ends with frame right before next I-frame n “Open” ends in B-frame, “Closed” in P-frame u (What is the difference?) n MPEG Encoding parameter, but ‘typical’: u I B B P B B P B B I u I B B P B B P B B P B B I n Why not have all P and B frames?

40. 40 How to Compress? n Removal of statistical redundancy u Spatial redundancy: intra coding u Temporal redundancy: inter coding n Human visual system F We are more sensitive to luminance than color. F We are more sensitive to low spatial frequencies than high spatial frequencies n Lossless compression vs. lossy compression

41. 41 Perceptual Coding n Encode source signal using lossy compression u Lossless algorithms typically reduce signal by 3:1 u Must use lossy algorithm to get adequate compression n Hide errors where humans will not see or hear it u Study hearing and vision system to understand how we see/hear u Masking refers to one signal overwhelming/hiding another (e.g., loud siren or bright flash) u Audio perception is 20-20 kHz but most sounds in low frequencies (e.g., 2 kHz to 4 kHz) u Visual perception strongly influenced by edges and low frequencies

42. 42 Spatial Redundancy: Intra Coding Block-based schemes - Transform coding - Vector quantization (VQ) Non block-based schemes - Subband/Wavelet coding - Pyramid coding

43. 43 Block-Based Coding

44. 44 Color Transformation n RGB  YUV color coordinates n U/V chrominance components are downsampled in coding UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

45. 45 Color Formats UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002) Revised from R.Liu Seminar Course ’00 @ UMD

46. 46 Block-based Coding … … Sequence Picture GOB YY YY CBCB CRCR Block Macroblock (MB)

47. 47 MPEG2 Video Basics: Sequence (Display Order) GOP (Display Order, N=12, M=3) Picture Slice MacroBlock Y Cr Cb 45 01 23 Y Blocks Cr Block Cb Block BBBBBBBBIPP P Note: Y = Luma Cr = Red-Y Cb = Blue-Y Source:Sarnoff Corporation