1. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Media Compression

2. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) You are Here Network Encoder Sender Middlebox Receiver Decoder

3. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Why compress? “Bandwidth Not Enough” “Disk Space Not Enough” Size of Uncompressed DVD Movie =

4. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Why compress? “Bandwidth Not Enough” “Disk Space Not Enough” Size of Uncompressed DVD Movie = 720 x 576 x 3 x 25 x 60 x 120 = 208.6GB

5. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Optical Disc Formats (1) CD: ~650 MB DVD: 4.7 (4.38) GB (single layer) 8.5 (7.92) GB (dual layer) Single and dual sided (up to 18 GB) 1X max. read speed: ~10 Mb/s Video codec: MPEG-2

6. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) JPEG Compression

7. Original Image (1153KB) 1:1

8. Original Image (1153KB) 3.5:1

9. Original Image (1153KB) 17:1

10. Original Image (1153KB) 27:1

11. Original Image (1153KB) 72:1

12. Original Image (1153KB) 192:1

13. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Compression Ratio QualitySizeRatio Raw TIFF1153KB1:1 Zipped TIFF982KB1.2:1 Q=100331KB3.5:1 Q=7067KB17:1 Q=4043KB27:1 Q=1016KB72:1 Q=16KB192:1

14. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Magic of JPEG Throw away information we cannot see Color information “High frequency signals” Rearrange data for good compression Use standard compression

15. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Discard color information Y VU

16. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Color Sub-sampling 4:4:4 4:2:2 4:2:0 4:1:1

17. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 4:2:2 Sub-sampling Y VU

18. Original Image (1153KB) 4:2:0

19. Original Image (1153KB) “4:1:0”

21. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Discrete Cosine Transform Demo

22. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Quantization 24265 -54-10 235 -4-2 136 21 35 -2 88 88 88 816 88 8 32 64 308 -6 20 00 10 00 00 00 /= Quantization Table DC AC

23. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Differential Coding 308 6 20 00 10 00 00 00 253 21 10 00 40 00 10 00 273 21 10 00 40 00 10 00 308 6 20 00 10 00 00 00 -53 21 10 00 40 00 10 00 23 21 10 00 40 00 10 00

24. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Zig-zag ordering 273 21 10 00 40 00 10 00 27, 3, 2, 4, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0

25. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Run-Length Encoding 273 21 10 00 40 00 10 00 27, 3, 2, 4, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0 (27, 1) (3, 1) (2, 1), (4, 1), (1, 2), (0, 5), (1, 1), (0, 4)

26. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Idea: Motion JPEG Compress every frame in a video as JPEG DVD-quality video = 208.6GB Reduction ratio = 27:1 Final size = 7.7GB

27. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Video Compression

29. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Temporal Redundancy

30. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Motion Estimation

31. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Bi-directional Prediction

32. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Watch for Motion Vectors Demo

33. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) H.261 I-Frame P-Frame

34. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) MPEG-1 B-Frame

35. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) MPEG Frame Pattern Example display sequence: IBBPBBP … Example encoding sequence: IPBBPBB

36. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Compression Ratio Frame TypeTypical Ratio I10:1 P20:1 B50:1

37. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Sequence sequence header: width height frame rate bit rate :

38. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) GOP: Group of Picture gop header: time :

39. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Picture pic header: number type (I,P,B) :

40. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Picture

41. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Slice

42. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Macroblock

43. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Block YY YY U V 1 Macroblock =

44. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Structure Summary

45. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) For I-Frame Every macroblock is encoded independently (“I-macroblock”)

46. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) For P-Frame Every macroblock is either I-macroblock a motion vector + error terms wrt a prev I/P-frame (“P-macroblock”)

47. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) For B-Frame Every macroblock is either I-macroblock P-macroblock a motion vector + error terms wrt a future I/P-frame 2 motion vectors + error terms wrt a prev/future I/P-frame

48. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) MPEG-1/2 File Formats (Packetized) Elementary streams, ES & PES Program streams PS (reliable mediums, e.g., DVD) Transport streams TS (for lossy mediums, e.g., on-air broadcast) MPEG-2 Elementary Encoder Packetizer Systems Layer MUX Transport Stream Video Source Audio Source MPEG encoded streams Data Source Packetizer Flow chart © Manish Karir PES: *.m2v PES: *.m2a TS: *.ts *.m2t *.mpg

49. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Review: MPEG structure ES, PS, TS Sequence GOP Picture Slice Macroblock Block

50. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) MPEG Decoding (I-Frame) Entropy Decoding Dequantize IDCT 101000101

51. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) MPEG Decoding (P-Frame) Entropy Decoding Dequantize IDCT 101000101 Prev Frame +

52. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Future Frame MPEG Decoding (B-Frame) Entropy Decoding Dequantize IDCT 101000101 Prev Frame + AVG

53. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) There is more.. Half-pel Motion Prediction Skipped Macroblock etc.

54. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) MPEG in Daily Life MPEG Standards Bit-rateUsage MPEG-11.5MbpsVCD MPEG-23-45 Mbps DVD, SVCD, HDTV MPEG-4Scalable QuickTime, DivX, AVCHD

55. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Camcorders in Daily Life Different formats used DV25 (MiniDV, DVCAM, DVCPRO) Capacity: 1 hour ~ 13 GB Speed: 25 Mb/s (user data) Color sampling: 4:1:1 Compression ratio: ~10:1

56. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Codec Comparison “M-JPEG” (e.g., DV) versus “MPEG” No “perfect” codec -> application dependent Compression Technique “M-JPEG” (I-frames only) “MPEG” (Temporal compression) Compression ratioLow (10:1 to 30:1)High (>100:1) Editing (frame-accurate)EasyDifficult Encoding/decoding complexity SymmetricAsymmetric Processing latencyLow to MediumHigh Multi-generation lossMediumHigh

57. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) High-Definition Standard by ATSC 18 different sub-formats 720p and 1080i are the most interesting 1280x720x60p, 1920x1080x60i (30p) 1080p is non-standard, but available 1.4 Gb/s raw bandwidth 10 – 20 Mb/s compressed (distribution, broadcast) 100 – 135 Mb/s compressed (pro tapes: DVCPROHD, HDCAM; for editing)

58. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Consumer HD HDV: MPEG-2 19 (720p) / 25 Mb/s (1080i) Tape format http://www.hdv-info.org AVCHD: H.264 5 to 20 Mb/s Hard disk format http://www.avchd-info.org/

59. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Optical Disc Formats (2) HD DVD (now dead) Capacity: 15 GB and 30 GB 1X speed: 36 Mb/s Video codec: VC-1, H.264, MPEG-2 Blu-ray Capacity: 25 GB and 50 GB 1X speed: 36 Mb/s Video codec: VC-1, H.264, MPEG-2

60. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Recent Codec: H.264 Same quality at half the rate Encoding complexity: ~4X How: Variable block size motion compensation Multiple reference frames Deblocking filter... Also called MPEG-4 Part 10 or AVC

61. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Hands-On Download source code, compile and play with ffmpeg mpeg_stat Video ‘Surfing_short.m2t’ from course web site (98 MB, HDV, transport stream) Try different MPEG-1/2 encoding parameter

62. NUS.SOC.CS5248-2009 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Impact on Systems Design How to package data into packets? How to deal with packet loss? How to deal with bursty traffic? How to predict decoding time? : :