1. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 1Lecture 9 ECEC-453 Image Processing Architecture Lecture 9, 2/12/ 2004 MPEG 1 Oleh Tretiak Drexel University

2. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 2Lecture 9 Review JPEG  Modes  Sequential DCT  Progressive DCT  Lossless  Hierarchical  (Lossy) DCT  Multiple color components  Quantization tables  Entropy coding: DC coefficients ZZ scan, run-length coding, Huffman coding

3. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 3Lecture 9 Review: Data Interleaving with Subsampling Example: a color image with Y (intensity), Cb, Cr, (color) components is subsampled so that one color block corresponds to four Y blocks MCU 1 = Y 00 Y 01 Y 10 Y 11 Cr 00 Cb 00, MCU 2 = Y 02 Y 03 Y 12 Y 13 Cr 01 Cb 01

4. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 4Lecture 9 Color Conversion (from JFIF)

5. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 5Lecture 9 Resolution Reduction Trials FullY down by 64Cb, Cr down by 64

6. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 6Lecture 9 RGB reduction FullR, B reduced by 64

7. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 7Lecture 9 Coding AC Coefficients AC coefficients are coded in zig-zag order to maximize possible runs of zeros. Code unit consist of run length followed by coefficient size. Baseline coding of size is the same as for DC differences (Table 2.9) Example: run of 6 zeros, size = -18. In the table, -18 is in category 5. Code is (6/5, 01101). If the Huffman code for 6/5 is 1101, codeword = 110101101

8. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 8Lecture 9 Huffman Coding - Block Diagram

9. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 9Lecture 9 Lecture Outline Prediction and motion compensation MPEG-1 and relatives — history Video coding - how MPEG-1 works Details Wrapup Teleconferencing MPEG-2

10. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 10Lecture 9 Predicting sequential images f(t-1)f(t)f(t) f(t)–f(t–1)

11. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 11Lecture 9 Motion Compensation Macroblock size  MxN Matching criterion  MAE (mean absolute error) Search window  ±p pixel locations Search algorithm  Full search  Logarithmic search  Parallel Hierarchical One-Dimensional Search  Pixel subsampling and projection  Hierarchical downsampling

12. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 12Lecture 9 Motion Estimation Methods No compensation Full search logarithmic search 3 level hierarchical

13. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 13Lecture 9 Video Coding History & Standards

14. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 14Lecture 9 Video Coding Chronology Late 1970: High bandwidth digital links drop in price  H.120, H.130, CCITT standards for video telephony, not popular in US and Japan because of technical shortcomings Late 1980's:  H.261 (also known as Px64, reads as P times 64): videoconferencing over ISDN —> ratified in 1990 Mid 1980's:  Sarnoff lab develops system for recording video on CD's (1.5 Mbit). Others follow, ratified as MPEG-1 in 1991. 1990 — start work on MPEG-2, ratified as H.262 in 1994.  Build on the ideas of MPEG-1, but added features for broadcasting 1994 — start work on MPEG-4,  Object-based standard (multimedia).

15. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 15Lecture 9 MPEG Home Official web site  http://www.chiariglione.org/mpeg/ Information site  http://www.mpeg.org/ History  MPEG-1, the standard for storage and retrieval of moving pictures and audio on storage media (approved Nov. 92)  MPEG-2, the standard for digital television (approved Nov. 94)  MPEG-4 version 1, the standard for multimedia applications (approved Oct. 98), version 2, (approved Dec. 99)  MPEG-4 versions 3&4  MPEG-7 the content representation standard for multimedia information search, filtering, management and processing.  Started MPEG-21, the multimedia framework.  http://www.chiariglione.org/mpeg/standards/mpeg-21/mpeg-21.htm

16. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 16Lecture 9 MPEG-1: How it works Goals What the standard specifies MPEG-1 decoder block diagram

17. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 17Lecture 9 MPEG-1: ‘1.5’ Mbps Sample rate reduction in spatial and temporal domains Spatial  Block-based DCT  Huffman coding (no arithmetic coding) of motion vectors and quantized DCT coefficients  352 x 340 pixels, 12 bits per pixel, picture rate 30 pictures per second —> 30.4 Mbps  Coded bit stream 1.15 Mbps (must leave bandwidth for audio)  Compression 26:1  Quality better than VHS! Temporal  Block-based motion compensation  Interframe coding (two kinds)

18. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 18Lecture 9 MPEG-1 Facts Decoder only is specified (encoder is up to implementers) Layered specification Must work in real time over fixed bandwidth media: bit rate control Must satisfy diverse externally imposed requirements  NTSC vs. PAL  Recorded media vs. Broadcast

19. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 19Lecture 9 Block Diagram of MPEG Decoder

20. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 20Lecture 9 Details and buzzwords Interlace, frame and field, picture NTSC and PAL CCIR 601 SIF Constrained parameter bit stream I, P, & B pictures Bit stream, GOP

21. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 21Lecture 9 Legacy Video Standards CRT technology, analog modulation and transmission NTSC (America and Japan)  2 interlaced fields = 1 frame  Frame contains 525 lines, about 10% not visible (vertical retrace)  30 frames per second, 60 fields per second  RGB in video camera and on CRT display, converted to composite video (luminance and chrominance in same frequency band) PAL (Europe)  Interlace, etc  625 lines per frame, 25 frames (50 fields) per second  Different (better) modulation of color (newer standard)

22. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 22Lecture 9 Frames and Fields MPEG 1 works with pictures (~ frames)

23. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 23Lecture 9 Input to MPEG-1 Standard allows many formats (up to 4095x4095 pixels) Standard optimized for CCIR 601 video formats: two source input formats (SIF’s) are specified (NTSC & PAL)  Coded color video has three components: Y, Cb, Cr A MPEG-1 macroblock has 16x16 Y and 8x8 Cb, Cr pixels

24. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 24Lecture 9 Picture Types MPEG-1 is designed to support random access & editing  I — intraframe coding only  P — predictive coding  B — bi-directional coding

25. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 25Lecture 9 Typical MPEG coding parameters Typical sequence  IPBBPBBPBBPBBPBB (16 frames) Average compression 26.3

26. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 26Lecture 9 Video Coder Preprocessing  Color conversion, format translations (interlaced to picture), downsampling Motion estimation, compensation and coding  I pictures — code directly (DCT)  Buffer regulator adjusts quantizer for constant bit rate  Entropy code, then decode and IDCT for further use  P pictures — estimate motion, take difference, code difference  B pictures — estimate two motion vectors  Form average of two predictive pictures  Code difference between current picture and (a) past picture, (b) future picture or (c) average picture, whichever produces least MAE Reorder pictures for transmission  Suppose we have sequence I1, B2, B3, P4, B5, B6, P7.  Send I1, P4, B2, B3, P7, B5, B6.

27. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 27Lecture 9 Coded Video Bit Stream Layered representation 1 Sequence layer May include tables 2 Group of Pictures (GOP) layer 3 Picture layer 4 Slice layer 5 Macroblock layer 6 Block layer

28. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 28Lecture 9 Picture of Layers

29. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 29Lecture 9 MPEG-1 Performance

30. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 30Lecture 9 Coding constraints (minimum) Constrained parameter bit stream. Every MPEG-1 decoder should support these parameters

31. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 31Lecture 9 Macroblock Coding: I & P I pictures  Divided into slices and macroblocks  No motion compensation  Each macroblock can have different quantization  DC and AC coded differently, as in JPEG  Different coding tables from JPEG P pictures  Divided into slices and macroblocks  Option: no motion compensation  Option: can code block as inter or intra (like I picture)  Can skip macroblock (replace with previous). Great compression

32. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 32Lecture 9 Coding Image Blocks B pictures  Inter or intra?  Forward, backward, interpolational?  Code block or skip?  Quantization step?

33. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 33Lecture 9 MPEG-1 Wrap-up Data below for decoder, SIF pictures, 2 B pictures per P IDCT must be precise, because of inter-frame coding MPEG-1 does not deliver quality acceptable for broadcast —> MPEG-2

34. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 34Lecture 9 MPEG-1: ‘1.5’ Mbps Sample rate reduction in spatial and temporal domains Spatial  Block-based DCT  Huffman coding (no arithmetic coding) of motion vectors and quantized DCT coefficients  352 x 340 pixels, 12 bits per pixel, picture rate 30 pictures per second —> 30.4 Mbps  Coded bit stream 1.15 Mbps (must leave bandwidth for audio)  Compression 26:1  Quality better than VHS! Temporal  Block-based motion compensation  Interframe coding (two kinds)

35. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 35Lecture 9 Video Teleconferencing Comprehensive Standard: H.320 Components of H.320  H.261: Video coding, 64 to 1920 kbits/sec  G.722, G.726, G.728: Audio coding from 16 kbits/sec to 64 kbits/sec  H.221: Multiplexing of audio and video (frame based rather than packet based)  H.230 and H.242: Handshaking and control  H.233: encryption

36. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 36Lecture 9 Generic Video Telephone System

37. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 37Lecture 9 H.261 Features Common Interchange Format  Interoperability between 25 fps and 30 fps countries  252 pix/line, 288 line, 30 fps noninterlace  Terminal equipment converts frame and line numbers  Y Cb Cr components, color sub-sampled by a factor of 2 in both directions Coding  DCT, 8x8, 4 Y and 2 chrominance per masterblock  I and P frames only, P blocks can be skipped  Motion compensation optional, only integer compensation  (Optional) forward error correction coding

38. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 38Lecture 9 H.261 vs MPEG-1 Similarities  CIF, SIF, non-interlaced  DCT technology Differences  H.261 uses mostly P frames, no B frames  H.261 typical bit rates much lower (down to 64 kbits/sec)  Low bit rates achieved by reducing frame rate and picture count  Simpler motion compensations  End-to-end coding delay must be low Conclusion: Same technology, different design to meet different needs

39. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 39Lecture 9 MPEG 2 i, i = 0, 1 History & Goals Expanding universe of video coding What are MPEG-2 profiles? Features of MPEG-2

40. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 40Lecture 9 MPEG-2 Goals Compatibility with MPEG-1 Good picture quality Flexibility in input format Random access capability (I pictures) Capability for fast forward, fast reverse play, stop frame Bit stream scalability Low delay for 2-way communications (videoconferencing) Resilience to bit errors

41. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 41Lecture 9 MPEG-2 Implications No reason to restrict to CCIR 601  High resolution can be included (HDTV) No single standard can satisfy all requirements  Family of standards Most applications use a small set of the features  Toolkit approach

42. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 42Lecture 9 MPEG-2 profiles A profile is a subset of the entire MPEG-2 bit-stream syntax  Simple  Main  4:2:2  SNR  Spatial  High  Multiview Each profile has several levels (resolution quality)  Low — MPEG1  Main — CCIR 601  High-1440 (Video Editing)  High (HDTV)

43. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 43Lecture 9 Features of MPEG-2 Support of both non-interlaced and interlaced pictures Color handling  Y Cb Cr color space  Several subsampling schemes are used  4:2:0, 4:2:2, 4:4:4 MPEG-2 sequence can be either frames or fields  Both frame prediction and field prediction are supported  There can be motion between two fields in a frame, so that frame prediction is more tricky  In frame prediction, both fields constitute one picture  In field prediction, either field in the previous frame or the previous field in this frame can be used as reference  Robustified coding of motion vectors to protect against bit errors  Special prediction modes: 16x8, dual-prime