2. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 1Lecture 15 ECEC-453 Image Processing Architecture 3/11/2004 Exam Review Oleh Tretiak Drexel University

3. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 2Lecture 15

4. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 3Lecture 15 Announcements Final on March 20 Cumulative Extra credit problem - write plugin for ImageJ (everybody does a different plugin)

5. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 4Lecture 15 Block Diagram — JPEG DCT For color images, each component is coded separately

6. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 5Lecture 15 Architecture for the DCT Separable DCT Options - Fast DCT ~ conventional computer - Vector DCT ~ parallel hardware 8x8 1-D DCT Unit operation: Multiply 8x8 matrix with 8x1 matrix ~ 64 ops

7. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 6Lecture 15 Computational Complexity 1D DCT - N input and output samples ~ N 2 = 64 operations (additions + multiplications) 2D DCT - direct implementation - M = N 2 input values, M output values -> M 2 = N 4 2D DCT - separable implementation, Y = TXT T = ZT T, where Z = TX, all matrices are NxN -> 2N 3 operations For N = 8 - 2D DCT direct — 4096 operations, 64 operations per pixel - 2D DCT separable — 1024 operations, 16 ops/pixel Big savings due to separable transform Inverse DFT — same story.

8. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 7Lecture 15 DCT: Encoding in JPEG, MPEG Take 8x8 blocks of pixels Subtract range mean value Compute 8x8 DCT Quantize the DCT coefficients - Typically, many of the samples are equal to zero Lossless entropy coding of the quantized samples Different quantization step is used for different DCT coefficients - y kl — DCT coefficients, q kl — quantizer steps - z kl — quantized values

9. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 8Lecture 15 Optimized (fast) DCT 1-D Chen DCT diagram. Dashed lines indicate subtraction, — multi- plication by a constant, — multiplication by 0.5 (shift). Characteristics of optimized DCT algorithms

10. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 9Lecture 15

11. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 10Lecture 15 Huffman Coding - Block Diagram

12. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 11Lecture 15 Coding AC Coefficients AC coefficients are coded in zig-zag (called ZZ in standard) order to maximize possible runs of zeros. Code unit consist of run length followed by coefficient size. Baseline coding of size category is the same as for DC differences (Table 2.9) Example: run of 6 zeros, size = -18. In the table, -18 is in category 5. Code is (6/5, 01101). If the Huffman code for 6/5 is 1101, codeword = 110101101

13. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 12Lecture 15 Example of JPEG compression Very high quality: compression = 2.33 Photoshop Image Very low quality: compression = 115 Produced by MATLAB with Quality = 0

14. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 13Lecture 15 Compression = 64 JPEG JPEG2000

15. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 14Lecture 15 Predictive Coding of Video E(x, y, t) = I(x, y, t) - P(x, y, t) - I ~ image, P ~ prediction, E ~ error P(x, y, t+1) = P(x, y, t) + Code(E(x, y, t)) At receiver, I e (x, y, t) = P(x, y, t+1) - I e (x, y, t) ~ estimate of image at time t Encoder Decoder

16. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 15Lecture 15 Generic Encoder - simplified

17. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 16Lecture 15 Motion Estimation Methods No compensation Full search logarithmic search 3 level hierarchical

18. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 17Lecture 15 Full-Search Method Compute for (2p+1) 2 values of (i, j). Each location requires 3MN operations Picture dimensions IxJ, F pictures per second - 3IJF(2p + 1) 2 operations per second - I = 720, J = 480, F = 30, p = 15 —> 30 GOPS Guaranteed to find best (MAE) displacement How to do it? - Special computers - Smaller p - Faster (suboptimal) algorithm

19. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 18Lecture 15 Hierarchical Search Prepare downsampled versions of current and reference images - Full macroblock 16x16 - Down 2 macroblock 8x8 - Down 4 macroblock 4x4 Full search in Down 4 reference image - 16 x speedup, smaller macroblock - 16 x speedup, fewer displacement vectors op = ±16, p’ = ±4 Around point of best match, do local search in Down 2 reference image (3x3 search zone) Repeat for Full reference image (3x3 search zone) Full Down 2 Down 4

20. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 19Lecture 15 Comparison

21. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 20Lecture 15 MPEG-1: ‘1.5’ Mbps Sample rate reduction in spatial and temporal domains Spatial - Block-based DCT - Huffman coding (no arithmetic coding) of motion vectors and quantized DCT coefficients o352 x 340 pixels, 12 bits per pixel, picture rate 30 pictures per second —> 30.4 Mbps oCoded bit stream 1.15 Mbps (must leave bandwidth for audio) oCompression 26:1 oQuality better than VHS! Temporal - Block-based motion compensation - Interframe coding (two kinds)

22. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 21Lecture 15 Picture Types MPEG-1 is designed to support random access & editing - I — intraframe coding only - P — predictive coding - B — bi-directional coding

23. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 22Lecture 15 Picture of Layers

24. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 23Lecture 15 Coding Image Blocks B pictures - Inter or intra? - Forward, backward, interpolational? - Code block or skip? - Quantization step?

25. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 24Lecture 15 MPEG-1 Wrap-up Data below for decoder, SIF pictures, 2 B pictures per P IDCT must be precise, because of inter-frame coding MPEG-1 does not deliver quality acceptable for broadcast —> MPEG-2

26. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 25Lecture 15 Typical MPEG coding parameters Typical sequence - IPBBPBBPBBPBBPBB (16 frames)

27. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 26Lecture 15 MPEG-2 Goals Compatibility with MPEG-1 Good picture quality Flexibility in input format Random access capability (I pictures) Capability for fast forward, fast reverse play, stop frame Bit stream scalability Low delay for 2-way communications (videoconferencing) Resilience to bit errors

28. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 27Lecture 15 MPEG-2 profiles A profile is a subset of the entire MPEG-2 bit-stream syntax - Simple - Main - 4:2:2 - SNR - Spatial - High - Multiview Each profile has several levels (resolution quality) - Low — MPEG1 - Main — CCIR 601 - High-1440 (Video Editing) - High (HDTV)

29. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 28Lecture 15 MPEG2 - Alternate Scan Zig-zag scanAlternate scan

30. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 29Lecture 15 MPEG2 — Subsampling Suppose picture is 720x480 - 4:4:4 oLuminance and chrominance @ 720x480 - 4:2:2 oLuminance @ 720x480, chrominance 360x480 - 4:2:0 oLuminance 420x480, chrominance 360x240 Weird terminology

31. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 30Lecture 15 Teleconferencing Standards Digital video areas - Broadcast television - Recorded programs - Two-way communications

32. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 31Lecture 15 Review: Video Telephone System H.320 H.200/AV.250 -Series H.221H.261

33. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 32Lecture 15 Review: H.261 Features Common Interchange Format - Interoperability between 25 fps and 30 fps countries - 252 pix/line, 288 line, 30 fps noninterlace - Terminal equipment converts frame and line numbers - Y Cb Cr components, color sub-sampled by a factor of 2 in both directions Coding - DCT, 8x8, 4 Y and 2 chrominance per masterblock - I and P frames only, P blocks can be skipped - Motion compensation optional, only integer compensation - (Optional) forward error correction coding

34. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 33Lecture 15 Picture Formats for H.263

35. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 34Lecture 15 Encoder: Where’s the meat? 63% 10%

36. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 35Lecture 15 Advanced Video Coding H.263 and MPEG-4 based on ~1995 technology After 1995, MPEG and VCEG (video coding) started working on a new low-rate standard (H.26L) Rec H.264 released in September 2002 Information on http://www.vcodex.com/ (some is on our web site)http://www.vcodex.com/ Site maintained by Ian Richardson, who has written books about video coding

37. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 36Lecture 15 New Features Prediction in I pictures Different block transform Different Block Sizes Changes in motion compensation VLC and arithmetic coding

38. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 37Lecture 15 I Picture Prediction System operates with 4x4 blocks and 16x16 macroblocks

39. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 38Lecture 15 9 Prediction Modes for 4x4 Blocks

40. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 39Lecture 15 4 Modes for 16x16 Macroblocks Mode 0: Vertical, extrapolate from upper samples Mode 1: Horizontal, extrapolate from left samples Mode 2: DC, mean of upper and left-hand samples Mode 3: Plane, linear fit to left and upper samples

41. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 40Lecture 15 Different Block Transform Basically, 4x4 DCT Scanning sequence for 16x16 macroblock is shown below 4x4 and 2x2 DC coefficients transformed (again)

42. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 41Lecture 15 4x4 DCT Tricks Y = AXA T a = 1/2, b = 0.707 cos(π/8), c = cos(3π/8) Trick: Y = (CXC T ).*E

43. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 42Lecture 15 Motion Compensation Ideas Adaptive motion compensation blocks: - 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, 4x4

44. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 43Lecture 15 Coding Ideas Constant quantizer value Zig-zag scan with novel run-length code Arithmetic coding an option Motion vectors to 1/4 pixel

45. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 44Lecture 15 Loop Filter Concept to overcome block artifacts Average across inter-block lines if difference is too big Difference threshold depends on coding mode (intra or inter) and quantization step size

46. Image Processing Architecture, © 2001, 2002 Oleh TretiakPage 45Lecture 15 Example of Loop Filter