1. EE591f Digital Video Processing 1 Roadmap Introduction Intra-frame coding –Review of JPEG Inter-frame coding –Conditional Replenishment (CR) –Motion Compensated Prediction (MCP) Object-based and scalable video coding* –Motion segmentation, scalability issues

2. EE591f Digital Video Processing 2 Introduction to Video Coding Lossless vs. lossy data compression –Source entropy H(X) –Rate-Distortion function R(D) or D(R) Probabilistic modeling is at the heart of data compression –What is P(X) for video source X? –Modeling moving pictures is more difficult than modeling still images due to temporal dependency

3. EE591f Digital Video Processing 3 Shannon’s Picture Rate (bps) Distortion Coder A Coder B Which coder wins, A or B?

4. EE591f Digital Video Processing 4 Distortion Measures Objective –Mean Square Error (MSE) –Peak Signal-to-Noise-Ratio (PSNR) –Measure the fidelity to original video Subjective –Human Vision System (HVS) based –Emphasize visual quality rather than fidelity We only discuss objective measures in this course

5. EE591f Digital Video Processing 5 Roadmap Introduction Intra-frame coding –Review of JPEG Inter-frame coding –Conditional Replenishment (CR) –Motion Compensated Prediction (MCP) Scalable video coding –3D subband/wavelet coding and recent trend

6. EE591f Digital Video Processing 6 A Tour of JPEG Coding Standard Key Components  Transform  Quantization  Coding -8×8 DCT -boundary padding -uniform quantization -DC/AC coefficients -Zigzag scan -run length/Huffman coding

7. EE591f Digital Video Processing 7 JPEG Baseline Coder Tour Example

8. EE591f Digital Video Processing 8 Step 1: Transform DC level shifting 2D DCT -128 DCT

9. EE591f Digital Video Processing 9 Step 2: Quantization Q-table Q Why increase from top-left to bottom-right?

10. EE591f Digital Video Processing 10 Step 3: Entropy Coding Zigzag Scan (20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,0,0,0, 0,1,1,0,1,EOB) Zigzag Scan End Of the Block: All following coefficients are zero

11. EE591f Digital Video Processing 11 Roadmap Introduction Intra-frame coding –Review of JPEG Inter-frame coding –Conditional Replenishment (CR) –Motion Compensated Prediction (MCP) Scalable video coding –3D subband/wavelet coding and recent trend

12. EE591f Digital Video Processing 12 Conditional Replenishment Based on motion detection rather than motion estimation Partition the current frame into “still areas” and “moving areas” –Replenishment is applied to moving regions only –Repetition is applied to still regions Need to transmit the location of moving areas as well as new (replenishment) information –No motion vectors transmitted

13. EE591f Digital Video Processing 13 Conditional Replenishment

14. EE591f Digital Video Processing 14 Motion Detection

15. EE591f Digital Video Processing 15 From Replenishment to Prediction Replenishment can be viewed as a degenerated case of prediction –Only zero motion vector is considered –Discard the history A more powerful approach of exploiting temporal dependency is prediction –Locate the best match from the previous frame –Use the history to predict the current

16. EE591f Digital Video Processing 16 Differential Pulse Coded Modulation _ + D xnxn xnxn x n-1 ynyn + ynyn xnxn D Encoder Decoder Q ^ ynyn ^ ^ ^ ^ x n-1 ^ ^ X n,y n : unquantized samples and prediction residues X n,y n : decoded samples and quantized prediction residues ^ ^

17. EE591f Digital Video Processing 17 Motion-Compensated Predictive Coding

18. EE591f Digital Video Processing 18 A Closer Look

19. EE591f Digital Video Processing 19 Key Components Motion Estimation/Compensation –At the heart of MCP-based coding Coding of Motion Vectors (overhead) –Lossless: errors in MV are catastrophic Coding of MCP residues –Lossy: distortion is controlled by the quantization step-size Rate-Distortion optimization

20. EE591f Digital Video Processing 20 Block-based Motion Model Block size –Fixed vs. variable Motion accuracy –Integer-pel vs. fractional-pel Number of hypothesis –Overlapped Block Motion Compensation (OBMC) –Multi-frame prediction

21. EE591f Digital Video Processing 21 Quadtree Representation of Motion Field with Variable Blocksize

22. EE591f Digital Video Processing 22 Rate-Distortion Optimized BMA Distortion alone Rate and Distortion counted bits using a VLC table

23. EE591f Digital Video Processing 23 Experimental Results Cited from G. Sullivan and L. Baker, “Rate-Distortion optimized motion compensation for video compression using fixed or variable size blocks”, Globecom’1991

24. EE591f Digital Video Processing 24 Fractional-pel BMA Recall the tradeoff between spending bits on motion and spending bits on MCP residues Intuitively speaking, going from integer-pel to fractional-pel is good for it dramatically reduces the variance of MCP residues for some video sequence. The gain quickly saturates as motion accuracy refines

25. EE591f Digital Video Processing 25 8-by-8 block, half-pel, var(e)=123.88-by-8 block, integer-pel, var(e)=220.8 Example MCP residue comparison for the first two frames of Mobile sequence

26. EE591f Digital Video Processing 26 Fractional-pel MCP

27. EE591f Digital Video Processing 27 Multi-Hypothesis MCP Using one block from one reference frame represents a single-hypothesis MCP It is possible to formulate multiple hypothesis by considering –Overlapped blocks –More than one reference frame Why multi-hypothesis? –The benefit of reducing variance of MCP residues outweighs the increased overhead on motion

28. EE591f Digital Video Processing 28 Example: B-frame f n-1 fnfn f n+1

29. EE591f Digital Video Processing 29 Generalized B-frame f n-1 fnfn f n+1 f n+2 f n-2

30. EE591f Digital Video Processing 30 Multi-Hypothesis MCP

31. EE591f Digital Video Processing 31 Key Components Motion Estimation –At the heart of MCP-based coding Coding of Motion Vectors (overhead) –Lossless: errors in MV are catastrophic Coding of MCP residues –Lossy: distortion is controlled by the quantization step-size Rate-Distortion optimization

32. EE591f Digital Video Processing 32 Motion Vector Coding 2D lossless DPCM –Spatially (temporally) adjacent motion vectors are correlated –Use causal neighbors to predict the current one –Code Motion Vector Difference (MVD) instead of MVs Entropy coding techniques –Variable length codes (VLC) –Arithmetic coding

33. EE591f Digital Video Processing 33 MVD Example MV MV 1 MV 2 MV 3 Due to smoothness of MV field, MVD usually has a smaller variance than MV

34. EE591f Digital Video Processing 34 VLC Example MV x /MV y symbolcodeword 0 -2 1 2 3 1 2 3 4 5 6 1 010 011 00100 00101 00110 Exponential Golomb Codes: 0…01x…x m m

35. EE591f Digital Video Processing 35 Key Components Motion Estimation –At the heart of MCP-based coding Coding of Motion Vectors (overhead) –Lossless: errors in MV are catastrophic Coding of MCP residues –Lossy: distortion is controlled by the quantization step-size Rate-Distortion optimization

36. EE591f Digital Video Processing 36 MCP Residue Coding TransformQuantizationCoding Conceptually similar to JPEG Transform: unitary transform Quantization: Deadzone quantization Coding: Run-length coding

37. EE591f Digital Video Processing 37 Transform Unitary matrix: A is real, A -1 =A T Unitary transform: A is unitary, Y=AXA T Examples –8-by-8 DCT –4-by-4 integer transform

38. EE591f Digital Video Processing 38 Deadzone Quantization 22  0 deadzone codewords

39. EE591f Digital Video Processing 39 Key Components Motion Estimation –At the heart of MCP-based coding Coding of Motion Vectors (overhead) –Lossless: errors in MV are catastrophic Coding of MCP residues –Lossy: distortion is controlled by the quantization step-size Rate-Distortion optimization

40. EE591f Digital Video Processing 40 Lagrangian Multiplier Method Motion estimation Mode selection QUANT: a user-specified parameter controlling quantization stepsize

41. EE591f Digital Video Processing 41 Summary How does MCP coding work? –The predictive model captures the slow-varying trend of the samples {f n } –The modeling of prediction residues {e n } is easier than that of original samples {f n } Fundamental weakness –Quantization error will propagate unless the memory of predictor is refreshed –Not suitable for scalable coding applications

42. EE591f Digital Video Processing 42 Roadmap Introduction Intra-frame coding –Review of JPEG Inter-frame coding –Conditional Replenishment (CR) –Motion Compensated Prediction (MCP) Scalable video coding –3D subband/wavelet coding and recent trend

43. EE591f Digital Video Processing 43 Scalable vs. Multicast What is scalable coding? Multicast Scalable coding foreman.yuv foreman128k.cod foreman256k.cod foreman512k.cod foreman1024k.cod foreman.yuv foreman.cod 1024512256128

44. EE591f Digital Video Processing 44 Spatial scalability 10111…0101000…110100

45. EE591f Digital Video Processing 45 Temporal scalability 10111…0101000…110100 Frame 0,1,2,3,4,5,… Frame 0,2,4,6,8,…Frame 0,4,8,12,… 30Hz15Hz 7.5Hz

46. EE591f Digital Video Processing 46 SNR (Rate) scalability 10111…0101000…110100 PSNR avg =30dB PSNR avg =35dB PSNR avg =40dB PSNR i : PSNR of frame i

47. EE591f Digital Video Processing 47 Scalability via Bit-Plane Coding A=  (a 0 +a 1 2+a 2 2 2 + … … +a 7 2 7 ) Least Significant Bit (LSB) Most Significant Bit (MSB) Example A=129  sign=+,a 0 a 1 a 2 …a 7 =10000001 sign=-, a 0 a 1 a 2 …a 7 =00110011  A=-(4+8+64+128)=-204 sign bit

48. EE591f Digital Video Processing 48 Why DPCM Bad for Scalability? Base layer Enhancement Layer 1 Enhancement Layer 2 I base PPP I enh1 I enh2 12 3 … Frame number P P P P P P suffer from drifting problem suffer from coding efficiency loss

49. EE591f Digital Video Processing 49 3D Wavelet/Subband Coding t x y 2D spatial WT+1D temporal WT

50. EE591f Digital Video Processing 50 Motion-Adaptive 3D Wavelet Transform Recall Haar transform Motion-adaptive Haar transform W,W -1 : forward and backward motion vector lifting-based implementation