1. Video Watermarking Real-time Labeling of MPEG-2 Compressed Video G. C. Langelaar, R. L. Lagendijk, and J. Biemond ITS, ICTG, Delft University of Technology Multimedia Security

2. 2 Outline Introduction to MPEG video compression standards Watermarking MPEG compressed video sequence –Labeling in the bit domain –Labeling in the coefficient domain Conclusion

3. Introduction to MPEG Video Compression Standard

4. 4 MPEG Video Encoding Zig-ZagQuantDCT DPCM RLE Huffman or Arithmetic Coding 01001... I-Frame R G B Y IQ To Other Color Space (Optional) Scan IQuant IDCT Reconstruct & Update Y IQ Forward Frame Buffer Y IQ Backward Frame Buffer Motion Estatimation P B-Frame Different Image Motion Vector

5. 5 Discrete Cosine Transform DCT –Convert the time domain signal into frequency domain 88 84 83 84 85 86 83 82 86 82 82 83 82 83 83 81 82 82 84 87 87 87 81 84 81 86 87 89 82 82 84 87 81 84 83 87 85 89 80 81 81 85 85 86 81 89 81 85 82 81 86 83 86 89 81 84 88 88 90 84 85 88 88 81 675 1 -6 2 -2 0 5 -5 -4 1 2 1 5 1 -3 0 2 3 4 6 -2 2 1 5 -3 -1 0 2 0 -2 2 -4 4 3 1 -1 -2 1 -3 1 1 -2 0 -3 2 -1 1 1 3 0 -1 0 -1 -1 0 -2 -1 -1 -5 5 2 -2 2 0 DCT

6. 6 Quantization –lossy data compression technique –e.g. 4/2 = 2, 5/2 = 2 675 1 -6 2 -2 0 5 -5 -4 1 2 1 5 1 -3 0 2 3 4 6 -2 2 1 5 -3 -1 0 2 0 -2 2 -4 4 3 1 -1 -2 1 -3 1 1 -2 0 -3 2 -1 1 1 3 0 -1 0 -1 -1 0 -2 -1 -1 -5 5 2 -2 2 0 168 0 -1 0 0 0 1 -1 -1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 -1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -1 1 0 0 0 0 Quant Xq(i,j)=X(i,j)/Q(i,j)

7. 7 ZigZag Scan –make all zeros to be concatenated –get compression gain 168 0 -1 0 0 0 1 -1 -1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 -1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -1 1 0 0 0 0 Zigzag 168,0,-1,0,0,0,0,0,0,0,0,1, 0,1,0,0,0,1,1,...,1,0,0,0,-1, 0,0,0,0,0,0,0,0,0,0

8. 8 Run Length Encode –Encode continuous zeros and a nonzero into a symbol RLE (1,-1) (8,1) (1,1) (3,1) (0,1) (9,1) (3,-1) (14,1) (4,-1) (0,1) (3,-1) EOB 168,0,-1,0,0,0,0,0,0,0,0,1, 0,1,0,0,0,1,1,...,1,0,0,0,-1, 0,0,0,0,0,0,0,0,0,0 RLE(x0,x1,x2,…) = (zeros,nozero)...

9. 9 DPCM Difference Pulse Coded Modulation –using the previous DC to predict the next DC value DC=138DC=140 DC=136DC=138 DPCM 138 => 2 => -4 => 2 X DPCM = X i - X i-1

10. 10 Variable Length Codes Variable Length Coding –c–code representation length is decided by its appearing possibility –s–short codeword to represent the high frequency event ABAC Fix length code A : 00 B : 01 C : 10 00010010 Variable length code A : 1 B : 01 C : 00 101100 8 bits 6 bits

11. 11 Motion Estimation 88 84 83 84 85 86 83 82 86 82 82 83 82 83 83 81 82 82 84 87 87 87 81 84 81 86 87 89 82 82 84 87 81 84 83 87 85 89 80 81 81 85 85 86 81 89 81 85 82 81 86 83 86 89 81 84 88 88 90 84 85 88 88 81 84 82 83 81 85 86 83 81 82 82 81 83 82 83 83 81 83 82 84 87 87 87 81 88 81 85 86 88 82 82 84 87 81 84 85 87 85 89 84 81 82 85 81 84 81 89 81 83 81 87 86 83 86 89 81 84 88 82 87 84 87 89 84 81 Frame N Frame N+1 -4 -2 0 -3 0 0 0 -1 -4 0 -1 0 0 0 0 0 1 0 0 0 0 0 0 4 0 -1 -1 -1 0 0 0 0 0 0 2 0 0 0 4 0 1 0 -4 -2 0 0 0 -2 -1 6 0 0 0 0 0 0 0 -6 -3 0 2 1 -4 0 - = Motion Vector

12. 12 Motion Compensation –using motion vector between current frame and reference frame to reconstruct the prediction of current frame Reference frame Current frame Reconstruct frame

13. 13 P,B-Frames The P-picture uses MC to de-correlate dependence between continuous frames The B-picture is introduced for increasing the frame rate without increasing too much bitrate I or P B Forward prediction Backward prediction

14. 14 GOP Layer I,P,B three type of picture to consist a GOP(group of picture) Temporal : 1 2 3 4 5 6 7 8 9 10 11 12 13 Picture type : I B B P B B P B B P B B P Coding seq : 1 3 4 2 6 7 5 9 10 8 12 13 11

15. 15 I,P,B Frame Difference between picture types

16. Watermarking MPEG Compressed Video Sequence

17. 17 Motivation Digital video recording devices is entering market. –The development and success of such devices depend on technological advances and the existence of adequate copy protection methods

18. 18 Simple One-Copy Only Scenario Label Presented? Yes Stop Recording No Add Label Store Labeled or Un-labeled Data Labeled Data Detection/Labeling Module in Video Recorder

19. 19 Requirements of Video Labeling Fidelity –The quality of video should not be significantly affected Robustness –Against re-encoding at a different bit-rate Blind-detection Directly embedding into the video data –For different platforms, interfaces, and data file formats Complexity –Real-time embedding/detection –Cost of labeling modules in consumer players Bit-stream length increasing is forbidden –To avoid hardware buffer overflow

20. 20 Labeling on Compressed Data Decoding, labeling, and re-encoding is computation- consuming A labeling scheme for MPEG video should avoid –Requiring a full-decoding operation –Drastically decreasing the quality of video –Increasing the size of labeled video data We need techniques which can be applied directly on the MPEG compressed data!!

21. 21 Labeling MPEG Video Streams A real-time labeling algorithm for compressed video should closely follow the compression standard to avoid computationally demanding operations –DCT/IDCT –Motion estimation The algorithm should work on the lowest layer –Quantized DCT-block

22. 22 DCT-block Representing Domains

23. Labeling in the Bit Domain

24. 24 Bit Domain Labeling To label bit-stream L, consisting of bits L i (i=1,2,…,l) in the MPEG stream –Suitable VLCs are selected –The LSB of their quantized level are forced to the value of L i

25. 25 Selecting Suitable VLCs Label-bit carrying VLC (lc-VLC) –Codewords which another codeword exists with the same run length a level difference of 1 the same codeword length Such a change of VLC will yield perceptually invisible degradations after decoding

26. 26 Example of lc-VLC

27. 27 Embedding in the Bit Domain Test codewords in a macroblock An lc-VLC codeword is found If the LSB of its level equal to the label bit L i ? The codeword is unchanged Yes No Replace the lc-VLC codeword with another one The procedure is repeated until all label bits are embedded

28. 28 Extraction in the Bit Domain The procedure is repeated until for i=1,…,l Test codewords in a macroblock An lc-VLC codeword is found Assign the value represented by its LSB to L i

29. 29 The Label Bit-rate The maximum number of label bits that can be added to the video stream per second –Determined by the number of lc-VLC codewords –Not known in advance –Experimental evaluation is required first

30. 30 Evaluating the Label Bit-rate A MPEG-2 video sequence is tested –10 seconds –720x576 pixels –25 frames per second –GOP length is 12 –Containing I, P, B frames

31. 31 Evaluating the Label Bit-rate (cont.) Intra coded macro-blocks only Intra and inter coded macro-blocks only

32. 32 Video Quality Degradation Informal subjective tests –The labeling does not introduce any visible artifacts in 4, 6, and 8 Mb/s coded streams –As for coding at a bit-rate less than 2 Mb/s, the unlabeled bit-stream already contain too many coding artifacts

33. 33 Video Quality Degradation (cont.) Numerically measurements –Difference images are amplified by 60 Unlabeled I-frame Frame difference (4 Mb/s) Frame difference (8 Mb/s) Most differences are located around the edges and in the textured areas

34. 34 Distributions of the lc-VLC Codewords According to the experiments, it appear that the lc-VLC are fairly uniformly distributed over the DCT spectrum – Each non-zero DCT-coefficient represented by a VLC have an equal probability of being modified Why this conclusion? Smooth areas has fewer nonzero DCT coefficients, and thus fewer VLC codewords

35. 35 Distributions of the lc-VLC Codewords (cont.) The maximum local degradation should be as low as possible –The number of lc-VLC codewords per block should be limited 87% of all blocks 64% of all lc-VLC 11% of all blocks

36. 36 Distributions of the lc-VLC Codewords (cont.) Using a threshold mechanism to limit the number of lc-VLC replacements per DCT block to T m The effect of limiting the number of lc-VLC replacements per block

37. 37 Drifts due to Prediction ∆MSE=MSE l -MSE u –MSE l : the MSE per frame between the uncompressed sequence and the labeled sequence coded at 8 Mb/s –MSE u : the MSE per frame between the uncompressed sequence and the sequence coded at 8 Mb/s

38. 38 Robustness of Bit Domain Embedding Bit domain is efficient and simple, but it can also be removed without significantly affecting the quality of video –Decoding and re-encoding using another bit- rate –Labeling the video stream again using another random label bit-stream Suitable for consumer applications

39. Labeling in the Coefficient Domain

40. 40 Coefficient Domain Labeling Coefficient domain labeling embeds the label bit-stream L consisting of bits L i, i=1,…,l, in the I-frames only. Each bit out of the label string has its own label-bit carrying region, lc-region, in a I- frame.

41. 41 Example of lc-region The first label bit is located in the top-left corner of a I frame

42. 42 Embedding in the Coefficient Domain A label bit is embedded in a lc-region by introducing an energy difference between the high-frequency DCT-coefficients of the top-half of the lc-region (denoted by lc-subregion A) and the bottom-half (denoted by B) Definition of the total energy in A The energy difference D S(c) is defined by the cut-off point c

43. 43 Embedding in the Coefficient Domain (cont.) Label bit ”0” is defined as D>0, and label bit “1” as D<0

44. 44 Embedding in the Coefficient Domain (cont.) If a bit “0” must be embedded, all energy after the cut-off points in the DCT-blocks of lc-subregion B is eliminated by setting the corresponding DCT- coefficeints to zero. Thus, If label bit”1” must be embedded, all energy after the cut-off-point in the DCT-blocks of lc-subregion A is elimated, so that:

45. 45 Why Using this Energy Difference? The coefficients can be forced to zero by shifting the EOB marker towards the DC-coefficient

46. 46 The Side Effects of Coefficient Elimination Since coefficients are removed to add a label, the labeled compressed video stream will always be smaller than the unlabeled video stream. If it is necessary to keep the original size of compressed video sequence, stuffing bits can be inserted.

47. 47 Factors Determining E A and E B The spatial contents of the sub-regions A and B The cut-off point C

48. 48 Determining the Cut-off Points Labeling Extraction D and n together determines the visibility and robustness 0<T<D

49. 49 Labeling Procedure

50. 50 Label Extracting Procedure

51. 51 Discarded Coefficients D=20, T=15 Not enough high frequency coefficients exist in the 1.4 Mb/s compressed stream, since only 75% of the extracted label bits are correct.

52. 52 Video Quality Degradation Unlabeled I-frameFrame difference (4 Mb/s)Frame difference (8 Mb/s) Numerically measurements –Difference images are amplified by 60 –Fewer difference per frame, but larger difference per block

53. 53 Histogram of the cut-off Points The lower the bit-rate is, the lower the cut-off points have to be because of the lack of high frequency components in the compressed video stream

54. 54 Discarded VLC Codewords Per Block 95% of all blocks Coded at 8 Mb/s 10% of blocks have energy above the cut-off point

55. 55 Limiting Discarded VLC Codewords Some coefficients will not be eliminated due to the limit Error-correction codes can be used

56. 56 MSE of different Frame Types The sequence is coded at 8 Mb/s The degradation now fades out for P and B frames

57. 57 Robustness of Coefficient Domain Labeling Possible attacks –Re-encoding –Trans-coding Trans-coding a 8Mb/s sequence at a lower bit-rate

58. 58 Conclusions Comparison of the two labeling schemes –The bit domain labeling method is computationally highly efficient and a high label bit-rate, but not robust against re- labeling or trans-coding –The coefficient domain labeling is more complex and has a lower bit-rate, but has higher robustness

59. 59 Conclusions (cont.) Both schemes are suitable for a real-time copy protection system for digital video recording devices

60. 60 The European ACTS SMASH Project The problem which labeling method should be used for a digital video recording device is currently under investigation within the European ACTS SMASH project

61. 61 References for video watermarking 1. Gerrit C. Langelaar and Reginald L. Lagendijk, “Optimal Differential Energy Watermarking of DCT Encoded Images and Video,” IEEE Trans. On Image Processing, vol.10, No.1, Jan. 2001, pp.148-158 2. Real-Time Labeling of MPEG-2 Compressed Video – http://ict.ewi.tudelft.nl/pub/inald/realtime.pdf