1. 6/9/20161 Video Compression Techniques Image, Video and Audio Compression standards have been specified and released by two main groups since 1985: International Telecommunications Union: H.261 — 264. International Standards Organization: JPEG, MEPG H. 261 ( Graphics, Video conferencing over ISDN) H.263 (Video conferencing and videotelephony over POST ) MPEG (Motion Picture Experts Group) Standards MPEG-1 ( Movie on CD-ROM ) MPEG-2 (Digital TV, Movie on DVD) MPEG-4 ( Internet and Mobile video )

2. 6/9/20162 Technique for Video Coding Simple Technique ( Spatial Redundancy ) - code individual frames as still images - use existing image compression method (such as JPEG)  Temporal encoding : Take advantage of frame coherence (Temporal redundancy )  usually minimal changes between successive frames  Only code the differences, the rest of the frame can be derived the previous

3. 6/9/20163 Spatial Compression  A video is a sequence of images – and images can be ompressed.  JPEG uses lossy compression – typical compression ratios are around 10:1 and, with deteriorating, quality up toward 20:1  We could just compress images and send these.  Time does not enter into the process.  This is called intra-coding (intra = within) –Generically called Motion JPEG (M-JPEG) –Typically used by digital cameras

4. 6/9/20164 Spatial Compression  Very similar to JPEG  Image divided into 8 by 8 pixel sub-blocks  for 352 frame width and 288 frame height  Number of blocks = 352/8 by 288/8 = 44 by 36 blocks  Each block DCT coded  Quantisation - dropping low- amplitude coefficients  Huffman coded  This produces a complete frame called an Intra frame (I)

5. 6/9/20165 Temporal Compression  Spatial compression does not take into account similarities between adjacent frames  Talking Heads - Backgrounds don’t change  Consecutive images (1/25th second apart) are very similar  Just send the difference between adjacent frames

6. 6/9/20166 Difference Coding  Only send difference between this frame and previous frame  Result is very sparse – high compression now possible using block-based DCT as before Frame n+1 Frame n

7. 6/9/20167 Frame nFrame n+1 - Residual

8. 6/9/2016 8 Difference Coding  Using the previous frame and the encoded difference frame we can recreate the original – this is called a predicted frame (P frame).  This recreated frame can then be used to form the next frame and the process is repeated.  Usually a sequence of P frames is created until the next I frame (maybe a scene change!) I P..... P P P I

9. 6/9/20169 Motion Compensation  Difference coding is good, but often an object will simply change position between frames.  DCT coding not as good as for ‘sparse’ difference image.

10. 6/9/201610 Motion Compensation  Motion compensation is a simple and efficient means of accounting for the motion of objects  Assumes the frame is split up into macroblocks of typically 16x16 pixels  Instead of transmitting the pixel colours for a macroblock in the current frame, we search for the position it in the previous frame (reference frame) and simply transmit the motion vector –Relies on simplistic assumptions –The search is computationally expensive Current frame Reference frame (dx,dy)

11. 6/9/201611 Motion Compensation  Block Matching--how to find the matching block? –Matching criteria:  In practice we couldn’t expect to find the exactly identical matching block, instead we look for close match.  Most motion estimation schemes look for minimum mean square error(MMSE) between block in the current frame and the block in the previous frame at some displacement (dx,dy). –Matching block size:  The block size will affect coding efficiency and computational complexity – For MPEG, a block size of 16×16 is used

12. 6/9/201612 MPEG-1 adopts the digital TV format also known as SIF (Source Input Format). 1 to 1.5 Mbps data rate 200 to 250 Kbps for audio : 1.15 to 1.2 Mbps for video suitable for traditional CD-ROM throughput rate MPEG-1 supports only non-interlaced video. Normally, its picture resolution is: – 352 × 240 for NTSC video at 30 fps – 352 × 288 for PAL video at 25 fps – It uses 4:2:0 Chroma subsampling MPEG-1

13. 6/9/201613 There are three different compression modes for the frames of an image sequence in video :  Intraframe compression: frames marked by I (I-frames) are strictly intraframe compressed. These frame serve as random access points to the sequence and can be decoded without any reference to other frames.  Interframe compression: frames marked by P (P-frames) are encoded relative to the closest I- frame or P-frame in the past. Motion vectors and prediction errors are coded. If motion estimation fails, blocks in P-frames can be encoded in I-frame mode.( contains only one motion vector backward motion vector ) frames marked by B (B-frames) are encoded relative to two frames, one in the past, one in the future, ( Contains two motion vectors forward and backward ) Modalities of Video Coding

14. 6/9/2016 MPEG Frame Types I- Frame (Intra – coded Images ) Coded as still image (no other image required for decoding ) giving a reference point for random access Macroblocks Y (16x16), C(8x8) 1 Macroblock = 4(8x8) Y block+1(8x8)Cb+1(8x8)Cr use JPEG for compression  D- Frame (DC- coded frames)  Save low frequency only  Used for fast forward or fast rewind (VCR control simulation)  Stored in addition to the regular data stream Y CrCr CbCb 8 8 8 8

15. 6/9/201615 Intra-Frame Coding (I-Frame ) Macroblocks are 16x16 pixel areas on Y plane of original image A macroblock usually consists of 4 Y blocks, 1 Cr block, and 1 Cb block Quantization is by constant value for all DCT coefficients (i.e., no quantization table as in JPEG).

16. 6/9/201616 P-Frame (predictive encode from previous I or P-frame) Blocks are encoded either predictively,or as I-frame blocks Encode motion vector  P-Frame coding  Previous image is called reference image  Image to code is called target image.  Actually, the difference is encoded. Inter – Frame Coding

17. 6/9/201617

18. 6/9/201618 How the macroblock is coded Many macroblocks will be exact matches (or close enough). So send address of each block in image --> Addr Sometimes no good match can be found, so send INTRA block --> Type Sometime, want to vary the quantization to fine tune compression, so send quantization value --> Quant Motion vector  vector Some blocks in macroblock will match well, others match poorly. So send bitmask indicating which blocks are present (Coded Block Pattern, or CBP). Send the blocks (4 Y, 1 Cr, 1 Cb) as in JPEG addrtypequantvectorCBPb0b1…b5

19. 6/9/201619 B- Frames ( Bi- directional predictive encoded) Sometimes it is necessary to use the both the preceding frame and following frame as reference images X Z Y Available from earlier frame (X) Available from later frame (Z)

20. 6/9/201620 Differences based on prediction of previous and next frame 2 motion vectors encoded with respect to previous and next I or P frames Real time and error control consideration, limit the number of B frames used in a sequence Preceding I or P ref. search area Target frame search area Next I or P ref. frame search area Best Match Macroblock (MP). Motion VP Macroblock being encoded (MT) Best Match Macroblock (MS). Motion VS B- Frames ( Bi- directional predictive encoded)

21. 6/9/201621 The Motion – Compensation -based B-frame coding idea is shown in fig below

22. 6/9/201622 B- Frames ( Bi- directional predictive encoded)

23. 6/9/201623 - Each MB from a B-frame will have up to two motion vectors (MVs) (one from the forward and one from the backward prediction). - If matching in both directions is successful, then two MVs will be sent and the two corresponding matching MBs are averaged (indicated by `%' in the figure) before comparing to the Target MB for generating the prediction error. - If an acceptable match can be found in only one of the reference frames, then only one MV and its corresponding MB will be used from either the forward or backward prediction. B- Frames ( Bi- directional predictive encoded)

24. 6/9/201624 Note : No defined limit to the number of consecutive B frames that may be used in a group of pictures, Optimal number is an application dependent. Most broadcast quality applications however, have tended to use 2 consecutive B frames (I,B,B,P,B,B,P,) as the ideal trade-off between compression efficiency and video quality. Advantage of the usage of B frames Coding efficiency. Most B frames use less bits. Quality can also be improved in the case of moving objects that reveal hidden areas within a video sequence. Better Error propagation: B frames are not used to predict future frames, errors generated will not be propagated further within the sequence. Disadvantage: Frame reconstruction memory buffers within the encoder and decoder must be doubled in size to accommodate the 2 anchor frames.

25. 6/9/201625 MPEG frame sequences I I I I …….. Best reproduction, poor compression Group of Pictures GOP : The number of frames /Pictures between successive I- frames is known as GOP. Its given the number N and typical value of N are from 3 through 12. Prediction Span: the number of frames between P- frame and the immediately preceding I or P-Frame. Its given the symbol M and typical value range from 1 to 3

26. 6/9/201626 For the following frame sequence I P P P I P P P M=1 Frame while N(GOP)=4 frames

27. 6/9/201627 M=3 and N (GOP)= 9 for the following frames sequence I BBP BB P BB I

28. 6/9/201628 Coding,Decoding and Display Order

29. 6/9/201629 MPEG-1 Bandwidth Requirements Examples: A digitized video is to be compressed using MPEG-1 standard assuming frame sequence of : I B B P B B P B B P B B I…And average compression ratio of 10:1 (I), 20: 1 (P), 50:1 (B). Derive the average bit rate that is generated by the encoder for both NTSC and PAL digitization format Note: The Frame size 352 x 240 NTSC; 352 x 288 PAL and each pixel represented by 8 bits Solution: Frame sequence= I B B P B B P B B P B B I ……….. Hence : 1/12 of frames are I –frames, 3/12 are P- frames, and 8/12 are B- frames Average compression ratio=(1x0.1+3x0.05+ 8x0.02)/12=0.0342 or 29.24:1 NTSC frame size : Without Compression =(352x240x8)+2(176x120x8)=1.01376Mbits/frame with compression=1.01376Mbits/frame x 1/29.24=34.67 kbits/frame Hence bit rate generated at 30 fps =1.040 Mbps

30. 6/9/201630 PAL frame size Without compression =352 x 288 x 8+2 (176 x 144 x 8)=1.216512 Mbits/frame with compression =1.216512 x 1/29.24 =41.604 Kbits / frame Hence bit-rate generated at 25 fps=1.040 Mbps