1. Improving Scene Cut Quality for Real-Time Video Decoding Giovanni Motta, Brandeis University James A. Storer, Brandeis University Bruno Carpentieri, Universita’ di Salerno

2. Outline  Introduction  H.263+ and TMN-8 Rate Control  Problem Description  Optimal Algorithm based on Dynamic Programming  Experimental Results  Conclusions and Future Research

3. Introduction  High variability in video sequences may cause the encoder to skip frames  Frame skipping occurs after a “scene cut” (i.e. when MC-prediction model fails)  If the encoder has some look-ahead capability it is possible to improve quality in proximity of scene cuts

4. H.263+ Video Encoding  State of the art Video Coding  MC-prediction and DCT coding  I and P macroblocks  Rate control

5. TMN-8 Rate Control  I/P Frame and MB decisions  Target bit rate for each frame  RD optimized bit allocation for MBs  Buffer control

6. Problem Description  Bits per frame (std100.qcif)

7. Problem Description  PSNR and Bits per frame across a scene cut

8. Problem Description  Frame n has several “I” macroblocks  Encoder is forced to skip n+1, n+2, n+3  Frame n-1 frozen on receiver’s display  Frame n+4 has a large prediction error  Encoder forced to skip frame n+5

9. Basic Idea  Avoid extra skipping and improve quality by selecting which frame should be encoded after a scene cut  Assumption: Encoder has look-ahead capability

10. Simplified approach TMN-8 behavior Last frame of the skipped sequence encoded

11. Simplified approach  PSNR and Bits per frame across a scene cut

12. Optimal Algorithm  Minimizes the number of skipped frames  Generalization of the text-paragraphing algorithm  Assumptions:  When the quality of F[i-j] is fixed to Q, the cost P[i, j] of predicting F[i] from F[i-j], is independent of how F[i-j] is encoded  P[i, j]  P[i, j+1]  P[i, 0], 1  j  d

13. Optimal Algorithm  Compute P[i, 0] for each frame  Compute P[i, j] for 1  j  d  Build (right to left) two matrices u R[i, j]: maximum residual capacity when F[i], …, F[n] are encoded so that the first frame that is not skipped is predicted by F[i-j] u S[i,j]: number of skipped frames corresponding to residual capacity R[i, j]  Time is O(d 2 n) = O(n) (constant d  7)

14. Test Sequences Std and Std100: concatenation of standard test sequences Commercials: Sampled TV commercials

15. Experimental Results  Gain in Bit/PSNR in proximity of scene cuts (simplified method)

16. Experimental Results  Gain on whole sequence (simplified method)

17. Conclusions  Simple yet effective method to improve quality in proximity of scene cuts  Experiments with simplified method show improvements of 14-30% (in Bit/PSNR)  Suitable for encoders of the MPEG family, provided that encoder has look-ahead capability  Decoding is unaffected

18. Future Research  Assess quality improvement when using optimal algorithm  Experiment with progressive transmission to eliminate frozen frame displayed by the decoder