Improving Scene Cut Quality for Real-Time Video Decoding Giovanni Motta, Brandeis University James A. Storer, Brandeis University Bruno Carpentieri, Universita’ di Salerno

Outline  Introduction  H.263+ and TMN-8 Rate Control  Problem Description  Optimal Algorithm based on Dynamic Programming  Experimental Results  Conclusions and Future Research

Introduction  High variability in video sequences may cause the encoder to skip frames  Frame skipping occurs after a “scene cut” (i.e. when MC-prediction model fails)  If the encoder has some look-ahead capability it is possible to improve quality in proximity of scene cuts

H.263+ Video Encoding  State of the art Video Coding  MC-prediction and DCT coding  I and P macroblocks  Rate control

TMN-8 Rate Control  I/P Frame and MB decisions  Target bit rate for each frame  RD optimized bit allocation for MBs  Buffer control

Problem Description  Bits per frame (std100.qcif)

Problem Description  PSNR and Bits per frame across a scene cut

Problem Description  Frame n has several “I” macroblocks  Encoder is forced to skip n+1, n+2, n+3  Frame n-1 frozen on receiver’s display  Frame n+4 has a large prediction error  Encoder forced to skip frame n+5

Basic Idea  Avoid extra skipping and improve quality by selecting which frame should be encoded after a scene cut  Assumption: Encoder has look-ahead capability

Simplified approach TMN-8 behavior Last frame of the skipped sequence encoded

Simplified approach  PSNR and Bits per frame across a scene cut

Optimal Algorithm  Minimizes the number of skipped frames  Generalization of the text-paragraphing algorithm  Assumptions:  When the quality of F[i-j] is fixed to Q, the cost P[i, j] of predicting F[i] from F[i-j], is independent of how F[i-j] is encoded  P[i, j]  P[i, j+1]  P[i, 0], 1  j  d

Optimal Algorithm  Compute P[i, 0] for each frame  Compute P[i, j] for 1  j  d  Build (right to left) two matrices u R[i, j]: maximum residual capacity when F[i], …, F[n] are encoded so that the first frame that is not skipped is predicted by F[i-j] u S[i,j]: number of skipped frames corresponding to residual capacity R[i, j]  Time is O(d 2 n) = O(n) (constant d  7)

Test Sequences Std and Std100: concatenation of standard test sequences Commercials: Sampled TV commercials

Experimental Results  Gain in Bit/PSNR in proximity of scene cuts (simplified method)

Experimental Results  Gain on whole sequence (simplified method)

Conclusions  Simple yet effective method to improve quality in proximity of scene cuts  Experiments with simplified method show improvements of 14-30% (in Bit/PSNR)  Suitable for encoders of the MPEG family, provided that encoder has look-ahead capability  Decoding is unaffected

Future Research  Assess quality improvement when using optimal algorithm  Experiment with progressive transmission to eliminate frozen frame displayed by the decoder