Computational Complexity Management of a Real-Time H.264/AVC Encoder C S Kannangara, I E Richardson, and A J Miller CSVT

Outline  Coding Complexity of H.264/AVC  Frame Rate vs. Coding Complexity  Objective  Architecture  Frame Level Control Algorithm  Per-frame Complexity Control Algorithm  Experimental Results 2

Coding Complexity of H.264/AVC Variable block size  Increase 2.5% complexity for each additional mode  Reduce 4~20% bit rate Hadamard transform  Increase 20% access frequency CABAC  Increase 25~30% access frequency  Reduce 16% bit-rate Multiple reference frame  Increase 20% access frequency for each added frame  Reduce 2~14% bit rate J. Ostermann, J. Bormans, P. List, D. Marpe, M. Narroschke, F. Pereira, T. Stockhammer, and T. Wedi, "Video Coding with H.264/AVC: Tools, Performance and Complexity," IEEE Circuits and Systems Magazine, vol. 4, pp. 7-28,

Frame Rate vs. Coding Complexity  Video encoding over limited computational resources Reduce frame rate or encoding complexity?  Subjective quality of high activity videos  Subjective quality of low activity videos J. A. F. Ronnie, T. Apteker, Valentin S. Kisimov, and Hanoch Neishlos, "Video Acceptability and Frame Rate," IEEE Multimedia, vol. 2, pp ,

Objective  Managed complexity (MC) coder Maintain a smooth video frame rate Frame level control algorithm  Calculate total delay  Calculate target encoding time for the next frame  Drop the frame if coding delay is too high Per-frame complexity control algorithm  Lagrangian cost  Complexity control 5

Architecture 6

Frame Level Control Algorithm (1/2)  Timing scenario of the buffer  Encoding delay Total_Delay n = T n st – T last + T f (Buffer_level-1)  T f + T f (B-1) (drop frames in this situation) F1F1 T1T1 T2T2 T3T3 T4T4 T5T5 TfTf TfTf TfTf TfTf T 1 st T 2 st T 3 st T 4 st T 1 enc T 2 enc T 3 enc F1F1 F2F2 F2F2 F3F3 F3F3 F3F3 F4F4 F4F4 F5F5 F2F2 F3F3 F4F4 F5F5 T 3 st – T 3 T 4 st – T 5 + T f The last input frame time The maximum buffer size 7

Frame Level Control Algorithm (2/2)  Target encoding time for the next frame Total_Delay n+k =Total_Delay n+k-1 –T f +T n+k-1 target  Total_Delay n+k =Total_Delay n –kT f +T n target +T n+1 target +…+T n+k-1 target Assume Total_Delay n+k = 0  (T n target +T n+1 target +…+T n+k-1 target )/k = T f – Total_Delay n /k Assume T n target T n+1 target …T n+k-1 target  T n target = T f –Total_Delay n /k Max Total_Delay n  Min T n target = 0  0 = T f - T f B/k  k = B T n target = T f – [T n st – T last + T f (Buffer_level- 1) ]/B 8 Delay due to F n+k-1 T n+k-1 st T last TfTf T n+k-1 target

Per-frame Complexity Control Algorithm (1/3)  Encode Fn within time T n target  Lagrangian cost (code or skip a MB) Code a MB  J code = D code + r R code + c C code Skip a MB  J skip = D skip (MV=0 without residual) J skip < J code ? skip : coding r  r  Aexp(BQP)  A=7.410 -8 F+5.210 -5 B=-3.710 -5 F+0.3  F = (Avg. MSE per MB)  (Avg. bits per MB) Set to 1 Detail 9

Per-frame Complexity Control Algorithm (2/3)  r (RD control) D(x i,M i )=F i  R -1 (X i,M i )   F = (1/N)F i   r  A  exp(B  QP)  A=7.410 -8 F+5.210 -5  B=-3.710 -5 F+0.3  (by linear regression) Carephone( ) Foreman( ) Foreman( ) C.S. Kannangara, I.E.G. Richardson, M. Bystrom, J.R. Solera, Y. Zhao, A. MacLennan and R. Cooney, “ Low Complexity Skip Prediction for H.264 Through Lagrangian Cost Estimation”, IEEE Trans. Circuits and Systems for Video Tech., vol. 16, no. 2, pp , February

Per-frame Complexity Control Algorithm (3/3)  c (Complexity control) c n = c n-1 +  c n  c n =   c 2 = 0.2(1-) 2 (T 1 enc – T 2 enc )   c 1 = 0 Limitation:  -2   c n  1 (handling Non-stationary behaviour)  Similar number of drop MBs between successive frames  c n  Limit c (limiting the number of drop MBs)  Drop frames rather than drop MBs Actual encoding time Target percentage of saved coding time 11 C. S. Kannangara and I. E. G. Richardson, "Computational Control of an H.264 Encoder through Lagrangian Cost Function Estimation", VLBV 2005, Sardinia, Italy, September 2005.

Experimental Results (1/6)  Environment JM 7.4  Baseline Profile  One reference frame, IPPP…  RDO disabled  Start at 15fps frame rate Multiple processing threads  Proposed algorithm  Encoding process 12

Experimental Results (2/6)  Initial environment 13

Experimental Results (3/6) 14

Experimental Results (4/6) 15

Experimental Results (5/6) 16

Experimental Results (6/6) 17