Rate-Control in Video Codec Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2008 Last updated

Heejune AHN: Image and Video Compressionp. 2 Agenda Basic Concepts Rate-Distortion Theory Practical Rate-Control Algorithms Other issues

Heejune AHN: Image and Video Compressionp Rate Control Concepts Key players coded video quality: (usu. Lower limits) bit-rate: some constraints (usu. upper limits) input video sequence: (noisy, degree of details, motion complexity) coding parameters (standards, coded mode, quantization step) Rate-optimization select the parameters for best video quality under constraints Poor algorithm : low quality, large fluctuation, dropped frames encoder Input Video Coded Video Bit-rate contraints network/ storage Rate Control Coding para. selection

Heejune AHN: Image and Video Compressionp. 4 Video stream Bit-rate characteristics Factors Encoding algorithms (intra/inter, forward/bi- direction, DCT/Wavelet, standards, etc) Input video Encoding parameters Q step, picture &mb mode, mv search area, GOP structures

Heejune AHN: Image and Video Compressionp. 5 Bit-rate Constraints Typical constraints parameters Average (mean) bit-rate (with duration) Peak (maximum) bit-rate Maximum variation in bit rate Prevention of underflow and overflows at encoder &decoder buffer Required latency Examples DVD video Total 4.7GB (play time 3 hours), peak rate => 3.5Mbps with some peak limits Video conferencing in ISDN or Circuit switching network Constant transmission bit-rate and delay => Based on buffer size, peak, average rate is calcuated Vide streaming over internet and packet switched network Network condition is variable. Should be adaptive rate control

Heejune AHN: Image and Video Compressionp Rate Control Theory Revisit to Quantization For Uniform distributed Input & Uniform quantizer Rate Distortion Theory General trends are same as above distortion ~ input signal variance and exp. desc.(bitrate) !!

Heejune AHN: Image and Video Compressionp. 7 Rate-Distortion Curve Rate decreases as distortion increases depends on input statistics (complex image => large variance) PSNR measure is 10log 10 (255 2 /D)

Heejune AHN: Image and Video Compressionp. 8 Rate-Distortion Optimization Motivation Which subject I have to study if I gets 90pts in Math, 50pts in Eng. ? Image coding Condition: we can use 10KBytes for one picture and each parts have different complexity (i.e, simpler and complex, variance) Which parts I have to assign more bits? simpler blocks More complex blocks

Heejune AHN: Image and Video Compressionp. 9 Lagrange multiplier Constrained Optimization Problem Minimize distortion with bits no larger than max bits Cannot use partial differential for minima and maxima Lagrange Multiplier technique Insert one more imaginary variable (called Lagrange) We have multivariable minimization problem

Heejune AHN: Image and Video Compressionp. 10 Side note: why not partial differential Simple example min subject to Solution By substitution minimum ½ at x = y = 1/sqrt(2) Wrong Solution With complex contraints, we cannot use substitution of variable method We cannot change partial different with ordinary difference If we check, x = y = 0 ? 1 1

Heejune AHN: Image and Video Compressionp. 11 An Example: Lagrange Multiplier Optimization Problem multiple independent Gaussian variables Lagrange multiplier optimization

Heejune AHN: Image and Video Compressionp. 12 Solution M 개를 모두 곱하고 M square 를 하면 Bit-allocation Lagrange value

Heejune AHN: Image and Video Compressionp. 13 RDO in Image Coding Bit allocation problem Modified Lagrange Method First proposed by H. Everett, then signal processing domain by Shoham and Gersho Theorem For any  the solution B*(  to the ‘unconstrained’ problem is also a solution to the ‘constrained’ problem subject to Here B = {r 1,r 2,..., r N }, ie. Bit allocaton each r i such that

Heejune AHN: Image and Video Compressionp. 14 RDO algorithm How to find such  that results to Algorithm Set Get R i for each i independently ie. tagent to slope of RD curve Get R (e.g. sum of r i ’s) Repeat until R = R c, changing  (note: large  small b) Diverse Constraints Sum Ri < R Embedded sum Ri < Rx Buffer controls Dependency condition

Heejune AHN: Image and Video Compressionp. 15 RDO algorithm Intuitive understanding Reverse Water filling theory larger bits for larger variance block The returns will be same at the equivalent point Large variance RD curve small variance RD curve R S * R L *

Heejune AHN: Image and Video Compressionp. 16 Practical Rate Control Problem in Lagrange optimization too much computational complexity for real-time use Rate control Model Buffer occupancy level B at the end of i-th frame B i PRE = B i-1 + Rv – Rc B i = B i PRE if 0< B i PRE <B max = B max if overflow (B i PRE >B max ) = 0 if underflow (B i PRE < 0) Rate Control (a negative feedback loop) : see Fig High level => large Q step => smaller Rv => level down Low level => smaller Q step => larger Rv => level up Rate Control Encoder Quant step Rv Rc Input Video

Heejune AHN: Image and Video Compressionp. 17 Practical Rate Control H263 Test Model Rate control Frame Level Bit allocation for preventing buffer underflow and overflow (some times even frame skip) Macroblock level Allocate bits for each MB A : # of pixles in a MB,   : standard deviation, Q i : quant. step Step 1: calculate the MB complexity   Step 2: calculate Q i from K, A, C, a ,   Step 3: encode MB Step 4: update the model parameters K, C

Heejune AHN: Image and Video Compressionp. 18 MPEG-2 Test Model 5 Bit allocation Assign a target bits to current GOP Assign to each current picture –Based on Complexity of previous I, P, B pictures (Q-Step and resultant bit) Rate control Compare coded bit with expected values, and adjust Q-Step Modulation block activity, increase Q-step

Heejune AHN: Image and Video Compressionp Other Rate Control Issue Variable Complexity Algorithm Constraints on computational power Motion search algorithm & search range Rate Control Mode selection Frame rate (skipping) Trade off Computation complexity v.s. video quality Also depends on input image (time-varying) Usage Computation limited Mobile device (usu. Software based codec) Iain E.G. Richardson, the athor of textbook,’s interest Performance Measure PSNR is not enough (for subject quality measure) Some objective measures are under development SSIM(Structural SIMiliarity)

Appendix Rate distortion theory

Heejune AHN: Image and Video Compressionp. 21 Rate-Distortion Theory Distortion of X’ from X D = H(X) – H(X|X’) X 전체의 정보양에서 X’ 을 통하여 확보한 X 의 정보양을 뺀것 (make sense? Yeh ~~~) 참고 : mutual information : I (X:X’) = H(X:X’) = H(X’:X) Information theoretic definition I(X:X’) H(X’) H(X) D = H(X) – I(X:X’) Note: 붉음색은 X’ = Quant(X) 인경우

Heejune AHN: Image and Video Compressionp. 22 Rate Distortion Rate-distortion function Inverse relation between Information/Data Rate and Distortion Example with Gaussian R.V. (known as most difficult distribution) Refer to element of Information theory

Heejune AHN: Image and Video Compressionp. 23 Derivation