Efficient Fine Granularity Scalability Using Adaptive Leaky Factor Yunlong Gao and Lap-Pui Chau, Senior Member, IEEE IEEE TRANSACTIONS ON BROADCASTING, VOL. 51, NO. 4, DECEMBER 2005

Outline FGS Introduction FGS Improvement –MCFGS –PFGS –RFGS Adaptive Leaky Prediction –Linear Model Experimental Results –RS Codes for Packet Loss Protection Experimental Results Conclusions

FGS Introduction Fine granularity scalability (FGS) is an amendment on the MPEG-4 standard. –This work aims to optimize the video quality over a given range of bit rate. –This was motivated by streaming video over the internet, where two critical assumptions come into play: the encoder does not know the channel capacity the decoder may not be able to decode all bits received over from the channel (or the bit stream may be truncated) –The base layer image can be received correctly. FGS is also resilient to packet losses, which are common over the Internet.

FGS Introduction Motion Compensation Motion Estmation Frame memory DCTQ VLC IDCT Clipping - Base Layer Bitstream Find MAXBitplane CodingVLCZigzag Scan - Enhancement Layer Bitstream Frame input Find MAXBitplane CodingVLCZigzag Scan

FGS Improvement FGS –Fine granularity, bandwidth adaptation, and error recovery ability, while still maintaining a simple and flexible coding structure. FGS Drawback –There is no temporal prediction in enhancement layer. –The prediction is always based on the lowest base layer reference. –FGS can provide good error recovery from data corruption or transmission errors in the enhancement layer. However, this also decreases the coding efficiency comparing to the non- scalable single-layer coding scheme due to the low quality reference image.

FGS Improvement There have been several methods proposed to improve the FGS coding scheme –Motion-Compensated Fine Granularity Scalability (MC-FGS) –Progressive Fine Granularity Scalability (PFGS) –The Robust Fine Granularity Scalable (RFGS)

FGS Improvement MC-FGS –Several enhancement bit-planes are directly introduced to the motion compensation loop for a high quality reference. –This method can achieve very high efficiency if the enhancement bit-planes used for prediction can be correctly transmitted, otherwise, severe drifting will happen because of the difference between the references in the encoder side and decoder side.

FGS Improvement MC-FGS Single loop Two loop

FGS Improvement PFGS –Use one more compensation loop in the enhancement layer, and keeps a prediction path from base layer to the higher quality layer that can gracefully recover from losses and errors. –The coding performance of PFGS can be further improved by macro block-based PFGS. –The performance of PFGS can ’ t get satisfying for its coarse assumption that the distortion is similar if different frames are truncated at the same bit plane.

FGS Improvement PFGS Base layer Base layer + enhancement1 Base layer + enhancement1,2 Base layer + enhancement1,2,3 Base layer + enhancement1,2,3,4

FGS Improvement RFGS –An improved high quality reference in the enhancement layer is constructed by combining the reconstructed base layer image and part of enhancement layer stream. –Leaky Prediction is designed for preventing from error propagation when enhancement data loss.

FGS Improvement RFGS Leaky Prediction: The reference frame is scaled by a factor 0 ≤α ≤ 1, when the prediction for the next frame. The leak factor is used to speed up the decay of error. Partial Prediction: The enhancement-layer loop can be built with an adaptive selection of number of bit Planes for the reference picture (denoted as β).

Adaptive Leaky Prediction RFGS uses a fixed leaky factor for all prediction –Enhancement bit stream has different importance for different layer –In order to have an optimal picture quality, different leaky factors shall be applied to different bit-planes instead of using a fixed leaky factor for all the bit-planes without considering the significance.

Adaptive Leaky Prediction Proposed coding scheme ELPi: Enhancement layer prediction image. BLPi: Base layer prediction Image. B: The base layer stream signal. ELRi: Enhancement layer reference image. BLRi: Base layer reference image. FB: Enhancement layer stream feedback

Adaptive Leaky Prediction

In the proposed method, the weighting factor γ actually controls the enhancement and base layer prediction image, and adaptive leaky factors α are applied for the bit-plane signals from the enhancement layer feedback. Comparing with that using a fixed leaky factor applied for the difference between the high quality reference image in the enhancement layer and the reconstructed base layer reference image (γ and α all have the same value), our method gives the flexibility to select adaptive values of leaky factors according to the rules.

Adaptive Leaky Prediction The proposed framework also gives the flexibility to select the value of weighting factor γ. –Just follow the method in RFGS to use a same value with the leaky factor, –Use the methods that can switch between the two predictions (low prediction and high prediction), where value is either 0 or 1. –In the proposed method, leaky factor with value of 1 is possible to provide better reference image. If these data using leaky value of 1 are lost, the error can be attenuated by the weighting factor and other leaky factors that are less than one in the following frames.

Adaptive Leaky Prediction How to decide the leaky factor? –Linear Model –Reed-Solomon Codes for Packet Loss Protection

Adaptive Leaky Prediction Linear Model: Since the data at the very beginning part is more likely to be correctly received and decoded, a larger leaky factor should be used and the leaky factors should decrease when the applied data is faraway from the beginning of the stream. The model can be more complex !

Experimental Results Linear Model –In the simulation, all the data in one bit-plane is applied for one leaky factor which is determined by the position of the last bit of that bit-plane. –The experiment setting The base layer was implemented the H.264 standard. All the sequences (100 frames) are coded at 10 fps. Only the 1st frame is coded as I-picture, followed by P- pictures, and no B-picture is used. base layer don ’ t use Leaky /partial prediction techniques The base layer stream is 32kbps The enhancement layer truncates at the defined length. The leaky factors are determined with T = 3.2kbits The weighting factor are fixed at one value. 4 bit-planes from the enhancement stream are used

Experimental Results Single layer: No FGS streaming

Experimental Results PSNR improvement versus bit rate of the proposed method over using a fixed leaky factor

Adaptive Leaky Prediction The linear model could be applied to assign the leaky factors without knowing any information from channel. We further enhance the framework by optimizing the leaky factors from the channel conditions, where enhancement layer is transmitted under UPP, and the data decodable probability can be derived from the protection code.

Adaptive Leaky Prediction Markov Model for Video Streaming The model are denoted G where the packet is received correctly and B where the packet is lost or damaged. Average loss probability: Average burst length:

Adaptive Leaky Prediction RS Codes for Packet Loss Protection –An (n, k) RS code The code words are formed across k information and ( n - k ) redundancy packets. The receiver can recover the message information from any subset of k packets which are correctly received. The probability that at least k packets are correctly received. –The probability can be computed based on the probability P(m,N) (block error density function) of m lost packets within the block k of N packets.

Adaptive Leaky Prediction The unequal packet-loss protection is desirable for FGS stream –More protection is applied for the more important data in the front blocks of the bit stream –The optimal protection codes can be found by following the local search hill-climbing algorithm to maximize the expected PSNR considering the packet loss. The decodable probability of block l:

Adaptive Leaky Prediction D : A block of data in the enhancement layer feedback α : Leaky factor αD: The image which is used to construct the high quality reference P : Decodable probability （ Loss when data is decodable ） （ Error when data is not decodable ） As error occurred, it could propagate to the following frames. So it is more important. Therefore w is a number over 1.

Experimental Results The experiment setting is the same with above The enhancement layer in one frame is transmitted using 80 packets with length of 47 bytes per packet (N = 80, L = 47) The optimal FEC codes for different blocks of data are found by reference [19] using hill-climbing approach. w=5, L B =9.57, P B =10%/20%

Experimental Results

Conclusions Video streaming over Internet has to face two main problems: –bandwidth variation –packet loss. FGS is one of the video scalable coding techniques that have attracted a lot of research to address these problems. This paper proposes a novel FGS coding scheme that applied adaptive leaky factors to further improve the coding efficiency and error robustness. –Two schemes have been proposed, i.e., linear model scheme and UPP-based scheme. Based on the network condition –The performance of the proposed methods has been verified over a wide range of bit rate and packet loss ratio.