Overview of Fine Granularity Scalability in MPEG-4 Video Standard Weiping Li Presented by : Brian Eriksson

Introduction - Problem Tradition System –Video encoded close to channel capacity –Decoded using all of the bits Internet Video System –Channel capacity changes –Partially decodable within a bit range

Introduction – Video Coding Performance LowHigh Channel Bitrate Received Quality Bad Good = Distortion- Rate Curve = Dialup Rate = Broadband Rate = Layered Coding = Desired Coding

LowHigh Channel Bitrate Received Quality Bad Good Previous Layered Coding Methods = Layered Coding

MPEG-2 Overview I-Frame – Intracoded Picture –Contains all frame information P-Frame – Predictive-Coded Picture –Uses previous frame and motion information to reconstruct frame B-Frame – Bidirectional-Coded Picture –Uses previous frame, next frame, and motion information to reconstruct the frame

Temporal Scalability Video is encoded into two layers with equal resolution but using different prediction. Base layer has lower frame rate BaseEnh. BaseEnh. BaseEnh. BaseXX XX XX With Enhancement Layer : Frame Rate = 30 fps Without Enhancement Layer : Frame Rate = 10 fps

MPEG-2 SNR Scalability Enhancement Layer Stream Base Layer Stream + IDCT Video Output Motion prediction includes enhancement information Efficiency dependent on two factors Encoder uses enhancement layer (drift can occur) Decoder receives enhancement layer Motion Compensation +

Spatial Scalability IDCT Enhancement Bitstream Base Bitstream (Motion Compensated) Upsampler + Enhancement Layer Video Base Layer Video Same frame rate, different resolutions Base frame = 128x128, enhancement = 256x256 Enhancement Layer not in prediction loop

LowHigh Channel Bitrate Received Quality Bad Good MPEG-4 Fine Granularity Scalability Technique = Desired Coding = Distortion- Rate Curve

Bitplane Coding Technique Given vector X: X = [-12, -53, 62, -7, 31, …,180,-43, …,5] Convert to signed magnitude Separate into sign bit and absolute values

Bitplane Coding Technique Sign(X) = [0, 0, 1, 0, 1, …,1,0, …,1] Abs(X) = [12, 53, 62, 7,31, …,180,43, …,5] To find the number of bitplanes needed, find the maximum value of abs(X) Max(abs(X)) = 180 = b 8 bitplanes Sign bit plane 1 bitplane

Bitplane Coding Technique c Sign(X) = [0, 0, 1, 0, 1, …,1,0, …,1] X = [-12, -53, 62, -7, 31, …,180,-43, …,5] 0,1,0,1,1, …,0,1, …,1 0,0,1,1,1, …,0,1, …,0 1,1,1,1,1, …,1,0, …,1 1,0,1,0,1, …,0,1, …,0 0,1,1,0,1, …,1,0, …,0 0,1,1,0,0, …,1,1, …,0 0,0,0,0,0, …,0,0, …,0 0,0,0,0,0, …,1,0, …,0 0,0,1,0,1, …,1,0, …,1

Run Level Encoding Symbol : (RUN, EOP) –RUN = Number of Consecutive Zeros before a 1 –EOP = 0 if there are more ones –EOP = 1 if the rest of the line are zeros. Example: –{ 1,0,1,0,0,0,1,0,0,0,…} = (0,0),(1,0),(3,1) –{0,0,0,1,0,0,0,…} = (3,1)

MPEG-4 Bitplane FGS Technique Base layer reaches lower bound of bit-range Divide image into 8x8 DCT blocks Divide blocks into Y,U,V color components Use bitplane run-level coding to encode/decode

Fine Granularity Scalability Encoder + - DCTBitplane Encoding Original Signal Base Layer Signal (Motion Compensated) Enhancement Layer Signal

Fine Granularity Scalability Decoder Bitplane Decoding IDCT + Base Layer Video (Motion Compensated) Enhancement Layer Video

Truncated Bitplane c Number of Truncated Layers = 0 Received X = [-12, -53, 62, -7,31, …,180,-43, …,-5] Original X = [-12, -53, 62, -7,31, …,180,-43, …,-5] 0,1,0,1,1, …,0,1, …,1 0,0,1,1,1, …,0,1, …,0 1,1,1,1,1, …,1,0, …,1 1,0,1,0,1, …,0,1, …,0 0,1,1,0,1, …,1,0, …,0 0,1,1,0,0, …,1,1, …,0 0,0,0,0,0, …,0,0, …,0 0,0,0,0,0, …,1,0, …,00,0,1,0,1, …,1,0, …,1 c Number of Truncated Layers = 1 Received X = [-12, -52, 62, -6,30, …,180,-42, …,-4] Original X = [-12, -53, 62, -7,31, …,180,-43, …,-5] 0,0,0,0,0, …,0,0, …,0 0,0,1,1,1, …,0,1, …,0 1,1,1,1,1, …,1,0, …,1 1,0,1,0,1, …,0,1, …,0 0,1,1,0,1, …,1,0, …,0 0,1,1,0,0, …,1,1, …,0 0,0,0,0,0, …,0,0, …,0 0,0,0,0,0, …,1,0, …,00,0,1,0,1, …,1,0, …,1 c Number of Truncated Layers = 2 Received X = [-12, -52, 60, -4,28, …,180,-40, …,-4] Original X = [-12, -53, 62, -7,31, …,180,-43, …,-5] 0,0,0,0,0, …,0,0, …,0 1,1,1,1,1, …,1,0, …,1 1,0,1,0,1, …,0,1, …,0 0,1,1,0,1, …,1,0, …,0 0,1,1,0,0, …,1,1, …,0 0,0,0,0,0, …,0,0, …,0 0,0,0,0,0, …,1,0, …,00,0,1,0,1, …,1,0, …,1 c Number of Truncated Layers = 3 Received X = [-8, -48, 56, -0,24, …,176,-40, …,-0] Original X = [-12, -53, 62, -7,31, …,180,-43, …,-5] 0,0,0,0,0, …,0,0, …,0 1,0,1,0,1, …,0,1, …,0 0,1,1,0,1, …,1,0, …,0 0,1,1,0,0, …,1,1, …,0 0,0,0,0,0, …,0,0, …,0 0,0,0,0,0, …,1,0, …,00,0,1,0,1, …,1,0, …,1

Advanced Bitplane Techniques

Frequency Weighting in Bitplane Normal Bitplane DCT Index Frequency Weighted Bitplane DCT Index

Selectively Enhanced Bitplane Detect visually significant area Shift upward in bitplane More likely to be included in truncated bitstream

Other Methods Error Resilience in Bitplane Random burst errors in the bitstream Resynchronization markers are used to resynch. Temporal Scalability in Bitplane Uses FGS to encode/decode the entire temporal enhancement frame

Simulation - DCT

Simulation - DWT

Selectively Enhanced / Weighted Bitplane Setup

Simulation – Enhanced Weighted Bitplane 1 Layer Removed 2 Layers Removed 3 Layers Removed4 Layers Removed5 Layers Removed6 Layers Removed

Simulation – Li

Conclusions FGS Bitplane method allows for quality parallel to distortion-rate curve A wavelet-based approach may yield better results Fairly simple implementation

Questions?