Methods of Video Object Segmentation in Compressed Domain Cheng Quan Jia

Presentation Outline Features for Segmentation in Compressed Domain Using Motion Vectors in Segmentation Confidence Measure Conclusion Q & A

Features for Segmentation in Compressed Domain An introduction to Compressed Domain

Compressed Domain: Definition Compressed Domain refers to video compression techniques that expliots Spatial and Temporal Redundancy using – DCT & Quantization – Motion Compensation Examples include MPEG-1/-2/-4, H.261 and H.263

Compressed Domain: Definition Opreations in the Compressed Domain involves processing of – DCT coefficients (from I-macroblocks) – Motion Vectors (from P-/B-macroblocks)

Compressed Domain: Parsing Unlike pixel domain, operations in the compressed domain do not require the input bitstream to be decoded Instead, they are Parsed

Compressed Domain: Parsing

Features for Segmentation After Parsing, we have – DCT coefficients (from I-macroblocks) – Motion Vectors (from P-/B-macroblocks) Which coresspond to – Frequencies of texture change – Motion of the macroblock

Using Motion Vectors in Segmentation

Acquiring Dense Motion Field Many video object segmentation methods attempt to acquire a dense smooth motion field in order to create object masks For this end spatial interpolation and motion accumulation are employed

Motion Accumulation

Due to the different magnitude and signs of motion vectors, the obtained MVs are normalized, e.g. MVs in B-macroblocks would have their signs reversed Filtering is applied to remove non-uniform MV and smooth the motion field

Motion Accumulation Chen and Bajic [chen2009] employs MV Integration block-wise and pixel-wise to enhance the Motion Field

Motion Accumulation Chen and Bajic [chen2009]Babu et al. [babu2004]

Porikli et al.’s Investigation The Compression Domain segmentation system published by Porikli et al. [porikli2010] experimented the effect of DCT coefficients and MV on segmentation performance – The DC parameters(for Y, U, V channels) of the I- frame – Low vertical and horizontal frequency AC values – A spatial energy term – Aggregated motion flow of the corresponding macroblock

Porikli et al.’s Investigation They create a Frequency-temporal data structure for each macroblock with the features and perform volume segmentation Their results show that using DCT terms in FT segmentation and using MV in the hierarchical clustering, on average, gives better results.

Porikli et al.’s Investigation

The Block Matching Process in encoding stage looks for only the best match for a macroblock rather than object motion Porikli et al.’s Investigation

Confidence Measure of Motion Vectors

Coimbra and Davies [coimbras2005] try to approximate Lucas–Kanade optical flow in MPEG-2 Compressed Domain Approximating Optical Flow

They argue that AC[1] and AC[8] in an I- macroblock can be used as confidence measure The confidence update step will have a 8×8 macroblock referencing a 16×16 image block in the I-frame, and the confidence of the motion vector of the macroblock is the weighted average of confidence in the 16×16 window Confidence Measure

Original imageMPEG-2 smooth motion field after confidence threshold

Conclusion

Due to block matching process, motion vectors in P-/B- frames do not necessary relate to object motion To ensure a motion vector is correlated to object motion, some sort of confidence measure is required [coimbras2005] demonstrated that edge strength can be an effective measure Conclusion

Problems not discussed here – Camera motion – Changes in illumination – Occlusions Conclusion

References 1.R. V. Babu, K. R. Ramakrishnan, and S. H. Srinivasan. Video Object Segmentation: A Compressed Domain Approach. IEEE Transactions on Circuits and Systems for Video Technology, 14(4):462–473, April Y.-M. Chen and I. V. Bajic. Compressed-Domain Moving Region Segmentation with Pixel Precision using Motion Integration. In IEEE Pacific Rim Conference on Computers and Signal Processing, 2009, pages 442 – 447, August M. T. Coimbra and M. Davies. Approximating Optical Flow Within the MPEG-2 Compressed Domain. IEEE Transactions on Circuits and Systems for Video Technology, 15(1):103–107, January F. Porikli, F. Bashir, and H. Sun. Compressed Domain Video Object Segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 20(1):2–14, January 2010.

Q & A SECTION Comments and Suggestions