1. Methods of Video Object Segmentation in Compressed Domain Cheng Quan Jia

2. Presentation Outline Features for Segmentation in Compressed Domain Using Motion Vectors in Segmentation Confidence Measure Conclusion Q & A

3. Features for Segmentation in Compressed Domain An introduction to Compressed Domain

4. Compressed Domain: Definition Compressed Domain refers to video compression techniques that expliots Spatial and Temporal Redundancy using – DCT & Quantization – Motion Compensation Examples include MPEG-1/-2/-4, H.261 and H.263

5. Compressed Domain: Definition Opreations in the Compressed Domain involves processing of – DCT coefficients (from I-macroblocks) – Motion Vectors (from P-/B-macroblocks)

6. Compressed Domain: Parsing Unlike pixel domain, operations in the compressed domain do not require the input bitstream to be decoded Instead, they are Parsed

7. Compressed Domain: Parsing

8. Features for Segmentation After Parsing, we have – DCT coefficients (from I-macroblocks) – Motion Vectors (from P-/B-macroblocks) Which coresspond to – Frequencies of texture change – Motion of the macroblock

9. Using Motion Vectors in Segmentation

10. Acquiring Dense Motion Field Many video object segmentation methods attempt to acquire a dense smooth motion field in order to create object masks For this end spatial interpolation and motion accumulation are employed

11. Motion Accumulation

12. Due to the different magnitude and signs of motion vectors, the obtained MVs are normalized, e.g. MVs in B-macroblocks would have their signs reversed Filtering is applied to remove non-uniform MV and smooth the motion field

13. Motion Accumulation Chen and Bajic [chen2009] employs MV Integration block-wise and pixel-wise to enhance the Motion Field

14. Motion Accumulation Chen and Bajic [chen2009]Babu et al. [babu2004]

15. Porikli et al.’s Investigation The Compression Domain segmentation system published by Porikli et al. [porikli2010] experimented the effect of DCT coefficients and MV on segmentation performance – The DC parameters(for Y, U, V channels) of the I- frame – Low vertical and horizontal frequency AC values – A spatial energy term – Aggregated motion flow of the corresponding macroblock

16. Porikli et al.’s Investigation They create a Frequency-temporal data structure for each macroblock with the features and perform volume segmentation Their results show that using DCT terms in FT segmentation and using MV in the hierarchical clustering, on average, gives better results.

17. Porikli et al.’s Investigation

18. The Block Matching Process in encoding stage looks for only the best match for a macroblock rather than object motion Porikli et al.’s Investigation

19. Confidence Measure of Motion Vectors

20. Coimbra and Davies [coimbras2005] try to approximate Lucas–Kanade optical flow in MPEG-2 Compressed Domain Approximating Optical Flow

21. They argue that AC[1] and AC[8] in an I- macroblock can be used as confidence measure The confidence update step will have a 8×8 macroblock referencing a 16×16 image block in the I-frame, and the confidence of the motion vector of the macroblock is the weighted average of confidence in the 16×16 window Confidence Measure

22. Original imageMPEG-2 smooth motion field after confidence threshold

23. Conclusion

24. Due to block matching process, motion vectors in P-/B- frames do not necessary relate to object motion To ensure a motion vector is correlated to object motion, some sort of confidence measure is required [coimbras2005] demonstrated that edge strength can be an effective measure Conclusion

25. Problems not discussed here – Camera motion – Changes in illumination – Occlusions Conclusion

26. References 1.R. V. Babu, K. R. Ramakrishnan, and S. H. Srinivasan. Video Object Segmentation: A Compressed Domain Approach. IEEE Transactions on Circuits and Systems for Video Technology, 14(4):462–473, April 2004. 2.Y.-M. Chen and I. V. Bajic. Compressed-Domain Moving Region Segmentation with Pixel Precision using Motion Integration. In IEEE Pacific Rim Conference on Computers and Signal Processing, 2009, pages 442 – 447, August 2009. 3.M. T. Coimbra and M. Davies. Approximating Optical Flow Within the MPEG-2 Compressed Domain. IEEE Transactions on Circuits and Systems for Video Technology, 15(1):103–107, January 2005. 4.F. Porikli, F. Bashir, and H. Sun. Compressed Domain Video Object Segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 20(1):2–14, January 2010.

27. Q & A SECTION Comments and Suggestions