1. A Progressive Tri-level Segmentation Approach for Topology-Change-Aware Video Matting Jinlong Ju 1, Jue Wang 3, Yebin Liu 1, Haoqian Wang 2, Qionghai Dai 1 1.Department of Automation, Tsinghua University, China 2.Graduate School at Shenzhen, Tsinghua University, China 3.Adobe Research, USA

2. The goal Interactive video object segmentation and matting Initial keyframe segmentation Local correction

3. Previous work 3D volume segmentation (global optimization) – Hard to converge Interactive video cutout [Wang et al., SIGGRAPH’05] Video object cut & paste [Li et al., SIGGRAPH’05]

4. Previous work Tracking & segmentation – Easy and intuitive workflow Frame t Frame t+1 Video SnapCut [Bai et al., SIGGRAPH’09] Unbiased Directional Classifiers [Zhong et al., SIGGRAPH Asia’12] Video Object Segmentation by Tracking Regions. ICCV’09. Object Tracking and Segmentation in a Closed Loop. ISVC'10.

5. Previous work Video matting Video Matting of Complex Scenes [Chuang et al., SIGGRAPH’02] Towards Temporally-Coherent Video Matting [Bai et al., Mirage’11]

6. Main ideas Previous methods Our approach Global, one-time optimization Progressive Binary segmentation -> trimap -> matting Tri-level segmentation -> matting

7. Framework Frame-to-frame propagation Frame t Frame t+1

8. Tri-level initial segmentation Step 1: coarse object alignment – SIFT feature tracking and alignment – Optical flow Frame t Frame t-1

9. Initial Tri-level segmentation Step 2: color models Frame t Frame t-1

10. Initial Tri-level segmentation Frame t Frame t-1

11. Initial Tri-level segmentation Comparison Frame t Frame t-1 Our initial labelingColor probability maps of Gaussian Mixtures

12. Initial Tri-level segmentation Frame tFrame t+1 Unmatched pixels

13. Initial Tri-level segmentation Step 5: local smoothing – Mean shift, same color models for each MS region Frame tFrame t+1 Initial map

14. Tri-level segmentation Refinement Goal: remove large unknown regions Before refinementAfter refinement

15. Tri-level segmentation Refinement Cross-frame window matching + shape prior Frame tFrame t+1

16. Tri-level segmentation Refinement Cross-frame window matching + shape prior Frame tFrame t+1

17. Final matting Directly use tri-level segmentation as trimap (ideally) After mattingBefore matting

18. Results Tri-level segmentation Final result Adobe After Effects RotoBrush

19. Results Tri-level segmentation Final result Adobe After Effects RotoBrush

20. Quantitative comparison

21. Conclusion Tri-label labeling Progressive segmentation Handles topology-change, fast-moving objects well Limitations: – Segmentation is a little bit too aggressive (not ideal for matting) – A few thresholds to tune