A Progressive Tri-level Segmentation Approach for Topology-Change-Aware Video Matting Jinlong Ju 1, Jue Wang 3, Yebin Liu 1, Haoqian Wang 2, Qionghai Dai 1 1.Department of Automation, Tsinghua University, China 2.Graduate School at Shenzhen, Tsinghua University, China 3.Adobe Research, USA

The goal Interactive video object segmentation and matting Initial keyframe segmentation Local correction

Previous work 3D volume segmentation (global optimization) – Hard to converge Interactive video cutout [Wang et al., SIGGRAPH’05] Video object cut & paste [Li et al., SIGGRAPH’05]

Previous work Tracking & segmentation – Easy and intuitive workflow Frame t Frame t+1 Video SnapCut [Bai et al., SIGGRAPH’09] Unbiased Directional Classifiers [Zhong et al., SIGGRAPH Asia’12] Video Object Segmentation by Tracking Regions. ICCV’09. Object Tracking and Segmentation in a Closed Loop. ISVC'10.

Previous work Video matting Video Matting of Complex Scenes [Chuang et al., SIGGRAPH’02] Towards Temporally-Coherent Video Matting [Bai et al., Mirage’11]

Main ideas Previous methods Our approach Global, one-time optimization Progressive Binary segmentation -> trimap -> matting Tri-level segmentation -> matting

Framework Frame-to-frame propagation Frame t Frame t+1

Tri-level initial segmentation Step 1: coarse object alignment – SIFT feature tracking and alignment – Optical flow Frame t Frame t-1

Initial Tri-level segmentation Step 2: color models Frame t Frame t-1

Initial Tri-level segmentation Frame t Frame t-1

Initial Tri-level segmentation Comparison Frame t Frame t-1 Our initial labelingColor probability maps of Gaussian Mixtures

Initial Tri-level segmentation Frame tFrame t+1 Unmatched pixels

Initial Tri-level segmentation Step 5: local smoothing – Mean shift, same color models for each MS region Frame tFrame t+1 Initial map

Tri-level segmentation Refinement Goal: remove large unknown regions Before refinementAfter refinement

Tri-level segmentation Refinement Cross-frame window matching + shape prior Frame tFrame t+1

Tri-level segmentation Refinement Cross-frame window matching + shape prior Frame tFrame t+1

Final matting Directly use tri-level segmentation as trimap (ideally) After mattingBefore matting

Results Tri-level segmentation Final result Adobe After Effects RotoBrush

Results Tri-level segmentation Final result Adobe After Effects RotoBrush

Quantitative comparison

Conclusion Tri-label labeling Progressive segmentation Handles topology-change, fast-moving objects well Limitations: – Segmentation is a little bit too aggressive (not ideal for matting) – A few thresholds to tune