1. Chuan Wang1, Haibin Huang1, Xiaoguang Han2, Jue Wang1
Megvii (Face++) USA1
Shenzhen Research Inst. of Big Data, The Chinese University of Hong Kong, Shenzhen, China2
June 4, 2019

2. Overview Contributions:
The first work of deep learning based video inpainting.
Include temporal structure preservation (3D CNN) and spatial detail recovery (2D CNN).
Jointly training of both networks improves the performance of the overall system.

3. Related Work Patch-based Image/Video Inpainting
Barnes et al.’s PatchMatch, 2009
Venkatesh et al.’s Efficient object-based video inpainting, 2009

4. Related Work Image Inpainting by 2DCNN
Satoshi et al.’s Globally and Locally Consistent Image Completion, 2017
Pathak et al.’s Context Encoders, 2016
Guilin Liu et al.’s Image Inpainting for Irregular Holes Using Partial Convolutions, 2018

5. Related Work Shape Completion by 3DCNN
Dai et al.’s Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis, 2017
Wang et al.’s Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks, 2017
Han et al.’s High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference, 2017

6. Algorithm Temporal structure inference by 3DCN
Spatial details completion by CombCN

7. Algorithm

8. Algorithm
L1 Loss =
L1 Loss

9. Algorithm Temporal structure inference by 3DCN
Spatial details completion by CombCN
Jointly Training:

10. Experiments - Comparisons with existing methods
Dataset FaceForensics
Input
3DCN
2DCN
CombCN GT

11. Experiments - Comparisons with existing methods
Dataset Caltech
Input
3DCN
2DCN
CombCN GT

12. Experiments - Comparisons with existing methods
Dataset FaceForensics (Random Holes)
Input
3DCN
2DCN
CombCN GT

13. Experiments - Ablation Studies
Performance of variants of 3DCN
V1. Feed 3DCN with videos in lower resolution
V2. Involve down-sampling in time axis in 3DCN
Performance of variants of training strategy
T1. Pre-train 3DCN, then train CombCN without fine-tuning it.
T2. Train 3DCN and CombCN jointly from scratch.
300VW Dataset

14. More Results
More results could be found at our project page:

15. Thank You