Temporally Coherent Completion of Dynamic Shapes AUTHORS:HAO LI,LINJIE LUO,DANIEL VLASIC PIETER PEERS,JOVAN POPOVIC,MARK PAULY,SZYMON RUSINKIEWICZ Presenter:Zoomin(Zhuming) Hao

Previous Work 1.Based on Template How to obtain the template? ① a separate rigid reconstruction step (e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008])

Previous Work Robust Single-View Geometry and Motion Reconstruction[Li et al. 2009]

Previous Work 1.Based on Template How to obtain the template? ① a separate rigid reconstruction step (e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008]) ② globally aggregating all surface samples through time (e.g., [Wand et al. 2009; Mitra et al. 2007; S¨ußmuth et al. 2008])

Previous Work Efficient Reconstruction of Nonrigid Shape and Motion from Real-Time 3D Scanner Data [Wand et al. 2009] input ==> a sequence of point clouds sampled at different time instances automatically assembles them into a common shape that best fits all of the input data a deformation field is computed that approximates the motion of this shape to match all the data frames limitations ： occurs if objects disappear in an acquisition hole and come out in a very different pose

Previous Work 1.Based on Template How to obtain the template? ① a separate rigid reconstruction step (e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008]) ② globally aggregating all surface samples through time (e.g., [Wand et al. 2009; Mitra et al. 2007; S¨ußmuth et al. 2008]) Disadvantage ？ fix the topology geometric details are limited to those in the template

Previous Work 2.Based on the assumption: Dynamic performance consists of rigid parts [Pakelny and Gotsman2008] manual segmentation,an optimal rigid motion is computed for each part [Chang and Zwicker 2009] limits to subjects that exhibit articulated motion [Zheng et al.2010] automatically extract a consensus skeleton to derive a consistent temporal topology

Previous Work Consensus skeleton for nonrigid space-time registration [Zheng et al.2010] input==>a sequence of point clouds acquired over time extract per-frame skeletons consolidate them into a skeleton structure (consistent across time and accounts for all the frames) Limitations ： It assumes that the underlying shape is clearly articulated which is not always the case for subjects wearing loose clothing Articulated Mesh Animation from Multi-view Silhouettes [Vlasic et al. 2008]

System Overview

Framework -- 1 Pairwise Correspondences Coarse-scale Correspondences ： non-rigid ICP algorithm[Li et al.2009]

Framework -- 1 Pairwise Correspondences Fine-scale Correspondences ： Improvement based on two observations: 1.far-away points can bias the local alignment(local-support) 2.stability of ICP matching algorithm depends on the local geometry Three-step Algorithm provided by this paper: 1.Sampling 2.Matching: non-rigid locally weighted ICP algorithm[Brown and Rusinkiewicz 2007] employ a CSRBF for point selection near feature point 3.Warping

Framework -- 1 Shape Accumulation f i ' (merged) and f i+1 (original)==> Corrsepondences f i+1 ‘ merge warp from first frame to the last frame from last frame to the first frame interleaved registration/merging scheme in a forward&backward fashion

System Overview

Framework -- 2 Hole Filling visual Hull prior [Vlasic et al. 2009] + weighted Poisson surface reconstruction [Kazhdan et al. 2006] Surface Fairing: Minimizing bending energy of the patch ’ s vertices using bi-Laplacian [Botsch and Sorkine 2008]

System Overview

Framework -- 3 Temporal Filtering 1.Warp two neighboring frames to current frame based on the pairwise correspondences 2.Combine them using Poisson reconstruction with different weight for different region Poisson reconstruction warp to

System Overview

Framework -- 4 Detail Resynthesis 1. Resynthesize high frequency detail [Nehab et al.2005] 2. Acquire normal maps [Vlasic et al.2009]

Conclusions Contribution: A framework to automatically fill holes with temporal coherent patches without relying on a geometrical template. some little improvements on previous algorithms Limitation: 1.Topology of our meshes will always match the(changing and sometimes incorrect)topology of the visual hull.(Ideally, we need to extract a single consistent topology) 2. Temporal correspondences are valid between nearby frames only 3. each frame should cover most part of the object surface ( limited to multi- view scans),the unobserved regions have no geometric details in them. Future work: To take physical properties into account