1. Deformable Object Behavior Reconstruction Derived Through Simultaneous Geometric and Material Property Estimation Shane Transue and Min-Hyung Choi University of Colorado Denver ISVC 2015 December 14 – 16 Las Vegas, NV

2. Introduction  Objective  Motivation  Related Work  Methodology  Results

3. Introduction  Physically-based Deformable Animation  Finite Element Methods (FEM) / Mass-spring Systems Represent most deformable models Depend on material coefficients Mass-spring (ks / kd) FEM (Young’s Modulus / Possion’s Ratio)  Physically Plausible Results Animation (Artistically controlled) Procedurally Generated Deformation Behaviors [Coros, et al. 2012]

4. Introduction  Current Research: Targeted Deformation  Generate deformation behavior  Artist targeted deformations Artificially impose deformations Arbitrarily assign materials, geometry Generate physically plausible behaviors  Replicate Real-world Deformations  Develop a methodology for automated deformation behavior extraction of real-world objects  Objective: Extract material and geometric properties of real- world material using sets of recorded depth-images Does geometric composition modify deformation behavior? [Transue, S. & Choi, M. 2015]

5. Motivation  Reproduce Real-world deformations  Automated reconstruction process Depth-imaging for surface deformation recording Iso-surface extraction (topology recording) Material property extraction  Geometric impact on simulated Finite Element Models Observation: Geometric density plays a pivotal role in the simulation of Finite Element Models

6. Motivation (cont’d)  Limited to discrete representations  Real-time requires tangible resolutions  Interact with Nodal structures (discontinuous force applications)  Collision Events are handled with discrete models  Does the highest resolution provide the closest replication?  Volumetric objects: Internal composition unknown  Vision-based recording: Limited to surface behaviors FEM Model (a) dropped onto a flat surface with resolutions of (b) n = 688, (c) n = 868, and (d) n = 4070 nodes. Young’s Modulus (1.0e5), Poisson’s Ratio (0.4) are held constant.

7. Related Work  Estimation of Elastic Material Parameters (FEM)  Learning elasticity parameters with a manipulation robot [B. Frank et al., 2010]  Estimation of elastic material parameters using FEM [M. Becker, et al., 2007]  Deformable surface tracking in multi-view setups [Cagniart et al., 2010]  Real-world Deformation Reconstruction  Finite element based tracking of deformable surfaces [Wuhrer et al., 2013]  Fitting solid meshes to animated surfaces [J. Choi and A. Szymczak, 2009]  Simultaneous topology and stiffness identification for mass-spring systems [Bianchi et al., 2004]

8. Method  Deformation Reconstruction  Reproduce Finite Element Model simulated deformations based on the observed deformation of a real-world model Scannable isotropic homogeneous elastic materials  Reconstruction Pipeline:  Depth-image Surface Recording: Records deformations of real-world objects using depth-images  Inverse Simulation: Approximate deformation forces by external surface displacements  Elastic Object Property Estimation: Determines optimal material and geometric parameters of simulate finite element model required to reconstruct the observed deformation

9. Method

10. Method: Depth-image Surface Recording  Deformable Surface Recording  High-fps Depth-imaging (Kinect2 ~70[fps])  Synchronized Imaging Devices / Synthetic Data (multi-view)  Record discrete surface states over time  Process: 1. Record deformation surface (multi-view) 2. Global Alignment (Ransac, FPFH, Mathematical, etc) 3. Refined Alignment (ICP) 4. Result: Point-cloud representation of the surface at time t

11. Method: Solid Model Surface Extraction  Surface Reconstruction  Input: Aligned point-cloud deformation state at time t Contains entire object surface Estimated surface normals [Rusu, 2009]  Output: Representative surface deformation based on the provided point-cloud [Kazhdan, 2005]  Two critical aspects: (1) Defines the deformation states of the real-world object over time (2) Defines the base mesh for generating the tetrahedral simulation mesh used within the FEM-based simulation

12. Method: Surface Animation Reconstruction  Depth Imaging  Surface Reconstruction  Reference Animation  Defines the set of surface states that defines the real world deformation  Initial surface reconstruction represents FEM rest-state  Subsequent states model surface displacements at a discrete interval (1) Used to generate displacement fields (2) Implicitly defines the forces that define the deformation

13. Method: Rest-state Mesh Generation  Tetrahedral Simulation Mesh Generation  Tetgen (Tetrahedral Generation based on Delaunay Triangulation)  Geometric Parameterization: Quality (q) - Shape Max-volume (v) – Density Parameterization: P(q, v)  Rest-state shell mesh used to generate parameterized internal geometry (opt) Low Quality Tetrahedra Tetrahedral mesh with 2 parameterizations (a) n = 400, (b) n = 600 nodes.

14. Method: Displacement Field Generation  Objective: Derive incremental deformation behaviors  Linearization (based on device sampling rate dt)  Used to approximate external forces (2 nd derivative)  Displacement Field  Given the surface states  Displacement of S from t to t+1  Generate displacement fields* Displacement Field between S(t) and S(t+1)

15. Displacement Field Generation (cont’d)  Displacement Generation*  Scanned surface nodes are inconsistent in time: Node n i at S(t) may not exist within S(t+1)  Displacement Generation through Non-rigid Registration  Coherent Point Drift (CPD) [Myronenko, 2010] Non-rigid alignment of surface nodes of S(t) to S(t+1) Required due to non-rigid topology deformations  Provides per-node displacements modeling the observed displacement  Defines the magnitude of a displacement field D for all points p i

16. Method

17. Method: Force Optimization  Displacement Field Force Generation (inverse dynamics)  Objective: Define external forces to make the simulated surface coplanar with the observed surface  Based on observed deformation  Defines estimated direction and magnitude  Direct calculation of corresponding velocity and force fields  Optimization: Minimization of the displacement field |D|  Generalized Deformation Behavior  Generalized deformation replication  Independent of applied external forces (unknown)  Resists variation and noise present within the depth-image

18. Deformation Optimization  Optimal Deformation:  Elastic material and geometric composition optimization  Constrained Optimization by Linear Approximations [Powell, 1994]  Constrained search (derivative free)  NLopt

19. Methodology Results Method Results: (a) generated displacement fields, (b) reconstructed surface animation, (c) simulated Finite Element Model

20. Quantitative Preliminary Results ParameterqvypD_anim Geometric Opt (max)2.132421.01930e-41.000e60.30000072.2366 Geometric Opt (opt)2.125001.01321e-41.000e60.30000053.0002 Material Opt (max)2.000001.00000e-45.000e50.28750075.4573 Material Opt( opt)2.000001.00000e-44.634e50.22121453.3687 Geo + Mat (max)2.136018.01578e-56.55411e50.29487675.7081 Geo + Mat (opt)2.127998.03421e-56.52945e50.29717751.6883

21. Visual Preliminary Results  Minimal example to illustrate the impact of the geometric structure as opposed to the influence of the force optimization over large deformations Resulting alignment of the optimal parameterization (a) of the FEM mesh superimposed onto the scanned surface. Geometric opt (b) and Material opt (c) illustrate the misalignment obtained during independent optimizations.

22. Evaluation  Optimization Hierarchy  Errors in force optimization propagate to deformation (drifting)  Both levels subject to local minima  Vision-based Recording  Multiple viewing angles to incorporate entire surface (limits utility)  Occlusions  Matte surface materials (ideal reflectance characteristics)  Illustrates the impacts of geometric structure on a simulated Finite Element Model and its deformation  Limited to isotropic homogeneous elastic materials  Optimization: Does not robustly handle discontinuous collision events

23. Conclusion  Introduce a method for automated deformation reconstruction  Reconstruction of scanned surfaces  Formulated an inverse dynamics problem from displacement fields  Provides initial parameters for a simulated FEM rather than best- guess estimates in geometry and elastic material coefficients Parameterized the FEM-based simulation mesh P(q, v, y, p) Optimization of Elastic material parameters and Tetrahedral Properties Automated method for estimating q, v, y, p  Illustrated the effect of fine-level geometric resolution on an FEM Preliminary work verifies the observation (in moderate cases) Future Work: Extension to concave models, collision

24. Thank you Questions?