Model-Driven 3D Content Creation as Variation Hao (Richard) Zhang – 张皓 GrUVi Lab, Simon Fraser University (SFU) HKUST, 04/21/11 TAUZJUNUDT SFU

3D content creation Inspiration  a readily usable digital 3D model Inspiration?

Realistic reconstruction Inspiration = real-world data [Nan et al., SIGGRAPH 2010]

Creative inspiration Creation of novel 3D shapes Inspiration = design concept, mental picture, … sketch High demand in VFX, games, simulation, VR, …

3D content creation is hard 2D-to-3D: an ill-posed problem ▫ Shape from shading, sketch-based modeling, … Creation from scratch is hard: job for skilled artists One of the most central problems in graphics; One of the most discussed at SIG’10 panel

Usable 3D content even harder Models created are meant for subsequent use Creation of readily usable 3D models

Usable 3D content even harder Models created are meant for subsequent use Creation of readily usable 3D models Higher-level information beyond low-level mesh ▫ Part or segmentation information ▫ Structural relations between parts ▫ Correspondence to relevant models, etc. Hard shape analysis problems!

Key: model reuse Reuse existing 3D models and associated info Model-driven approach: creation is driven by or based on existing (pre-analyzed) models

Key: model reuse Reuse existing 3D models and associated info Model-driven approach: creation is driven by or based on existing (pre-analyzed) models Two primary modes of reuse: ▫ New creation via part composition

Key: model reuse Reuse existing 3D models and associated info Model-driven approach: creation is driven by or based on existing (pre-analyzed) models Two primary modes of reuse: ▫ New creation via part composition ▫ New creation as variation or modification of existing model(s), e.g., a warp or a deformation

Modeling by example New models composed by parts retrieved from an existing data repository Key: retrieve relevant parts Many variants … [Funkhouser et al., SIGGRAPH 2004]

Pros and cons Pros: ▫ Significant deviation from existing models ▫ Exploratory modeling via part suggestions [Chaudhuri & Koltun., SIG Asia 2010]

Pros and cons Pros: ▫ Significant deviation from existing models ▫ Exploratory modeling with part suggestions Cons: ▫ Are models composed by parts readily usable?

Pros and cons Pros: ▫ Significant deviation from existing models ▫ Exploratory modeling with part suggestions Cons: ▫ Are models composed by parts readily usable?  structure lost by part composition; how to stitch?

Pros and cons Pros: ▫ Significant deviation from existing models ▫ Exploratory modeling with part suggestions Cons: ▫ Are models composed by parts readily usable?  structure lost by part composition; how to stitch? ▫ Does part exploration always reflect user design intent?

Model-driven creation as variation New creation as variation of existing model(s) Enrich a set; generate “more of the same” … Photo-inspired 3D model creation Inspiration = photographsInspiration = a model set

Model-driven creation as variation New creation as variation of existing model(s) Enrich a set; generate “more of the same” … Inspiration = a model set

Style-Content Separation by Anisotropic Part Scales Kai Xu 1,2, Honghua Li 2, Hao Zhang 2, Daniel Cohen-Or 3 Yueshan Xiong 2, and Zhi-Quan Cheng 2 1 Simon Fraser Universtiy 2 National Univ. of Defense Tech. 3 Tel-Aviv University

Motivation Enrich a set of 3D models with their derivatives Set belongs to the same family or class

Variations in shape parts in the set Geometric or content difference Part proportion (= style) difference

? Style transfer as a derivative Part proportion style

? Style transfer as a derivative Part proportion style

Difficulty with style transfer Style transfer needs part correspondence Part correspondence is difficult ▫ Unsupervised problem ▫ Both content and style variations Variations can be significant!

Work at part and OBB level Parts enclosed and characterized by tight oriented bounding boxes (OBBs)

Style content separation To address both shape variations in the set ▫ Separate treatment of “style” and “content” Style 1 Style 2 Style 3 Content Style

Style transfer as a derivative Creation = filling in the style-content table

Style vs. content Fundamental to human perception ContentStyle LanguageWordsAccents TextLettersFonts Human faceIdentitiesExpressions

Style content separation Previous works on faces, motion, etc. ▫ Prerequisite: data correspondence ▫ Correspondence dealt with independently ▫ Correspondence itself is the very challenge!

Our approach One particular style: Anisotropic part scales or part proportions

Our approach One particular style: Anisotropic part scales or part proportions The approach: Style-content separation with style clustering in a correspondence-free way

Algorithm overview Pipeline Style clustering Co-segmentation Inter-style part correspondence Content classification

Anisotropic part scales Measure style distance between two shapes

Anisotropic part scales Measure style distance between two shapes Part OBB graphs of given segmentation

Anisotropic part scales Measure style distance between two shapes Compute style signatures …… Part OBB graphs of given segmentation

Anisotropic part scales Measure style distance between two shapes …… Part OBB graphs of given segmentation Euclidean distance Compute style signatures

Style distance issues Unknown segmentation Unknown correspondence ? ?

Style distance Search over all part compositions and part counts ……

Style distance For each part count, find minimal distance …… A good signature will return min distance across all part counts to reflect corresponding part decompositions …

Correspondence-free style signature Binary relations: difference of part scales between adjacent OBBs Use Laplacian graph spectra: OBB graph

Correspondence-free style signature Unary attributes: anisotropy of parts Use Laplacian graph spectra: OBB graph linear planar spherical Graph spectra is permutation-free

Style clustering Spectral clustering using style distances

Pipeline Style clustering Co-segmentation Inter-style part correspondence Content classification

Co-segmentation Approach: ▫ Consistent segmentation [Golovinskiy & Funkhouser, SMI 09] ▫ Initial guess: global alignment (ICP) [Golovinskiy & Funkhouser 09]

Co-segmentation Approach: ▫ Consistent segmentation [Golovinskiy & Funkhouser, SMI 09] ▫ Initial guess: global alignment (ICP) We co-segment within a style cluster ▫ Removing non-homogeneous part scaling from analysis [Golovinskiy & Funkhouser 09]

Co-segmentation Approach: ▫ Consistent segmentation [Golovinskiy & Funkhouser, SMI 09] ▫ Initial guess: global alignment (ICP) We co-segment within a style cluster ▫ Removing non-homogeneous part scaling from analysis [Golovinskiy & Funkhouser 09] After style separation

Pipeline Style clustering Co-segmentation Inter-style part correspondence Content classification

Inter-style part correspondence Approach: deform-to-fit ▫ Deformation-driven correspondence [Zhang et al., SGP 08] ▫ Consider common interactions between OBBs 1D-to-1D 1D-to-2D2D-to-2D 2D-to-3D

Inter-style part correspondence Deform-to-fit: appropriate deformation energy Pruned priority-driven search

Pipeline Style clustering Co-segmentation Inter-style part correspondence Content classification

Content classification Use Light Field Descriptor [Chen et al. 2003] Compare corresponding parts Part-level LFDGlobal LFD

Synthesis by style transfer OBBs are scaled Underlying geometry via space deformation content style style transfer

Results: hammers

Results: goblets

Results: humanoids

Results: chairs

Pros and cons Pros: ▫ Automatic generation of many variations ▫ Unsupervised ▫ Deals with anisotropic part scales ▫ Variation = part scaling: structure preservation

Pros and cons Pros: ▫ Automatic generation of many variations ▫ Unsupervised ▫ Deals with anisotropic part scales ▫ Variation = part scaling: structure preservation Cons: ▫ Rely on sufficiently good initial segmentations ▫ Variation does not create new content

Interesting future work Learn and synthesize with generic styles

Model-driven creation as variation New creation as variation of existing model(s) Photo-inspired 3D model creation Inspiration = photographs

Photo-inspired 3D modeling Photo-Inspired Model-Driven 3D Object Modeling Kai Xu 1,2, Hanlin Zheng 4, Hao Zhang 2, Daniel Cohen-Or 3 Ligang Liu 4, and Yueshan Xiong 2 1 NUDT 2 SFU 3 TAU 4 ZJU Conditionally accepted

Overview Input: single photograph + pre-analyzed dataset

Overview 1. Model-driven labelled segmentation of photographed object

Overview 2. Choosing of a candidate model from the database

Overview 3. Silhouette-constrained deform-to-fit of candidate

Overview Output

Structure preservation Any higher-level structural info in the candidate models is preserved during deform-to-fit ▫ Symmetry relations ▫ Part-level correspondence in the set ▫ Controller structures [Zheng et HKUST, EG 11]

Structure preservation Any higher-level structural info in the candidate models is preserved during deform-to-fit ▫ Symmetry relations ▫ Part-level correspondence in the set ▫ Controller structures [Zheng et HKUST, EG 11] Structures also serve to constrain deformation of candidate model

Controller representations Controllers: cuboids and generalized cylinders Relations: symmetry, proximity, etc. Fitting primitives

Controller representations Controllers: cuboids and generalized cylinders Relations: symmetry, proximity, etc. Fitting primitives

Deformation of controllers photo

Controller primitives Deformation of controllers photocandidate model

Controller primitives Deformation of controllers Result of silhouette- driven deform-to-fit photocandidate model

Structure preservation at work symmetry

Structure preservation at work symmetry proximity

Structure preservation at work symmetry proximity optimization

Structure preservation at work symmetry proximity optimization output Short video

Results Guidance in single view but coherent 3D results

Results

The Google chair challenge

Not just chairs …

Pros and cons Pros: ▫ Photos: immensely rich source of inspiration ▫ Silhouette-driven deformation ▫ Variation is less “intrusive” to retain high-level info of source model  readily usable

Pros and cons Pros: ▫ Photos: immensely rich source of inspiration ▫ Silhouette-driven deformation ▫ Variation is less “intrusive” to retain high-level info of source model  more readily usable Cons ▫ Variation does not create new structures

Future work Photo-inspired model deformation only a start Further model refinement, e.g., via sketches

Future work Photo-inspired model deformation only a start Further model refinement, e.g., via sketches Model-driven structure modification

Future work Photo-inspired model deformation only a start Further model refinement, e.g., via sketches Model-driven structure modification Other inspirations for 3D content creation ▫ Sketch-inspired model variation

Future work Photo-inspired model deformation only a start Further model refinement, e.g., via sketches Model-driven structure modification Other inspirations for 3D content creation ▫ Sketch-inspired model variation Style transfer with unknown style in a set

Thank you, 谢谢 TAUZJUNUDT SFU