Model-Based Stereo with Occlusions Fabiano Romeiro and Todd Zickler TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAA

Introduction Varying illumination Varying pose Occlusions Varying expressions

Introduction Past Work: Image-based For example: Eigenfaces [Turk and Pentland, 1991] Fisherfaces [Belhumeur et al, 1997]

Introduction Past Work: Model-based 3D Morphable Models (3DMMs) [Blanz et al, 1999; Blanz et al, 2003; Blanz et al, 2005; Smet et al, 2006] 2D AAMs [Cootes et al, 1998; Baker et al, 2004; Mathews et al, 2004; Gross et al, 2006] 2D+3D AAMs [Xiao et al, 2004]

Introduction Past Work: 3DMMs 3D Morphable Models (3DMMs) Pros [Blanz et al, 1999; Blanz et al, 2003; Blanz et al, 2005; Smet et al, 2006] Pros - Self-occlusion handled by model itself - Allows direct modeling of illumination Cons - Difficult and expensive fitting process

Introduction Past Work: Stereo 3DMMs Our work [Fransens et al, 2005] - Stereo based cost - Texture model not needed Our work Stereo fitting with both shape and texture → Robust to foreign-body occlusions → Improved Accuracy

Outline 3DMM Background Joint Shape and Texture Stereo Fitting Handling Occlusions Conclusions

Background 3DMMs Vectorization of laser scans: PCA performed: Basis for shape - Basis for texture [Blanz and Vetter, 1999]

Background 3DMMs Representation of face shape and texture: [Blanz and Vetter, 1999] Prior probabilities on the coefficients:

Stereo Match

Texture Match

Joint Shape and Texture Stereo Fitting of 3DMMs

Robust Stereo Fitting of 3DMMs

Optimization Procedure Initial fit - Fit Shape, Pose to minimize reprojection error on selected feature points - Rough initial estimates of Shape and Pose Optimization procedure 4 experiments Stereo and texture Stereo Robust Stereo and texture Robust Stereo

Results First 2 experiments: Stereo and Texture vs. Stereo 480 recovered shape models (60 individuals, 8 poses) K.U. Leuven Stereo face database [Fransens et al, 2005]

Results Qualitative Results Stereo and Texture Stereo

Results Qualitative Results Stereo+texture Stereo

Results Quantitative Results

Results Under Occlusions Half-Occlusion Full Occlusion near Full-Occlusion far

Results Under Occlusions Input Robust Stereo Robust S+T Robust Stereo Shape Estimate Occlusion Map

Conclusions Robust stereo fitting of 3DMMs - Uses both stereo constraint, texture information Increased accuracy of fit - Ability to handle occlusions Future Work - More sophisticated stereo matching term - Different feature spaces - Break model into segments respecting occlusion boundaries