Image-Based Modeling of Complex Surfaces Todd Zickler DEAS, Harvard University

Image-Based Modeling of Complex Surfaces Appearance

Image-Based Modeling of Complex Surfaces Modeling Appearance: Face Recognition [Harvard Face Database]

Image-Based Modeling of Complex Surfaces Modeling Appearance: Tracking Spacetime Faces [Zhang et al., 2004] Detection/tracking of facial features [Colmenarez et al., 1999]

Image-Based Modeling of Complex Surfaces Modeling Appearance: Applications  Object recognition  Face recognition  Visual tracking  Human-computer interfaces  Compression  Navigation  Surveillance  Metrology  Inspection  Vision for Graphics

Image-Based Modeling of Complex Surfaces Appearance

Image-Based Modeling of Complex Surfaces Non-parametric Appearance: Uniform Sampling? 5 º sampling 1,000,000 images >1x10 6 Mb 1 º sampling 625,000,000 images >1x10 9 Mb

Image-Based Modeling of Complex Surfaces Appearance = Shape + Reflectance SHAPEREFLECTANCE

Image-Based Modeling of Complex Surfaces Reflectance

Image-Based Modeling of Complex Surfaces Outline I = f ( shape, reflectance, illumination ) 1. SHAPE f -1 ( I ) = ?

Image-Based Modeling of Complex Surfaces Outline 1. SHAPE I = f ( shape, reflectance, illumination )

Image-Based Modeling of Complex Surfaces Outline 1. SHAPE + 2. REFLECTANCE

Image-Based Modeling of Complex Surfaces Shape: Restrictive Reflectance Assumptions Shape from shading [Tsai and Shaw, 1994] Variational Stereo [Faugeras and Keriven, 1998] Space Carving [Kutulakos and Seitz, 1998] Multiple-window stereo [Fusiello et al., 1997]

Image-Based Modeling of Complex Surfaces Simple (and Common!) Reflectance Model LAMBERTIAN: IDEALLY DIFFUSE

Image-Based Modeling of Complex Surfaces Example: Conventional Stereo

Image-Based Modeling of Complex Surfaces Example: Conventional Stereo

Image-Based Modeling of Complex Surfaces Reflectance: BRDF

Image-Based Modeling of Complex Surfaces Reflectance: BRDF

Image-Based Modeling of Complex Surfaces Helmholtz Reciprocity [Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977]

Image-Based Modeling of Complex Surfaces Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO

Image-Based Modeling of Complex Surfaces Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO

Image-Based Modeling of Complex Surfaces Reciprocal Images  Specularities “fixed” to surface IlIl IrIr  Relation between I l and I r independent of BRDF

Image-Based Modeling of Complex Surfaces Reciprocity Constraint ^ n vlvl ^ vrvr ^ p olol oror = vlvl ^ vrvr ^ p olol oror ^ n

Image-Based Modeling of Complex Surfaces Reciprocity Constraint ^ n vlvl ^ vrvr ^ p olol oror = vlvl ^ vrvr ^ p olol oror ^ n  Arbitrary reflectance  Surface normal

Image-Based Modeling of Complex Surfaces Reciprocal Acquisition CAMERA LIGHT SOURCE

Image-Based Modeling of Complex Surfaces Recovered Normals [ECCV 2002]

Image-Based Modeling of Complex Surfaces Recovered Surface [ECCV 2002]

Image-Based Modeling of Complex Surfaces Helmholtz Stereopsis: Recent Work Uncalibrated Helmholtz Stereopsis Binocular Helmholtz Stereopsis [CVPR 2003][ICCV 2003]

Image-Based Modeling of Complex Surfaces Outline 1. SHAPE + 2. REFLECTANCE

Image-Based Modeling of Complex Surfaces Spatially-varying Reflectance 5 º sampling: 1,000,000 images; >1x10 6 Mb 1 º sampling: 625,000,000 images; >1x10 9 Mb x 

Image-Based Modeling of Complex Surfaces Point-wise Reflectance Measurement (& Light Fields) Stanford Computer Graphics LaboratoryUSC Institute for Creative Technologies Graphics Lab Mitsubishi Electric Research LaboratoriesUniversity of Illinois Beckman Institute

Image-Based Modeling of Complex Surfaces Reflectance Sharing = Scattered Data Interpolation )()θ,(qfxf     q  x θ  

Image-Based Modeling of Complex Surfaces Application: Human Face

Image-Based Modeling of Complex Surfaces Application: Human Face DIFFUSESPECULAR          N i iikkk qqqplxaqf 1 λ)()()( ~ 

Image-Based Modeling of Complex Surfaces PREDICTED Application: Human Face ACTUALPREDICTED

Image-Based Modeling of Complex Surfaces Predicting Appearance in Real Time

Image-Based Modeling of Complex Surfaces Conclusion SHAPEREFLECTANCE

Image-Based Modeling of Complex Surfaces Future Work 1. Face/object recognition 2. Surface structure at shape/reflectance interface, transparency 3. Appearance over time  weathering, aging 4. Dynamic appearance  Reflectance under deformation