Capturing Facial Details by Space- time Shape-from-shading Yung-Sheng Lo *, I-Chen Lin *, Wen-Xing Zhang *, Wen-Chih Tai †, Shian-Jun Chiou † CAIG Lab, Dept. of CS, National Chiao Tung University * Chunghwa Picture Tubes, LTD. † 1

Outline Introduction Acquisition of facial motion Space-time shape-from-shading Experiment and results Conclusion 2

Introduction Performance-driven method is one of the most straightforward method for facial animation. Expression details, e.g. wrinkles, dimples, are key factors but difficult for motion capture. Original captured imagesDeformation without details 3

Introduction (cont.) Physics-based simulation and blend shape methods try to mimic the details. But, the synthesized details are not the exact expressions. Muscle-based [E. Sifakis et al. 2005] Blend shape [Z. Deng et al. 2006] 4

Introduction (cont.) Our goal is to enhance existing motion capture tech. and capture facial details. SFS With the captured images and directional lighting, our optimization-based shape-from-shading (SFS) can estimate details from shading in video. With facial details Captured video 5

The proposed method Combines the benefits of motion capture and shape-from-shading. Motion capture and stereo reconstruction general geometry accurate on feature points and general geometry. Unreliable corresponding matching at textureless regions Shape-from-shading don’t needcorrespondencefor textureless regions don’t need detailed point correspondence for textureless regions. relative undulation Estimate relative undulation. Sensitive to noise. Space-time shape-from-shading Motion capture + Space-time shape-from-shading. 6

The proposed method 7

Approximate geometry by Mocap Tracking by block matching and stereo reconstruction. Deforming a generic face model by radial-basis functions (RBF). 8

Facial details by SFS Estimating time-varying details by iterative approximating shape V and reflectance R. Input image 9

Space-time constraints Only SFS is not enough. For more reliable detailed motions, we proposed using space-time constraints. 10 Highly sensitive to noise After applying our spatial constraints

Spatial constraints Mostly continuous surface High spatial coherence 11 ∆ For J Є Neighbor(p) ∆ Neighbor(p) denotes the 8-neighbor pixel set ∆ Wj is an adaptive weight ∆ Kcs is the weight for spatial constraints. Reduce the noise noise Ztp Ztj Ztp Ztj noise Ztp Ztj

Still flicker According to biomechanics properties: A human facial surface should gradually transit between expressions. Temporal constraints 12 T0T1 flicker T2 T0 A video image sequence

Space-time shape-from-shading Finally, our objective function becomes 13 spatial constraints + temporal constraints + shading constraints = Space-time shape-from-shading

Performance issue if applied our optimization to the whole face. DOF is too large Assigned some small windows. We preferred areas with more wrinkles and creases. 14 D.O.F=N*M (pixels) *i(frames) N M Video image sequence Fi … F2 F1 F3

Experiment Illumination-controlled (single light source) Two video streams. (HDV, 1280*720,30 fps) We pasted 25 to 30 markers on human face. 15 { C1 } {C2 }

Facial detailed results and Comparison 16

Result of synthesis Generic model: 6078 vertices 6315 polygons 17 deformation (RBF) subdivision per-pixel normal mapping

Result of animation 18

Conclusion & Future work We propose capturing detailed motion by conventional Mocap and advanced shape-from- shading. Doesn’t need additional devices, paint pigments, or restrict the wrinkle shape. With spatial and temporal constraints, our optimal shape-from-shading is more reliable. Reflectance parameters are also estimated. 19

Conclusion & Future work In addition to Phong model, we will extend the concept to other reflectance models. E.g Cook-Torrance BRDF model, BSSRDF, etc) Currently, SFS is only applied to designated segments. An more efficient SFS for the whole face will make our animation more realistic. 20

Thank for your attention! 21 forehead details between eyebrows smile