1. Presented by Matthew Cook INFO410 & INFO350 S2 2015 INFORMATION SCIENCE Paper Discussion: Dynamic 3D Avatar Creation from Hand-held Video Input Paper Discussion: Dynamic 3D Avatar Creation from Hand-held Video Input A Ichim, S Bouaziz & M Pauly

2. INFO410 S2 2015 PAPER DISCUSSION SLIDE 2 INFORMATION SCIENCE Overview  Pipeline for creating rigged, personalised face from hand-held video  Makes use of  Feature tracking  Multi-view stereo depth estimation  Blendshape animation/modelling  New contributions  Factor out environment light to create face albedo map  Build blendshape rig from single camera video input

3. INFO410 S2 2015 PAPER DISCUSSION SLIDE 3 INFORMATION SCIENCE Overview - Stages

4. INFO410 S2 2015 PAPER DISCUSSION SLIDE 4 INFORMATION SCIENCE Static Modelling  From mobile phone video input, generate 2 meshes  High res model of face detail from neutral pose  Low res blendshape model for animating expressions  Later will create one blendshape pose per facial expression

5. INFO410 S2 2015 PAPER DISCUSSION SLIDE 5 INFORMATION SCIENCE Static Modelling  Point cloud generated with ‘Structure from Motion’  Applies features of captured face to a ‘Geometric Morphable Model’

6. INFO410 S2 2015 PAPER DISCUSSION SLIDE 6 INFORMATION SCIENCE Static Modelling  The user is asked to mark complex ridges for increased accuracy  Eyes are created simply by applying the detected iris colour to an existing mesh

7. INFO410 S2 2015 PAPER DISCUSSION SLIDE 7 INFORMATION SCIENCE Albedo Map  An albedo map is a texture map of just the diffuse reflections of a surface  i.e. it tries to eliminate the effect of environment light

8. INFO410 S2 2015 PAPER DISCUSSION SLIDE 8 INFORMATION SCIENCE Dynamic Modelling  Could stop at static modelling and include only ‘virtual’ emotions in virtual environment  But what if you want to use the user’s own expressions?

9. INFO410 S2 2015 PAPER DISCUSSION SLIDE 9 INFORMATION SCIENCE Blendshapes  A mesh of vertices which are weighted to vertices in another mesh  As vertices from the blendshape mesh are moved, the attached vertices from the target are moved, according to their weights  Used to smoothly animate ‘organic’ motions http://area.autodesk.com/tutorials/dragon_slayer

10. INFO410 S2 2015 PAPER DISCUSSION SLIDE 10 INFORMATION SCIENCE Blendshapes  The blendshape models are generated using a combination of feature tracking and model-fitting  As with point registration, users can manually adjust the detected points

11. INFO410 S2 2015 PAPER DISCUSSION SLIDE 11 INFORMATION SCIENCE Detail Synthesis  Additional details can be added after animation from known information such as surface normals

12. INFO410 S2 2015 PAPER DISCUSSION SLIDE 12 INFORMATION SCIENCE Results

13. Potential Exam Questions

14. INFO410 S2 2015 PAPER DISCUSSION SLIDE 14 INFORMATION SCIENCE Exam Questions  Describe how you can estimate the depth of a feature using only a single video stream  What is an albedo map?  What is a blendshape?

15. INFO410 S2 2015 PAPER DISCUSSION SLIDE 15 INFORMATION SCIENCE References  The Paper: http://lgg.epfl.ch/~bouaziz/pdf/Avatars_SIGG15.pdf The Paper: http://lgg.epfl.ch/~bouaziz/pdf/Avatars_SIGG15.pdf