1. Computer Vision in Graphics Production Adrian Hilton Visual Media Research Group Centre for Vision, Speech and Signal Processing University of Surrey http://www.ee.surrey.ac.uk/CVSSP/VMRG

2. Overview Where is computer vision useful in content production? Where is vision used? What can and can’t current computer vision do?

3. Computer Vision in Content Production Computer vision: analysis & interpretation of real images/video Match moving Matting Model building Human motion capture Studio Production Facial animation Image-based illumination

4. Match Moving Reconstruction of camera movement to composite CG Products: 2d3, RealViz … (semi-automatic camera tracking) … standard tool in film production Oxford University/2d3

5. Matting Separation of foreground and background objects - actor/background separation - wire/set removal Studio: chroma-key (solved) Post-production: Imagineer, RealViz, …. (open-problem)

6. Model Building Capture of real shape for CG models objects environments characters Shape capture active sensors (laser/structured-light) - accurate 3D surface measurements - static objects or environments structure from images - low-accuracy - static objects Problem: unstructured surface measurements

7. Model Building (2) Walking with Dinosaurs - FrameStore

8. Structured Models

9. Courtesy Stanford Computer Graphics Lab. Animation of David

10. Human Motion Capture Capture of real motion Marker based systems widely used in performance animation whole-body/face character animation (Golum) ‘realistic’ motion characteristics accurate real-time? widely used in film production (with post-production)

11. Human Motion Capture(2) Markerless human motion capture advantages: unintrusive; simultaneous appearance capture model-based visual tracking low-accuracy visual ambiguity (uniform apperance, non-rigid shape)

12. Human Motion Capture(3)

13. Studio Production Modelling actors 3D animated models animation re-illumination loss of visual realism to captured images Free-viewpoint video post-production of arbitrary camera views/paths ‘matrix’ flowmo shots aim: quality equivalent to captured video

14. Studio Production (2)

16. Limitations: - shape detail face/hands/hair - not video quality Studio Production (3)

17. Studio Production: Free-view video

21. Facial Animation Production of photo-realistic faces 2D video-based approaches video rewrite [Bregler’97] concatenative synthesis [Cossato’98,Ezzat’02] photo-realistic limited viewpoint, illumination 3D markers Performance animation [Phigin’98] shape only 3D video (shape+appearance) concatenative synthesis photo-realistic control of viewpoint, illumination

22. 3D Video

23. 3D Video Database 51 people: Expressions + Speech (short sentence)

24. Initial Face Synthesis from Speech Input: Output:

25. Image-based illumination Illumination of CG scenes with real illumination high-dynamic range capture of illumination environment mapping illumination to CG scene widely used in production Debevec SIGGRAPH’00 FiatLux

26. Computer Vision in Content Production Match moving - yes, automatic camera tracking available Matting - partial, many unsolved problems Model capture – yes, tools for semi-automatic restructuring Human motion capture – no, inaccurate Studio Production – partial, free viewpoint video Facial animation – yes, 3D video Image-based illumination – yes, widely used Other applications of computer vision….

27. Future Directions Video-based rendering photo-realistic, free-viewpoint rendering of dynamic scenes relighting required Video-based animation animation from captured video (face,whole-body…) control of motion, viewpoint, illumination

28. People Jon Starck, Joel Mitchelson, Gordon Collins, Eng-Jon Ong, Ioannis Ypsilos, Rob Dilks, Michael Kalkavouras Collaborators BBC, BT, Sony, Canon, Philips, Hensons, Framestore, Snell&Wilcox, 3D Scanners, AvatarMe EPSRC – DTI