1. A Prototype System for 3D Dynamic Face Data Collection by Synchronized Cameras Yuxiao Hu Hao Tang

2. Problem Statement Collect multi-view face video with expressions; Potential researches  Co-articulation of facial expression and lip movement:  Non-frontal view audio/visual speech recognition-lip reading

3. Relevant Works Static 2D face databases: FERET, CMU PIE, ORL, Yale Database, UMIST, etc Static 3D face databases: 3D-RMA, GavabDB, YorkDB, XM2VTS database, FRGC database,etc 3D Dynamic face databases:  CMU FIA, no markers, no audio  Intel Research China Database, not synchronized

4. Highlights Total Solution: Both hardware and software MultiView+Synchronization+RealTime Flexibility: Flexibly extended from 2 cameras to 5 cameras; Supplementary Tools:  camera calibration,  color space conversion,  2D facial feature tracking  3D face shape recovery

5. Physical Setup FoamHead

6. Data Specification Video Quality  Multi-Views Video (2~5 Cameras)  Resolution: 640*480, 100*100 for face region at least  Frame Rate: at least 30 frame/second Synchronization  Cross Cameras  Cross IEEE Buses Color Representation  GRBG Bayer Pattern  24 bitRGB

7. System Diagram Camera Calibration Video Data Capture Color De-mosaicing Facial Feature Tracking 3D Shape Reconstruction

8. Synchronization-Hardware Configuration DragonFly Camera

9. Y Synchronization-Software Implementation … Buffers Buffer Overrun? Buffer Overrun? Buffer Overrun? Buffer Overrun? Time Stamp Matched? Time Stamp Matched? … Y Y Y N N Re-sync Compression AVI

10. Offline Color De-mosaic Raw Data: Color represented in Sparse (Stippled) Pattern Reconstructed RGB color image Raw Data Reconstructed RGB color video

11. Camera Calibration Find the intrinsic and extrinsic parameters Use Camera Calibration Toolbox for Matlab Two-step procedure  Find projection matrix using Direct Linear Transformation  Use as initialization for nonlinear minimization of mean squared re-projection error

12. Camera Calibration (cont ’ ed) Camera 1Camera 2 Camera 1 Camera 2

13. Camera Calibration (cont ’ ed)

14. Camera 1

15. Camera Calibration (cont ’ ed) Camera 1 Camera 2 Average re-projection error < 0.2 pixels (0.16279,0.13482) and (0.16439,0.12685)

16. Facial Marker Tracking Simple but effective tracking algorithm

17. Facial Marker Tracking (cont ’ ed) Statistical marker collocation model

18. Facial Marker Tracking (cont ’ ed)

20. 3D Reconstruction: Stereo Triangulation A bit of theory

21. 3D Reconstruction: Stereo Triangulation (cont ’ ed)

22. Deliveries The data acquisition system of camera array Tools for Color De-mosaicing The calibration data and tools Some sample data result 3D ground truth data and labeling tool Technical Report

23. Outline (4Ws+2Hs) Why (do we do this?) Who (has done the related work?) What (we proposed to do?) How (did we achieve our goal?) Why (we need to do so?) How (we evaluate our work?)