A Prototype System for 3D Dynamic Face Data Collection by Synchronized Cameras Yuxiao Hu Hao Tang.

Slides:

Advertisements

Similar presentations

The fundamental matrix F

Advertisements

Lecture 11: Two-view geometry

Image acquisition using sparse (pseudo)-random matrices Piotr Indyk MIT.

A Unified Approach to Calibrate a Network of Camcorders & ToF Cameras M 2 SFA 2 Marseille France 2008 Li Guan Marc Pollefeys {lguan, UNC-Chapel.

For Internal Use Only. © CT T IN EM. All rights reserved. 3D Reconstruction Using Aerial Images A Dense Structure from Motion pipeline Ramakrishna Vedantam.

Some problems... Lens distortion  Uncalibrated structure and motion recovery assumes pinhole cameras  Real cameras have real lenses  How can we.

www-video.eecs.berkeley.edu/research

Face Alignment with Part-Based Modeling

Computer vision: models, learning and inference

Learning to estimate human pose with data driven belief propagation Gang Hua, Ming-Hsuan Yang, Ying Wu CVPR 05.

Multiple View Reconstruction Class 24 Multiple View Geometry Comp Marc Pollefeys.

Amir Hosein Omidvarnia Spring 2007 Principles of 3D Face Recognition.

Mean transform, a tutorial KH Wong mean transform v.5a1.

RECOGNIZING FACIAL EXPRESSIONS THROUGH TRACKING Salih Burak Gokturk.

Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.

Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.

Flexible Bump Map Capture From Video James A. Paterson and Andrew W. Fitzgibbon University of Oxford Calibration Requirement:

Outdoor Motion Capturing of Ski Jumpers using Multiple Video Cameras Atle Nes Faculty of Informatics and e-Learning Trondheim University.

CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai.

W M AM A I AI IM AIM Time (samples) Response (V) True rating Predicted rating  =0.94  =0.86 Irritation Pleasantness.

Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.

Single-view geometry Odilon Redon, Cyclops, 1914.

Camera Calibration from Planar Patterns Homework 2 Help SessionCS223bStanford University Mitul Saha (courtesy: Jean-Yves Bouguet, Intel)

CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.

Smart Traveller with Visual Translator for OCR and Face Recognition LYU0203 FYP.

CSE473/573 – Stereo Correspondence

Jacinto C. Nascimento, Member, IEEE, and Jorge S. Marques

Capturing the Motion of Ski Jumpers using Multiple Stationary Cameras Atle Nes Faculty of Informatics and e-Learning Trondheim University.

Camera Parameters and Calibration. Camera parameters From last time….

CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.

Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2006 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.

ICBV Course Final Project Arik Krol Aviad Pinkovezky.

Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.

Automatic Camera Calibration

Computer vision: models, learning and inference

Camera Calibration & Stereo Reconstruction Jinxiang Chai.

A Brief Overview of Computer Vision Jinxiang Chai.

Geometric and Radiometric Camera Calibration Shape From Stereo requires geometric knowledge of: –Cameras’ extrinsic parameters, i.e. the geometric relationship.

Structure from images. Calibration Review: Pinhole Camera.

Abstract Some Examples The Eye tracker project is a research initiative to enable people, who are suffering from Amyotrophic Lateral Sclerosis (ALS), to.

Ondřej Rozinek Czech Technical University in Prague Faculty of Biomedical Engineering 3D Hand Movement Analysis in Parkinson’s Disease

Advanced Computer Technology II FTV and 3DV KyungHee Univ. Master Course Kim Kyung Yong 10/10/2015.

Scientific Writing Abstract Writing. Why ? Most important part of the paper Number of Readers ! Make people read your work. Sell your work. Make your.

CSCE 643 Computer Vision: Structure from Motion

Presented by Daniel Khashabi Joint work with Sebastian Nowozin, Jeremy Jancsary, Andrew W. Fitzgibbon and Bruce Lindbloom.

Multiview Geometry and Stereopsis. Inputs: two images of a scene (taken from 2 viewpoints). Output: Depth map. Inputs: multiple images of a scene. Output:

Video Data Hiding using Forbidden Zone and Selective Embedding Submitted Under Team Members.

Lec 22: Stereo CS4670 / 5670: Computer Vision Kavita Bala.

Single-view geometry Odilon Redon, Cyclops, 1914.

Source: Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on Author: Paucher, R.; Turk, M.; Adviser: Chia-Nian.

CSE 140: Computer Vision Camillo J. Taylor Assistant Professor CIS Dept, UPenn.

3D Reconstruction Using Image Sequence

Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs.

Single-view geometry Odilon Redon, Cyclops, 1914.

Observing Lip and Vertical Larynx Movements During Smiled Speech (and Laughter) - work in progress - Sascha Fagel 1, Jürgen Trouvain 2, Eva Lasarcyk 2.

An Introduction to Digital Image Processing Dr.Amnach Khawne Department of Computer Engineering, KMITL.

Facial Recognition By Lisa Tomko.

Lec 26: Fundamental Matrix CS4670 / 5670: Computer Vision Kavita Bala.

Bayer Color Filter Array Demosaicing

Design and Calibration of a Multi-View TOF Sensor Fusion System Young Min Kim, Derek Chan, Christian Theobalt, Sebastian Thrun Stanford University.

Visual Information Processing. Human Perception V.S. Machine Perception  Human perception: pictorial information improvement for human interpretation.

CS4670 / 5670: Computer Vision Kavita Bala Lec 27: Stereo.

PRESENTED BY Yang Jiao Timo Ahonen, Matti Pietikainen

Overview What is Multimedia? Characteristics of multimedia

Coding Approaches for End-to-End 3D TV Systems

Mean transform , a tutorial

Image Segmentation Techniques

Single-view geometry Odilon Redon, Cyclops, 1914.

Fig. 2 System and method for determining retinal disparities when participants are engaged in everyday tasks. System and method for determining retinal.

Fig. 2 System and method for determining retinal disparities when participants are engaged in everyday tasks. System and method for determining retinal.

Presentation transcript:

A Prototype System for 3D Dynamic Face Data Collection by Synchronized Cameras Yuxiao Hu Hao Tang

Problem Statement Collect multi-view face video with expressions; Potential researches  Co-articulation of facial expression and lip movement:  Non-frontal view audio/visual speech recognition-lip reading

Relevant Works Static 2D face databases: FERET, CMU PIE, ORL, Yale Database, UMIST, etc Static 3D face databases: 3D-RMA, GavabDB, YorkDB, XM2VTS database, FRGC database,etc 3D Dynamic face databases:  CMU FIA, no markers, no audio  Intel Research China Database, not synchronized

Highlights Total Solution: Both hardware and software MultiView+Synchronization+RealTime Flexibility: Flexibly extended from 2 cameras to 5 cameras; Supplementary Tools:  camera calibration,  color space conversion,  2D facial feature tracking  3D face shape recovery

Physical Setup FoamHead

Data Specification Video Quality  Multi-Views Video (2~5 Cameras)  Resolution: 640*480, 100*100 for face region at least  Frame Rate: at least 30 frame/second Synchronization  Cross Cameras  Cross IEEE Buses Color Representation  GRBG Bayer Pattern  24 bitRGB

System Diagram Camera Calibration Video Data Capture Color De-mosaicing Facial Feature Tracking 3D Shape Reconstruction

Synchronization-Hardware Configuration DragonFly Camera

Y Synchronization-Software Implementation … Buffers Buffer Overrun? Buffer Overrun? Buffer Overrun? Buffer Overrun? Time Stamp Matched? Time Stamp Matched? … Y Y Y N N Re-sync Compression AVI

Offline Color De-mosaic Raw Data: Color represented in Sparse (Stippled) Pattern Reconstructed RGB color image Raw Data Reconstructed RGB color video

Camera Calibration Find the intrinsic and extrinsic parameters Use Camera Calibration Toolbox for Matlab Two-step procedure  Find projection matrix using Direct Linear Transformation  Use as initialization for nonlinear minimization of mean squared re-projection error

Camera Calibration (cont ’ ed) Camera 1Camera 2 Camera 1 Camera 2

Camera Calibration (cont ’ ed)

Camera 1

Camera Calibration (cont ’ ed) Camera 1 Camera 2 Average re-projection error < 0.2 pixels ( , ) and ( , )

Facial Marker Tracking Simple but effective tracking algorithm

Facial Marker Tracking (cont ’ ed) Statistical marker collocation model

Facial Marker Tracking (cont ’ ed)

3D Reconstruction: Stereo Triangulation A bit of theory

3D Reconstruction: Stereo Triangulation (cont ’ ed)

Deliveries The data acquisition system of camera array Tools for Color De-mosaicing The calibration data and tools Some sample data result 3D ground truth data and labeling tool Technical Report

Outline (4Ws+2Hs) Why (do we do this?) Who (has done the related work?) What (we proposed to do?) How (did we achieve our goal?) Why (we need to do so?) How (we evaluate our work?)