1. Real-Time Audio-Visual Automatic Speech Recognition Demonstrator TSI-TUC, Greece (A. Potamianos, E. Sanchez-Soto, M. Perakakis) NTUA, Greece (P. Maragos, G. Papandreou) INRIA, France (G. Gravier, P. Gros) MUSCLE Showcase: IBC 2007

2. MUSCLE TSI - TUC IBC, Amsterdam, September 2007 Audio-Visual Automatic Speech Recognition Audio Video Recognized Speech Audio-only Automatic Speech Recognition (ASR) degrades under noise Use video for lip-reading to boost ASR performance

3. MUSCLE TSI - TUC IBC, Amsterdam, September 2007 Showcase Main Points Shortcomings of current AV-ASR systems  Research-level set-ups  videos shot under carefully controlled conditions  processing is performed off-line Goal: build a proof-of-concept practically deployable laptop-based AV-ASR prototype which:  uses consumer microphone and camera to capture the speaker  performs visual/audio feature extraction, as well as speech recognition on the laptop in real-time  is robust to failures of a single modality, such as visual occlusion of the speaker's face

4. MUSCLE TSI - TUC IBC, Amsterdam, September 2007 AV-ASR Prototype Architecture Image Acquisition Firewire color camera, 640x480 @25 fps Face detector Adaboost-based, @5 fps HMM-based backend Face tracking & feature extraction Real-time AAM fitting algorithms (Re)initialization System Overview GPU-accelerated processing OpenGL implementation Transcription

5. MUSCLE TSI - TUC IBC, Amsterdam, September 2007 Visual Front-End = = Analyze face expression and appearance Real-time feature extraction algorithms Excellent performance in AV-ASR experiments

6. MUSCLE TSI - TUC IBC, Amsterdam, September 2007 Audio-Visual Speech Recognition Results AVA

7. MUSCLE TSI - TUC IBC, Amsterdam, September 2007

8. Real-Time Audio-Visual Automatic Speech Recognition Demonstrator MUSCLE Network of Excellence More Info: http://cvsp.cs.ntua.gr Funding by EU’s I.S.T. / FP6 Programme IBC 2007