1Seminar „Multimodale Räume“ Uni Karlsruhe, The FAME Project Acronym:Facilitating Agent for Multicultural Exchange Partners: Universität Karlsruhe,INPG Grenoble, UJF Grenoble,ITC-irst Trento, UPC Bacelona,SONY Europe, Stuttgart ATLAS Barcelona Project volume: 5.5 M Euro Duration: 40 months, started October 2001 More info:

2Seminar „Multimodale Räume“ Uni Karlsruhe, The FAME Projekt Facilitating Agent for Multicultural Exchange Volume: Duration: 40 months (since October 2001) Financial Volume:  5,5 Mio. € currently approx. 30 scientists Partners: Uni Karlsruhe, INPG Grenoble, UJF Grenoble, UPC Barcelona, ITC-irst Trento, Stuttgart, Barcelona

3Seminar „Multimodale Räume“ Uni Karlsruhe, Project Goals long term vision : facilitate communication between humans reduce the workload on the users of technical equipment observe humans and their activities in an intelligent room and serve as a context-aware information butler FAME project goal : provide and integrate core technologies (video and speech perception, augmented reality, translation, information retrieval) to show feasibility of the concept demonstrate system at fair scenario 1 (lecture scenario): one person is giving a talk or lecture or presentation scenario 2 (meeting scenario): several people are discussing / working on a common task

4Seminar „Multimodale Räume“ Uni Karlsruhe, The FAME Showcases scenario 1 (presentation)scenario 2 (meeting) use A/V equipment intelligent cameraman presentation tracking summarisation + archiving translation, crosslingual IR augmented reality video-based activity tracking topic spotting information butler service: planning of fair visit

5Seminar „Multimodale Räume“ Uni Karlsruhe, The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) FAME outside viewreception by FAME-guy meeting insidepeople mention topics room gives information about spotted topics

6Seminar „Multimodale Räume“ Uni Karlsruhe, The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) at the phicon wallgesturesthe projection table multimodal input on table output also on the wall interactive visit planning borrow a camera for photographs of the visit

7Seminar „Multimodale Räume“ Uni Karlsruhe, The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) Back from the visit in the FAME room: dowload and look at photos select, print, save photos using phicon interaction record testimony intelligent cameraman, presentation tracker take home photos and information about FAME

8Seminar „Multimodale Räume“ Uni Karlsruhe, multimodal environment context-aware intelligent camera-man automatically track people and their activities augmented reality environment move physical icons (phicons) on table/wall, and interact with projection on table/wall spontaneous speech recognition (with distant microphones) translation and crosslingual information retrieval in European-English, Catalan, and Spanish dialog and context model Important Components

9Seminar „Multimodale Räume“ Uni Karlsruhe, Microphon-Array (Speaker Lokalization) Microphon-Array (Speaker Lokalization) Smartboard as Projection Wall Smartboard as Projection Wall Livingroomg IR-Remote Control X-10 Illumination Audio Signals Several Beamers 4 Cameras TV/Video Loudspeakers Microphones Multimodal Environment at UKA

10Seminar „Multimodale Räume“ Uni Karlsruhe, Augmented Reality Table project virtual reality on real table move around physical icons (multiple users) interact with projection select, move, rotate, resize, delete, change color write on table, pass notes to others, point to items

11Seminar „Multimodale Räume“ Uni Karlsruhe, Intelligent Camera Man follow speaker while talking and moving around detect interaction from audience zoom on area of interest e.g. when pointing somewhere or showing something

12Seminar „Multimodale Räume“ Uni Karlsruhe, Lecture Supporter track lecture or presentation operate FAME room equipment by speech commands automatically switch slides during presentation automatically create transcript of lecture create summary, translate to other languages record and store all lectures in searchable database retrieve and browse through previously recorded lectures

13Seminar „Multimodale Räume“ Uni Karlsruhe, Links wichtige Wörter wichtige Wörter wichtige Wörter TO THE RECOGNITION OF THE CONTINUOUS SPEECH RECOGNITION IN NOISY AND PATTERN RECOGNITION NOT IN SECURING DUE RECOGNITION AND RESPECT CLASS 32 CLASS 14 CLASS 57 CLASS 6 CLASS 70 top n most frequent 40k words least frequent 20k words 60k vocabulary HUB-4 corpus trigramm- model P ( w n | w n -1, w n -2 ) 72 classes P ( C n | w n -1, w n -2 ) · P ( w n | C n ) add to classes estimate presentation- slides important words tf-idf 100 links ± 2 contexts perplexity scores Adaptation Overview 20% fewer errors

14Seminar „Multimodale Räume“ Uni Karlsruhe, Welcome in Barcelona in Summer 2004