ACL, ECCAI and the Verbmobil/SmartKom Consortia German Research Center for Artificial Intelligence Stuhlsatzenhausweg 3, Geb Saarbrücken Tel.: (0681) /4162 Fax: (0681) WWW: Wolfgang Wahlster ACL Workshop, Hong Kong 7 October 2000

Hosted by DFKI (Thierry Declerck) links to software systems (shareware, freeware, open source, trial versions) all registered software packages are classified software from academic and industrial groups Classification according to NSF/EU „State of the Art in Language Technology“ Cooperation with ELRA, LDC, ELSNET, CLASS, TELRI, BAS The International NL Software Registry of ACL

Russia RAAI Sweden SAIS Switzerland SGAICO Slovenia SLAIS Slovak Republic SSKI SAV Spain AEPIA Catalonia ACIA ECCAI MEMBERS

Founded in 1982, currently 26 member societies Represents more than 6000 AI researchers The world‘s largest AI society Web site: Every two years: ECAI conference and ACAI Summer school ETAI electronic journal, neo-classical reviewing Hardcopy distributed by Royal Swedish Academy of Sciences ECCAI: European Coordination Committee on Artificial Intelligence

UNIVERSITÄT DES SAARLANDES RUHR-UNIVERSITÄT BOCHUM Phase 2 UNIVERSITÄT HAMBURG UNIVERSITÄT KARLSRUHE UNIVERSITÄT BIELEFELD TECHNISCHE UNIVERSITÄT MÜNCHEN FRIEDRICH- ALEXANDER- UNIVERSITÄT ERLANGEN-NÜRNBERG UNIVERSITÄT STUTTGART RHEINISCHE FRIEDRICH WILHELMS-UNIVERSITÄT BONN LUDWIG MAXIMILIANS UNIVERSITÄT MÜNCHEN TU-BRAUNSCHWEIG EBERHARDT-KARLS UNIVERSITÄT TÜBINGEN  W. Wahlster, DFKI The Verbmobil Consortium

Multilingual and Mobile Communication Assistants Multimodal Interfaces SmartKom Speech-based Web Access to Multilingual Web pages WAP Phones WebTV Multilingual Audio Retrieval and Audio Mining Discussions Lecture Notes Organizers Multilingual Indexing and Annotation of Videos Video Archives News Archives Call Centers ECommerce Mobile Travel Assistance Telephone Translations Verbmobil Dialog Translation International Research Trends in Multilingual Systems Multilingual Language Technology S peech Recognition, Language Understanding, Language Generation, and Speech Synthesis Multilingual Language Technology S peech Recognition, Language Understanding, Language Generation, and Speech Synthesis Spontaneous Speech, Robust Processing and Translation, Semantic and Pragmatic Understanding

Verbmobil‘s Massive Data Collection Effort Transliteration Variant 1 Transliteration Variant 2 Lexical Orthography Canonical Pronounciation Manual Phonological Segmentation Automatic Phonological Segmentation Word Segmentation Prosodic Segmentation Dialog Acts Noises Superimposed Speech Syntactic Category Word Category Syntactic Function Prosodic Boundaries The so-called Partitur (German word for musical score) orchestrates fifteen strata of annotations 3,200 dialogs (182 hours) with 1,658 speakers 79,562 turns distributed on 56 CDs, 21.5 GB

Treebank: 85,000 Trees 79,562 turns 1,520,000 words 76,210 dialog acts Collection Sites: All over Germany American English: CMU Japanese: ATR Mandarine: Philips Taiwan The Verbmobil Corpora

Machine Learning for the Integration of Statistical Properties into Symbolic Models for Speech Recognition, Parsing, Dialog Processing, Translation Transcribed Speech Data Segmented Speech with Prosodic Labels Annotated Dialogs with Dialog Acts Treebanks & Predicate- Argument Structures Aligned Bilingual Corpora Hidden Markov Models Neural Nets, Multilayered Perceptrons Probabilistic Automata Probabilistic Grammars Probabilistic Transfer Rules Extracting Statistical Properties from Large Corpora

MediaInterface European Media Lab Uinv. Of Munich Univ. of Stuttgart Saarbrücken Aachen Dresden Berkeley Stuttgart MunichUniv. of Erlangen Heidelberg Main Contractor Project Management Testbed Software Integration DFKI Saarbrücken The SmartKom Consortium: Project Budget: $ 34 M Project Duration: 4 years D AIMLER C HRYSLER Ulm SmartKom: Intuitive Multimodal Interaction  W. Wahlster, DFKI

User(s) Media Analysis Design Media Fusion Output Rendering Representation and Inference User Model Discourse Model Domain Model Task Model Media Models Interaction Management Media Analysis Input Processing Information Applications People Intention Recognition Media Design Application Interface Discourse Modeling User Modelin g Presentation Design Language Graphics Gesture Biometrics Language Graphics Gesture Animated Presentation Agent The Architecture of the SmartKom Agent (cf. Maybury/Wahlster 1998)  W. Wahlster, DFKI

SmartKom-Home/Office: A Versatile Agent-based Interface SmartKom-Public: A Multimodal Communication Booth SmartKom-Mobile: A Handheld Communication Assistant Media Analysis Kernel of SmartKom Interface Agent Interaction Management Application Manage- ment Media Design SmartKom: A Transportable and Transmutable Interface Agent  W. Wahlster, DFKI

Smartcard/ Credit Card for authentication and billing Docking station for PDA/Notebook/ Camcorder high speed and broad bandwidth Internet connectivity High-resolution scanner Loudspeaker Room microphone Face-tracking camera Virtual touchscreen protected against vandalism Multipoint video conferencing SmartKom-Public: A Multimodal Communication Booth  W. Wahlster, DFKI

SmartKom Domains: EPG and Cinema Selection Multimodal Selection Of Cinema Multimodal Selection of Seats

Camera GPS Microphone Loudspeaker Stylus-Activated Sketch Pad Wearable Compute Server Docking Station for Car PC Biosensor for Authentication & Emotional Feedback GSM for Telephone, Fax, Internet Connectivity SmartKom-Mobile: A Handheld Communication Assistant  W. Wahlster, DFKI

SmartKom‘s Data Collection of Multimodal Dialogs User Side-view Camera Face-tacking Camera with Microphone Environmental Noise Microphone Array Screen Projected Webpage Face-tacking Camera Loudspeaker Microphone Array User Bird’s-eye Camera LCD Beamer SIVIT- Camera

BAS in Munich (Prof. Tillmann, Dr. Schiel) 68 GB of transliterated speech data 590 hours of speech, 2500 different speakers works for Verbmobil and SmartKom consortium treebanks, gesture and dialog act annotations very large corpora of GSM phone dialogs multi-channel recordings of spontaneous dialogs WOZ data one year exclusive use for German labs and industry after one year distribution by ELRA and then by LDC The German Center for Language Resources