Talk and Look: Tools for Ambient Linguistic Knowledge A Project funded by the European Community under the Sixth Framework Programme for Research and Technological Development (IST ) D7.3 Public Project Presentation (update 2006)

2 © TALK Consortium, 2006 The problem

3 © TALK Consortium, 2006 The problem New technologies should make life easier. But... our high-tech environment makes ever greater demands on people We need … natural conversational interaction instead of complex controls and operating instructions

4 © TALK Consortium, 2006 TALK objectives Natural communication requires  Content: users say what they want  Flexibility: users say it the way they want, with no need to learn the specific commands a device “expects”  Adaptivity: system adapts to preferences, knowledge, and ability of user, and to context  Learning: system uses dialogue strategies learned from experience, and continues to learn from its interactions with users

5 © TALK Consortium, 2006 Background of the TALK project TALK ISU theory: Information State Updates for dialogue context management Reconfigurable systems Adaptivity, Learning of dialogue strategies Multimodal dialogue management Task domains: in-car, smart home

6 © TALK Consortium, 2006 TALK consortium Overall Coordination Scientific Coordination

7 © TALK Consortium, 2006 User categories Industrial users  Developers of dialogue applications who need better dialogue design and more efficient development techniques  Information about industrial user requirements through feedback by industrial partners in the consortium End users  anybody with a car or a home  Information about end user requirements through  Industrial partners  Market studies (e.g. JDPower & Assoc, Cisco Systems)  Project-internal wizard-of-oz experiments and usability tests

8 © TALK Consortium, 2006 Major exploitable results - Technologies Methods for designing better – flexible and adaptive – dialogue systems that learn from interactions with users, based on ISU technology Methods for rapid and cost-effective deployment of new dialogue applications through reconfigurable dialogue systems:  Separating domain-specific information from generic communicative behavior  Separating central aspects of dialogue structure from modality- and language-specific realisation

9 © TALK Consortium, 2006 Subsidiary exploitable resuls  Reusable software tools (e.g. TrindiKit, ATK, GF, DIPPER)  Reusable annotated data archives and databases of application-specific knowledge, multimodal grammar library  Design and methodology for conducting multimodal wizard- of-oz experiments, user testing of baseline in-car systems  Contributions to standards (W3C, ISO, standards workshop with AMI project and W3C 12/12/05)  Framework and system for ontology-based, generic presentation planning (DFKI/USAAR In-Car showcase, PATE/RDFS)  Skills development: training PhD and Master students

10 © TALK Consortium, 2006 Research results and resources (December 2005) Corpora (annotated data archive):  SACTI 1 and 2 (UCAM)  MIMUS (USEV)  Sammie 1 and 2 corpora (DFKI/USAAR): MP3 WOZ corpora annotated at various levels  ISU-annotated COMMUNICATOR data (UEDIN) Research results (highlights):  Multilingual & multimodal grammars (e.g Tram Info and AgendaTalk dialogue systems, D1.2b)  The Hidden Information State model (D4.3)  Reinforcement Learning of dialogue strategies that outperform all COMMUNICATOR systems (D4.1) Software: ATK, Trindikit 4, GF 2.4, Reinforcement Learner, User Simulations, Automatic ISU annotator, several dialogue systems

11 © TALK Consortium, 2006 Exploitation strategy Open source software  530 TrindiKit downloads, 413 GF downloads, 2985 ATK downloads, DIPPER 66 downloads  approx 43,000 hits on TALK website in december 05 Continuous transfer of exploitable results to Industrial partners Business development executive (UEDIN)  Funded by Scottish Enterprise (national economic development agency)  Full time effort devoted to transfer of research in speech and language technology from University to industry (Licensing, Consulting, Joint projects, Spinouts)  Similar service by Contact Office for Knowledge and Technology Transfer (USAAR)

12 © TALK Consortium, 2006 Dissemination Strategy: Three target groups Research community: via scientific publications  11 refereed journal papers, 38 refereed conference papers  18 talks and invited lectures  Standards workshop: TALK/AMI/W3C December 2005 General public:  More than 15 press articles  Press release, internet news releases, TV presentations, public demonstrations Industry:  Uptake by industrial partners, testing by BMW and BOSCH  Engagement with commercial R&D teams  Planned presentations at Trade Fairs (e.g. CeBIT) Training:  5 PhD (4 completed), 19 MSc (6 completed)  Anzere Spring School, ESSLLI 2006 advanced lecture course Project website:

13 © TALK Consortium, 2006 Milestones Major milestones achieved in 2005:  In-car systems Version 1.0  Adaptive dialogue system trained with human data  International press release  Standardization workshop with AMI and W3C (12/12/05)  User testing of the baseline in-car systems Future milestones:  Reconfigurable systems (early 2006)  TALK workshop at a major conference (2006)  Advanced lecture course at ESSLLI 2006  Evaluation of learned versus hand-coded systems  Final evaluation of the in-car showcase (late 2006) Long-term effects:  Scientific community will build on TALK results, standards, tools and resources (as happened with TRINDI and SIRIDUS)  Industry will use TALK methods and tools for research and development