Multi-modal Interfaces “ “The focus of IST in FP6 is on the future generation of technologies in which computers and networks will be integrated into the everyday environment, rendering accessible a multitude of services and applications through easy-to-use human interfaces“ IST WP

Our Mission The challenge is: To make interfaces as simple as possible, and interaction as easy as possible For citizens to be able to access, receive and use information in their own language We support the research, development and integration of: Advanced technologies for interfaces & interaction Integrated multilingual information systems and services We focus on: Technology-mediated interaction and communication between people, between their devices, and between people and content Combining traditional human language services, like human translation and interpretation, and advanced language technologies and resources E1 Interfaces

The Facts Call Title: IST Call 5 Call Identifier: FP IST-5 Strategic Objective: Multi-modal Interfaces Open: 17 May 2005 Close: 21 September 2005 at (Brussels local time) Budget: 54 M€ Instruments: IP’s, STReP’s New/Traditional Instruments: 60/40

Recommendations for Work Programme 05-06: Multi-modal interaction Multi-modal integration, e.g. fusion Robustness and acceptability of “mature” modalities Unconventional modalities Multilingual communication New learning paradigms in (spoken) language translation and portability to new languages, including language resources Online, real-time search/detection and off-line browsing of multi-modal information in multiple languages Multi-modal Interfaces Multi-modal Interfaces, 2004 emphasise multi-user scenarios and non-business applications, e.g. entertainment, learning and home applications

Quotes: IT is “in a state that we should be ashamed of, it’s embarrassing” To be truly successful, a complex technology needs to “disappear” People will no longer be sitting 2 feet away from a computer screen with a keyboard, but 10 feet away with a remote control Both Google and the iPod are “interfaces” – metaphorical gateways through which humans enter and navigate around a technology. Both are examples of simplicity concealing complexity underneath. In computing “the holy grail of simplicity is I-just-wanna-talk-to- my-computer, and it should simply anticipate my needs” Make it simple The Economist, 30 th Oct. 2004

Main challenges Need to provide information artefacts to be used by increasingly diverse user groups Expanding context of use, from scientific and business use to residential and nomadic use Increasing variety of media and devices used to access a community-wide pool of services and information resources Demonstrate added-value of multi-modality and performance advances through multi-modal interaction Go beyond system integration for demonstration purposes and target real world solutions Emphasise multi-user scenarios and non-business applications Stronger focus on defining the expected impact Need to promote cooperation between small European companies Promote research that could lead to spin-offs

An R&D agenda Key interaction technologies (speech, natural language input, haptics, vision, sensors, gesture, agents, …, including unconventional modalities) and their modelling and integration into interaction platforms Interface architectures, tools for cost effective integration, e.g. fusion, and toolkit interoperability Computational models of multi-modality, and advanced corpora for research Re-develop a serious effort on automatic translation Reinforce work on robustness and acceptability of speech recognition and research on speaker-independent software New learning paradigms in (spoken) language translation and portability to new languages, including greater focus on language resources as computer processable and reusable resources Design processes, methods and tools that support scalability and modality independence Empirical science base for understanding end-users, e.g. computational models, test suites and benchmarks

Multi-modal Interfaces Objective: natural and easy to use interfaces that use several modalities or are multilingual Natural interaction between humans and the physical or virtual environment Multilingual communication systems IST WP

Natural interaction between humans and the physical or virtual environment Interfaces that are autonomous, and learn and adapt to user intentions and behaviour, in dynamically changing environments Featuring unconstrained, robust interaction, recognise user reactions and respond to them in a natural way Selecting the right combinations of modalities according to the preferences and context Looking at both the fusion of information related to different modalities and their channelling to multiple modalities Multi-modal Interfaces IST WP

Natural interaction between humans and the physical or virtual environment interfaces must demonstrate their ability to hide complex technologies, improve ease of use and accessibility, and support much more diverse and less expert user communities making key enabling technologies (speech, vision, haptics,...) much more tolerant of inconsistencies and sufficiently robust for wide usage ensure that voice recognition based upon natural language understanding becomes a routine form of user-computer interaction 5-year Targets draft

Multi-modal Interfaces Multilingual communication systems For unrestricted domains, including task-oriented, real-time understanding of spontaneous spoken and gesture input Address novel learning paradigms that exploit contextual information, human and linguistic knowledge in a more effective way than currently done Portability of new languages taking advantage of methods and techniques developed for languages already covered is a further challenge to be addressed, e.g. in the context of new EU languages IST WP

5-year Targets Multilingual communication systems establish a productive research community on language understanding and machine translation move from manual encoding of linguistic information to automatic acquisition, analysis and annotation of information supporting autonomy in the way systems develop linguistic abilities develop interoperable written and spoken resources and new applications supporting the seamless integration of the new languages, within the context of EU policies on multilingualism and enlargement focus on systems that handle multiple languages draft

Multi-modal Interfaces This covers: User modelling, system design, visual recognition and tracking, language understanding and spoken language translation Applications should focus on proof of concepts Includes consumer, nomadic, creative, artistic and gaming applications IP - system-level objectives in natural interaction and multilingual communication STReP – can also include work on language understanding and spoken language translation IST WP

Partnerships The WP tells us we: can span components, devices, infrastructures and services should bring services and technology developments closer together And in addition: bring together research communities working on speech and sound recognition and generation, computer vision, graphical animation, language understanding, touch-based sensing and feedback (haptics), machine learning, ergonomics, psycholinguistic, user modelling and dialogue management fewer participants with more commitment visible leadership from a group of industrial partners

Instruments draft Integrated Project (IP): Result driven with strong industry/academic consortium and users well integrated Focus on impact, exploitation, and replication Multi-faceted with R&D, demonstration, training, awareness building, standardisation, … partners, 5+ countries, 4 years, 5-10 M€ Targeted Project (STReP): Problem oriented with the right team for the S&T challenge Single-problem R&D with high innovation/risk/reward Investigating new research options, proof of concept, or competitive approaches 5-8 partners, 4+ countries, 3 years, 2-4 M€

The Evaluation Criteria draft Relevance: Clarity – make it a clear yes/no decision Reference national and European foresight results Quality of consortium: Tell them what your principle qualities are – quality of R&D, industrial participation, exploitation potential, SME involvement, uniqueness of consortium, track record, … Well balanced in terms of expertise, effort and exploitation All partners must have substantial and well defined roles to play (no window dressing, no hangers on) Users and end-users are present and have clearly defined roles Resources: Clear task allocation and no unnecessary overlaps Justify costs, effort allocation, and link explicitly to results

The Evaluation Criteria draft Management and organisation: Clear work-plan and good progress indicators and tracking are key signs Indicate some valuable intermediate milestones Indicate how risks will be assessed and what are you back- up options or contingency plans How decisions are made, and can they be made in a timely way – conflict resolution and consensus building Good interlinking between objectives, activities and work- packages (Gantt) Exploitation/dissemination: Protect and manage IPR and provide access to partners Start actions early and be precise about options Go beyond just “publishing results” – make an impact and above all be visible

The Evaluation Criteria draft European added-value: Show added value to existing national and European programmes Compliment ongoing efforts of most partners S&T excellence: Position the project vis-à-vis wider strategic objectives Define clear challenging but attainable objectives Describe state-of-the-art and go beyond (no catch up) Commit to some key deliverables/impact measures Justify balance between research and development Stress visible industry-scale validation through experiments, prototypes, demonstrators… Show that you can scale to industrialisation Don’t just be interesting, be manufacturable

Its all about generating, demonstrating and validating new knowledge Be ambitious, but clearly define your scientific and technological objectives – think Be unique, innovate but be industrially relevant - include key industry players and ensure broad industrial impact Be evaluator friendly, make it a compelling read, easy to understand, a give them a reason to remember it at the end of the week Be credible, one proposal, no “template” proposals, do what is needed to get the result, not more, but not less – remember 54 M€ represents ~1,200 person-years of effort Check your ideas with us! Remember…

IST infodesk fax: General FP6: Bernard Smith Further information