Multi-modal Interfaces “ “The focus of IST in FP6 is on the future generation of technologies in which computers and networks will be integrated into the everyday environment, rendering accessible a multitude of services and applications through easy-to-use human interfaces“ IST WP

Our Mission The challenge is: To make interfaces as simple as possible, and interaction as easy as possible For citizens to be able to access, receive and use information in their own language We support the research, development and integration of: Advanced technologies for interfaces & interaction Integrated multilingual information systems and services We focus on: Technology-mediated interaction and communication between people, between their devices, and between people and content Combining traditional human language services, like human translation and interpretation, and advanced language technologies and resources E1 Interfaces

What do we mean? Human-Computer Interaction “is concerned with the design, implementation and evaluation of interactive computer-based systems, as well as with the multi-disciplinary study of various issues affecting that interaction. The aim of HCI is to ensure the safety, utility, effectiveness, efficiency, accessibility and usability of such systems.” So its all about “radical changes in the way people work and interact with each other and with information” and systems “conceived and designed as community-centred, sharable, expandable, cooperative, collaborative and responsive media, catering through user and environmental monitoring, for a board range of human needs” Constantine Stephanidis, ICS-FORTH, ERCIM News, July 2001 Constantine Stephanidis, International Journal of HCI, 1999

Focus so far… Interaction intuitive multi-modal interfaces that are autonomous and capable of learning and adapting to the user environment in dynamically changing contexts recognise emotive user reaction robust dialogue capability with unconstrained speech and language input Human-to-human: technology mediated human communication Human-to-things: virtual and physical Human-to-self: health, well-being Human-to-content: information retrieval/browsing/navigation Device-to-device: human mediated device-device communication Human-to-embodied robots: assistance, autonomous, doing-things Multilingual systems unrestricted spontaneous speech-to-speech translation for specific application scenarios statistical/hybrid (integrating linguistic knowledge) approaches to automatic translation E1 Interfaces

The story so far! International Scientific Forum “Towards and Information Society for All”, International Journal of Human-Computer Interaction, Vol. 10(2), 1998 and Vol. 11(1), 1999 Special Theme: Human Computer Interaction, ERCIM NEWS, No. 46, July 2001 Coordination and Fusion in Multi-modal Interaction - a research roadmap, Dagstuhl Seminar 01441, Nov Interface Technologies Workshop, Luxembourg, May 13-14, 2002 IST Work Programme , Strategic Objective Internal Reflection Group on Interface Technologies – multi-sensorial, multilingual interactivity and virtual environments, 28 May 2002 Impact Study of IST – section on HLT, contract C28262, March 2004 Multi-modal Interfaces – consultation meeting, Luxembourg 23 April 2004 IST Work Programme , Strategic Objective “Make it simple” - a survey of information technology, The Economist, Oct. 30, 2004 “Building the Europe of Knowledge” a proposal for 7 th Framework Programme, COM(2005) 119 final, 6 April 2005

Main challenges Need to provide information artefacts to be used by increasingly diverse user groups Expanding context of use, from scientific and business use to residential and nomadic use Increasing variety of media and devices used to access a community-wide pool of services and information resources Demonstrate added-value of multi-modality and performance advances through multi-modal interaction Go beyond system integration for demonstration purposes and target real world solutions Emphasise multi-user scenarios and non-business applications Stronger focus on defining the expected impact Need to promote cooperation between small European companies Promote research that could lead to spin-offs

An R&D agenda Key interaction technologies (speech, natural language input, haptics, vision, sensors, gesture, agents, …, including unconventional modalities) and their modelling and integration into interaction platforms Interface architectures, tools for cost effective integration, e.g. fusion, and toolkit interoperability Computational models of multi-modality, and advanced corpora for research Re-develop a serious effort on automatic translation Reinforce work on robustness and acceptability of speech recognition and research on speaker-independent software New learning paradigms in (spoken) language translation and portability to new languages, including greater focus on language resources as computer processable and reusable resources Design processes, methods and tools that support scalability and modality independence Empirical science base for understanding end-users, e.g. computational models, test suites and benchmarks

Future orientations? Focus more on adaptability, learning capabilities and re- configurable interfaces Collaborative technologies and interfaces in the arts, including games, design, new media Less explored modalities, e.g. handwriting, haptics, bio- sensing Affective computing, including character and facial expression recognition and animation Innovative aspects of user centred design and ergonomics, and more generally design theory as an enabling discipline, e.g. product design Multilingualism and machine translation A real need for more product innovation and industrial impact Extracted from the Call 1 evaluation, 2004

Multi-modal Interfaces Objective: natural and easy to use interfaces that use several modalities or are multilingual Natural interaction between humans and the physical or virtual environment Multilingual communication systems IST WP

Natural interaction between humans and the physical or virtual environment Interfaces that are autonomous, and learn and adapt to user intentions and behaviour, in dynamically changing environments Featuring unconstrained, robust interaction, recognise user reactions and respond to them in a natural way Selecting the right combinations of modalities according to the preferences and context Looking at both the fusion of information related to different modalities and their channelling to multiple modalities Multi-modal Interfaces IST WP

Natural interaction between humans and the physical or virtual environment interfaces must demonstrate their ability to hide complex technologies, improve ease of use and accessibility, and support much more diverse and less expert user communities making key enabling technologies (speech, vision, haptics,...) much more tolerant of inconsistencies and sufficiently robust for wide usage ensure that voice recognition based upon natural language understanding becomes a routine form of user-computer interaction 5-year Targets draft

Multi-modal Interfaces Multilingual communication systems For unrestricted domains, including task-oriented, real-time understanding of spontaneous spoken and gesture input Address novel learning paradigms that exploit contextual information, human and linguistic knowledge in a more effective way than currently done Portability of new languages taking advantage of methods and techniques developed for languages already covered is a further challenge to be addressed, e.g. in the context of new EU languages IST WP

5-year Targets Multilingual communication systems establish a productive research community on language understanding and machine translation move from manual encoding of linguistic information to automatic acquisition, analysis and annotation of information supporting autonomy in the way systems develop linguistic abilities develop interoperable written and spoken resources and new applications supporting the seamless integration of the new languages, within the context of EU policies on multilingualism and enlargement focus on systems that handle multiple languages draft

Overall Context And in addition: Leadership & Ambition: your project should be the most important one in the field in Europe, if not in the world Success: tangible, easy to understand and measurable indicators, with highly visible results and project “products” Industrialisation: strong focus on exploitable results, well structured industrial participation, and planned and sustainable commercialisation Standardisation: increased effort to bring together industrial players, academia, user groups and policy makers Visibility and dissemination: fewer, more valuable and visible deliverables, workshops, products, demonstrators… Instruments: pick the right one – an IP is not an inflated STReP

The Evaluation Criteria draft European added-value: Show added value to existing national and European programmes Compliment ongoing efforts of most partners S&T excellence: Position the project vis-à-vis wider strategic objectives Define clear challenging but attainable objectives Describe state-of-the-art and go beyond (no catch up) Commit to some key deliverables/impact measures Justify balance between research and development Stress visible industry-scale validation through experiments, prototypes, demonstrators… Show that you can scale to industrialisation Don’t just be interesting, be manufacturable