1. 15-Nov-2002MUMIN Workshop1 Multimodal research at UIAH Kristiina Jokinen MediaLab University of Art and Design Helsinki Kristiina.Jokinen@uiah.fi

2. 15-Nov-2002 MUMIN Workshop2 University of Art and Design Education and research in the fields of design, new media, audiovisual communication, art education, arts Largest art school in Scandinavia Nearly 1600 students, 15% from abroad Lume, the Finnish centre for media research and development, was opened in 2000 Departments of film and television, design for theatre, film and television, and new media

3. 15-Nov-2002 MUMIN Workshop3 Media Lab Formed in 1993 Explore, discover and comprehend the new digital technology Impact in society Possibilities for communication, interaction and expression Challenges to new media and information design 2-year masters programme MA in New Media (full time) New Media program for professionals (flexible study method) 20 full-time, 15 professionals, and approx. 30 minor subject students annually to the MA programmes

4. 15-Nov-2002 MUMIN Workshop4 Adaptive Systems for Complex Interaction Research on natural interaction between humans and computers Interaction strategies, cooperative response planning Various input modalities Concepts, models Apply machine-learning techniques to dialogue processing Compare and test applicability of the techniques Support Design-for-all principles in designing intelligent interfaces

5. 15-Nov-2002 MUMIN Workshop5 Human-computer interaction Computer as a tool Passive and transparent Supports the human goals, human control Computer as an agent Models of beliefs, desires, intentions (BDI) Intelligent software mediating between the human and an application Cooperation, negotiation Complex interaction Multimodal communication

6. 15-Nov-2002 MUMIN Workshop6 Projects USIX-Interact: Natural Interaction and Adaptive Methods http://www.mlab.uiah.fi/interact/http://www.mlab.uiah.fi/interact/ DUMAS: Dynamic User Modelling for Adaptive Speech Interfaces http://www.sics.se/dumas/http://www.sics.se/dumas/ MUMMI: Multi-Modal Museum Interfaces (Study project together with Marjo Mäenpää and Antti Raike, Design for All, Virtual Art Exhibition) http://mlab.uiah.fi/mummi/ http://mlab.uiah.fi/mummi/

7. 15-Nov-2002 MUMIN Workshop7 Natural interaction Language that suits to computers Language that humans use to communicate => Language that humans and computers use when interacting with each other - Different ? How? - How it emerges from interaction?

8. 15-Nov-2002 MUMIN Workshop8 Interact: key aspects for adaptivity Conversational ability Dialogue modelling and natural communication Learning systems Various methods and techniques Various interface techniques (speech, text, map) Language technology Finnish and multilingual Agent-based architecture Jaspis development platform http://www.mlab.uiah.fi/interact/

9. 15-Nov-2002 MUMIN Workshop9 Interact Partners University of Art and Design, Media Lab University of Helsinki, Language Technology University of Tampere, TAUCHI unit Helsinki University of Technology, Neural Networks Research Centre Fujitsu Invia oyj Tecnomen oyj Lingsoft oy The Arla Institute Finnish Association for the Deaf Finnish Technology Agency

10. 15-Nov-2002 MUMIN Workshop10 DUMAS - Dynamic Universal Mobility for Adaptive Speech interfaces EU 5th framework R&D project Swedish Institute of Computer Science UIAH, Media Lab University of Tampere, TAUCHI- unit UMIST, Manchester ETEX, Frankfurt Conexor oy, Helsinki Timehouse oy, Helsinki KTH, Stockholm http://www.sics.se/dumas

11. 15-Nov-2002 MUMIN Workshop11 DUMAS Objectives Interactive email application Dynamic – various capabilities Universal – various situations and formats Mobility – various mobile applications for Adaptive – learning systems Speech – spoken and text input Interfaces – intelligent interaction

12. http://www.sics.se/dumas/ Goals Development of speech-based applications multilingual: Finnish, Swedish, English adapts to the user’s needs and habits Main application: AthosMail Experiments at the end of the project: Athos-radiostation Athos-text-TV UIAH responsibility: user modelling components

13. http://www.sics.se/dumas/ Challenges for User Modelling Learn from the user-computer interaction those aspects that are important in making the use flexible and enjoyable: cognitive load speaking habits dialogue strategies User Model is involved in almost all decision making from speech recognition to dialogue management to speech synthesis Classification and learning methods e.g. neural networks, Bayes-nets, reinforcement learning

14. 15-Nov-2002 MUMIN Workshop14 Learning via Interaction Situation: language is activity between rational agents (cf. Allwood) Contact + perception + understanding + reaction Task: achieve a communicative goal maximise mutual comprehensibility minimise ambiguity Constraints: language is possessed by a group of agents => cooperation limited resources => adaptation to new situations Reinforcement learning agent takes an action a, finds itself in a state s, and receives a reward r The task is to find a policy that maximizes the agent's reward in an environment

15. 15-Nov-2002 MUMIN Workshop15 Adaptive multimodal interfaces What and when to adapt? User-centred parameters: Habits and preferences Attitudes and intentions Environmental parameters: E.g. speech recognizer accuracy

16. 15-Nov-2002 MUMIN Workshop16 Action paths for an average user

17. 15-Nov-2002 MUMIN Workshop17 MUMMI: Multimodality and Museum Interfaces Study project together with Marjo Mäenpää, Antti Raike Cooperation with the Finnish National Gallery: Marjatta Levanto, Riikka Haapalainen New ways of relating the arts that are both visually interesting and accessible in terms of contents: Virtual art exhibition, interactive guiding of the user through the exhibition Text, speech, signing avatar Design for all Accessibility to the virtual visitors on museum web sites

18. 15-Nov-2002 MUMIN Workshop18 Design for All Places the user in the centre (user-centred design) Cognitive factors (perception, memory, learning, problem-solving, etc.) as they come into play during interactions with things Usefulness: what is relevant do the functions, information, etc., match what the user actually needs? Usability: ease-of-use a simple concept, but not always easy or intuitive to implement New ways to interact with computers?

19. 15-Nov-2002 MUMIN Workshop19 Other Multimodal Projects at MediaLab QuiQui’s Giant Bounce (Kukakumma Muumaassa) Perttu Hämäläinen, Johanna Höysniemi http://www.kukakumma.net/ use your body to play interaction with your body child-centred design

20. 15-Nov-2002 MUMIN Workshop20 Other Multimodal Projects at MediaLab Cinemasense (Elokuvantaju) Antti Raike http://elokuvantaju.uiah.fi/ web portal for film production (learning material) organise cinematic concepts in the student’s mind especially sign language

21. 15-Nov-2002 MUMIN Workshop21 Other Multimodal Projects at MediaLab Experimental, affective interfaces Jukka Ylitalo, Heidi Tikka http://mlab.uiah.fi/eia/ interactive media, media and art

22. 15-Nov-2002 MUMIN Workshop22 Burning Issues Conversational interfaces Dialogue processing: turn taking, feedback, repairs, non- verbal elements Architectures Learning in agent-based architectures How to plug-&-play? Processing techniques Cognitive models of language understanding Machine learning (supervised vs. unsupervised) Design for all Usability: for whom, why, what Evaluation

23. 15-Nov-2002 MUMIN Workshop23 References Cinemasense http://elokuvantaju.uiah.fi/http://elokuvantaju.uiah.fi/ DUMAS http://www.sics.se/dumashttp://www.sics.se/dumas Experimental interfaces http://mlab.uiah.fi/eia/http://mlab.uiah.fi/eia/ Interact http://www.mlab.uiah.fi/interact/http://www.mlab.uiah.fi/interact/ MUMMI http://mlab.uiah.fi//mummi/http://mlab.uiah.fi//mummi/ QuiQui http://www.kukakumma.net/http://www.kukakumma.net/ Jokinen et al. (2002). Adaptive Dialogue Systems – Interaction with Interact. Proceedings of the 3rd SIGDial Workshop, Philadelphia, US. Jokinen, K., J. Rissanen, H. Keränen, and K. Kanto (2002). Learning interaction patterns for adaptive user interfaces. The 7th ERCIM UI4All Workshop, October, Paris, France. Jokinen, K. and A. Raike (2002). Multimodality – the latest technology and visions and demands for the future. Multimodality IT-seminar, Castberggård, Denmark.