1. Multimodal Interaction

2. Modalities vs Media Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular image

3. How Do Multimodal Systems Differ? Domain/application Available media Modeling of context/environment Modeling of user Focus of research

4. Example Multimodal Systems Not speech-centric MIT paintbrush, soundbrush http://www.youtube.com/watch?v=04v_v1gnyO8 http://www.youtube.com/watch?v=iZbe3t8YSf4 http://www.youtube.com/watch?v=18RY8Jgid20 Wearables http://www.gatech.edu/innovations/wearable/

5. Example Multimodal Systems Speech-centric MSOIP COMIC SmartKom

6. MSOIP Keywords Multimodal mobile dialog Integration of speech and pen input User modeling for presentations Johnston et al. 2001

7. MATCH Video http://www.research.att.com/~johnston/ Scroll down to the bottom of the page

8. About MATCH What input modalities? What output modalities? What application(s)? What aspects of context?

9. COMIC Keywords Ambient intelligence HHI/HCI research Collaborative problem solving User modeling Avatar Alexandersson et al. 2004

10. COMIC Video http://www.hcrc.ed.ac.uk/comic/demos/facial-animation/

11. COMIC Video http://www.hcrc.ed.ac.uk/comic/demos/slot/

12. About COMIC What input modalities? What output modalities? What applications? What aspects of context?

13. SmartKom Keywords Multimodal dialog across applications devices and situations Avatar Situation aware Alexandersson et al., Reithinger et al. 2003

14. SmartKom Video http://www.smartkom.org/start_en.html I showed the SK-Mobile one, but the other one is also interesting.

15. About SmartKom What input modalities? What output modalities? What applications? What aspects of context?

16. Parts of a Multimodal System Interpreter Dialog Manager Generator Knowledge Base Speech In Speech Out Gesture In Present Out Text In Text Out

17. HCI and Multimodal Systems Input integration/fusion Representations Effective help Quality presentations Managing context Understanding the user

18. Different Uses of Modalities Concurrent or sequential Redundant or Complementary or Contradicting

19. Input Integration/Fusion Key elements: Time Multiple uses of some modalities Error rates Typical approach is to map straight to semantics if possible

20. Representation Increasing use of XML-based languages (SMIL, EMMA) But these don’t solve the semantic problems Keep ‘backbone’ knowledge separate from ‘peripheral’ information (Alexandersson et al.)

21. Effective Help How do each of the systems provide the user with: Explicit help? Implicit help?

22. Quality Presentations Talking heads Advantages Disadvantages Informative presentations are key User modeling/adaptive presentations are a bonus These systems go beyond scripts

23. Managing Context What kinds of context are there in a mobile multimodal interaction?

24. Understanding the User What kinds of information can we gather about users in general? About one user in particular? How can we use this information?

25. Commercial Multimodal Systems Most are for research Military Training and battlefield Education Tutoring systems Commercial ones include: Wii: http://www.youtube.com/watch?v=n4nZVAE eitU http://www.youtube.com/watch?v=n4nZVAE eitU Microsoft surface: http://www.youtube.com/watch?v=rP5y7yp0 6n0 http://www.youtube.com/watch?v=rP5y7yp0 6n0

26. TradeOffs You get: More intuitive technology More information, more easily Less (dumb stuff) for you to do You trade: Privacy Control

27. Towards the Future Design Multimodal systems in virtual worlds, or crossing over from virtual to real worlds Ambient multimodal interaction Implementation Mashups – user controlled Pervasive multimedia

28. Towards the Future http://www.youtube.com/watch?v=FM JwURqpFWs http://www.programmableweb.com/m ashups

29. SciFi? Lathe of Heaven by Ursula LeGuin Summa Technologiae by Stanislaw Lem Fast Times at Fairmont High by Vernor Vinge The Human Machine Merger, talk by Raymond Kurzweil (at http://www.kurzweilai.net/meme/frame.html? main=memelist.html?m=6%23581) http://www.kurzweilai.net/meme/frame.html? main=memelist.html?m=6%23581

30. Additional Info http://search.techrepublic.com.com/se arch/multimodal+system.html http://www.w3.org/2002/mmi/