1. Marco Blumendorf I Marco.Blumendorf@dai-labor.de July 21th, 2009 Towards a Model-Based Framework for the Development of Adaptive Multimodal User Interfaces for Ambient Assisted Living Environments DAI-Labor Fakultät IV – Elektrotechnik und Informatik Technische Universität Berlin Marco Blumendorf and Sahin Albayrak

2. HCI International 20092 July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

3. HCI International 20093 July 21th, 2009 Ambient Assisted Living Environments are Smart Environments and reflect Ubiquitous Computing Paradigms

4. HCI International 20094 July 21th, 2009 “Ubiquitous Computing” (Mark Weiser, 1991) “In the 21st century the technology revolution will move into the everyday, the small and the invisible.”

5. HCI International 20095 July 21th, 2009 Smart Objects Smart Devices Smart Networks Smart Environments

6. HCI International 20096 July 21th, 2009 Interaction in Smart Environments used interaction modalities

7. HCI International 20097 July 21th, 2009 Challenges Complex, distributed system to interact with Different capabilities and skills of users Personalization of UIs Context adaptivity Continuous, flexible and coherent interaction Excellent interaction experience  Lack of development tools, methodologies and runtime support

8. HCI International 20098 July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

9. HCI International 20099 July 21th, 2009 Requirements Connect interaction resources Integrate context Host the applications Support UI features (Multimodality, Distribution, Adaptation, …)

10. HCI International 200910 July 21th, 2009 Connecting Interaction Resources

11. HCI International 200911 July 21th, 2009 Integrating Context Sensors

12. HCI International 200912 July 21th, 2009 Application Hosting Sensors

13. HCI International 200913 July 21th, 2009 Model State Sensors

14. HCI International 200914 July 21th, 2009 Monitoring State Sensors Monitoring

15. HCI International 200915 July 21th, 2009 Stimulating Model Execution Sensors Stimulation

16. HCI International 200916 July 21th, 2009 Processing Components Sensors

17. HCI International 200917 July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

18. HCI International 200918 July 21th, 2009 Mediating between human and computer 1001101010 1111100100 “turn the light on”

19. HCI International 200919 July 21th, 2009 Mediating between human and computer 1001101010 1111100100 ? “turn the light on”

20. HCI International 200920 July 21th, 2009 Mediating between human and computer 1001101010 1111100100 “turn the light on”

21. HCI International 200921 July 21th, 2009 Abstraction Layers See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003) Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004) and TERESA (Berti et al. 2004)

22. HCI International 200922 July 21th, 2009 Mediation Process See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003) Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004) and TERESA (Berti et al. 2004)

23. HCI International 200923 July 21th, 2009 Adding State and Execution Logic Dynamic Executable Model See e.g. Breton & Bézivin 2001 for details Definition Elements - static structure and constant elements Situation Elements - state of the model and changing elements Execution Elements - interpretation process altering state (and definition) at runtime

24. HCI International 200924 July 21th, 2009 Adding State and Execution Logic

25. HCI International 200925 July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

26. HCI International 200926 July 21th, 2009 Implementation The Multi-Access Service Platform (MASP) implements the presented concepts Features Multimodal interaction Dynamic distribution Adaptation Different applications have been implemented.

27. HCI International 200927 July 21th, 2009 A Meta UI for Smart Environments

28. HCI International 200928 July 21th, 2009 The Multimodal Cooking Assistant

29. HCI International 200929 July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

30. HCI International 200930 July 21th, 2009 Results We can build models of multimodal applications that dynamically change the used modalities. Applications can be distributed and dynamically configured at runtime. A follow me mode shows automatic reconfiguration based on context information. Dynamic layout changes according to the distance of the user to the screen have been realized.

31. HCI International 200931 July 21th, 2009 Open issues and future work Technology and software engineering centric Next Step: Study how to deploy the features properly to provide integration into environment and the required excellent user experience! Additional Aspects: –multi-user and multi-application scenarios have not been adequately addressed yet –tools also covering runtime issues and moving towards end- user development and customization –integration of natural language processing and models-based development

32. HCI International 200932 July 21th, 2009 References [Berti 2004]Silvia Berti, Francesco Correani, Giulio Mori, Fabio Paternò, and Carmen Santoro. Teresa: A transformation-based environment for designing and developing multi-device interfaces. In ACM CHI 2004, volume II, pages 793–794, 2004. ACM Press. [Breton&Bézivin 2001]Erwan Breton and Jean Bézivin. Towards an understanding of model executability. In FOIS '01: Proceedings of the international conference on Formal Ontology in Information Systems, pages 70–80, New York, NY, USA, 2001. ACM. [Calvary 2003]Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouillon, and Jean Vanderdonckt. A unifying reference framework for multi-target user interfaces. Interacting with Computers, 15(3):289–308, 2003. [Limbourg 2004]Quentin Limbourg, Jean Vanderdonckt, Benjamin Michotte, Laurent Bouillon, and Víctor López-Jaquero. Usixml: A language supporting multi-path development of user interfaces. In EHCI/DS-VIS, volume 3425 of Lecture Notes in Computer Science, p. 200–220. Springer, 2004. [Szekely 1996]Pedro A. Szekely. Retrospective and challenges for model- based interface development. In DSV-IS, pages 1–27. Springer, 1996. [Weiser 1991] Mark Weiser, The computer for the 21st century, Scientific American, Morgan Kaufmann Publishers Inc., 1991, 265, 66-75

33. HCI International 200933 July 21th, 2009 The End … Thank you for your attention! Your questions please … marco.blumendorf@dai-labor.de http://masp.dai-labor.de Please visit our booth at the break! Dynamic Executable Model