Marco Blumendorf I July 21th, 2009 Towards a Model-Based Framework for the Development of Adaptive Multimodal User Interfaces for Ambient Assisted Living Environments DAI-Labor Fakultät IV – Elektrotechnik und Informatik Technische Universität Berlin Marco Blumendorf and Sahin Albayrak

HCI International July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

HCI International July 21th, 2009 Ambient Assisted Living Environments are Smart Environments and reflect Ubiquitous Computing Paradigms

HCI International July 21th, 2009 “Ubiquitous Computing” (Mark Weiser, 1991) “In the 21st century the technology revolution will move into the everyday, the small and the invisible.”

HCI International July 21th, 2009 Smart Objects Smart Devices Smart Networks Smart Environments

HCI International July 21th, 2009 Interaction in Smart Environments used interaction modalities

HCI International July 21th, 2009 Challenges Complex, distributed system to interact with Different capabilities and skills of users Personalization of UIs Context adaptivity Continuous, flexible and coherent interaction Excellent interaction experience  Lack of development tools, methodologies and runtime support

HCI International July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

HCI International July 21th, 2009 Requirements Connect interaction resources Integrate context Host the applications Support UI features (Multimodality, Distribution, Adaptation, …)

HCI International July 21th, 2009 Connecting Interaction Resources

HCI International July 21th, 2009 Integrating Context Sensors

HCI International July 21th, 2009 Application Hosting Sensors

HCI International July 21th, 2009 Model State Sensors

HCI International July 21th, 2009 Monitoring State Sensors Monitoring

HCI International July 21th, 2009 Stimulating Model Execution Sensors Stimulation

HCI International July 21th, 2009 Processing Components Sensors

HCI International July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

HCI International July 21th, 2009 Mediating between human and computer “turn the light on”

HCI International July 21th, 2009 Mediating between human and computer ? “turn the light on”

HCI International July 21th, 2009 Mediating between human and computer “turn the light on”

HCI International July 21th, 2009 Abstraction Layers See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003) Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004) and TERESA (Berti et al. 2004)

HCI International July 21th, 2009 Mediation Process See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003) Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004) and TERESA (Berti et al. 2004)

HCI International July 21th, 2009 Adding State and Execution Logic Dynamic Executable Model See e.g. Breton & Bézivin 2001 for details Definition Elements - static structure and constant elements Situation Elements - state of the model and changing elements Execution Elements - interpretation process altering state (and definition) at runtime

HCI International July 21th, 2009 Adding State and Execution Logic

HCI International July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

HCI International July 21th, 2009 Implementation The Multi-Access Service Platform (MASP) implements the presented concepts Features Multimodal interaction Dynamic distribution Adaptation Different applications have been implemented.

HCI International July 21th, 2009 A Meta UI for Smart Environments

HCI International July 21th, 2009 The Multimodal Cooking Assistant

HCI International July 21th, 2009 Agenda Motivation A Runtime Architecture for Adaptive Multimodal User Interfaces User Interface Models The Multi Access Service Platform Conclusion

HCI International July 21th, 2009 Results We can build models of multimodal applications that dynamically change the used modalities. Applications can be distributed and dynamically configured at runtime. A follow me mode shows automatic reconfiguration based on context information. Dynamic layout changes according to the distance of the user to the screen have been realized.

HCI International July 21th, 2009 Open issues and future work Technology and software engineering centric Next Step: Study how to deploy the features properly to provide integration into environment and the required excellent user experience! Additional Aspects: –multi-user and multi-application scenarios have not been adequately addressed yet –tools also covering runtime issues and moving towards end- user development and customization –integration of natural language processing and models-based development

HCI International July 21th, 2009 References [Berti 2004]Silvia Berti, Francesco Correani, Giulio Mori, Fabio Paternò, and Carmen Santoro. Teresa: A transformation-based environment for designing and developing multi-device interfaces. In ACM CHI 2004, volume II, pages 793–794, ACM Press. [Breton&Bézivin 2001]Erwan Breton and Jean Bézivin. Towards an understanding of model executability. In FOIS '01: Proceedings of the international conference on Formal Ontology in Information Systems, pages 70–80, New York, NY, USA, ACM. [Calvary 2003]Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouillon, and Jean Vanderdonckt. A unifying reference framework for multi-target user interfaces. Interacting with Computers, 15(3):289–308, [Limbourg 2004]Quentin Limbourg, Jean Vanderdonckt, Benjamin Michotte, Laurent Bouillon, and Víctor López-Jaquero. Usixml: A language supporting multi-path development of user interfaces. In EHCI/DS-VIS, volume 3425 of Lecture Notes in Computer Science, p. 200–220. Springer, [Szekely 1996]Pedro A. Szekely. Retrospective and challenges for model- based interface development. In DSV-IS, pages 1–27. Springer, [Weiser 1991] Mark Weiser, The computer for the 21st century, Scientific American, Morgan Kaufmann Publishers Inc., 1991, 265, 66-75

HCI International July 21th, 2009 The End … Thank you for your attention! Your questions please … Please visit our booth at the break! Dynamic Executable Model