1 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Distributed User Interfaces in Ambient Environment Jean Vanderdonckt 1, Hildeberto Mendonça 1, José Pascual Molina Massó 1,2 1 Université catholique de Louvain (UCL) Louvain School of Management (LSM), Information Systems Unit (ISYS) Belgian Laboratory of Computer-Human Interaction (BCHI) 2 Universidad de Castilla-La Mancha (UCLM) Laboratory of User Interaction & Software Engineering (LoUISE) Instituto de Investigación en Informática de Albacete (I3A)

2 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Introduction (1/3) We live in an ambient, but distributed, world –Multiple devices, multiple locations, multiple users A real distributed worldA future (?) distributed world Harris poll for multiple PCs and displays (July 2002)

3 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Introduction (2/3) Distributed User Interfaces (DUIs) are a particular type of UIs whose portions are distributed across places or locations in a particular physical environment. Designing a DUI remains a complex problem which may prevent designers from exploring design issues –Too expensive to develop many alternatives –Hard to modify once coded  Rapid prototyping –Rapid prototyping of DUIs turns to be an important issue in AmI –Could be used as a vehicle for developing and demonstrating visions of innovative DUIs –Could show various distribution configurations before going to full implementation

4 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Introduction 3/3 Related work –DYNAMO-AID –MIGRATION project –Everywhere –3DSim –VAQUITA  Our approach is different from previous pieces of work –It relies on a physical environment model, produced as a virtual reality scene for rapid prototyping purposes –The environment is decomposed into surfaces, some being interactive –These surfaces are attached to a service model, characterizing the capabilities they can offer

5 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 The underlying model (1/5) Context of use: is decomposed into 3 aspects –User: population is expressed in terms of stereotypes sharing the same attributes (e.g., preferences, age, skills) –Platform: sofware –computing- and hardware – viewing- (e.g. mobile, laptop, desktop, wall screen) –Environment: the physical context is represented as a virtual reality scene, inspired from/based on VRML/X3D Context UserStereotypePlatformLocationEnvironment Hardware platform Interaction surface Service Event Condition Message Role

6 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 The underlying model (2/5) Interaction surface –Refers to any physical surface which can be “acted on or observed” so as to support user interaction with a system, whether visible or embedded. –2 types: Digital: a screen, a monitor, a wall display Physical: a table equipped with a camera tracking system, a real device [Carpendale, 2006] Context UserStereotypePlatformLocationEnvironment Hardware platform Interaction surface Service Event Condition Message Role

7 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 The underlying model (3/5) Location –Positioning of interaction surfaces and platforms in the environment –To be represented by potential positions of platforms in the virtual reality scene Digital interaction surface on the wall Digital interaction surface on the desk Context UserStereotypePlatformLocationEnvironment Hardware platform Interaction surface Service Event Condition Message Role

8 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 The underlying model (4/5) Service –Any basic function provided by the interaction surface to the designer who is prototyping the DUI Services provided by this interaction surface: - cuiRendering: renders a CUI on this surface - cuiUnrendering: unrenders a previously rendered CUI from this surface - cuiAdaptation: adapts a CUI for this surface Context UserStereotypePlatformLocationEnvironment Hardware platform Interaction surface Service Event Condition Message Role

9 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 The underlying model (5/5) Event: Occurrence during service execution, e.g. widgetRenderingError Condition: Precondition, post-condition and integrity constraints Message: A container of information, used to signal events Role: Responsible for perfoming services and raising events Event: CUI rendered on this surface Condition: resolution is large enough to accommodate the CUI Message: To cuiAdaptation service: needs to adapt to the new constraints Context UserStereotypePlatformLocationEnvironment Hardware platform Interaction surface Service Event Condition Message Role

10 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Software Tool for Rapid Prototyping 1/6 A software tool named VUIToolkit has been developed above UsiXML and expanded with the previous environment model. VUIToolkit offers a set of VRML97/X3D PROTOs that transform the standard plain widgets into a truly 3D representation, referred as Virtualized User Interface.

11 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Software Tool for Rapid Prototyping 2/6 With this tool, the environment model is produced as a Virtual Reality scene, thus allowing the rendering of both software (e.g. widgets) and hardware (e.g. physical buttons), but just for rapid prototyping purposes. The tool provides basic services such as copy a CUI from one surface to another one, thus migrating it.

12 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Software Tool for Rapid Prototyping 3/6 When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: –Direct mapping. It is not possible to set a one-to-one mapping, as there is no VRML/X3D primitives that can be matched with a CIO. These Web3D languages specify basic elements that must be used together to create interactive elements.

13 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Software Tool for Rapid Prototyping 4/6 When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: –Direct mapping. –New mapping. There is a need to fill a gap by introducing a new widget by appropriate implementation. Some attributes defined in the CUI model are not used in the 3D world, while new attributes must be added in order to describe the interface elements in the 3D world, such as position and size.

14 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Software Tool for Rapid Prototyping 5/6 When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: –Direct mapping. –New mapping. –No possible mapping. There are CIOs whose VRML/X3D counterpart have not been done yet or are impossible or difficult to implement. An example of this case is the CIO box.

15 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Software Tool for Rapid Prototyping 6/6 Input/output mapping from desktop-oriented 2D GUI to VRML/X3D virtual world. –Digital Interaction Surface / Display = “Screen” PROTO. Rectangular area where screen units (pixels) are converted into world units (meters). Serves as a container of UI elements (children field). –Mouse = User’s pointing device. VRML/X3D sensors. –Keyboard? VRML: requires non-standard extensions (PG’s KbdSensor). X3D: StringSensor. Virtual keyboard.

16 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Examples 1/6 The user’s environment is a small office, with five different platforms

17 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Examples 2/6 User Interface Rendering: cuiRendering

18 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Examples 3/6 User Interface Migration: cuiMigration cuiStoring cuiUnrendering cuiRetrieving cuiRendering

19 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Examples 4/6 User Interface Adaptation: cuiAdaptation Aspire ”, 16: x800 e750 3,8”, 3:4 240x320

20 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Examples 5/6 Other Services: uiPhysicalization, uiDigitization

21 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Examples 5/6 Other Services: uiPhysicalization, uiDigitization Physical interaction surface Digital interaction surface

22 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Conclusion Benefits –Rapid prototyping of DUIs by manipulation in a virtual reality scene –Permanent maintaining of a model behind (CUI + context) –Rendering as a model-to-code service –Adaptation as a model-to-model service –Transition between digital and physical worlds is easy Shortcomings –Possible composition of services, but not implemented –Composition should be based on messages, but also on quality of services E.g., what is the closest surface to me that could render this UI? –Manual operations trigger services: no algorithm for distributing UI pieces on interaction surfaces –Rendering limited by widgets provided by VUIToolkit

23 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 How to model the quality of services provided by interaction surfaces Allow utilization of QoS at various stages Specifications of –Users –providers Existing models –Q-WSDL, WSLA, DAML-QoS, etc. UML Profile for Modeling Quality of Service –Metamodel –Characteristics quantified by dimensions –Constraints specifying expected values –Context in which quality expressions are involved –Levels that can be achieved under constraints and context [OMG, 2006 ; Hersens, 2007]

24 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Upcoming events… Albacete (Spain), June

25 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7 th, 2007 Danke shön! – Merci! – Gracias! For more information and downloading, User Interface eXtensible Markup Language European network on Multimodal UIs Special thanks to all members of the team!