1. Rapid Prototyping of Distributed User Interfaces J. P. Molina 1,2, J. Vanderdonckt 1, P. González 2 A. Fernández 2 and M. D. Lozano 2 1 Université catholique de Louvain 2 University of Castilla-La Mancha www.usixml.org 6th International Conference on Computer-Aided Design of User Interfaces CADUI´2006 Bucharest, Romania, June 5-8, 2006

2. June 5-8 2CADUI 2006, Bucharest, Romania Contents Introduction Introduction Foundations Foundations Distributed UI Design Space Distributed UI Design Space Software Tool for Rapid Prototyping Software Tool for Rapid Prototyping Examples Examples Conclusion Conclusion

3. June 5-8 3CADUI 2006, Bucharest, Romania Contents Introduction Introduction Foundations Foundations Distributed UI Design Space Distributed UI Design Space Software Tool for Rapid Prototyping Software Tool for Rapid Prototyping Examples Examples Conclusion Conclusion

4. June 5-8 4CADUI 2006, Bucharest, Romania Introduction Distributed User Interfaces (DUIs) apply the notion of distributing parts or whole of a UI across several places or locations like Distributed Systems. Distributed User Interfaces (DUIs) apply the notion of distributing parts or whole of a UI across several places or locations like Distributed Systems. Designing a DUI remains a complex problem which may prevent designers from exploring design issues. Designing a DUI remains a complex problem which may prevent designers from exploring design issues.

5. June 5-8 5CADUI 2006, Bucharest, Romania Introduction First, advances in DUIs do not necessarily rely on the same concepts of distribution, and the axes according to which the distribution may take place vary from one research to another. First, advances in DUIs do not necessarily rely on the same concepts of distribution, and the axes according to which the distribution may take place vary from one research to another. Second, as developing DUIs require a sophisticated architecture, it is not surprising DUIs are slow to obtain, expensive to produce, and probably equally complex to use. Second, as developing DUIs require a sophisticated architecture, it is not surprising DUIs are slow to obtain, expensive to produce, and probably equally complex to use.

6. June 5-8 6CADUI 2006, Bucharest, Romania Introduction Rapid prototyping of DUIs turns to be an important issue, not only it could be used as a vehicle for developing and demonstrating visions of innovative DUIs but also… Rapid prototyping of DUIs turns to be an important issue, not only it could be used as a vehicle for developing and demonstrating visions of innovative DUIs but also… … it could help showing various distribution configurations before going to full implementation. … it could help showing various distribution configurations before going to full implementation.

7. June 5-8 7CADUI 2006, Bucharest, Romania Introduction However, rapid prototyping is also challenging, as the design space of DUIs covers a wide range of different possibilities. However, rapid prototyping is also challenging, as the design space of DUIs covers a wide range of different possibilities. In order to tackle this complexity, the software tool presented here supports rapid prototyping of DUIs based on a limited, but significant, design space of four design dimensions. In order to tackle this complexity, the software tool presented here supports rapid prototyping of DUIs based on a limited, but significant, design space of four design dimensions.

8. June 5-8 8CADUI 2006, Bucharest, Romania Contents Introduction Introduction Foundations Foundations Distributed UI Design Space Distributed UI Design Space Software Tool for Rapid Prototyping Software Tool for Rapid Prototyping Examples Examples Conclusion Conclusion

9. June 5-8 9CADUI 2006, Bucharest, Romania Foundations The software tool and the design space presented here relies on the following concepts: The software tool and the design space presented here relies on the following concepts: Cameleon framework Cameleon framework Context of use Context of use Interaction surface Interaction surface Environment model Environment model

10. June 5-8 10CADUI 2006, Bucharest, Romania Cameleon Reference Framework Defines UI development steps for multi-context interactive applications [Calvary et al., 2003]. Defines UI development steps for multi-context interactive applications [Calvary et al., 2003]. Final UI (FUI). Final UI (FUI). UI running on a particular computing platform (HTML, Java, VRML). UI running on a particular computing platform (HTML, Java, VRML). Concrete UI (CUI). Concrete UI (CUI). A mock-up that is independent of any computer platform. A mock-up that is independent of any computer platform. Abstract UI (AUI). Abstract UI (AUI). Defines abstract containers and individual components, independent of any modality of interaction. Defines abstract containers and individual components, independent of any modality of interaction. Task & Concepts (T&C). Task & Concepts (T&C). Describe tasks and domain-oriented concepts. Describe tasks and domain-oriented concepts.

11. June 5-8 11CADUI 2006, Bucharest, Romania Cameleon reference framework Multi-path UI development: Abstraction, Reification and Translation. Multi-path UI development: Abstraction, Reification and Translation.

12. June 5-8 12CADUI 2006, Bucharest, Romania Cameleon reference framework The selected language for this work is UsiXML, which stands for User Interface eXtensible Markup Language. The selected language for this work is UsiXML, which stands for User Interface eXtensible Markup Language. www.usixml.org IdealXML [Montero et al., 2005] GrafiXML [B. Michotte]

13. June 5-8 13CADUI 2006, Bucharest, Romania Context of use The context of use describes all the entities that may influence how the user’s task is carried out. The context of use describes all the entities that may influence how the user’s task is carried out. Takes into account three aspects: Takes into account three aspects: User type, e.g. system experience, task experience, task motivation. User type, e.g. system experience, task experience, task motivation. Computing platform type, e.g. mobile platform, stationary platform. Computing platform type, e.g. mobile platform, stationary platform. Physical environment type, e.g. office conditions, outdoor conditions. Physical environment type, e.g. office conditions, outdoor conditions.

14. June 5-8 14CADUI 2006, Bucharest, Romania Interaction surface This term, introduced by [Coutaz et al., 2003], refers to any physical surface which can be “acted on or observed” so as to support user interaction with a system, whether visible or embedded. This term, introduced by [Coutaz et al., 2003], refers to any physical surface which can be “acted on or observed” so as to support user interaction with a system, whether visible or embedded. An interaction surface could be a screen, a monitor, a wall display, a table equipped with a camera tracking system, etc. An interaction surface could be a screen, a monitor, a wall display, a table equipped with a camera tracking system, etc.

15. June 5-8 15CADUI 2006, Bucharest, Romania Environment Model UsiXML’s existing physical environment model has been expanded with the concept of interaction surface. UsiXML’s existing physical environment model has been expanded with the concept of interaction surface.

16. June 5-8 16CADUI 2006, Bucharest, Romania Environment Model The physical environment is described as a scene which is in turn decomposed into surfaces, to be connected together or not by position and orientation. Each surface can be defined as an interaction surface. The physical environment is described as a scene which is in turn decomposed into surfaces, to be connected together or not by position and orientation. Each surface can be defined as an interaction surface.

17. June 5-8 17CADUI 2006, Bucharest, Romania Environment Model An environment also comprises one or several computing platforms, holding one or more hardware platforms. An environment also comprises one or several computing platforms, holding one or more hardware platforms. Hardware platforms are considered as rectangular interaction surfaces that can be acted on (by using pointers) and/or observed (by looking at the screen). Hardware platforms are considered as rectangular interaction surfaces that can be acted on (by using pointers) and/or observed (by looking at the screen).

18. June 5-8 18CADUI 2006, Bucharest, Romania Contents Introduction Introduction Foundations Foundations Distributed UI Design Space Distributed UI Design Space Software Tool for Rapid Prototyping Software Tool for Rapid Prototyping Examples Examples Conclusion Conclusion

19. June 5-8 19CADUI 2006, Bucharest, Romania Distributed UI Design Space The proposed DUI design space is decomposed into four design axes or dimensions: The proposed DUI design space is decomposed into four design axes or dimensions: Type of computing platform Stationarization … Mobilization Type of computing platform Stationarization … Mobilization Amount of interaction surfaces Decentralization … Centralization Amount of interaction surfaces Decentralization … Centralization Type of interaction surface Physicalization … Digitization Type of interaction surface Physicalization … Digitization Type of user interface Materalization … Demateralization Type of user interface Materalization … Demateralization

20. June 5-8 20CADUI 2006, Bucharest, Romania Distributed UI Design Space

21. June 5-8 21CADUI 2006, Bucharest, Romania Contents Introduction Introduction Foundations Foundations Distributed UI Design Space Distributed UI Design Space Software Tool for Rapid Prototyping Software Tool for Rapid Prototyping Examples Examples Conclusion Conclusion

22. June 5-8 22CADUI 2006, Bucharest, Romania Software Tool for Rapid Prototyping A software tool named VUIToolkit has been developed above UsiXML and expanded with the previous environment model. A software tool named VUIToolkit has been developed above UsiXML and expanded with the previous environment model. With this tool, the environment model is produced as a Virtual Reality scene, thus allowing the rendering of both software (e.g. widgets) and hardware (e.g. physical buttons), but just for rapid prototyping purposes. With this tool, the environment model is produced as a Virtual Reality scene, thus allowing the rendering of both software (e.g. widgets) and hardware (e.g. physical buttons), but just for rapid prototyping purposes. The tool provides basic operations such as copy a CUI from one surface to another one, thus migrating it. The tool provides basic operations such as copy a CUI from one surface to another one, thus migrating it.

23. June 5-8 23CADUI 2006, Bucharest, Romania Software Tool for Rapid Prototyping VUIToolkit offers a set of VRML97/X3D PROTOs that transform the standard plain widgets into a truly 3D representation, referred as Virtualized User Interface. VUIToolkit offers a set of VRML97/X3D PROTOs that transform the standard plain widgets into a truly 3D representation, referred as Virtualized User Interface.

24. June 5-8 24CADUI 2006, Bucharest, Romania Mapping from CUI to VRML/X3D When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: Direct mapping. Direct mapping. It is not possible to set a one-to-one mapping, as there is no VRML/X3D primitives that can be matched with a CIO. These Web3D languages specify basic elements that must be used together to create interactive elements. It is not possible to set a one-to-one mapping, as there is no VRML/X3D primitives that can be matched with a CIO. These Web3D languages specify basic elements that must be used together to create interactive elements.

25. June 5-8 25CADUI 2006, Bucharest, Romania Mapping from CUI to VRML/X3D When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: Direct mapping. Direct mapping. New mapping. New mapping. There is a need to fill a gap by introducing a new widget by appropriate implementation. There is a need to fill a gap by introducing a new widget by appropriate implementation. Some attributes defined in the CUI model are not used in the 3D world, while new attributes must be added in order to describe the interface elements in the 3D world, such as position and size. Some attributes defined in the CUI model are not used in the 3D world, while new attributes must be added in order to describe the interface elements in the 3D world, such as position and size.

26. June 5-8 26CADUI 2006, Bucharest, Romania Mapping from CUI to VRML/X3D When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed: Direct mapping. Direct mapping. New mapping. New mapping. No possible mapping. No possible mapping. There are CIOs whose VRML/X3D counterpart have not been done yet or are impossible or difficult to implement. An example of this case is the CIO box. There are CIOs whose VRML/X3D counterpart have not been done yet or are impossible or difficult to implement. An example of this case is the CIO box.

27. June 5-8 27CADUI 2006, Bucharest, Romania Mapping from CUI to VRML/X3D Input/output mapping from desktop-oriented 2D GUI to VRML/X3D virtual world. Input/output mapping from desktop-oriented 2D GUI to VRML/X3D virtual world. Display = “Screen” PROTO. Display = “Screen” PROTO. Rectangular area where screen units (pixels) are converted into world units (meters). Rectangular area where screen units (pixels) are converted into world units (meters). Serves as a container of UI elements (children field). Serves as a container of UI elements (children field). Mouse = User’s pointing device. Mouse = User’s pointing device. VRML/X3D sensors. VRML/X3D sensors. Keyboard? Keyboard? VRML: requires non-standard extensions (PG’s KbdSensor). VRML: requires non-standard extensions (PG’s KbdSensor). X3D: StringSensor. X3D: StringSensor. Virtual keyboard. Virtual keyboard.

28. June 5-8 28CADUI 2006, Bucharest, Romania Contents Introduction Introduction Foundations Foundations Distributed UI Design Space Distributed UI Design Space Software Tool for Rapid Prototyping Software Tool for Rapid Prototyping Examples Examples Conclusion Conclusion

29. June 5-8 29CADUI 2006, Bucharest, Romania Examples The user’s environment is a small office, with five different platforms The user’s environment is a small office, with five different platforms

30. June 5-8 30CADUI 2006, Bucharest, Romania Across Multiple Computing Platforms Stationarization … Mobilization Stationarization … Mobilization Moving along this axis means rendering a FUI on a target platform which is more stationary / mobile than the source platform. Moving along this axis means rendering a FUI on a target platform which is more stationary / mobile than the source platform. It can be achieved by performing an abstraction from CUI to AUI, followed by a reification from AUI to CUI. A simple translation from CUI to CUI can also work. It can be achieved by performing an abstraction from CUI to AUI, followed by a reification from AUI to CUI. A simple translation from CUI to CUI can also work.

31. June 5-8 31CADUI 2006, Bucharest, Romania Across Multiple Computing Platforms Stationarization  Mobilization Stationarization  Mobilization

32. June 5-8 32CADUI 2006, Bucharest, Romania Across Multiple Interaction Surfaces Decentralization … Centralization Decentralization … Centralization Moving along this axis means rendering a FUI on more / less interaction surfaces than previously, the displays being connected to the same computing platform or not. Moving along this axis means rendering a FUI on more / less interaction surfaces than previously, the displays being connected to the same computing platform or not. It can be achieved by performing a decomposition of the source UI (by graceful degradation, fragmentation, etc) or a re-composition of gathered pieces (by union, merging, etc). It can be achieved by performing a decomposition of the source UI (by graceful degradation, fragmentation, etc) or a re-composition of gathered pieces (by union, merging, etc).

33. June 5-8 33CADUI 2006, Bucharest, Romania Across Multiple Interaction Surfaces Decentralization   Centralization Decentralization   Centralization

34. June 5-8 34CADUI 2006, Bucharest, Romania Across Different Interaction Surfaces Physicalization … Digitization Physicalization … Digitization Moving along this axis means rendering a FUI on a target platform which is more physical / digital than the source platform. Moving along this axis means rendering a FUI on a target platform which is more physical / digital than the source platform. Projection surfaces are considered as physical ones since the UI is rendered by projection onto a surface, and user’s actions are tracked by camera recognition techniques. Projection surfaces are considered as physical ones since the UI is rendered by projection onto a surface, and user’s actions are tracked by camera recognition techniques.

35. June 5-8 35CADUI 2006, Bucharest, Romania Across Different Interaction Surfaces Physicalization  Digitization Physicalization  Digitization

36. June 5-8 36CADUI 2006, Bucharest, Romania Across Different Types of User Interfaces Materialization … Dematerialization Materialization … Dematerialization Moving along this axis means changing the distribution of the FUI towards more physical / immaterial components. Moving along this axis means changing the distribution of the FUI towards more physical / immaterial components. It can be achieved by re-assigning pieces of the FUI to physical or software objects. Materialization obviously touches the area of tangible UIs. It can be achieved by re-assigning pieces of the FUI to physical or software objects. Materialization obviously touches the area of tangible UIs.

37. June 5-8 37CADUI 2006, Bucharest, Romania Across Different Types of User Interfaces Materialization   Dematerialization Materialization   Dematerialization

38. June 5-8 38CADUI 2006, Bucharest, Romania Contents Introduction Introduction Foundations Foundations Distributed UI Design Space Distributed UI Design Space Software Tool for Rapid Prototyping Software Tool for Rapid Prototyping Examples Examples Conclusion Conclusion

39. June 5-8 39CADUI 2006, Bucharest, Romania Conclusions A design space for DUIs consisting of four dimensions has been introduced to better understand this field. A design space for DUIs consisting of four dimensions has been introduced to better understand this field. That design space is supported in a tool for rapid prototyping of DUIs, using VUIToolkit to render in a virtual world the concrete UI specified in UsiXML. That design space is supported in a tool for rapid prototyping of DUIs, using VUIToolkit to render in a virtual world the concrete UI specified in UsiXML. A richer environment model has been defined so as to represent the world in which distribution may occur. A richer environment model has been defined so as to represent the world in which distribution may occur. The functionality given by this combination enable designers to explore different design issues and keeping the distribution which is found adequate to the goals. The functionality given by this combination enable designers to explore different design issues and keeping the distribution which is found adequate to the goals.

40. Mulţumesc! Thank you Merci Gracias More info: www.usixml.org Contact: jpmolina@dsi.uclm.es The UsiXML Consortium