Direct Manipulation of User Interfaces for Migration J. P. Molina 1,2, J. Vanderdonckt 1 and P. González 2 1 Université catholique de Louvain 2 University of Castilla-La Mancha Intelligent User Interfaces 2006 Jan 29 - Feb , Sydney, Australia

January 29 - February 1 2Intelligent User Interfaces 2006, Sydney, Australia Contents Introduction Introduction MigriXML MigriXML Case study Case study Migration process in detail Migration process in detail Conclusion Conclusion

January 29 - February 1 3Intelligent User Interfaces 2006, Sydney, Australia Contents Introduction Introduction MigriXML MigriXML Case study Case study Migration process in detail Migration process in detail Conclusion Conclusion

January 29 - February 1 4Intelligent User Interfaces 2006, Sydney, Australia Introduction End users of modern interactive systems face a wide variety of computing platforms to support their interactive tasks, from mobile phones to wall screens End users of modern interactive systems face a wide variety of computing platforms to support their interactive tasks, from mobile phones to wall screens The specific needs of the user’s environment at a given moment, or its evolution, can make necessary to remove one application from one platform and make it run in another one, that is, to migrate the application The specific needs of the user’s environment at a given moment, or its evolution, can make necessary to remove one application from one platform and make it run in another one, that is, to migrate the application

January 29 - February 1 5Intelligent User Interfaces 2006, Sydney, Australia Introduction Migration can be: Migration can be: Total, when the whole application is migrated Total, when the whole application is migrated Partial, only the UI is migrated; if it is decomposed into control and presentation, then migration can be: Partial, only the UI is migrated; if it is decomposed into control and presentation, then migration can be: Control-oriented, migrates only the control component Control-oriented, migrates only the control component Presentation-oriented, the control remains Presentation-oriented, the control remains Mixed, parts of both control and presentation are migrated Mixed, parts of both control and presentation are migrated A user interface (UI) is said to be migratable if it holds the migration ability A user interface (UI) is said to be migratable if it holds the migration ability

January 29 - February 1 6Intelligent User Interfaces 2006, Sydney, Australia Introduction A special UI is required for controlling the run- time migration of the UI of the interactive systems A special UI is required for controlling the run- time migration of the UI of the interactive systems This UI is referred to as the meta-user interface, implemented in very different ways: This UI is referred to as the meta-user interface, implemented in very different ways: Sytem-initiated, the system initiates the migration Sytem-initiated, the system initiates the migration User-initiated, the user initiates the migration User-initiated, the user initiates the migration Mixed-initiated, user and system collaborate to perform the migration Mixed-initiated, user and system collaborate to perform the migration

January 29 - February 1 7Intelligent User Interfaces 2006, Sydney, Australia Introduction Related work Related work MigrationInitiated by app user interface systemuser controlpresentation Bharat and Cardelli, 95 Song, Chu and Kurakake, 02 MIGRATION Project Bandelloni et al., 04 CamNote Balme et al., 04 Drag&Drop, Drag&Pick Baudish et al., 03 Aura de Sousa and Garlan, 02

January 29 - February 1 8Intelligent User Interfaces 2006, Sydney, Australia Introduction Elements represented in in the meta-UI Elements represented in in the meta-UI Representation of source platform target platform space between Detachable Interfaces Grolaux et al., 2004 Pick&Drop Rekimoto, 97 Augmented Surfaces Rekimoto and Masanori, 99 physical Proximal Interactions Rekimoto et al., 03 screenshot ARIS Biehl and Baily, 04 icon 2D map

January 29 - February 1 9Intelligent User Interfaces 2006, Sydney, Australia Introduction Tasks are usually distributed among platforms, but without any consistent way to distribute them, the space between platforms introduces a discontinuity that may disrupt the fluency of the interactive task Tasks are usually distributed among platforms, but without any consistent way to distribute them, the space between platforms introduces a discontinuity that may disrupt the fluency of the interactive task Direct manipulation is proposed to tackle that discontinuity, and thus avoid possible usability problems Direct manipulation is proposed to tackle that discontinuity, and thus avoid possible usability problems

January 29 - February 1 10Intelligent User Interfaces 2006, Sydney, Australia Contents Introduction Introduction MigriXML MigriXML Case study Case study Migration process in detail Migration process in detail Conclusion Conclusion

January 29 - February 1 11Intelligent User Interfaces 2006, Sydney, Australia MigriXML In order to make user interface migration as natural as direct manipulation, a meta-UI in the form of a virtual environment has been developed In order to make user interface migration as natural as direct manipulation, a meta-UI in the form of a virtual environment has been developed MigriXML enables the user to carry out that migration process without the physical presence of the involved platforms, in a virtual and interactive way, relaying on UsiXML language and a set of models MigriXML enables the user to carry out that migration process without the physical presence of the involved platforms, in a virtual and interactive way, relaying on UsiXML language and a set of models

January 29 - February 1 12Intelligent User Interfaces 2006, Sydney, Australia MigriXML To generate the virtual reality scene representing the migration environment, a suite of models is used describing relevant aspects of the problem in terms of UsiXML: To generate the virtual reality scene representing the migration environment, a suite of models is used describing relevant aspects of the problem in terms of UsiXML: The Concrete User Interface The Concrete User Interface The User Model The User Model The Platform Model The Platform Model The Environment Model The Environment Model Describes properties of interest, which can be physical, psychological and organizational Describes properties of interest, which can be physical, psychological and organizational

January 29 - February 1 13Intelligent User Interfaces 2006, Sydney, Australia MigriXML: Environment Model The physical part consists of: The physical part consists of: a scene model based on VRML97/X3D, a scene model based on VRML97/X3D, the multi-surface interaction ontology, the multi-surface interaction ontology, and the Stanford topology model. and the Stanford topology model. Each scene is composed of physical resources and a series of planes, which are decomposed into areas Each scene is composed of physical resources and a series of planes, which are decomposed into areas Each area may be an interaction surface: a monitor on a table, a wall screen, or any area where the UI is projected and/or recognized with computer vision Each area may be an interaction surface: a monitor on a table, a wall screen, or any area where the UI is projected and/or recognized with computer vision Each area is described by attributes like dimensions, relative position, relationships with other objects, etc Each area is described by attributes like dimensions, relative position, relationships with other objects, etc

January 29 - February 1 14Intelligent User Interfaces 2006, Sydney, Australia MigriXML architecture Interactive system architecture Interactive system architecture GUI application GUI application Arch meta-model or ‘Slinky’ meta-model Arch meta-model or ‘Slinky’ meta-model Platform Platform Layered structure Layered structure

January 29 - February 1 15Intelligent User Interfaces 2006, Sydney, Australia MigriXML architecture General MigriXML architecture General MigriXML architecture User’s environment User’s environment GUI application GUI application Arch meta-model fits the UsiXML models Arch meta-model fits the UsiXML models Platform Platform Virtual Hardware (VHW) Virtual Hardware (VHW) Virtual Machine (VM) Virtual Machine (VM) Virtual Window Manager (VWM) Virtual Window Manager (VWM) Migration Manager (MM) Migration Manager (MM)

January 29 - February 1 16Intelligent User Interfaces 2006, Sydney, Australia Implementation This work focuses on the original part of providing a virtual control environment for migrating parts or whole of a UI from one platform to another This work focuses on the original part of providing a virtual control environment for migrating parts or whole of a UI from one platform to another To address its implementation, the VRML97/X3D languages have been used To address its implementation, the VRML97/X3D languages have been used User Interfaces and the Arch meta-model: User Interfaces and the Arch meta-model: Interaction toolkit component: VUIToolkit Interaction toolkit component: VUIToolkit Domain-specific component: Java and Javascript Domain-specific component: Java and Javascript

January 29 - February 1 17Intelligent User Interfaces 2006, Sydney, Australia Implementation VUIToolkit VUIToolkit A set of VRML97/X3D PROTOs that transforms the standard plain widgets into a truly 3D representation (Virtualized User Interface) A set of VRML97/X3D PROTOs that transforms the standard plain widgets into a truly 3D representation (Virtualized User Interface) This toolkit was developed starting from the object classes described in the Concrete Model of UsiXML language This toolkit was developed starting from the object classes described in the Concrete Model of UsiXML language

January 29 - February 1 18Intelligent User Interfaces 2006, Sydney, Australia Implementation Java and Javascript Java and Javascript Most VRML97/X3D browsers can interpret Javascript code and execute Java code Most VRML97/X3D browsers can interpret Javascript code and execute Java code VRML97/X3D specifications describe two programming interfaces to access the scene graph: VRML97/X3D specifications describe two programming interfaces to access the scene graph: SAI (Scene Authoring Interface) SAI (Scene Authoring Interface) EAI (External Authoring Interface) EAI (External Authoring Interface) Allowed us to leave aside, for this first implementation of MigriXML, the complex integration of emulators for each specific platform Allowed us to leave aside, for this first implementation of MigriXML, the complex integration of emulators for each specific platform

January 29 - February 1 19Intelligent User Interfaces 2006, Sydney, Australia Implementation Interactive Java/Javascript application, virtualized user interface rendered in the 3D window of VRML97/X3D browser Interactive Java/Javascript application, virtualized user interface rendered in the 3D window of VRML97/X3D browser

January 29 - February 1 20Intelligent User Interfaces 2006, Sydney, Australia Implementation Interactive application, virtualized user interface and virtual platform in MigriXML Interactive application, virtualized user interface and virtual platform in MigriXML

January 29 - February 1 21Intelligent User Interfaces 2006, Sydney, Australia Contents Introduction Introduction MigriXML MigriXML Case study Case study Migration process in detail Migration process in detail Conclusion Conclusion

January 29 - February 1 22Intelligent User Interfaces 2006, Sydney, Australia Case study The user’s environment is a small office, with five different platforms The user’s environment is a small office, with five different platforms

January 29 - February 1 23Intelligent User Interfaces 2006, Sydney, Australia Case study Display characteristics of each platform Display characteristics of each platform

January 29 - February 1 24Intelligent User Interfaces 2006, Sydney, Australia Case study UsiXML was used to describe the environment UsiXML was used to describe the environment The virtual world was the result of a structured process of creation: The virtual world was the result of a structured process of creation: Requirements: from UsiXML specifications, designers knew the details of the environment Requirements: from UsiXML specifications, designers knew the details of the environment Preparation: compilation of audio-visual material; sketches and mock-ups were created Preparation: compilation of audio-visual material; sketches and mock-ups were created Design, test and optimize: model creation, edit appearance, enhance world Design, test and optimize: model creation, edit appearance, enhance world

January 29 - February 1 25Intelligent User Interfaces 2006, Sydney, Australia Case study User interfaces were specified in UsiXML too User interfaces were specified in UsiXML too Different tools have been developed to help the UI designer to visually create this models Different tools have been developed to help the UI designer to visually create this models IdealXML [Montero et al., 05] IdealXML [Montero et al., 05]

January 29 - February 1 26Intelligent User Interfaces 2006, Sydney, Australia Case study User interfaces were specified in UsiXML too User interfaces were specified in UsiXML too Different tools have been developed to help the UI designer to visually create this models Different tools have been developed to help the UI designer to visually create this models IdealXML IdealXML GrafiXML [B. Michotte] GrafiXML [B. Michotte]

January 29 - February 1 27Intelligent User Interfaces 2006, Sydney, Australia Case study Two different user interfaces Two different user interfaces

January 29 - February 1 28Intelligent User Interfaces 2006, Sydney, Australia Case study Migrating from one laptop to the other Migrating from one laptop to the other Aspire ”, 16: x800 PT-LB10SU 4:3 800x600 Latitude 15”, 4:3 1024x768

January 29 - February 1 29Intelligent User Interfaces 2006, Sydney, Australia Case study Migrating from a laptop to the Pocket PC Migrating from a laptop to the Pocket PC Aspire ”, 16: x800 e750 3,8”, 3:4 240x320

January 29 - February 1 30Intelligent User Interfaces 2006, Sydney, Australia Contents Introduction Introduction MigriXML MigriXML Case study Case study Migration process in detail Migration process in detail Conclusion Conclusion

January 29 - February 1 31Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail The migration process is divided in four stages (A, B, C and D), each one having a finite set of steps, representing a total amount of 14 steps The migration process is divided in four stages (A, B, C and D), each one having a finite set of steps, representing a total amount of 14 steps

January 29 - February 1 32Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage A: Select an interactive application Stage A: Select an interactive application

January 29 - February 1 33Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage A: Select an interactive application Stage A: Select an interactive application The user presses button (M) to grab the window, which follows the user wherever she points at

January 29 - February 1 34Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage A: Select an interactive application Stage A: Select an interactive application The action is transmitted to the corresponding Virtual Window Manager (VWM) by an event

January 29 - February 1 35Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage A: Select an interactive application Stage A: Select an interactive application The VWM forwards the information to the Migration Manager (MM)

January 29 - February 1 36Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage B: Select target platform Stage B: Select target platform

January 29 - February 1 37Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage B: Select target platform Stage B: Select target platform The MM broadcasts a message to all platforms

January 29 - February 1 38Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage B: Select target platform Stage B: Select target platform Platforms change their state to ‘wait-for-selection’, a TouchSensor is activated

January 29 - February 1 39Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage B: Select target platform Stage B: Select target platform The user selects the target platform by clicking on its screen

January 29 - February 1 40Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage B: Select target platform Stage B: Select target platform The Virtual Hardware (VHW) casts an event to inform the VWM

January 29 - February 1 41Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage B: Select target platform Stage B: Select target platform The VWM forwards the message to the MM

January 29 - February 1 42Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage C: Migrate to target platform Stage C: Migrate to target platform

January 29 - February 1 43Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage C: Migrate to target platform Stage C: Migrate to target platform The MM commands to detach the application from source platform

January 29 - February 1 44Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage C: Migrate to target platform Stage C: Migrate to target platform The MM broadcast a message to the rest of platforms, which change to normal execution 10

January 29 - February 1 45Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage C: Migrate to target platform Stage C: Migrate to target platform The MM takes the UsiXML specification and re- generates the app UI

January 29 - February 1 46Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage D: Adapt to target platform Stage D: Adapt to target platform

January 29 - February 1 47Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage D: Adapt to target platform Stage D: Adapt to target platform The MM asks the target platform to host the re- generated application

January 29 - February 1 48Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage D: Adapt to target platform Stage D: Adapt to target platform The application is launched, and its virtualized UI added to the scene graph of the target VHW

January 29 - February 1 49Intelligent User Interfaces 2006, Sydney, Australia Migration process in detail Stage D: Adapt to target platform Stage D: Adapt to target platform The virtualized UI is adapted to the target screen and windowing system

January 29 - February 1 50Intelligent User Interfaces 2006, Sydney, Australia Contents Introduction Introduction MigriXML MigriXML Case study Case study Migration process in detail Migration process in detail Conclusion Conclusion

January 29 - February 1 51Intelligent User Interfaces 2006, Sydney, Australia Conclusion MigriXML and previous work MigriXML and previous work MigrationInitiated by app user interface systemuser controlpresentation Bharat and Cardelli, 95 Song, Chu and Kurakake, 02 MIGRATION Project Bandelloni et al., 04 CamNote Balme et al., 04 Drag&Drop, Drag&Pick Baudish et al., 03 Aura de Sousa and Garlan, 02 MIGRIXML

January 29 - February 1 52Intelligent User Interfaces 2006, Sydney, Australia Conclusion MigriXML and previous representations MigriXML and previous representations Representation of source platform target platform space between Detachable Interfaces Grolaux et al., 2004 Pick&Drop Rekimoto, 97 Augmented Surfaces Rekimoto and Masanori, 99 physical Proximal Interactions Rekimoto et al., 03 screenshot ARIS Biehl and Baily, 04 icon 2D map MIGRIXML 3D virtual

January 29 - February 1 53Intelligent User Interfaces 2006, Sydney, Australia Conclusion MigriXML highlights: MigriXML highlights: As an environment for simulating migration, there is no need to have the physical device to carry on the process of adapting a previous UI to a new platform As an environment for simulating migration, there is no need to have the physical device to carry on the process of adapting a previous UI to a new platform As a meta-UI for migration, it could also render remote spaces, moving us forward tele-presence applications As a meta-UI for migration, it could also render remote spaces, moving us forward tele-presence applications In both cases, all involved elements are graphically rendered and the migration is continuously depicted during the process, avoiding disruption in the user’s task, and thus improving usability In both cases, all involved elements are graphically rendered and the migration is continuously depicted during the process, avoiding disruption in the user’s task, and thus improving usability

January 29 - February 1 54Intelligent User Interfaces 2006, Sydney, Australia Conclusion As future work, evaluations are desirable: As future work, evaluations are desirable: On the one hand, expert and user evaluation in order to assess the influence of the meta-display problem, that is, how the resolution of the display that renders the meta-UI can affect utility and usability of this approach On the one hand, expert and user evaluation in order to assess the influence of the meta-display problem, that is, how the resolution of the display that renders the meta-UI can affect utility and usability of this approach On the other hand, summative evaluation with other meta-UI representations, such as the iconic 2D map of ARIS, in order to assess which approach is better for a given scenario On the other hand, summative evaluation with other meta-UI representations, such as the iconic 2D map of ARIS, in order to assess which approach is better for a given scenario

Thank you Mercí Gracias More info: Contact: