User Interfaces Composition with UsiXML Sophie LEPREUX (1), Jean VANDERDONCKT (2), Christophe KOLSKI (1) (1) LAMIH “Automated Reasoning and Human-Computer Interaction” Research Group University of Valenciennes and Hainaut-Cambrésis, France (2) BCHI University of Louvain-La-Neuve, Belgium UsiXML-EICS Berlin, Germany - June 20, 2010

- To take the user caracteristics, the platform used, the environment into account, to allow an universal access to information, knowledge, services… in numerous different contexts Introduction  New UI design problems, focus on UI Composition Brain-Computer Interface

- Tree-algebra based UI Composition -UsiXML framework -Composition/decomposition Operators -Compose to support the 7 Dimensions -Conclusion Summary

Tree-algebra based UI Composition Window (id=window, name=window, width=« 500 » height=« 350 ») Box (type=« vertical ») Button (DefaultContent = Save) Button (DefaultContent=Close) Output (DefaultContent =« Fiche Client ») Box (type = horizontal) Box (type = horizontal) Output (…) Input (…) Box (type = horizontal) Output (…) Input (…) tag content <window id="window_component_0" name="window_component_0" defaultContent=« Nouveau client" width="500" height="350"> <outputText id="output_text_component_2" name="output_text_component_2" defaultContent=« Fiche Client" isVisible="true" isEnabled="true" isBold="true" textColor="#000000"/> <outputText id="output_text_component_2" name="output_text_component_2« defaultContent=« Fiche Client" isVisible="true" isEnabled="true" isBold="true" textColor="#000000"/> …

- Tree-algebra based UI Composition -UsiXML framework -Composition/decomposition Operators -Compose to support the 7 Dimensions -Conclusion Summary

UsiXML framework Task & Concepts Abstract UI Concrete UI Final UI Source = Target = Task & Concepts Abstract UI Concrete UI Final UI

Summary - Tree-algebra based UI Composition -UsiXML framework -Composition/decomposition Operators -Compose to support the 7 Dimensions -Conclusion

Composition/decomposition Operators Cut( T1, ) T1 T2 Unary operators Binary operators Similarity Equivalence Left difference Right difference Intersection Unique union Normal union Fusion ∩ Complementary( T1, ) Projection( T1, ) Selection( T1, ) T1

Composition/decomposition Operators Example: Normal Union operator applied on two user interfaces U horizontal =

Composition/decomposition Operators Illustrations of operators on CUI models T1 = tourist application UI T2 = event management application UI (a) Selection (T1, set of elements) (b) Projection (T1, {outputText, Button}) (c) Intersection (T1, T2)

- Tree-algebra based UI Composition -UsiXML framework -Composition/decomposition Operators -Compose to support the 7 Dimensions -Conclusion Summary

Compose to support the 7 Dimensions - Use of (de)composition operators to adapt the UI according to the 7 dimensions: (1) Multi-device Usage and/or (2) Multi-User interface, (3) Multi-linguality, (4) Multi- organization, (5) Multi-context usage, (6) Multi-Modality usage, (7) Multi-platform Usage

Compose to support the 7 Dimensions (1) Multi-device usage Task & Concepts Abstract UI Concrete UI Final UI Source = Task & Concepts Abstract UI CUI ’ 1 FUI’ 1 Target = FUI’ n CUI’ n … … CUI’ 1 = Selection (set elements of Music controller task, Concrete UI) CUI’ 2 = Selection (set elements of Sounder player task, Concrete UI) CUI’ 3 = Selection (set elements of volume controller, Concrete UI) Adaptation 1: decomposition of the CUI Source in several CUIs Adaptation 2: Fusion of several decompositions of the CUI source CUI’ 1 = Selection (set elements of Music controller task, Concrete UI) CUI’ 2 = Selection (set elements of Sounder player task, Concrete UI) Union Selection (set elements of volume controller, Concrete UI) CUI’ 3 = Selection (set elements of volume controller, Concrete UI) - Illustration: control of a music player

Compose to support the 7 Dimensions (2) Multi-user interface - Two interesting cases: Case #1: a shared UI is co-localized, the users can simultaneously interact on one UI Ex.: tactile multi-touch interface or tangible objects used Case #2: a user arrives in a collective work situation (or leaves it): the user needs a work space, with personal data and/or functions - Other interesting problems to consider : Distant shared UI Consideration of the user rights or roles (ex: moderator UI in a brainwriting session at a distance: composed differently as the UI of the other participants)

Compose to support the 7 Dimensions (3) Multi-linguality - if two similar UI are modeled at the CUI level (or higher) with specifications given for one (or more) language and another UI modeled with others languages, it can be possible to compose these UI to obtain one UI modeled for all these languages - Importance to consider translation considerations leading to necessities about composition/decomposition due to socio-cultural aspects Ex: source language read from left to right, and downwards (english) target language read from right to left, and downwards (arabic), or upwards (for instance: Chinese) Consequence: different zones of the UI have to be composed/decomposed differently

Compose to support the 7 Dimensions (4) Multi-organization T1 & C T’& C T2 & C T’ = T1 U T2 Source = Target = Source = - Importance to consider how the tasks have to be performed in different organizations or groups - Importance to consider the organization models described in the literature (ex. in social sciences or multi-agent system domain)

Compose to support the 7 Dimensions (5) Multi-context usage - Several composition/decompositions strategies are possible: Ex.: development methodology based on patterns, business component and machine learning (plastic UI approach, [PhD Hariri 08]) Ex: integration of the task model in the business component in order to compose complete application and to facilitate the co-evolution of the system [Bourguin G., Lewandowski A. & Tarby J.C., TAMODIA’2007] - A context modification can generate evolutions at each abstraction level Ex: a lighting change acts on the luminosity of the device (FUI) whereas the background noise change acts (or not) on the choice of the vocal modality. Ex: a modification of the user work brings modification of tasks and/or on the tasks planning  Interesting composition/decomposition problems to consider

Compose to support the 7 Dimensions (6) Multi-Modality usage - The AUI layer allows specifying the UI independently of the modality  According to needs brought by context change, a UI composition at a high level allows merging UI which will be concretized by different modalities in the lower levels - Ex : Food order realized with several modalities AUI 1 AUI’ AUI T’ = T1 U T2 AUI’ =AUI 1 U AUI 2 Normal Union (AUI (order a pizza), AUI (Order Chinese food))

Compose to support the 7 Dimensions (7) Multi-platform Usage - Analogy: example of multi-platform language is: Java  Execution on several platforms without changing model or implemention - Our objective: to do the same from the UI point of view  to generate code from CUI model to FUI model  For example, graphical editor GraphiXML allows generating java or XHTML code So the graphical UI which is modeled with GraphiXML is multi-platform The composition operators: not needed into this process

- Tree-algebra based UI Composition -UsiXML framework -Composition/decomposition Operators -Compose to support the 7 Dimensions -Conclusion Summary

Conclusion - Presentation of operators developed initially to compose at design time and at the CUI level - Currently: operators used to each abstraction level of UI - Operators: could be used at the runtime - These operators: a support to UI adaptation - Illustrations in the paper for the 7 dimensions: Multi-device Usage, Multi-User interface, Multi-linguality, Multi- organization, Multi-context usage, Multi-Modality usage, Multi-platform Usage  But: adaptations to criterion (dimension) can be done to the detriment of the others  As a result, the adaptation must be global i.e. the adaptation has to take into account all the criteria simultaneously and not considered one by one - Research questions are open concerning:  (1) the one by one or simultaneous consideration of these dimensions,  (2) the order in taking them into consideration …

User Interfaces Composition with UsiXML Sophie LEPREUX (1), Jean VANDERDONCKT (2), Christophe KOLSKI (1) (1) LAMIH “Automated Reasoning and Human-Computer Interaction” Research Group University of Valenciennes and Hainaut-Cambrésis, France (2) BCHI University of Louvain-La-Neuve, Belgium UsiXML-EICS Berlin, Germany - June 20, 2010 Ajout logo1 JVD Ajout logo2 JVD THANK YOU FOR YOUR ATTENTION