1. Université catholique de Louvain (UCL) Belgian Laboratory of Computer-Human Interaction (BCHI) Place des Doyens, 1 B-1348 Louvain-la-Neuve (Belgium) Presented by CAiSE'2005 Keynote address - Porto, June 16, 2005 1 A MDA-Compliant Environment for Developing User Interfaces of Information Systems Jean Vanderdonckt vanderdonckt@isys.ucl.ac.be Head of BCHI Lab, http://www.isys.ucl.ac.be/bchi

2. CAiSE'2005 Keynote address - Porto, June 16, 2005 2 « Everything you can imagine is real » (Picasso) « Everything you model can be turned into a real interface »

3. CAiSE'2005 Keynote address - Porto, June 16, 2005 3 Outline 1. Conceptual modeling of user interfaces 2. Forward engineering 3. Reverse engineering 4. Lateral engineering 5. Conclusion

4. CAiSE'2005 Keynote address - Porto, June 16, 2005 4 1.1 Why is conceptual modeling of user interfaces desirable? “The presentation layer outside the scope of CSCD” (Antoni Olivé) –A domain model is certainly not a presentation model, but there could be a mapping between UIs remained for many years the poor parent of conceptual modeling and software engineering –It emerged during late 80’s –Still growing The complexity and the variety of user interfaces (UI) is dramatically increasing The paradigm of « one system fits all » is no longer valid

5. CAiSE'2005 Keynote address - Porto, June 16, 2005 5 1.1 Diversity of users Traditional users People with disabilities –Visual, motor, cognitive

6. CAiSE'2005 Keynote address - Porto, June 16, 2005 6 1.1 Variety of tasks Users want to determine their path on each platform [Forrester Research, Inc., 2003]

7. CAiSE'2005 Keynote address - Porto, June 16, 2005 7 1.1 Heterogeneousness of platforms

8. CAiSE'2005 Keynote address - Porto, June 16, 2005 8 1.1 Multiplicity of contexts of use LocationRoleDeviceExperience SportingMultimedia Travel programme WorkingTravel booking site Powerful interface for complex operations TravellingBooking notification Everywhere connectivity for simple data exchange FamilyTV is multi-media family device #1

9. CAiSE'2005 Keynote address - Porto, June 16, 2005 9 UI #12UI #11UI #10UI #9Application 3 UI #8UI #7UI #6UI #5Application 2 UI #4UI #3UI #2UI #1Application 1 Platform #4Platform #3Platform #2Platform #1 Application 1 Application 2 Application 3 UI model #1 UI model #2 UI model #3 Platform #1 Platform #2 Platform #3 Platform #4 Platform model #1 Platform model #2 Platform model #3 Platform model #4 1.1 Consequence Proliferation of developments

10. CAiSE'2005 Keynote address - Porto, June 16, 2005 10 1.1 Consequence Why is it difficult? –For the designer: Multiplicity of contexts of use No systematic design method –For the user: Poor usability of resulting UI UI not adapted to the genuine context of use –For the developer: Increase of development and maintenance costs Increasing development complexity Little of no factoring out of common elements Fold & Drop techniqueFold & Drop technique: P. Dragicevic

11. CAiSE'2005 Keynote address - Porto, June 16, 2005 11 1.2 What does it cover? Conceptual modeling should cover –Presentation: external manifestation of the UI that is perceivable to the user (static) –Dialog: ordering in time and space of operations performed by the user/system (dynamic) –Modalities of interaction Graphical Vocal Haptic, tactile

12. CAiSE'2005 Keynote address - Porto, June 16, 2005 12 1.2 What does it cover? User Platform Domain Task Environment

13. CAiSE'2005 Keynote address - Porto, June 16, 2005 13 1.3 Abstraction #1: the concrete UI Starting point: FUI = Final User Interface –Mark-up languages: HTML, WML, cHTML –Programming languages: Visual Basic, Java Many manifestations of the same object Abstraction with respect to the toolkit –Presentation aspects –Events: generating, passing, controlling –Attributes: external (e.g., font) vs internal (e.g. state) –Primitives: life cycle

14. CAiSE'2005 Keynote address - Porto, June 16, 2005 14 PR_IO_CreatePR_IO_Display IO createdIO displayedIO activatedIO deactivatedIO undisplayedIO destroyed PR_IO_Deactivate PR_OI_ErasePR_OI_DestroyPR_IO_Activate PR_IO_ChangeAttribute 1.3 Abstraction #1: the concrete UI Definitions –Original: Abstract Interaction Object = abstraction of the same Concrete Interaction Objects independently of their underlying presentation and dialog –We have introduced this abstraction in 1993! –Current: Concrete Interaction Object = abstraction of interaction objects with respect to the underlying toolkit [Vanderdonckt & Bodart, 1993]Vanderdonckt & Bodart, 1993

15. CAiSE'2005 Keynote address - Porto, June 16, 2005 15 1.3 Abstraction #1: the concrete UI Each graphicalCIO has specific properties such as isVisible, isEnabled, fgColor, and bgColor. Each graphicalCIO is sub-typed into one of the two possible categories: graphicalContainer, and graphicalIndividualCom ponent CUI Model CIOgraphicalCIO graphicalContainergraphicalIndividualComponent CUI Model CIO graphicalCIO graphicalContainergraphicalIndividualComponent graphicalContainer graphicalIndividualComponent GrafiXML [Limbourg, 2004]Limbourg, 2004

16. CAiSE'2005 Keynote address - Porto, June 16, 2005 16 1.3 Abstraction #2: the abstract UI Different CIOs can be used for the same purpose, but with different interaction modalities Definition –Abstract Container = set of Abstract Individual Component –AIC = abstraction of CIOs of the same type, but independently of any interaction modality –Abstract User Interface (AUI) = decomposition into AC+AIC

17. CAiSE'2005 Keynote address - Porto, June 16, 2005 17 1.3 Abstraction #2: the abstract UI [Limbourg, 2004]Limbourg, 2004

18. CAiSE'2005 Keynote address - Porto, June 16, 2005 18 1.3 Abstraction #2: the abstract UI Notation: based on canonical abstract prototypes: Larry Constantine, Helmut Windl, James Noble, & Lucy Lockwood: http://www.forUse.comhttp://www.forUse.com Abs.container A. component input output navigation control select IdealXML [Montero et al., 2005]Montero et al., 2005

19. CAiSE'2005 Keynote address - Porto, June 16, 2005 19 1.3 Abstraction #3: the task Task = set of actions carried out by a user in a given context to reach a goal Logical decomposition of task into sub-tasks Temporal ordering: LOTOS operators –T1 >> T2Enabling –T1 [ ]>> T2Enabling + information passing –T1 |> T2Suspend/resume –T1 [ ] T2Choice –T1 [> T2Disabling (e.g. Form submit) –T1 |=| T2Independence (any order, but finished) –T1*Iteration –T1{n}Finite iteration –T1 ||| T2Concurrency –T1 |[x]| T2Concurrency + information passing –[T] Optional –TRecursion [Paternò et al., 1997] CTTE Editor

20. CAiSE'2005 Keynote address - Porto, June 16, 2005 20 1.3 Abstraction #3: the task Task definition = action + object –Action types Acquire, render, modify, publish, compute, derive,… –Object types: Element, list, table, collection, compound,…

21. CAiSE'2005 Keynote address - Porto, June 16, 2005 21 1.3 Abstraction #3: the task [Limbourg, 2004]Limbourg, 2004

22. CAiSE'2005 Keynote address - Porto, June 16, 2005 22 1.3 Abstraction #4: the context of use Context of use= triple (U,P,E) where –U is a user model: from cognitive psychology –P is a platform model: subset of UAProf (W3C) –E is an environment model: from ubiquitous computing Pick & Drop techniquePick & Drop technique: J. RekimotoJ. Rekimoto [Cameleon project, 2004]Cameleon project, 2004

23. CAiSE'2005 Keynote address - Porto, June 16, 2005 23 1.3 Abstraction #4: the context of use [Limbourg, 2004]Limbourg, 2004

24. CAiSE'2005 Keynote address - Porto, June 16, 2005 24 triggers (tg):{, } x updates (up): x observes (ob): x isExecutedIn (ex): x manipulates (ma):{, } x These mappings can be established: 1.3 Mapping the models [Montero et al., 2005]Montero et al., 2005

25. CAiSE'2005 Keynote address - Porto, June 16, 2005 25 1.3 Mapping the models Mapping the models with a mapping model (!!) [Limbourg, 2004]Limbourg, 2004

26. CAiSE'2005 Keynote address - Porto, June 16, 2005 26 1.4 What do we have so far? Final User Interface (FUI) ConcreteUser Interface (CUI) Abstract User Interface (AUI) Task& Domain Rendering Code Modality-independent Abstract Individual Component Task Classes HTML push button Download Down Load Toolkit-independent Concrete Interaction Object Final User Interface (FUI) ConcreteUser Interface (CUI) Abstract User Interface (AUI) Task& Rendering Code Task Classes <input type=submit value=“Download" name= Download Down Load Down Load Windows push button Method triggered: download a file Object: computer file OSF/Motif XmButton Graphical 2D push button Digital control AIC VRML97/X3D button Software key Function key Graphical 3D push buttonPhysical push button Physical control AIC Control AIC

27. CAiSE'2005 Keynote address - Porto, June 16, 2005 27 1.4 What do we have so far? Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use Reification UsiXML unsupported model UsiXML supported model User S Platform SEnvironment S

28. CAiSE'2005 Keynote address - Porto, June 16, 2005 28 2. Forward engineering 2.1 Task and domain models 2.2 From T&D to AUI –Model-to-model transformation 2.3 From AUI to CUI –Model-to-model transformation 2.4 From CUI to FUI [Limbourg, 2004]Limbourg, 2004

29. CAiSE'2005 Keynote address - Porto, June 16, 2005 29 2.1 Task and domain models

30. CAiSE'2005 Keynote address - Porto, June 16, 2005 30 2.1 Task and domain models [Limbourg, 2004]Limbourg, 2004

31. CAiSE'2005 Keynote address - Porto, June 16, 2005 31 2.1 Task and domain models

32. CAiSE'2005 Keynote address - Porto, June 16, 2005 32 2.2 From T&D to AUI Definition of AUI structure –Which objects should be logically grouped? Definition of AIC types – Which « functionnality » should assume AICs and what data do they manipulate ? Definition of spatio-temporal arrangement –How should AIC be arranged in space and/or in time ? Definition of dialog control –What is the valid flow of action on AIOs ? [Limbourg, 2004]Limbourg, 2004

33. CAiSE'2005 Keynote address - Porto, June 16, 2005 33 2.2 From T&D to AUI Definition of AUI structure [Limbourg, 2004]Limbourg, 2004

34. CAiSE'2005 Keynote address - Porto, June 16, 2005 34 2.2 From T&D to AUI Definition of AIC types AC = Abstract Component AIC = Abstract Individual Component CIC = Concrete Interaction Component [Limbourg, 2004]Limbourg, 2004

35. CAiSE'2005 Keynote address - Porto, June 16, 2005 35 2.2 From T&D to AUI Definition of spatio-temporal arrangement [Limbourg, 2004]Limbourg, 2004

36. CAiSE'2005 Keynote address - Porto, June 16, 2005 36 2.2 From T&D to AUI All transformations are in UsiXML –Each model = instance of meta-model –Each model = graph as instance of graph type –Each model-to-model transformation = graph transformation Set of productions

37. CAiSE'2005 Keynote address - Porto, June 16, 2005 37 2.2 From T&D to AUI Typed model-to-model transformation Uses language Meta-Meta-Model Graph Structure is instance of Meta-Model Our Meta-Model Meta-Model Subset 1 e.g., Task+Domain Model is instance of Meta-Model Subset 2 e.g., Concrete UI Model is instance of Initial UI Model e.g., MyTaskAndDomainModel Resultant UI Model e.g., MyConcreteUIModel Transformation Rule Our transformation catalog [Limbourg, 2004]Limbourg, 2004

38. CAiSE'2005 Keynote address - Porto, June 16, 2005 38 2.2 From T&D to AUI TransformiXML

39. CAiSE'2005 Keynote address - Porto, June 16, 2005 39 2.3 From AUI to CUI Definition of CUI structure –Which AIC is a window? Definition of CIO type –Which « widget » should represent which AIO ? Definition of placement –What layout can be specified between CIOs –Definition of navigation –Which container can be started or closed from which container? Definition of dialog control –What is the dialog local to each CIO? –What is the valid flow of action on CIOs? [Limbourg, 2004]Limbourg, 2004

40. CAiSE'2005 Keynote address - Porto, June 16, 2005 40 2.3 From AUI to CUI Definition of CIO structure NACLHSRHS ::= NACLHSRHS ::= [Limbourg, 2004]Limbourg, 2004

41. CAiSE'2005 Keynote address - Porto, June 16, 2005 41 2.3 From AUI to CUI Definition of placement [Limbourg, 2004]Limbourg, 2004

42. CAiSE'2005 Keynote address - Porto, June 16, 2005 42 2.4 From CUI to FUI Model-to-code transformation –By rendering (interpretation) Java (InterpiXML)InterpiXML Tcl/tk (QtkiXML)QtkiXML Flash (FlashiXML)FlashiXML –By code generation HTML Visual Basic

43. CAiSE'2005 Keynote address - Porto, June 16, 2005 43 2.4 What do we have so far? Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use Reification Abstraction UsiXML unsupported model UsiXML supported model User S Platform SEnvironment S

44. CAiSE'2005 Keynote address - Porto, June 16, 2005 44 3. Reverse engineering 3.1 From FUI to CUI 3.2 From CUI to AUI, task & domain

45. CAiSE'2005 Keynote address - Porto, June 16, 2005 45 3.1 From FUI to CUI Do you have the source code? –Markup languages (e.g., HTML): static code analysis (ReversiXML)ReversiXML Do you have the compiled code? –Programming languages (e.g., compiled C): resource file extraction and static code analysis Do you have the running application? –Video analysis: computer vision Do you have only the documentation? –Image analysis: image processing [Bouillon & Vanderdonckt, 2004]Bouillon & Vanderdonckt, 2004

46. CAiSE'2005 Keynote address - Porto, June 16, 2005 46 3.1 From FUI to CUI ReversiXML [Bouillon & Vanderdonckt, 2004]Bouillon & Vanderdonckt, 2004

47. CAiSE'2005 Keynote address - Porto, June 16, 2005 47 3.2 From CUI to AUI, task & domain Code-to-model and model-to-model –CUI to AUI [Limbourg, 2004]Limbourg, 2004

48. CAiSE'2005 Keynote address - Porto, June 16, 2005 48 3.2 From CUI to AUI, task & domain Re-engineering = reverse + forward Possible re- interpretation by QtkiXMLQtkiXML [Bouillon & Vanderdonckt, 2004]

49. CAiSE'2005 Keynote address - Porto, June 16, 2005 49 3.2 What do we have so far? Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use Reification Abstraction UsiXML unsupported model UsiXML supported model User S Platform SEnvironment S

50. CAiSE'2005 Keynote address - Porto, June 16, 2005 50 Environment T 3.2 What do we want to get? Final user Interface T Concrete user Interface T Task and Domain T Abstract user Interface T T=Target context of use Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use Reification Abstraction Reflexion Translation http://www.plasticity.org UsiXML unsupported model UsiXML supported model User S Platform SEnvironment SPlatform T User T [Cameleon project, 2004]Cameleon project, 2004

51. CAiSE'2005 Keynote address - Porto, June 16, 2005 51 3.3 Lateral engineering Definition = model-to-model transformation applied at the same level of abstraction –Same context of use –Across various contexts of use Examples –Simple CUI adaptation: widget replacement –Design-time vs run-time adaptation

52. CAiSE'2005 Keynote address - Porto, June 16, 2005 52 Example 1: widget replacement (1) 7 Youri... <radioButton groupName=“grupo01" defaultContent="Femme" defaultState="false" id="radiobutton_0"/> <radioButton groupName="grupo01" defaultContent="Homme" defaultState="true" id="radiobutton_1"/>... Excerpt for an usiXML CUI specification. [Limbourg et al., 2004]Limbourg et al., 2004

53. CAiSE'2005 Keynote address - Porto, June 16, 2005 53 Example 1: widget replacement (2)

54. CAiSE'2005 Keynote address - Porto, June 16, 2005 54 Example 1: widget replacement (3) The UsiXML graph before applying any rule

55. CAiSE'2005 Keynote address - Porto, June 16, 2005 55 Example 1: widget replacement (4) LHS RHS NAC Rule 1: Create a new comboBox with the same id and name as the name of the group of radioButtons. [Limbourg et al., 2004]Limbourg et al., 2004

56. CAiSE'2005 Keynote address - Porto, June 16, 2005 56 Example 1: widget replacement (5) Rule 1: Create a new comboBox with the same id and name as the name of the group of radioButtons. The usiXML graph after aplying the first rule

57. CAiSE'2005 Keynote address - Porto, June 16, 2005 57 Example 1: widget replacement (6) LHSRHS ::= Rule 2: Convert every radioButton within the group “x” into an item for the comboBox “x”, we have just created. [Limbourg et al., 2004]Limbourg et al., 2004

58. CAiSE'2005 Keynote address - Porto, June 16, 2005 58 Example 1: widget replacement (7) Rule 2: Convert every radioButton within the group “x” into an item for the comboBox “x”, we have just created. The usiXML graph after aplying the second rule

59. CAiSE'2005 Keynote address - Porto, June 16, 2005 59 7 Youri...... Example 1: widget replacement (8) Excerpt from the final transformated usiXML specification

60. CAiSE'2005 Keynote address - Porto, June 16, 2005 60 Example 1: widget replacement (9)

61. CAiSE'2005 Keynote address - Porto, June 16, 2005 61 Example 2: Adaptation Two forms of adaptation –Adaptability: user-initiated –Adaptivity: system-initiated Four steps: user, system, third party, combination –Detection –Computation –Decision –Execution 2 Adaptivity forms –UI remain the same: vectorial UIs –UI change at run-time: plastic UIs

62. CAiSE'2005 Keynote address - Porto, June 16, 2005 62 Example 2: Plastic UI [Grolaux, Van Roy, Vanderdonckt, 2002]Grolaux, Van Roy, Vanderdonckt, 2002

63. CAiSE'2005 Keynote address - Porto, June 16, 2005 63 Example 3: multimodal UI Need for additional abstraction Additional transformation rules [Stanciulescu, Limbourg, Michotte, Vanderdonckt, 2005]Stanciulescu, Limbourg, Michotte, Vanderdonckt, 2005

64. CAiSE'2005 Keynote address - Porto, June 16, 2005 64 Example 4: Crazy UIs Use the same model –for 3D UI: Cube –for migration across platforms in virtual reality: MigriXML [Molina, Vanderdonckt, Gonzalez, 2005]Molina, Vanderdonckt, Gonzalez, 2005

65. CAiSE'2005 Keynote address - Porto, June 16, 2005 65 5. Conclusion A multi-path support of MDA UsiXML model: Abstract user interface UsiXML model: Concrete user interface Rendering Final user interface UsiXML models: task, domain Generative programming Graph transformations Graph transformations Derivation rules IdealXML ReversiXML FlashiXML VisualiXML TransformiXML GrafiXML VisiXML SketchiXML FormiXML KnowiXML

66. CAiSE'2005 Keynote address - Porto, June 16, 2005 66 5. Conclusion Conceptual modeling of UIs –Separation of concerns is feasible –Correlability of models is required –It is possible to have multiple abstractions on demand –Not all models should be involved –Everything is in UsiXML –Extreme modeling or extreme engineering Everything is in the model: relatively easy Model-to-model transformation: harder Model-to-code generation: hardest Industrial practise –Multi-path development –Sketching of UI

67. CAiSE'2005 Keynote address - Porto, June 16, 2005 67 Future trend: sketching? [Coyette & Vanderdonckt, 2005]Coyette & Vanderdonckt, 2005

68. CAiSE'2005 Keynote address - Porto, June 16, 2005 68 I would like to dedicate this talk to Prof. Em. François Bodart (Univ. of Namur, BE) for –Inoculating me the virus of (meta-)modeling and structured design of information systems –Sharing with me his vision about user interfaces and interactive information systems

69. CAiSE'2005 Keynote address - Porto, June 16, 2005 69 I would like to thank BCHI team members for their involvement and their constant effort in the UsiXML initiative among others –Laurent Bouillon: ReversiXML, UsiXMLLaurent Bouillon –Benoît Collignon: PlastiXMLBenoît Collignon –Adrien Coyette: SketchiXMLAdrien Coyette –Murielle Florins: Graceful degradationMurielle Florins –Monica Gemo: multi-platform UIs and annotationsMonica Gemo –Juan Gonzalez: 3D UsiXMLJuan Gonzalez –Donatien Grolaux: detachable UIs, DistriXMLDonatien Grolaux –Josefina Guerrero: UIs for workflowsJosefina Guerrero –Quentin Limbourg: multi-path development, UsiXML, all toolsQuentin Limbourg –Céline Mariage: MetroWeb, guidelinesCéline Mariage –Benjamin Michotte: GrafiXML, TransformiXML, UsiXMLBenjamin Michotte –José Pascual Molina: MigriXML, VUIToolkitJosé Pascual Molina –Francisco Montero: IdealXMLFrancisco Montero –Adrian Stanciulescu: ModaliXML, UsiXMLAdrian Stanciulescu –Daniela Trevisan: augmented realityDaniela Trevisan –…

70. CAiSE'2005 Keynote address - Porto, June 16, 2005 70 I would like to thank João Falcão e Cunha, Oscar Pastor, Nuno Jardim Nunes The CAiSE Advisory Committee –Janice Bubenko –Collette Roland –Arne Solvberg

71. CAiSE'2005 Keynote address - Porto, June 16, 2005 71 … and you for your attention Free download and register USer Interface eXtensible Language http://www.usixml.org SIMILAR, the European task force making user interfaces similar to human-to-human communication http://www.similar.cc Home Page of BCHI Lab http://www.isys.ucl.ac.be/bchi