A model-based development for 3D User Interfaces Juan Manuel Gonzalez Calleros Université catholique de Louvain, School of Management (IAG) Information Systems Unit (ISYS) Louvain-la-Neuve, 13th June, 2006

June 13thDEA Louvain-la-Neuve, Belgium2 Contents  Introduction  Related Work  Model-based development for 3D User Interfaces  Case Study  Conclusion

June 13thDEA Louvain-la-Neuve, Belgium3 Introduction - Problem  Current methods: rarely provide the design knowledge rarely provide the design knowledge are more focusing on the implementation are more focusing on the implementation  Available tools for 3D UIs are: Toolkits, interface builders, rendering engines

June 13thDEA Louvain-la-Neuve, Belgium4 Introduction – proposal  We propose: a user-centered approach a user-centered approach model-driven architecture model-driven architecture separation of concerns separation of concerns  Our goal:  “to demonstrate the feasibility of a MDE- compliant method that is user-centered as opposed to contents-centric for developing 3D UIs”

June 13thDEA Louvain-la-Neuve, Belgium5 Contents  Introduction  Related Work  Model-based development for 3D User Interfaces  Case Study  Conclusion

June 13thDEA Louvain-la-Neuve, Belgium6 Related work – 3D UIs generation  Programmatic approach: OpenGL, Direct3D, C++, MCNL, VRML, X3D, …  Toolkit programming: Maya, Max3D, Crazy Eddie's GUI System, Anark, Alice  Render engines: Microsoft task gallery, infinite 3D Cube, Sphere XP, looking glass, Clara browser, 3DNA  Model based: ontoWEb (VR-Wise), ConGenIVE (VRIXML), InTML, Contigra

June 13thDEA Louvain-la-Neuve, Belgium7 Related work – Programmatic Approach, VRML

June 13thDEA Louvain-la-Neuve, Belgium8 Related work – Toolkit Approach, Anark

June 13thDEA Louvain-la-Neuve, Belgium9 Related work – Render Engines, 3DNA

June 13thDEA Louvain-la-Neuve, Belgium10 Related work – Model-based approach, CoGenIVE-VRIXML    <Tickstyle orientation =" horizontal“ position =" both "  frequency ="16"/>      

June 13thDEA Louvain-la-Neuve, Belgium11 Related work – Observations  Methodological diversity  Not user centered approaches  Single entry and single output  Transformations are hidden

June 13thDEA Louvain-la-Neuve, Belgium12 Related work – Shortcomings  Lack of a methodology for developing 3D UIs  Lack of user task models  Lack of models independent to the final implementations  Lack of a toolkit for such work  Lack of genuine 3D UIs

June 13thDEA Louvain-la-Neuve, Belgium13 Contents  Introduction  Related work  Model-based development for 3D User Interfaces  Case Study  Conclusion

June 13thDEA Louvain-la-Neuve, Belgium14 Model-based development of 3DUIs requirements  Expressivity  Machine processable  Human readable  Methodological explicitness, flexibility, separation of concerns

June 13thDEA Louvain-la-Neuve, Belgium15 We rely on…  Cameleon Framework  UsiXML  UML class diagrams  Graph transformations

June 13thDEA Louvain-la-Neuve, Belgium16 Method Outline

June 13thDEA Louvain-la-Neuve, Belgium17 Development paths

June 13thDEA Louvain-la-Neuve, Belgium18 Graph transformation tool support AGG

June 13thDEA Louvain-la-Neuve, Belgium19 Method Step1 – Task and concepts description

June 13thDEA Louvain-la-Neuve, Belgium20 Method Step1 - User Action Task Extension actionTypeDefinition TriggerSpecifies the triggers of an action TerminateSpecifies the end to an action IndicateSpecifies some sort of indication CreateSpecifies the creation an item ModifyAn action of modifying an item ToggleThe existence of two different states of an item CommunicateThe action to communicate the user to the system or vice versa MediateThe action of mediate user actions PerceiveThe action of identifying any user action

June 13thDEA Louvain-la-Neuve, Belgium21 Method Step1– 3D Task Patterns  A pattern is not a finished design that can be transformed directly into code  It is a description or template for how to solve a problem that can be used in many different situations  We propose some patterns for common 3D user tasks

June 13thDEA Louvain-la-Neuve, Belgium22 Method Step1 – 3D Task Patterns  Navigation pattern Travel Travel Wayfinding Wayfinding

June 13thDEA Louvain-la-Neuve, Belgium23 Method Step1– 3D Task Patterns  Navigation pattern Travel pattern Travel pattern Wayfinding Wayfinding

June 13thDEA Louvain-la-Neuve, Belgium24 Method Step1– 3D Task Patterns  Navigation pattern Travel Travel Wayfinding pattern Wayfinding pattern

June 13thDEA Louvain-la-Neuve, Belgium25 Method Step1– Domain Model

June 13thDEA Louvain-la-Neuve, Belgium26 Method Step1 – Software tool

June 13thDEA Louvain-la-Neuve, Belgium27 Method Step2 – Abstract Model  An Abstract Individual Component (AIC) is an abstraction that allows the description of interaction objects in a way that is independent of the modality  An Abstract Container (AC) is an entity allowing a logical grouping of other abstract containers or abstract individual components.

June 13thDEA Louvain-la-Neuve, Belgium28 Method Step2 – Tool support

June 13thDEA Louvain-la-Neuve, Belgium29 Method Step3 –To obtain a Concrete specification

June 13thDEA Louvain-la-Neuve, Belgium30 Method Step3 –Taxonomy of 3D UIs

June 13thDEA Louvain-la-Neuve, Belgium31 Method Step3 -Taxonomy of 3D UIs   Pure Reality   Augmented reality   Augmented Virtuality   Virtual 3D GUI   Digital 3D GUI   3D rendering of 2D GUI   2D GUI

June 13thDEA Louvain-la-Neuve, Belgium32 Method Step3 - Lesson learnt from Taxonomy  Source of inspiration  Abstract characteristics  Design options  Gaps to fulfill  Concrete User Interface Model

June 13thDEA Louvain-la-Neuve, Belgium33 Method Step3 - Tool support - Alice

June 13thDEA Louvain-la-Neuve, Belgium34 Method Step3 - Tool support - Maya

June 13thDEA Louvain-la-Neuve, Belgium35 Method Step3 - Tool support - VUIToolkit

June 13thDEA Louvain-la-Neuve, Belgium36 Method Step4 – Final User Interface  VRML  X3D  Java 3D

June 13thDEA Louvain-la-Neuve, Belgium37 Contents  Introduction  Related Work  Model-based development for 3D User Interfaces  Case Study  Conclusion

June 13thDEA Louvain-la-Neuve, Belgium38 Case Study – Task and domain definition

June 13thDEA Louvain-la-Neuve, Belgium39 Case Study – Concrete Definition

June 13thDEA Louvain-la-Neuve, Belgium40 Case Study – Concrete Model B

June 13thDEA Louvain-la-Neuve, Belgium41 Case Study – From Abstract to concrete Model NAC LHS:=RHS x>2

June 13thDEA Louvain-la-Neuve, Belgium42 Case Study – From Abstract to concrete Model “Show Questionnaire” Output (value unknown) Attribute, data type, domain characteristics An output text “Select Answer”Select attribute value + repetitive (selection values not known) Data type, domain characteristics A dropdown list, a group of option buttons “Send Questionnaire”ControlFeedbackA trigger “Navigation”NavigationFeedbackA trigger

June 13thDEA Louvain-la-Neuve, Belgium43 Mock-up of the Concrete User Interface

June 13thDEA Louvain-la-Neuve, Belgium44 Case Study – From Abstract to concrete Model <SphereTrigger defaultContent="Start" radious="1.5" solid="True" isVisible="true" isEnabled="true"> isEnabled="true"> <Material diffuseColor=" " specularColor=" " emissiveColor=" " shininess="0.3"/> emissiveColor=" " shininess="0.3"/>

June 13thDEA Louvain-la-Neuve, Belgium45 Case Study – Editing the FUI in Maya

June 13thDEA Louvain-la-Neuve, Belgium46 Case Study – Editing the FUI in Alice

June 13thDEA Louvain-la-Neuve, Belgium47 Case Study – FUI generated with Alice

June 13thDEA Louvain-la-Neuve, Belgium48 Case Study – FUI generated with the VUIToolkit

June 13thDEA Louvain-la-Neuve, Belgium49 Case Study – FUI generated with the Maya

June 13thDEA Louvain-la-Neuve, Belgium50 Case Study – FUI generated with the Maya

June 13thDEA Louvain-la-Neuve, Belgium51 Use of the virtual polling system Case Study in a virtual world

June 13thDEA Louvain-la-Neuve, Belgium52 Contents  Introduction  Related Work  Model-based development for 3D User Interfaces  Case Study  Conclusion

June 13thDEA Louvain-la-Neuve, Belgium53 Conclusion –Model-based approaches Models t= task, Do = Domain Di = dialog AUI=abstract presentation CUI=concrete user interface U = user, C = context. Inter Model Transformation  Bidirectional derivation → Derivation link ≈→ Manual Derivation ≈  Manual Bidirectional der. FUI = Final User Interface FUI target languages *Indicates that it is done manually VR-Wise CUI CUI → FUI VRML, X3D CoGenIVE T, Di, CUI ( T, Di, CUI )  FUI C++ InTML Di, CUI T ≈→ (Di, CUI), (Di, CUI) ≈→ FUI Contigra CUI, Di (CUI, Di) → FUI X3D, Behavior3D, Audio3D Our methodology T, Do, C, AUI, CUI T  Do, T  AUI, AUI  CUI, CUI  AUI, T  CUI, (T, Do,AUI,CUI)  C, CUI ≈→ FUI Java, XHTML, Flash, HTML, Voice XML* Java 3D*, X3D*, VRML*.

June 13thDEA Louvain-la-Neuve, Belgium54 Conclusion  The method based on graph transformation has been introduced, defined and illustrated  We proposed a method for 3D UI development whose life cycle was divided into four steps ranging from the most abstract to the most concretes according to the principles of Model- Driven Engineering.  The feasibility of the approach is much depending on the amount and the quality of the design rules that are encoded in UsiXML.

June 13thDEA Louvain-la-Neuve, Belgium55 Conclusion  Future work. Exploring more design options and encode them in UsiXML Models Models Transformational rules Transformational rules Software tools Software tools More FUI More FUI  Consolidate concepts

June 13thDEA Louvain-la-Neuve, Belgium56 Thank you Juan Manuel Gonzalez Calleros