1. Université catholique de Louvain (UCL) Belgian Laboratory of Computer-Human Interaction (BCHI) Place des Doyens, 1 B-1348 Louvain-la-Neuve (Belgium) Presented by August 24th, 2004WCC'2004 - Topical day on Multimodal interaction1 Multimodality and context-aware adaptation Quentin Limbourg, Jean Vanderdonckt

2. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction2 Context-awareness What is in the context of use? –User –Platform –Environment Organisation Socio-psychological factors Two examples of adaptation due to context- awareness –Adaptability = adaptation by the user –Adaptivity = adaptation by the system

3. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction3 Example of adaptability Personalisation –Requires many editing –Is hard to achieve –Does not necessarily factor out common parts –Does not necessarily separate fixed parts from variable parts Goal –To have a system for automated generation of UIs With support of personalisation That can generate multiple variants of the same UI –For this purpose, we use A feature model Generative Programming

4. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction4 C C F1 C C F2 C C F1 F2 Dependencies 1. 2. What‘s a Feature Model? Optional Exclusive Alternate

5. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction5 Example of a Feature Model

6. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction6 Feature model C C F1 F2 F3 F2a F2b F3a F3b C C F1 F2 F3 F2a F2b F3a F3b 01 1 1100 Model Specifications 0 1 0 1 0 1 USIXML Specifications USIXML Specifications

7. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction7 Generative Programming void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT & channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT & chann elMix ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT & channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT & chann elMix ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); V oid channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT & channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT &); virtual ~ImgProcessT(); virtual char* getClassName() const; virtual char getClassID () const; ImgProcessT & operator+=(const ImgProcessT &); ImgProcessT & operator-=(const ImgProcessT &); ImgProcessT & operator^=(const ImgProcessT &); ImgProcessT operator+ (const ImgProcessT &); ImgProcessT operator- (const ImgProcessT &); ImgProcessT operator^ (const ImgProcessT &); ImgProcessT & channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL); mgProcessT & operator+=(const ImgProcessT &); ImgProcessT & operator-=(const ImgProcessT &); ImgProcessT & operator^=(const ImgProcessT &); ImgProcessT operator+ (const ImgProcessT &); ImgProcessT operator- (const ImgProcessT &); ImgProcessT operator^ (const ImgProcessT &); ImgProcessT & channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL); V oid channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT & channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT &); virtual ~ImgProcessT(); virtual char* getClassName() const; virtual char getClassID () const; ImgProcessT & operator+=(const ImgProcessT &); ImgProcessT & operator-=(const ImgProcessT &); ImgProcessT & operator^=(const ImgProcessT &); ImgProcessT operator+ (const ImgProcessT &); ImgProcessT operator- (const ImgProcessT &); ImgProcessT operator^ (const ImgProcessT &); ImgProcessT & channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL); mgProcessT & operator+=(const ImgProcessT &); ImgProcessT & operator-=(const ImgProcessT &); ImgProcessT & operator^=(const ImgProcessT &); ImgProcessT operator+ (const ImgProcessT &); ImgProcessT operator- (const ImgProcessT &); ImgProcessT operator^ (const ImgProcessT &); ImgProcessT & channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT & channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT & channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); USIXML-Specification ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL) Components void areaOperation(const int&, const int&, ImgProcessT & (ImgProcessT ::*pimgpr)(const int&, Pl*), const int& nAngle, Pl* ppl ); void areaOperation(const int&, const int&, ImgProcessT & (ImgProcessT ::*pimgpr)(Pl*), Pl* ppl ); void areaOperation(const int&, const int&, ImgProcessT & (ImgProcessT ::*pimgpr)(const int&, const int&, const int& ), const int&, const int& ); void channelShowHistogram( int*, const int&, const int& n = 0 ) const; void channelFft ( T*, T*, const unsigned int&, Pl* ppl = NULL ); void channelIfft ( T*, T*, const unsigned int&, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT & channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT &); virtual ~ImgProcessT(); virtual char* getClassName() const; virtual char getClassID () const; ImgProcessT & operator+=(const ImgProcessT &); I void areaOperation(const int&, const int&, ImgProcessT & (ImgProcessT ::*pimgpr)(const int&, Pl*), const int& nAngle, Pl* ppl ); void areaOperation(const int&, const int&, ImgProcessT & (ImgProcessT ::*pimgpr)(Pl*), Pl* ppl ); void areaOperation(const int&, const int&, ImgProcessT & (ImgProcessT ::*pimgpr)(const int&, const int&, const int& ), const int&, const int& ); void channelShowHistogram( int*, const int&, const int& n = 0 ) const; void channelFft ( T*, T*, const unsigned int&, Pl* ppl = NULL ); void channelIfft ( T*, T*, const unsigned int&, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT & channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT & channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT &); virtual ~ImgProcessT(); virtual char* getClassName() const; virtual char getClassID () const; ImgProcessT & operator+=(const ImgProcessT &); I Final code ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT & channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT & channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL );

8. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction8 Example of adaptivity Computer-aided surgery by mixed reality

9. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction9 Multi-platform Extending approaches for multiple contexts of use –Context is now restricted to one computing platform at a time Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1: Web terminal

10. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction10 Multi-platform Traditional approach: last level only Final User Interface Platform 1: Web terminal Platform 2: Personal computer Final User Interface Platform 3: Pocket PC Final User Interface

11. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction11 Multi-platform Traditional approach: Atomica HTML/Java UI code Platform 1: Web terminal Platform 2: Personal computer Visual C++ UI code Platform 3: Pocket PC Mobile Visual Basic UI

12. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction12 Multi-platform Shortcomings of the traditional approach (ct’d) –Repetition of development and maintenance [UIML,1999] Platform 1: Web terminal Platform 2: PC Platform 3: Pocket PC Java UI #1C++ UI #1Mobile UI #1 Application #1 Java UI #2C++ UI #2Mobile UI #2 Application #2 Java UI #3C++ UI #3Mobile UI #3 Application #3

13. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction13 Multi-platform Physical level deployment Platform 1: Final User Interface Physical User Interface Platform 2: Final User Interface Physical User Interface Platform 3: Final User Interface Physical User Interface = Translation relationship = Transformation at the same level of abstraction

14. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction14 Multi-platform Physical level deployment: Galaxy Windows UI code Physical User Interface Platform 1: Personal Computer Platform 2: Macintosh Mac OS Macintosh UI code Physical User Interface Platform 3: Workstation OSF/Motif OSF/Motif UI code Physical User Interface

15. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction15 Multi-platform Physical level deployment: Galaxy Windows UI code Platform 1: Personal Computer Platform 2: Macintosh Mac OS Macintosh UI code Platform 3: Workstation OSF/Motif OSF/Motif UI code

16. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction16 Multi-platform Logical level deployment Final User Interface Physical User Interface Logical User Interface Platform 1: Personal Computer Platform 2: Macintosh Mac OS Final User Interface Physical User Interface Logical User Interface Platform 2: Workstation OSF/Motif Final User Interface Physical User Interface Logical User Interface

17. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction17 Multi-platform Logical level deployment: UIML HTML UI code HTML Renderer UIML Specification Platform 1: Personal Computer with Web browser Platform 2: Personal Computer With XML browser XML UI code XML Renderer Platform 3: Mobile Phone WML UI code WML Rendered

18. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction18 Multi-platform: UIML Skeleton – Part 1 What parts comprise the UI & what’s their relationship? …... [Abrams et al., 1999]

19. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction19 Multi-platform Task & Concepts level deployment Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1Platform 2 Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 3 Final User Interface Physical User Interface Logical User Interface Task & Concepts

20. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction20 Multi-platform Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1Platform 2 Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 3 Final User Interface Physical User Interface Logical User Interface Task & Concepts Multiple levels of translation/reification [Thevenin,2001]

21. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction21 Multi-platform Translation at Task & Concept level Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1 Platform 2 Final User Interface Physical User Interface Logical User Interface Task & Concepts

22. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction22 Multi-platform Translation at Task & Concept level –Task model primitives Sub-tasks (nodes) –Remove sub-task: for different reasons –Add sub-task: unlikely to appear –Modify sub-task: switch, reshuffle, relax constraints Hierarchical & temporal operators (edges) –Modify temporal operator: reordering –Cutting sub-trees –Concepts model primitives

23. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction23 Register to CHI’2001 Conference Pay registration By credit cardBy fax [ ] RecordPayment mode >> Specify all registration needs Enter personal informationSpecify special needsSpecify attendee type | | | >> Multi-platform Task translation | | |

24. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction24 Multi-platform Concepts removal: CT-UIMS [Märtin,1990]

25. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction25 Multi-platform Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1 Platform 2 Final User Interface Physical User Interface Logical User Interface Task & Concepts Translations T&C + logical levels

26. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction26 Multi-platform Translations T&C + logical levels Specify all registration needs Enter personal informationSpecify special needsSpecify attendee type | | |

27. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction27 Multi-platform Translations T&C + logical levels Specify all registration needs Enter personal informationSpecify special needsSpecify attendee type | | | >>

28. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction28 Multi-platform Translations T&C + logical levels

29. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction29 Multi-platform Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1 Platform 2 Final User Interface Physical User Interface Logical User Interface Task & Concepts Translations T&C + logical + physical levels

30. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction30 Multi-platform Translations T&C + logical + physical levels –T&C + logical remain constant over time or coordinated with changes –Graceful degradation of presentation Widgets: replacement, reduction, resizing Text: text summarization, sentence elision, automatic summary Images: color reduction, cropping, resizing,… Animation: reduction to still images,… Video: reduction of frame, refreshing,…

31. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction31 Multi-platform Graceful degradation of presentation –Widgets

32. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction32 Multi- platform Graceful degradation of presen- tation –Images

33. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction33 Multi-platform Graceful degradation of presentation –Images

34. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction34 Multi-platform Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1 Platform 2 Final User Interface Physical User Interface Logical User Interface Task & Concepts Translations combined at all levels

35. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction35 Multi-platform Multiple transformations possible Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1 Platform 2 Final User Interface Physical User InterfaceLogical User InterfaceTask & Concepts

36. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction36 Multi-platform Plastic UI = UI that can accomodate multiple changes of context (here, platforms) while preserving some usability properties Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1 Platform 2Platform 3 Final User Interface [Calvary, Coutaz & Thevenin, 2001]

37. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction37 Multi-platform Plastic UI: Virtual keyboard [Grolaux, 2001]

38. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction38 Multi-platform Plastic UI: FlexClock [Grolaux, Van Roy, Vanderdonckt, 2002]

39. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction39 Multi-platform Plastic UI: adaptable bounded value

40. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction40 Multi-platform Reverse engineering Final User Interface Physical User Interface Logical User Interface Task & Concepts Platform 1Platform 2 Final User Interface Physical User Interface Logical User Interface = Abstraction relationship = Transformation into a more abstract level of abstraction ReversiXML WebRevenge ReversiXML

41. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction41 XIML2WML generator

42. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction42 Multi-platform Reverse engineering Final User Interface Platform 1Platform 2 Final User Interface Physical User Interface = Crossing relationship = Transformation between different levels of abstraction Logical User Interface Retargeting [Bouillon & Vanderdonckt,2002]

43. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction43 Multi-directional User Interfaces A UI is represented by a suite of models that are –Analysable –Editable –Transformable Transformational approach –Any UI is subject to any transformation at any time From one language to another From one computing platform to another From reverse engineering to forward engineering …

44. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction44 Multi-directional User Interfaces Model engineering –Forward engineering (top-down) –Reverse engineering (bottom-up) –Bidirectional engineering (both ways) Model transformations –Reification, abstraction –Translation –Crossing (retargeting, revamping,…)

45. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction45 Contexts of use Channels Modalities Multiple levels of consideration Channels plus: User behaviourUser behaviour User expectationsUser expectations User experienceUser experience User groupsUser groups Physical constraintsPhysical constraints EnvironmentEnvironment Modalities plus : Market segmentsMarket segments Time of usageTime of usage Place of usagePlace of usage Network, Wi-fiNetwork, Wi-fi Various tasksVarious tasks Platforms Devices Platforms plus : Modalities of interactionModalities of interaction Mono vs multimodalityMono vs multimodality Devices plus : Software characteristicsSoftware characteristics Hardware characteristicsHardware characteristics Languages: programming, markupLanguages: programming, markup Tools: renderer, interpreter, compilerTools: renderer, interpreter, compiler

46. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction46 Contexts of use Channels Modalities USIXML Platforms Devices

47. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction47 An overview of USIXML tools Running prototypes –GrafiXML: CUI Hi-Fi editor + Code Generator (Java Swing, XHTML) –FormiXML: CUI Hi-Fi editor for Forms (XForms, HTML) –SketchiXML: CUI Lo-Fi Sketching –VisiXML: CUI Mid-Fi (MS Visio Plug-in) –FlashiXML: CUI renderer for SVF –ReversiXML: reverse engineering from HTML to CUI, AUI –TransformiXML API: transformation tool –Teresa (F. Paternò, CUI level to multi FUI generation) Demo

48. August 24th, 2004WCC'2004 - Topical day on Multimodal interaction48 Thank you very much for your attention! www.usixml.org Credits www.plasticity.org www.similar.cc