Download presentation
Presentation is loading. Please wait.
Published byMay James Modified over 9 years ago
1
A Transformational Approach for Developing Multimodal Web User Interfaces Adrian Stanciulescu
2
Plan Context Motivations Thesis statement Requirements for multimodal web UIs Proposal Reference framework for multimodal web UIs Design space for multimodal web UIs Transformational method for producing multimodal UIs Tool support Case studies Conclusion & Future work
3
Context Graphical user interfaces – classical human-computer interface paradigm Users with limited literacy and typing skills Handheld devices Need for hands and eyes free Difficult to support mouse and keyboard interaction Web access anyplace, anywhere, anytime Need for newer, better interface paradigm Multimodal user interface is the answer… Adaptation to the context of use (user, platform, environment) More natural and efficient human-computer interaction
4
Motivations Modality interaction flexibility Faster interaction Lower incidence of errors and esier error recovery True device mobility Usability improvement Robust systems Enhancement of device effectiveness Improved experience Lack of multimodal applications
5
Thesis statement Development of multimodal web user interfaces Three types of user interfaces: Graphical Vocal Multimodal Conceptual and methodological aspects for a model-to-model transformational approach Development life cycle based on a design space composed of design options
6
Requirements Support for multimodal input and output CARE properties support for input and output modalities Approach based on design space Openness to new modalities Separation of modalities
7
Requirements (cont’d) Ontological independence of modalities Transformation-based development Machine processable Human readable Ontological homogeneity Reuse of specification Methodological explicitness Methodological extendibility Support for tool interoperability
8
Reference framework Camelon reference framework UsiXML – selected language: Structured according to four levels of abstraction Progressive transformation from Task & Concept level to Final UI Support for the extension of the transformational methodology (graph transformation) Underlying unique formalism (graph-based syntax) Support for reusability of specification and transformations thanks to the XML structure Support for modality independence Allows introduction of new interaction modality Cross-toolkit development
9
Vocal Concrete Interaction Objects Shortcomings of UsiXML previous version: No specialized containers for user-system dialog No specialized container for multiple selection No differentiations between the system’s output types No object dedicated to system interruptions
10
Vocal Concrete Interaction Objects (cont’d) VocalContainers: VocalGroup VocalForm VocalMenu VocalConfirmation VocalIndividualComponents: VocalOutput: VocalFeedback VocalPrompt VocalMenuItem VocalInput VocalNavigation Break Exit EventTypes: error, help, noInput, noMatch
11
Vocal Concrete Interaction Objects (cont’d) Example: virtual ice cream shop
12
Multimodal Concrete Interaction Objects MultimodalCIO: combination of graphical CIO and vocalCIO outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance groupBox + radioButtons vocalMenu + vocalItems + vocalInput groupBox + radioButtons + vocalMenu + vocalItems + vocalInput + guidance Graphical Vocal Multimodal
13
Concrete UI Relationship GraphicalRelationships GraphicalTransition GraphicalAdjacency GraphicalContainment VocalRelationships VocalTransition VocalAdjacency VocalContainment Synchronization: between vocalCIOs and graphicalCIOs
14
Multimodal Concrete Interaction Objects Syntax example: multimodal text field with equivalent input and redundant output
15
Specification of transformation Graph-based patterns Syntactically typed patterns Textual concrete syntax of patterns: UsiXML syntax
16
How transformation rules work… LHS: Find an occurrence of LHS into G NAC: Check that NAC does not match into G. If there is a match then skip to another occurrence of LHS RHS: Replace LHS by RHS Transformation rule Initial Model Resultant Model
17
UsiXML syntax of transformation rules NAC RHS mappings
18
The four steps of the transformational approach Step 1: Construct the Task&Domain Models Step 2: From Task & Domain to AUI Step 3: From AUI to CUI Step 4: From CUI to FUI
19
Sub-steps of the transformational approach
20
Design space for web UIs Support for the progressive development life cycle towards a final system Composed of a set of design options Each design option has a finite set of design values Advantages: Clarifies the development process in terms of options Facilitates the incorporation in the development life cycle of design options as an abstraction covered by a software Supports the tractability of more complex design problems
21
Design options for graphical web UIs
23
Design options for multimodal web UIs Prompt: vocal (A), graphical (A), multimodal (E, C, R) Grouping/distinction of items: Input: vocal (A), graphical (A), multimodal (E) Output: vocal (A), graphical (A), multimodal (E, C, R) Immediate feedback: vocal (A), graphical (A), multimodal (E, C, R) Guidance: Input: textual (A), Iconic (A), Acoustic (A), Speech (A), Multimodal (E, C, R) Immediate feedback: textual (A), Iconic (A), Acoustic (A), Speech (A), Multimodal (E, C, R)
24
Step 2: From Task & Domain to AUI Separated sub-task presentation Combined sub-task presentation T11T12T13 T1 AC11 AC12 AC13 AC1 T11T12T13 T1 AC11 AC12 AC13 NACLHSRHS NACLHSRHS
25
Step 3: From AUI to CUI Multimodal text field Prompt: multimodal (R) Input: multimodal (E) Immediate feedback: multimodal (R) Guidance: Input: iconic (A) Output: iconic (A) Prompt: multimodal Please say your name (vocal +graphical) 1 Input: multimodal Juan Gonzale z 2 (vocal +graphical) Juan Gonzalez Guidance for input: iconic Guidance for feedback: iconic Feedback:multimodal Your answer was: Juan Gonzalez 3 (graphical +vocal)
26
Transformation rule NAC: LHS: RHS: Multimodal prompt Guidance for input Multimodal input Guidance for output Vocal feedback
27
Case studies 2 case studies: Virtual Polling system and Car rental system
28
Tool support IdealXML TransformiXML GrafiXML CFB Generator Communication Flow Builder XHTML+Voice Generator tool
29
Conclusions Model-drive approach for the development of multimodal web user interfaces Requirements fulfillment: Support for multimodal input/output CARE properties support for input/output modalities Approach based on design options Methodology covers a subset of the UIs which can be developed with X+V
30
Conclusions Advantages of combining design options into a design space: Descriptive Generative Comparative
31
Conclusions Work positioned at the intersection of SE and HCI Focus on the feasibility of code generation (2 case studies) Usability point of view: Inexistent published studies on multimodal UI design in SE and HCI Few ongoing works caused by the reduced number of multimodal applications Internal validation of the Ph.D. (study based on qualitative principles derived from the cognitive psychology)
32
Future work Investigate the implementation of a new transformation engine: MOF QVT Reinforce existing vocal components Extend the ontology with new vocal components Design space improvement Extend transformation catalogs Adaptation of multimodal web user interface to the context of use
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.