Design Options for Multimodal Web Applications Adrian Stanciulescu and Jean Vanderdonckt {stanciulescu, UCL/IAG/BCHI

Outline Introduction Motivations Reference framework Design space for multimodal web user interfaces Design options for graphical web UIs Design options for multimodal web UIs Tool support Examples Conclusions

Introduction Software development life cycle should naturally evolve from early requirements to detailed ones until final system is developed and deployed Evolution inevitably goes through identifying, defining, analyzing, comparing and deciding between different, potentially contradictory alternatives that may effect the entire process The UI of this software does not escape from the aforementioned observations

Introduction Design space that supports effectively and efficiently the progress of the development life cycle towards a final system while assuring some form of quality Design space consists of an n-dimensional space where each dimension is denoted by a single design option Each design option has a finite set of design values E.g., label location (left-aligned, centered, right-aligned)

Introduction Advantages: Clarifies the development process in terms of options Facilitates the incorporation in the development life cycle of design options as an abstraction covered by a software Supports the tractability of more complex design problems Shortcomings: In theory: design options could be very numerous (even infinite) In practice: impossible to consider a large amount of design options Too complex or expensive to implement Design options do not necessarily addressed users’ needs and requirements Out of the designer’s scope of understanding, imagination or background Design decision is not always clear (may violate some usability principles or guidelines)

Motivations The languages in which they are implemented (e.g., XHTML, VoiceXML, X+V): Restrict the amount of possible UIs to obtain Set directly the CARE properties to Assignment, Equivalence and Redundancy Supported interaction style for the above mentioned languages make them appropriate for certain types of applications (e.g., information systems) but totally inadequate for other types (e.g., air traffic control) Independence of any implementation or tool support is a contribution which could be useful to any designer of a multimodal user interface

Reference framework Cameleon reference framework UsiXML language: Graphical interaction Vocal interaction Multimodal interaction Design option-based transformational approach: Step 1: Construct the Task & Domain Models Step 2: From Task & Domain to AUI Step 3: From AUI to CUI Step 4: From CUI to FUI

Design options for graphical web UIs Design options for graphical UIs Sub-task presentation Sub-task navigation Concretization of navigation & control Navigation concretization Control concretization

Design options for graphical web UIs Sub-task presentation: specifies the appearance of each sub-task in the final user interface. Sub-task presentation separated combined one by one combined box tabbed dialog box float window many at once float window all in one areas with separators group boxes bulleted list numbered list Abstract level Concrete level

Sub-task presentation – example Separated sub-task presentation T11T12T13 T1 AC11 AC12 AC13 AC1 T11T12T13 T1 AC11 AC12 AC13 NACLHSRHS NACLHSRHS Combined sub-task presentation

Design options for graphical web UIs Navigation concretization: identifies the placement and the cardinality of the objects/logically grouped set of objects that ensure the navigation. Navigation concretization placement cardinality localglobal simple multiple

Sub-task navigation - example Local placement for navigation objects AC1 T11T12T13 T1 AC11 AC12 AC13 AIC111AIC121AIC131 Global placement for navigation object AC1 T11T12T13 T1 AC11 AC12 AC13 AIC11 NACLHSRHS NACLHSRHS

Design options for graphical web UIs Sub-task navigation: specifies the way in which the dialog control is transferred from one sub-task to another. Sub-task navigation asynchronous sequential (synchronous)

Design options for graphical web UIs Concretization of navigation and control: specifies if the navigation and control are ensured or not by the same object. Concretization of navigation & control separatedcombined

Design options for graphical web UIs Control concretization: specifies the placement and the cardinality of the objects that ensure the control. Control concretizations placement cardinalitylocalglobal multiplesimple

Design options for multimodal web UIs Consider the ergonomic criteria for the evaluation of human-computer interfaces (Scapin, 1993) Associate a corresponding CARE property to each design value. Design options for multimodal UIs Prompt Grouping / Distinction of items Immediate feedback Guidance

Design options for multimodal web UIs Prompt: refers to the interaction channels available in order to lead the users to take specific actions whether it be data entry or other tasks. vocal (A) graphical (A) multimodal (E, C, R)

Design options for multimodal web UIs Grouping/Distinction of items: concerns the organization of information in relation to one another. Input: any information input from the user to the system vocal (A) graphical (A) multimodal (E, C, R) Output: any information output from the system to the user vocal (A) graphical (A) multimodal (E, C, R)

Design options for multimodal web UIs Immediate feedback: concerns the system responses to users’ action vocal (A) graphical (A) multimodal (E, C, R)

Design options for multimodal web UIs Guidance: refers to the means available to advise, orient, inform, instruct and guide the users throughout their interactions with the system Guidance for input: any guidance offered to the user in order to guide him with the input Textual (A) Iconic (A) Acoustic (A) Speech (A) Multimodal (E, C, R) Guidance for output: any guidance offered to the user in order to guide him with the feedback Textual (A) Iconic (A) Acoustic (A) Speech (A) Multimodal (E, C, R)

Multimodal example Multimodal text field Prompt: multimodal (R) Grouping for input: multimodal (E) Immediate feedback: multimodal (R) Guidance: Input: iconic (A) Output: iconic (A) Prompt: multimodal Please say your name (vocal +graphical) 1 Input: multimodal Juan Gonzale z 2 (vocal +graphical) Juan Gonzalez Guidance for input: iconic Guidance for feedback: iconic Feedback:multimodal Your answer was: Juan Gonzalez 3 (graphical +vocal)

Examples DEMO

Tool support MultiXML – five software modules: IdealXML TransformiXML GrafiXML CFB Generator XHTML+Voice Generator tool

Conclusions Model-driven approach for the development of multimodal web user interfaces Currently…approach based on design space composed of 13 design options Design space covers a subset of the total number of UIs which can be developed with X+V

Conclusions Advantages of combining design options into a design space: Descriptive Comparative Generative