1EMODE workshop – September 2007 Automatic Usability Assessment of Multimodal User Interfaces Based on Ergonomic Rules Adrian Stanciulescu Jean Vanderdonckt Benoit Macq Université catholique de Louvain (UCL)

2EMODE workshop – September 2007 Outline Introduction Context –OpenInterface platform –UsiXML language –UsiXML for OpenInterface Navigation in large scale images –Existing solutions –Proposed solution Structure of the navigational tasks –Reaching the structure –Working hypopothesis Expanded UsiXML model for vocal interaction Conclusions & future work

3EMODE workshop – September 2007 Introduction Information vizualization: succesful paradigm in HCI Zoomable user interfaces –Organize information in space and scale –Interaction techniques: translation and scale –E.g.: storytelling, web browsing, browsing of maps –Few usability studies, inconclusive results Interaction devices –Traditional interaction: mouse and keyboard –New emergent interaction: vocal and tactile –The problem: optimize the interaction –Combining vocal and tactile vs. Vocal-only: faster, less error prone, less linguistic complexity, less time to complete the task

4EMODE workshop – September 2007 Introduction Our objectives: –Automatic usability assessment of MM navigational tasks over large scale images –Interactions types: translation and zooming –Interaction modalities: graphical (keyboard, mouse), vocal(speech input), tactile (finger, stylus pen) –Specification of composing elements: sequential or order-independent interaction

5EMODE workshop – September 2007 OpenInterface platform Provide an open source platform for the design and rapid development of multimodal prototyped applications –Components: pieces of software that provide services (CIDL) –Pipelines: inter-connections between components (PDCL) Advantages: –Allow seamless integration of heterogeneous software (C/C++, Java, Matlab) –Allows rapid prototyping of multimodal applications (easy software connection) –Reusability of delivered software To address: –No OI module for usability guidance at design time –A knowledge base of multimodal ergonomic rules

6EMODE workshop – September 2007 UsiXML language Structured according to four levels of abstraction (Cameleon framework) Advantages: –Support for multimodal I/O –Separation of modalities –Support for CARE properties –Extendibility to new modalities Shortcomings: –Non-hierarchical structure of the grammars –Undefined order of specification for grammar elements –No means to specify the cardinality of grammar elements

7EMODE workshop – September 2007 UsiXML for OpenInterface Usability adviser tool Multimodal OpenInterface Applications UsiXML Specification Usability Evaluation Ergonomic rules Abstraction

8EMODE workshop – September 2007 Usability adviser tool Shortcomings of the existing tools: –No tool to assess the usability of both web and non-web UIs –Formalized language to specify the ergonomic rules: limitations with respect to the quality of the evaluation Our objectives: –Real time guidance during the development life cycle –Evolutional knowledge base –Multiplatform ergonomic rule validation (UsiXML specifications) –Ergonomic rules expressed in a formal language close to natural language –No need to learn the formal language (rule specification assistant) –Portability of the knowledge base (single file storing the rules)

9EMODE workshop – September 2007 Navigation in large scale images Existing solutions –Combine multiple smaller displays into a single one –Multiple LCDs butted together into a single one –Cost-intensive, space-intensive, or both Our proposal: –Navigational techniques are required –3 X 3 overlaying grid –Meaningful symetrical pairs (Top/Bottom, Left/Right)

10EMODE workshop – September 2007 Taxonomy of the navigational tasks Translation: top / bottom / left / right / main diagonal up (Bottom Left) / main diagonal down (Top Right)/ second diagonal up (Bottom Right)/ second diagonal down (Top Left) –Translate without specifying a value: the user decides when to stop the translation –Translate by specifying the number of grid units Zooming: zoom in / zoom out –Zooming without specifying the zoom factor: it is inferred from the selected zoomed area and the size of the display window –Zooming with a zoom factor: Fixed zoom factor: zoom by steps Specified zoom factor: the value is specified by the user – Zooming in a single selected grid unit (e.g., zoom in Top Left / zoom in Centre, zoom in Bottom Right) – Zooming in multiple selected grid units: Adjacent horizontal grid unit selection: zoom in Horizontal Top/ zoom in Horizontal Centre / zoom in Horizontal Bottom Adjacent vertical grid unit selection: zoom in Vertical Left / zoom in Vertical Centre / zoom in Vertical Right Diagonal grid unit selection: (e.g., zoom in Left Centre to Bottom Right / zoom in Centre to Top Right.

11EMODE workshop – September 2007 Reaching the structure General structure: Instruction:= {Action, Object, Parameter} EBNF specific format: Instruction:= {{Action}- [Object] [{Parameter}]}- Instanciation of the instruction:

12EMODE workshop – September 2007 Working hypothesis A.Hypothesis 1: Action applied over 1 optional Object which has 2 Parameters B.Hypothesis 2: Action-Parameter order Ordered instruction: Instruction: = Action >> Parameters Order-independent instruction: Instruction: = Action |=| Parameters

13EMODE workshop – September 2007 Working hypothesis C.Hypothesis 3: Equivalent navigational instructions: { Sub-instruction} : groups one or more elements of an instruction specified by employing a monomodal/multimodal interaction

14EMODE workshop – September 2007 Expanded UsiXML model VUI support added: –Grammar –Part –Item Advantages: –Flexibility –Reusability

15EMODE workshop – September 2007 Conclusions & future work Conclusions: –Large number of choices when deciding in the favour of a particular set of navigational tasks and the employed modality –Usability studies required: speed, error rates and subjective satisfaction –Usability guideline are limited and no yet validated –Automatic usability assesment based on a knowledge base of ergonomic rules Future work: –Address the tactile interaction –Expand UsiXML to support tactile interaction –Adding new ergonomic rules related to MM interaction

16EMODE workshop – September 2007 Thank you very much for your attention BCHI Lab Similar netork of Execellence User Interface eXtensible Markup Language OpenInterface