Multimodal Interfaces in a Ubiquitous Computing Environment 3 rd UK-Ubinet Workshop —————— 9 th – 11 th February 2005 —————— Fausto. J. Sainz Salces, Dr. David Llewellyn-Jones, Prof. Madjid Merabti —————— School of Computing and Mathematical Statistics Liverpool John Moores University James Parsons Building Byrom Street Liverpool, L3 3AF, UK {cmsfsain, D.Llewellyn-Jones,

Disappearing Computers – Disappearing Interfaces The Ubiquitous Computing paradigm presents difficult user interface problems –Enforces a minimal interface –Requires increased usability Current user interfaces tend to concentrate on visual display techniques Need to explore alternative methods for transferring information and allowing control

Harnessing the Senses We felt the most appropriate sense for information presentation after vision was sound –Can alert the user without requiring their specific attention –Ambient interface ideal in a pervasive environment Considerable work exists in the field of audio interfaces –Earcons: tones or sequences of tones as a basis for building messages [1,2] –Auditory icons: everyday sounds that convey information about events by analogy with everyday sound producing events [3] –Voice: instructions are read either as a recorded or simulated voice Our work looks at the use of earcons in a Ubiquitous Computing Environment [1] M. Blattner, Sumikawa, D. & Greenberg, R., "Earcons and icons: Their structure and common design principles," Human Computer Interaction, vol. 4(1), pp , [2] Brewster, S., Wright, P.C., Edwards A.D.N. (1994). “A detailed investigation into the effectiveness of earcons,” in Auditory Display. Sonification, audification and auditory display., vol. XVIII, G. Kramer, Ed. Santa Fe, U.S.A.: Addison-Wesley, 1994, pp [3] W. W. Gaver, "Using and creating auditory icons.," in Auditory Display. Sonification, audification and auditory display., vol. XVIII, G. Kramer, Ed. Santa Fe, U.S.A.: Addison-Wesley, 1994, pp

Multimodality The use of several means to present information simultaneously Combine the use of visual icons and earcons –Makes sure the message arrives at the receiver –Redundant info –Multiple channels (also increases accessibility) –More natural presentation method Undertook experiments to compare devices used to control household appliances –Small handheld device with a restricted form factor –Large screen ‘fixed’ laptop device Both devices were used with various combinations of visual, audio and multimodal interfaces

Interface Design We compared people’s perceptions and ability to use each device using 6 permutations of the interface –Visual– Four device control –Earcons– Six device control –Multimodal

Design Process An initial design stage was used to establish effective designs for each of the interfaces modalities –Human constraints –Machine constraints –Cognitive knowledge –User preference An iterative design method was used

Earcon Design Each earcon is comprised of two parts –The first motif represents the household appliance –The second motif represents the new state of the device (e.g. “on” or “off”) Motif 1 Motif 2 Hob off Hob on

Experiments Experiments were undertaken individually with two groups of users –20–33 year olds –60–84 year olds The intention was to observe a broad range of users, both relatively proficient users and computer novices

Results: Comparing Large & Small Overall, users made favourable comments about the smaller handheld devices The most common remark concerned the positive usability aspect of the palmtop “Small portable, clear, handy able to carry in one hand” However, people cited the small screen as being a problem The larger screen was seen as a positive aspect of the laptop device, but this was countered by its fixed nature Interestingly, the only comment concerning security, was in favour of the palmtop

Results: Device Ranking The comments are echoed by the preference ranking that subjects gave for the devices Screen Visual Palmtop Visual Screen Audio Palmtop Audio Palmtop Multimodal Percent Ranked Lowest Screen Visual Palmtop Visual Screen Audio Screen Multimodal Palmtop Multimodal Percent Ranked Highest

Results: NASA TLX Mental Demand measurement “How much mental demand and perceptual activity was required (e.g. thinking, deciding, calculating, remembering, looking, searching, etc.)? Was the task easy or demanding, simple or complex, exacting or forgiving?” AudioVisualMultimodal Mean Mode 305 & 105 Std. Deviation Frequency Audio Frequency Visual Frequency Multimodal 100

Results: NASA TLX Performance measurements “How successful do you think you were in accomplishing the goals of the task set by the experimenter? How satisfied were you with your performance in accomplishing these goals?” AudioVisualMultimodal Mean Mode 555 Std. Deviation Frequency Audio Frequency Visual Frequency Multimodal 100

Results: NASA TLX Frustration measurement “How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?” AudioVisualMultimodal Mean Mode 555 Std. Deviation Frequency Audio Frequency Visual Frequency Multimodal 100

Conclusions Users generally found the handheld device to be better than the larger screen laptop Many of the benefits were described in terms that might apply in a Ubiquitous Computing environment Multimodal designs were preferred –Earcons can be used to augment a limited user interface in a beneficial way –Earcons alone performed significantly worse than visual icons alone These results applied across both groups of subjects Further results concerning reaction times are in the process of being analysed

Multimodal Interfaces in a Ubiquitous Computing Environment 3 rd UK-Ubinet Workshop —————— 9 th – 11 th February 2005 —————— Fausto. J. Sainz Salces, Dr. David Llewellyn-Jones, Prof. Madjid Merabti —————— School of Computing and Mathematical Statistics Liverpool John Moores University James Parsons Building Byrom Street Liverpool, L3 3AF, UK {cmsfsain, D.Llewellyn-Jones,