Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Mikael B. Skov Department of Computer Science Aalborg University, Denmark Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context- Aware Mobile Systems in the Field

2 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Motivation We have to investigate into the criteria, methods, and data collection techniques for usability evaluation of mobile systems (Johnson 1998) Often it is assumed that usability evaluations of mobile devices should be done in the field “… the scaling dimensions that characterize context-aware systems makes it impossible to use traditional, contained usability laboratories …” Abowd and Mynatt (2000) Kjeldskov and Graham (2003) found that 71% of mobile device evaluations were done in laboratory experiments

3 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Motivation

4 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Aim to compare the outcome of evaluating the usability of a mobile system in a laboratory respectively in the field to describe techniques used for improving the realism of laboratory settings by including mobility and context and support high-quality video data collection in the field

5 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh System: MobileWARD Wireless access to EPR on handheld computer Information and functionality adapted to location, time and the nurse’s assignments

6 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh System: MobileWARD In the corridor Overview of all patients, assigned patients and pending tasks Direct access to reading details about each individual patient’s history Entering a ward Overview of the patients in the ward Scanning patient’s wrist band Access to entering new measures General: Button size to allow interaction w. finger or pen

7 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Method Laboratory evaluation Lab at Aalborg University, Denmark 6 test subjects (trained nurses) Tasks derived from user study Laboratory furnished as hospital, divided into two wards + corridor Field evaluation Frederikshavn Hospital, Denmark 6 test subjects (trained nurses) No specified tasks Involving real work activities

8 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Method Mobile usability equipment enabling the capturing of video and audio The usability problems were classified as cosmetic, serious or critical (Molich, 2000) All sessions were analyzed in random order by two teams of trained usability evaluators The two teams produced two lists of usability problems and these were merged into one complete list.

9 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Findings (1) 37 different usability problems Lab evaluation resulted in 36 problems 8 critical, 18 serious, and 10 cosmetic Field evaluation resulted in 23 problems 7 critical, 10 serious, and 6 cosmetic Primarily more serious and cosmetic problems

10 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Findings (2) More problems per session 18.8 (2.0) problems versus 11.8 (3.3) problems (U=2.651, p<0.01) Critical: 5.3 (1.2) and 4.5 (2.2) problems Serious: 7.5 (1.0) and 4.5 (0.8) problems Cosmetic: 6.0 (0.9) and 2.8 (1.0) problems Identified significantly more serious (U=2.79, p<0.01) and cosmetic problems (U=2.84, p<0.01)

11 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Field Evaluations Revisited (1) Little added value of taking the evaluation into the field Same problems in the laboratory Field contribution: Validity of data entered into the system Lack of control undermined the extendibility of the field None of the field subjects used the note taking facility The higher number of identified problems in the lab condition could be a result of irrelevant usability problems

12 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Field Evaluations Revisited (2) Both the lab and field revealed context-aware problems All seven context-aware related problems in both conditions All field subjects got confused when the system automatically updated information or functionality according to the physical location The clip-on camera facilitated data collection of mobile use The configuration allowed subjects to move freely in the environment while at the same time still providing a close-up view of the interaction However, problems of placing the devices between use

13 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Conclusions Was it worth the hassle? Not really, at least not for usability problem identification However, the real use situation provided additional information on use Replicating the context – always possible? Lab evaluation without context replication Field evaluation with task assignments

14 Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Questions…