PS50118 – Interacting with Technology Laboratory vs. Field Usability Evaluation Jason Cooper

Outline What is Usability? Carrying out Usability Testing Usability Evaluation of Mobile Devices The Comparison Conclusion 2

What is Usability? Commonly considered to be a way of ensuring that systems that promote interactivity with a user are easy to learn, effective to use, and enjoyable from the users perspective (Preece et al 2002) ISO states that usability refers to the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user 3

Usability Goals (Preece et al 2002) Effectiveness: This goal refers to how good the system itself is at doing what it is supposed to do. Efficiency: Refers to how the system supports the users in carrying out their activities and whether they are able to use the system productively once they have gained enough experience. Safety: Is concerned with the way in which the user is protected from dangerous conditions and undesirable situations Utility: Refers to the way that the system is capable of providing the correct kind of functionality to the user at the right time, such that they are able to do what they need or want to. Learnability: Considers how easy it is for a user to become competent in the use of a system. Memorability: Refers to how straightforward it is for the user of the system to remember how to use it once it is initially learned. 4

Usability Testing – The Lab 1 Traditionally usability testing is carried out in a controlled environment where the product is tested to determine whether it can be considered usable. The controlled environment usually consists of a laboratory where a set of pre-planned activities or scenarios can be run and repeatedly measured. Its goal is to assess whether the product will do what it is intended to do. 5

Usability Testing – The Lab 2 Data collected includes opinions of users of the system and performance on the set of activities. Quantitative performance measures are gathered which all for the following types of data to be produced: Time taken by the user to complete a specific activity. Time to complete an activity after a specified time away from the product Number and type of errors per activity Number of errors per specified unit of time Number of times the user had to navigate to the online help or manuals Number of users making a particular error Number of users completing a specific activity successfully 6

Usability Testing – The Field Usually conducted to determine how a product or prototype is adopted and use by people in their working and everyday lives. Length of time that testing can taken within the field can vary from just a few minutes to months or in some cases even years depending on what the product that that is being tested. Provides predominantly qualitative data such descriptions of peoples behaviours and activities. Collected by observing and interviewing users, collecting video, audio and field notes that attempt to detail what has occurred in the environment during the testing, 7

Usability Evaluation of a Mobile Device 1 Shift in focus from the lab to the field. Nielsen et al (2002) asserted that all mobile devices should always been evaluated with a realistic and natural setting. Initial reason for this were that it was thought lab testing was unlikely to be able to find all problems that occur in real mobile usage (Johnson P. 1998) There also appears to be an implicit assumption that usability of a mobile device could only be properly evaluated in the field. (Gregory et al 2000 & Brewster 2002) 8

Usability Evaluation of a Mobile Device 2 However there still remained a significant preference for lab based evaluation with 71% being undertaken in the lab and 19% being conduced within the field. (Kjeldskov 2005) Reasons for this included Field was considered to be time consuming in terms of organisation and collection of data. Complication of data recording Lack of control Hard to know whether all was evaluated that should of been evaluated. Whereas laboratory evaluations: Controlled conditions Clear set tasks Peaceful space that enabled concentration Control over activities and monitoring Special equipment 9

The Comparison Different empirical studies were found that attempt to compare usability evaluation of mobile systems in different settings, however they provide different results, but all attempt to focus on the number, type and severity of mobile device usability problems that are found in the relative settings. Kjeldskov et al (2004): Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. Kaikkonen et al (2005) Usability testing of mobile applications: A comparison between laboratory and field testing Nielsen et al (2006). It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. 10

Field, its not worth the hassle! 1 Laboratory evaluation discovered the exact same number of usability problems as was discovered in the field. There was a lack of control that was experienced within the field. Both the field and laboratory evaluations were able to deliver context-aware related problems, which contradict with some literature that suggest context-aware problems are better acquired in a field setting. 11

Field, its not worth the hassle! 2 In conclusion the two studies reported that: Realistic aspects were not a problem. Possibility of Lab problems being false positives. Proposal of use of field studies in other areas of the development lifecycle, providing a better insight into what was needed from the system in the first place. 12

Field, its worth the hassle! 1 Nielsen et al (2006) proposed that the contradictory results in Kjeldskov et al (2004) and Kaikkonen et al (2005) reports were possibly due to: Low number of test subjects. Same data collection techniques were not employed in the field as it was in the lab. Conflicting procedures. 13

Field, its worth the hassle! 2 Nielsen et al (2006) carried out a comparison study of field and evaluation usability evaluation of mobile device using the similar conditions and same data collection equipment it showed: Usability problems categorised as relating to either cognitive load or interaction style were identified only in the field evaluation Reason given this was that field enabled realistic setting which in turn meant the user become frustrated easier. The nature of laboratory setting was also said to increase the mental demands and frustration level of the participants significantly. When both the evaluations were conducted in the same way, field was more successful at identifying the more significant usability problems In conclusion although the cost, complexity and amount of time it takes to carry out a field evaluation is a down side, Nielsen et al consider the added value gained in terms of the capability of field evaluation to provide usability issues not detected in the laboratory setting makes field evaluation worthwhile. 14

Conclusion 1 So is it worth it? Yes – If we are able to detect usability problems in the field that are not detected in the lab then we must undertake a field study. But why are there so little people doing it then? This is for all the reasons outlined in the presentation. Costly in terms of money and time Little Control People wonder still is it worth while Complicated We must also remember that mobile systems are relatively new, so people are still use to doing it in the lab, but as more take place within the field I believe it is likely more advantages will emerge. 15

Conclusion 2 However alternatives do exist, whereby the field is simulated in the lab. (D. Svanæs ???). Hybrid Research Strategy with a full-scale simulated ward environment created with the help of health workers. With the aid of video recording they were able to observe details in patient-doctor interaction and in technology that were overlooked in the field study. However the field study gave a much richer picture. Conclusion is that both the lab and field supplement each other and it is the combination that provide valuable insights that can not be gained from one method alone. 16

References Preece, J., Rogers, Y., Sharp, H., (2002) Interaction design beyond human-computer interaction. West Sussex: Wiley Nielsen, C. M., Overgaard, M., Pedersen, M. B., Stage, J., and Stenild, S It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. In Proceedings of the 4th Nordic Conference on Human-Computer interaction: Changing Roles (Oslo, Norway, October , 2006). A. Mørch, K. Morgan, T. Bratteteig, G. Ghosh, and D. Svanaes, Eds. NordiCHI '06, vol ACM, New York, NY, DOI= Kjeldskov, J., Skov, M. B., Als, B. S. and Høegh, R. T. (2004) Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. In Proceedings of the 6th International Mobile HCI 2004 conference. LNCS, Springer-Verlag. Kaikkonen, A., Kallio, T., Kekäläinen, A., Kankainen, A. and Cankar, M. (2005) Usability testing of mobile applications: A comparison between laboratory and field testing. Journal of Usability Studies, 1(1): Baillie, L. (2003) Future Telecommunication: Exploring actual use, In Proceedings of IFIP TC13 International Conference on Human-Computer Interaction, (INTERACT '03). IOS Press Abowd G, D,. Mynatt, E,D Charting past, present, and future research in ubiquitous computing, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.1, p.29-58, March 2000 [doi> / ] /