1. Mobile Usability Testing Inst. f. Softwaretechnik und Interaktive Systeme qse.ifs.tuwien.ac.at, thurnher@qse.ifs.tuwien.ac.at, christian.fruehwirth@qse.ifs.tuwien.ac.at Topics I.Motivation & Definition II.Mobile Devices III.Challenges in Mobile Usability Testing IV.Methods & Approaches V.Heuristics & Guidelines VI.Examples & Ideas Lecture :: „Mobile Usability Testing“

2. Mobile Usability Testing Goals :: What do we want to achieve in this lecture? 2 *vgl.: Schilit and Theimer (1994) What we want: Enable you to set up a Mobile Usability Test based on Quantitative as well as Qualitative Methods. Give you Inspirations & Ideas about what can be done to make mobile applications more usable.

3. Mobile Usability Testing Methods :: How do we want to achieve our goals? 3 *vgl.: Schilit and Theimer (1994) How we want to do it: Creating an awareness for the challenges of mobile- Usability Compare common Usability Testing Methods Overview about possible Heuristics & Guidelines for Mobile Usability Testing

4. I. Motivation & Definition Considerations for Mobile Testing: Results are strongly influenced by surrounding environment Results are influenced by devices used for testing Collected Data will be „richer“ (Gesture, Voice, …) Definition :: „Mobile Usability Testing“ 4

5. Mobile Usability Testing I. Motivation

6. I. Motivation & Definition For every dollar spent acquiring a customer you will spend $100 dollars reacquiring them after they leave because of poor usability or bad customer service. (*) Costs of late Usability research 6 *vgl.: MauroNewMedia (2002)

7. I. Motivation & Definition Usability Engineering in the Software Development Process 7 *vgl.: INTERACT 2001 Workshop, Jan Gulliksen, Inger Boivie)

8. Mobile Usability Testing II. Mobile Devices

9. Are used “on -the-run and for activities that may last only a few seconds or are highly context dependent” (*) Mobile Devices :: Definition 9 *vgl.: (Vetere et al., 2003, p.1)

10. II. Mobile Devices „Mobile Devices: One Generation From Useful“(*) Tighter Integration needed: Devices do not work together well with each other.  Synchronization with other Applications Different Features packed into one device do not act as one entity  PDAs with GSM modules Ad-hoc Networking is still far from realisation  Network coverage & Roaming (WLAN, UMTS, GPRS, GSM)  Industry Standards (Bluetooth compatibility, vulnerability) Mobile Devices :: Realistic View 1 10 *vgl.: (Jakob Nielsen 2003, Alertbox, Aug. 18th.)

11. II. Mobile Devices Design / Interface Problems still to be solved: Deck-of-Card Size(*) limits the Screen Size  Higher Screen-Resolutions, better viewing angle. One Dimensional Interfaces (Scroll wheels) are not suitable for 2D – Screens. Text-input is still a great Problem. Small Devices  small Buttons.  new Button Alignments? Mobile Devices :: Realistic View 2 11 *vgl.: (Jakob Nielsen 2003, Alertbox, Aug. 18th.)

12. II. Mobile Devices Fundamental Problems: Quality of Service of local Network Providers. Things that „could be done“ just can‘t because of local Service- Limitations or lack of network coverage. (Broadband, UMTS, etc.) Online Services must specialize for Mobile use Much shorter Articles, more use of XML, simplyfied Navigation Reconsider the way email is used  not just forward every mail to the PDA (Attachments, Executeables) Mobile Devices :: Realistic View 3 12

13. Mobile Usability Testing III. Challenges

14. 1. Device Proliferation –Handling many different Devices, Rendering Methods 2. Application Modality –Handling simultaneous voice / Data User interactions 3. User Mobility –Users are likely to be distracted during use 4. Data Collection –Recording eye-movement and video taping will not work everywhere Challenges :: of Mobile Usability Testing (*) 14 *vgl.: http://www.littlespringsdesign.com/analysis/utest.html

15. III. Challenges Devices can be As small as possible, optimized for voice communication Quite large, optimized for data display Optimized for gaming Optimized for multimedia Challenges :: 1 : Device Proliferation (*) 15 Applications are perceived differently Reading News-Bulletin on a point-matrix phone display  content is forgotten 3 hours later. Reading News-Bulletin on a java-enabled 19“ CRT Monitor  Will be stored in long-term memory *vgl.: http://www.littlespringsdesign.com/analysis/utest.html

16. III. Challenges Mobile Applications often combine Graphic / Data and Voice Elements. Difficult to test in an early stage of development (Software isn‘t fully functional / not yet written.) Test must be able to provide simultaneous experiences Challenges :: 2 : Application Modality(*) 16 *vgl.: http://www.littlespringsdesign.com/analysis/utest.html

17. III. Challenges Mobile Users are very likely to be distracted. Natural Environment is not always manageable/affordable to be simulated in a lab Distractions and „Normal anomalies“ (Waiter interrupting you in a restaurant to take your order) have to be part of the test Challenges :: 3 : User Mobility 17

18. III. Challenges Acquiring Data outside the Lab Eye tracking will hardly work on tiny screens and under mobile conditions. Recording tools interfere with the users‘ interaction with the device. (Cameras mounted on a cell phone make the user hold it in an unnatural position) Challenges :: 4: Data Collection 1 18 “It is cold and snowing and you do not know from where your bus leaves in 5 minutes. You pick up your WAP phone to check: The mobile user runs to catch her bus, after her run three researchers with cameras and microphones..(*)“ *(vgl.: Per-Ola Rasmussen ExarbII 2003)

19. III. Challenges Mobile Users interact not just with the screen and keyboard. Test needs to record, gestures, face-expressions, voice, body- language, etc.  Much richer Data (People leaning left and right while playing a Formula-1 racing Game on their Java-enabled Phone.) Record what is the user doing, what is he/she NOT doing. Challenges :: 4: Data Collection 2 19

20. Mobile Usability Testing IV. Methods & Approaches

21. The common Methods Methods :: Approaches to Mobile Usability Testing 1 21 Heuristic Evaluation (Nielsen & Mack, 1994) Guidelines vs. Design. Used in early development Cognitive Walkthrough (Rowley&Rhoades, 1992) How or why a person would react in a certain situation. Based on assumptions on the user's mental model *(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)

22. IV. Methods & Approaches The common Methods Methods :: Approaches to Mobile Usability Testing 2 22 Feature Inspection (Nielsen & Mack, 1994) Does the product meet the users needs and demands? Used in middle stages of development Consistency Inspection (Nielsen‚ 1995) Checks consistency across multiple products from the same product – family. *(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)

23. IV. Methods & Approaches The common Methods Methods :: Approaches to Mobile Usability Testing 3 23 Standards Inspection (Wixon et. Al. 1994, Nielsen 1995) Ensures compliance with industry standards. Best used in middle-stages of development Guideline Checklist (Wixon et. Al. 1994, Nielsen 1995) Used in conjunction with other usability methods. The Checklists give the tester a basis by which to compare the application *(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)

24. IV. Methods & Approaches The common Methods Methods :: Approaches to Mobile Usability Testing 4 24 Thinking Aloud (Nielsen‚1994) Lets the evaluator understand how the user views the system. Contextual Inquiry (Holzblatt & Beyer ‚1993) Used to get a broad knowledge about the environment that you are producing for. More a discovery process than an evaluative process. *(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)

25. IV. Methods & Approaches Methods :: Lab vs. natural environment 25 *(vgl.: Mason 1988, Järvinen, 1999) The larger the number of factors that is under control in a test, the more scientific rigour is emphasized. The more natural like the test setting is, the more relevant and applicable the results will be.

26. IV. Methods & Approaches The Ideal Test Natural situation / environment Application is fully functional All possible forms of devices are being tested Users are free to do what they would normally do Users don't feel „tested“ Tester can record every audio / visual / voice / movement / screen data from the user without affecting the users behavior. Tester sees the Application the way the users see it. Methods :: Approaches to Mobile Usability Testing 5 26

27. IV. Methods & Approaches Preparation Define goals, methods and tasks/scenarios for the test. Introduction, Warm-up Introduce the test to the user Start with easier tasks, give time for a short warm-up phase. Testing Perform the actual test with as less interaction between user and tester as possible Methods :: General Mobile Usability Testing Process 1 27

28. IV. Methods & Approaches Test situation Give the user time to get out of the test-situation. Then start reviewing his/her opinions, impressions and suggestions. Make sure to discuss special occurrences that may have happened during the test with the user. Methods :: General Mobile Usability Testing Process 2 28

29. IV. Methods & Approaches 1. Device Proliferation 2. Application Modality 3. User Mobility 4. Data Collection Remember :: Challenges to deal with (*) 29 *vgl.: http://www.littlespringsdesign.com/analysis/utest.html

30. IV. Methods & Approaches For informal, problem identifying tests Ask participants to use the application maybe over lunch Offering a compensation helps „motivating“ the participant Don‘t forget to have them sign an informed consent statement Interruptions (waiter, etc.) are welcome Watch what happens when the users resumes the task and see what difficulties occur. Methods :: Handling User Mobility 1 30

31. IV. Methods & Approaches For formal, statistically precise tests Don‘t try to introduce distractions into the test unless you are testing with a greater number of participants Referring to Nielson, the marginal benefit will decrease if you are testing with more then 10 Users Thesis and formula is questioned from many researchers. Methods :: 3 : Handling User Mobility 2 31

32. IV. Methods & Approaches Chances(*): Reproducible conditions Easier / more complete documentation Use of more sophisticated tools Risks Unusual environment for the user Restrictions due to simulation Non recording of the original work surrounding field (office atmosphere, disturbances, etc.) Methods :: 3 : Simulate the natural environment? 1 32 *(vgl.: akziv. Requirements from users point of view. 2004)

33. IV. Methods & Approaches Which environmental factors are reproduceable in a laboratory surrounding. Environmental Conditions BT 041118.xls Interactive Example 33

34. IV. Methods & Approaches „Mobile Devices are build to be mobile so take them out into the field“ (*) Take the lab to the user, not the user to the lab. Methods :: 3 : Simulate the natural environment? 2 34 *(vgl.: akziv, „wearability“. 2004

35. IV. Methods & Approaches Methods :: 3 : Lab Test vs. Field Test(*) 1 35 *(vgl.: Jesper Kjeldskov, Aalborg University Denmark) Example Application: Using SMS Service on a PDA while walking In the lab: on a treadmill In the field: on a pedestrian street

36. IV. Methods & Approaches Methods :: 3 : Lab Test vs. Field Test(*) 2 36 *(vgl.: Jesper Kjeldskov, Aalborg University Denmark) Attention needed to navigate NoneConscious Body Motion None1. Sitting at a table or standing n/a Constant2. Walking on a treadmill with constant speed 4. Walking at constant speed on a changing track Varying3. Walking on a treadmill with varying speed 5. Walking at varying speed on a changing track

37. IV. Methods & Approaches Usability problems identified by the test subjects Numbers are basically equal Notice that while sitting on a desk the cosmetic problems identified by the users were far more! Methods :: 3 : Lab Test vs. Field Test(*) 3 37 TechniquesTotal Lab 1Lab 2Lab 3Lab 4Lab 5Field Critical4434334 Serious11 999817 Cosmetic1988861232 Total34232021182353 *(vgl.: Jesper Kjeldskov, Aalborg University Denmark)

38. IV. Methods & Approaches Are Lab tests superior? Consider Cost / benefit of different techniques and settings Time and effort per problem found Can you afford NOT to find a problem? Costs of missing Usability: lost of repurchases increased calls at helpdesk lost of repurchases lost of brand reputation necessary redesign in late state or next version law suits …… Methods :: 3 : Lab Test vs. Field Test(*) 4 38 *(vgl.: Jesper Kjeldskov, Aalborg University Denmark)

39. IV. Methods & Approaches Mobile devices are extremely personal. Users may pick them up, gesture, or lean back with them.(*) Record Device Screen + Users Face at the same time Use wireless tracking & recording technology (WLAN, Bluetooth, small radio cameras, etc.)  Users should not be handicapped by the testing equipment Methods :: 4 : Handling Data Collection 1 39 *(vgl.: Little Springs Inc. 2004)

40. IV. Methods & Approaches Recording a mobile phone‘s screen and the users face at the same time with two cameras mounted on the phone(*) Methods :: 4 : Handling Data Collection 2 40 *(vgl.: Little Springs Inc. 2004)

41. IV. Methods & Approaches Recording „Soft Information“ Define methods to integrate „soft-information“ E. g.: Users starts shaking the Phone to make it start the application faster; user‘s thumb tends to cover up parts of the device‘s display) Many qualitative information can be quantized.  Measuring heart-beats / second to determine the stress-level  Recording subconscious hand and leg movements.  Record number of extra-applicational interactions (user answered 2 phone calls and asked his colleague for help during the use of the application for 5. min.) Methods :: 4 : Handling Data Collection 3 41 *(vgl.: Little Springs Inc. 2004)

42. Mobile Usability Testing V. Possible Heuristics & Guidelines

43. V. Heuristics & Guidelines Heuristics :: General Guidelines for mobile Applications 1 43 *(vgl.: Little Springs Inc. 2004) Highly functional design Don‘t use fancy designs if they don‘t bring a real benefit for the User Consistent usage of icons / buttons / names and labels Consider the users mental-models when you introduce new functions or name buttons. Integrated content navigation Help the user tracking it‘s way through the information, always provide a clear exit – point. Consider Shortcuts

44. V. Heuristics & Guidelines Heuristics :: General Guidelines for mobile Applications 2 44 *(vgl.: Little Springs Inc. 2004) Reduced HCI interactions Especially when done on mobile devices interactions with the user are often difficult and time-consuming (text-input on a mobile phone) Reduce Interactions by any means possible (Location based services, Heuristics, default-values, etc.) Offer intelligent search-functions Assist the user in finding the information, as any unnecessary interaction makes the applications less usable for the user.

45. V. Heuristics & Guidelines Heuristics :: Mobile Games 1 45 For Mobile Games the Rules are a little different Navigation Consistency? The User should not feel like using his/her phone, he/she should experience the Game World *(vgl.: NOKIASeries 60 Developer Platform 2.0: Usability Guidelines For J2ME™ Games

46. V. Heuristics & Guidelines Heuristics :: Mobile Games 2 46 Game experience vs. Social acceptable behavior Sound, Light and Vibration enhance the users Game experience  Typically, games are played in locations where it is not suitable or socially acceptable to have the sound on. During Mobile Usab. Testing consider there are usually other people close by when the user plays the game. *(vgl.: NOKIASeries 60 Developer Platform 2.0: Usability Guidelines For J2ME™ Games

47. Mobile Usability Testing VI. Examples & Ideas

48. Text Interface :: Projecting the image 48 *(vgl.: http://www.ibizpda.com) A standard-sized Query-Keyboard is projected by laser on any given surface. The Users input is recognised by a small camera in the cigarette-pack sized device. Note: Being announced in 1999 the product is still under development and may never reach market maturity. (2004)(*) www.virtualdevices.net

49. VI. Examples & Ideas Text Interface :: Breaking the Qwerty Paradigm 49 *(vgl.: www.frogpad.com/) Standard-sized keys aligned for one-handed use. Can be used under mobile conditions (doesn't need chair + desk environment as similar fold-up keyboards for PDAs)

50. VI. Examples & Ideas Remote Controll :: Force Push 50 *(vgl.: Koji Tsukada, mobiquitous.com/pub/apchi2002-ubi-finger.pdf/) Operate household devices with a gesture of your fingertip. IR-Led points at Device to be controlled Touch & Acceleration Sensors combined with Software recognise gestures and execute command

51. VI. Examples & Ideas Natural Interaction :: Stick with the basics 51 *(vgl.: http://news.bbc.co.uk/2/hi/technology/3774747.stm) Real „Pick and Drop“ 1.PDA – User picks up a file on his screen by tapping on it with a digital pen. 2.He passes on the pen to his colleague who drops the pen on his PDA-screen. 3.The file is copied.

52. Mobile Usability Testing For more information see: http://qse.ifs.tuwien.ac.at/courses/Usability/VO_Usability_Engineering.htm