Usability Testing I. Omaima Al-Matrafi

Why Should You Care About Usability? Have you ever… gotten lost in a Web site? left a site without finding the information you wanted? waited too long for a page to download? gone to a site you can’t view or read? visited a site with outdated information? Do you want people to visit and return to your site?

What is usability? “A function of particular users performing particular tasks in a particular environment” (Smith et al. 68) The “people who use the product can do so quickly and easily to accomplish their own tasks” (Dumas and Redish 4)

ISO Definition (9241-11) for Usability “...the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”

Definitions (cont.) Ease of learning Efficiency of use Usability is a combination of factors that affect the user's experience with the product or system, including: Ease of learning How fast can a user who has never seen the user interface before learn it sufficiently well to accomplish basic tasks? Efficiency of use Once an experienced user has learned to use the system, how fast can he or she accomplish tasks?

Definitions (cont.) Memorability Error frequency and severity If a user has used the system before, can he or she remember enough to use it effectively the next time or does the user have to start over again learning everything? Error frequency and severity How often do users make errors while using the system, how serious are these errors, and how do users recover from these errors? Subjective satisfaction How much does the user like using the system? Confusing, frustration and satisfaction

Usability Is not functionality Is not beta testing Functionality is how the system works Is not beta testing Beta testing is unfocused and at end of process Is not market research Market research is about customer demands

Types of Usability Problems Product doesn’t match job or task Poor organization/layout Unexpected occurrence of events Product not self-evident Requires recall rather than recognition Inconsistent screens, messages, terminology Design is inefficient Cluttered or unattractive design No feedback or poor feedback about status or errors No exit or undo Help or documentation is not helpful

Usability Problem Example: Inconsistent

Usability Problem Example: Poor Organization/Layout

Usability Problem Example: Product Not Self-evident

What is Usability Testing? An empirical study of a product’s usability by observing actual users do real tasks with the product The primary goal is to improve the usability of the product 5 characteristic: Real users Real tasks Specific usability goals/concerns Observing and recording the testing Data analysis

Planning: Define the Purpose Look at goals of your organization. Which goals is the Web suited to meet? Prioritize the goals for your site. Have one primary goal. Optionally have secondary goals. Example goals: Provide a positive image of NASA / your organization Provide information/function that the user needs

Five Steps for test process Step 1: User Analysis & Profiles Step 2: Decide what to Test Step 3: Preparing for the Testing Step 4: Conducting the test Step 5: Analyzing the Data

Step 1: User Analysis & Profiles Knowing Your Users What are your users’ characteristics? Age, computer literacy, domain knowledge, access methods, browsers, work environments, handicaps, etc. Collect this information through Surveys / Questionnaires Visits to their environment A typical test includes 6-12 participants

Step 2: Decide what to Test Choose an overall purpose Example: How useable is our new website? Determine objectives or what you are testing for. Examples: Does our search engine provide usable results in the first 5 links returned? Are search results clear to the users? Choose type of test: Performance: Can they do it? Understandability: Can they understand it? Read-and-locate: Can they find it?

Step 2: Decide what to Test con. Select tasks: Consider tasks with a high chance of user failure (complex tasks, one-of-a-kind tasks, highly abstract or technical tasks) Consider tasks with a high cost of user failure (tasks that require support, like help or support calls, to complete; tasks where data could be damaged or lost) Consider: First impressions (look and feel) First tasks Tasks most performed Critical tasks Specific problem areas New task for the product

Step 2: Decide what to Test con. Select performance objectives (should be individualized for each task) Time: How long to complete tasks, to find things, to performance procedures Error/Success: user errors, attempts to do/find something, numbers of times section re-read, if the task was completed successfully Five attributes of usability (Nielsen, 1994): Learnability: system is easy to learn so users can get started quickly Efficiency: system should be easy to use, resulting in high productivity Memorability: system should be easy to remember Errors: system should have low error rate and allow error recovery Satisfaction: system should be pleasant to use

The goal and the concerns of the test Example: General goal: users can use this program to read the documents easily and quickly. Quantitative usability goals: users will be able to find the menu choice they want in less than 30 second with no more than two wrong choices and Know the function of each icon from the first time they see it. General concern for this test: Can users find the correct menu choices and icons as they need them (ease of learn) Specific concern of this test: 1-unrestanding how can open a PDF file to read it. 2- Accessing particular information easy and quickly by searching the document. 3- Increase the ability of enhancing the vision by zooming the page. 4- The ability of copying a specific part from this document. 5- The ability of making a snapshot for any part of document

Planning: User Tasks Why will users come to your site? What’s in it for them? Best way to find out is to talk to them! Create use scenarios to drive your design. Example user tasks: Find information about a particular product Learn about newest projects Find out about future presentations Download copy of a document Contact organization employees Provide feedback

Planning: Example Scenario Scenarios try to capture a moment in time for the user. A sample scenario might read: John is working at XCompto make their systems more usable. He remembers that someone at MyComp is involved with usability engineering and wants to contact that person, either by phone or email, for more information on doing usability engineering at a government agency.

Step 3: Preparing for the Testing Choose order of tasks: start easy, go sequential, or be random Create written test materials: Task list for users Written welcome speech/ Intro to be read to user Consent forms Pre-task and post task questionnaires & interview questions Other materials Recruit participants & determine “payment” Define team member’s roles: Facilitator/Briefer (necessary): Often only team member to interact with users Observation recorder/note taker (necessary) Camera operator (optional) Help desk operator (optional) Test administer (optional)

Step 3: Preparing for the Testing Cont. Create written test plan Practice: conduct walkthroughs of the testing and if possible pilot test (the pilot test users could even be a team member) Prepare test environment (day of test)

Pre-test questionnaire Ensure that participant meets user profile Gather additional detail about the participants’ knowledge and experience

Post-task questionnaire Asks questions about a task as soon as it is completed Captures feedback when it is fresh Can help measure change in attitude toward the task Keep it short

Description of the test Tasks and their scenarios The path for each task Measuring usability Typical Criteria for measuring usability Consent Form Methods: Details how you will run the test Pre-Test ( user profile ), Post-Task questionnaires, Post test questioner, and Observation. Task resources and Procedure: time, HW, SW and procedure Preparing the test Test Team Members: define what each person will do Measure Excellent Acceptable Unacceptable Task1: open file Time for task <15 sec 15-20 sec More than 20 sec S = Selection error 1 More than 1 C= confusing 1 or more

Step 4: Conducting the test Greet & Brief participant: Read/say welcome Emphasize that you are not testing them, but the product and that they should act as natural as possible Explain think-aloud protocol (if using) Emphasize how user tells you she has completed a task Stress that the testing is “anonymous” Be unbiased (especially the Facilitator/Briefer) Intervene carefully (avoid as much as possible) Observe and record data Debrief user

Step 5: Analyzing the Data Collate data into findings: Choose an approach: Top-down approach: predetermine categories of findings (like navigation, design, terminology) and go through data looking for “hits” Bottom-up approach: put each observation on a sticky note/note card, sort into categories and label categories Determine time and errors/success Examine findings for each user, user profile, and task Use analysis techniques such as statistics (even averages help)

Your opinion about the program Choices Very bad 1 2 3 4 Very good 5 No. answers Percentage 0.40 0.20 Screen arrangement Not logical logically 0.60 Icons design ambiguity understandable .40 Easy of finishing task Very easy Very difficult Learn of the program Your opinion on the task Choices Very easy 1 2 3 4 Very difficult 5 No. answers Percentage .20 Did you finish the task Yes No .80 Do you face any problem during executing the task .40 .60 What is the kind of difficulties Difficult in navigating between menus Ambiguity icons The path to complete the task is long others .0 .5

Step 5: Analyzing the Data con. Analyze data: Determine cause of problems Determine scope/severity of problems Make recommendations/changes Report Findings

Usability Related Web Sites http://usability.gsfc.nasa.gov Goddard Usability Engineering Center http://www.usability.gov Usability resources by National Cancer Institute http://www.uie.com User Interface Engineering (Jared Spool) http://www.UPAssoc.org The Usability Professionals’ Association http://www.useit.com/alertbox The Alertbox: Current Issues in Web Usability (Jakob Nielsen) Book: A Practical Guide to Usability Testing (Dumas & Redish)