User Interface Evaluation Usability Testing Methods

 Conduct experiments to find specific information about a design and/or product.  Basis comes from experimental psychology.  Uses statistical data methods  Quantitative and Qualitative

Usability Testing Methods  During usability testing, users work on specific tasks using the interface/product and evaluators use the results to evaluate and modify the interface/product.  Widely used in practice, but not appropriately used.  Often abused by developers that consider themselves to be usability experts.  Can be very expensive and time consuming.

Usability Testing Methods  Performance Measurement  Thinking-aloud Protocol  Question-asking Protocol  Coaching Method

Usability Testing Methods  Co-discovery Learning  Teaching Method  Retrospective Testing  Remote Testing

Performance Measurement  Applicable Stages:  Design, Code, Test & Deployment  Personnel  Usability Experts, approximately 1.  Developers, 0.  Users, 6.

Performance Measurement  Usability Issues Covered  Effectiveness: Yes  Efficiency: Yes  Satisfaction: No  Quantitative Data is collected.  Can NOT be conducted remotely.  Can be used on any system.

Performance Measurement  What is it?  Used to collect quantitative data.  Typically, you will be looking for benchmark data.  Objectives MUST be quantifiable  75% of users shall be able to complete the basic task in less than 30 minutes.

Performance Measurement  How can I do it?  Define the goals that you expect users to perform

Performance Measurement  How can I do it?  Quantify the goals  The time users take to complete a specific task.  The Ratio between successful interactions and errors.  The time spent recovering from errors.  The number of user errors.  The number of commands or other features that were never used by the user.  The number of system features the user can remember during a debriefing after the test.  The proportion of users who say that they would prefer using the system over some specified competitor.

Performance Measurement  How can I do it?  Get participants for the experiments  Conduct very controlled experiments  All variables must remain consistent across users

Performance Measurement  Problems With Performance Measurement  No qualitative data.

Thinking-aloud Protocol  Applicable Stages:  Design, Code, Test & Deployment  Personnel  Usability Experts, approximately 1.  Developers, 0.  Users, 4.

Thinking-aloud Protocol  Usability Issues Covered  Effectiveness: Yes  Efficiency: No  Satisfaction: Yes  Quantitative Data is NOT collected.  Can NOT be conducted remotely.  Can be used on any system.

Thinking-aloud Protocol  What is it?  Technique where the participant is asked to vocalize his or her thoughts, feelings, and opinions while interacting with the product..

Thinking-aloud Protocol  How can I do it?  Select the participants, who will be involved?  Select the tasks and design scenarios.  Ask the participant to perform a task using the software.

Thinking-aloud Protocol  How can I do it?  During the task, ask the user to vocalize  Thoughts, opinions, feelings, etc. 

Thinking-aloud Protocol  Problem With Thinking-Aloud Protocol  Cognitive Overload  Can you walk & chew gum at the same time?  Asking the participants to do too much.

Question-asking Protocol  Applicable Stages:  Design, Code, Test & Deployment  Personnel  Usability Experts, approximately 1.  Developers, 0.  Users, 4.

Question-asking Protocol  Usability Issues Covered  Effectiveness: Yes  Efficiency: No  Satisfaction: Yes  Quantitative Data is NOT collected.  Can NOT be conducted remotely.  Can be used on any system.

Question-asking Protocol  What is it?  Similar to Thinking-aloud protocol.  Instead of participant saying what they are thinking, the evaluator prompts the participant with questions while using the system.

Question-asking Protocol  How can I do it?  Select the participants, who will be involved?  Select the tasks and design scenarios.  Ask the participant to perform a task using the software.

Question-asking Protocol  How can I do it?  During the task, ask the user to questions about the product  Thoughts, opinions, feelings, etc. 

Question-asking Protocol  Problem With Thinking-Aloud Protocol  Cognitive Overload++  Can you walk, chew gum & talk at the same time?  Asking the participants to do too much.  Added pressure when the evaluator asks questions.  Can be frustrating on novice users.

Coaching Method  Applicable Stages:  Design, Code, Test & Deployment  Personnel  Usability Experts, approximately 1.  Developers, 0.  Users, 4.

Coaching Method  Usability Issues Covered  Effectiveness: Yes  Efficiency: No  Satisfaction: Yes  Quantitative Data is NOT collected.  Can NOT be conducted remotely.  Can be used on any system.

Coaching Method  What is it?  A system expert sits with the participant and acts as a coach.  Expert answers the participant’s questions.  The evaluator observes their interaction.

Coaching Method  How can I do it?  Select the participants, who will be involved?  Select the tasks and design scenarios.  Ask the participant to perform a task using the software in the presence of a coach/expert.

Coaching Method  How can I do it?  During the task, the user will ask the expert questions about the product 

Coaching Method  Problem With Coaching Method  In reality, there will not be a coach present.  This is good for creating a coaching system, but not for evaluating the interface.

Co-Discovery Learning  Applicable Stages:  Design, Code, Test & Deployment  Personnel  Usability Experts, approximately 1.  Developers, 0.  Users, 6.

Co-Discovery Learning  Usability Issues Covered  Effectiveness: Yes  Efficiency: No  Satisfaction: Yes  Quantitative Data is NOT collected.  Can NOT be conducted remotely.  Can be used on any system.

Co-Discovery Learning  What is it?  Two test users attempt to perform tasks together while being observed.  They are to help each other in the same manner as they would if they were working together to accomplish a common goal using the product.  They are encouraged to explain what they are thinking about while working on the tasks.  Thinking Aloud, but more natural because of partner.

Co-Discovery Learning  How can I do it?  Select the participants, who will be involved?  Select the tasks and design scenarios.  Ask the participants to perform a task using the software.

Co-Discovery Learning  How can I do it?  During the task, the users will help each other and voice their thoughts by talking to each other. 

Co-Discovery Learning  Problem With Co-Discovery Learning  Neither is an expert  The blind leading the blind.

Teaching Method  Applicable Stages:  Design, Code, Test & Deployment  Personnel  Usability Experts, approximately 1.  Developers, 0.  Users, 4.

Teaching Method  Usability Issues Covered  Effectiveness: Yes  Efficiency: No  Satisfaction: Yes  Quantitative Data is NOT collected.  Can NOT be conducted remotely.  Can be used on any system.

Teaching Method  What is it?  You have 1 participant use the system.  Ask the participant to teach a new novice participant how to use the system.

Teaching Method  How can I do it?  Select the participants, who will be involved?  Select the tasks and design scenarios.  Ask the 1 st participant to perform a task using the software.  Ask the 1 st participant to teach a new participant.

Teaching Method  How can I do it?  Observe their interactions. 

Teaching Method  Problem With Teaching Method  Neither is an expert  The blind leading the blind.  Possible to discover some interesting things about the learn-ability of your interfaces.

Retrospective Testing  Applicable Stages:  Design, Code, Test & Deployment  Personnel  Usability Experts, approximately 1.  Developers, 0.  Users, 4.

Retrospective Testing  Usability Issues Covered  Effectiveness: Yes  Efficiency: Yes  Satisfaction: Yes  Quantitative Data can be collected.  Can NOT be conducted remotely.  Can be used on any system.

Retrospective Testing  What is it?  A videotape of the session is observed by the usability expert and the participants.

Retrospective Testing  How can I do it?  Select the participants, who will be involved?  Select the tasks and design scenarios.  Use one of the usability testing methods that we have discussed.  Videotape the session.

Retrospective Testing  How can I do it?  Review the videotape with the users. 

Retrospective Testing  Problem With Retrospective Testing  Extremely time consuming!

Remote Testing  Applicable Stages:  Design, Code, Test & Deployment  Personnel  Usability Experts, approximately 1.  Developers, 0.  Users, 5.

Remote Testing  Usability Issues Covered  Effectiveness: Yes  Efficiency: Yes  Satisfaction: Yes  Quantitative Data can be collected.  Can be conducted remotely.  Can be used on any system.

Remote Testing  What is it?  The participants are separated from the evaluators.  No formal observation.  No usability lab.

Remote Testing  How can I do it?  Give the product/software to participants.  Collect information about how they use your software/product.  Methods  Same-Time Different Place  Different-Time Different Place

Remote Testing  How can I do it?  Camtasia, SnagIt  Usability Logger   Journaled Sessions 

Remote Testing  Problem With Remote Testing  The evaluator is not there.  Can’t observe facial expressions.  Great for Web based systems.

Usability Testing Methods  Select the method that works best for you.  Select the method that fits your implementation.  Be thorough during your experiments.  The more data, the better.

Usability Testing Methods  Hawthorne Effect  The tendency for people to improve their performance after any change when they know their performance is being studied.