1. Some Usability Engineering Methods
Randolph Bias
2/17/2011

2. Today Divide the day into thirds:
First third – RB on usability evaluation methods
Second third – Yahoo speaker on accessibility
Third third – Reconvene for discussion about slash work on . . .
Wiki
Etc.

3. Remember . . .
Our approach, within usability engineering of web sites and other user interfaces, is:
Empirical
Iterative
User-Centered Design

4. The Methods of Usability Engineering . . .
Are employed in order to enable you to bring user data (empiricism) to bear on the emerging product design.
You (the usability engineer) become an advocate for the user, in the product development process.

5. One big problem
Cost

6. Three Methods to address cost
Remote End-user testing (lab testing)
Heuristic Evaluations
Usability Walkthroughs
Today let’s talk about WHY and WHEN we employ one method or another, and HOW to carry them out.

7. Next week -- End-user Testing
Also called “lab testing”
Can be done on paper-and-pencil design, prototype, early code, existing product

8. EUT - Benefits Gather performance and satisfaction data
Performance data: time on task, error rates, # calls to the help desk, # references to the documentation,
Satisfaction data: End-of-test questionnaire
Can be find-and-fix or benchmarking
Ensure coverage of certain parts of the UI – we have good control over the tasks

9. EUT - Limitations Artificial situation
Successful test doesn’t “prove” the product works
Aside – It’s ALL about confidence.
Need representative users!
Ease of learning vs. ease of use
Hard to test longitudinally

10. EUT -- What to test? Can rarely cover all the UI. I like to test:
critical tasks
frequent tasks
nettlesome tasks

11. Rubin’s 4 Types
Exploratory (working on the “skeleton” – maybe the Information Architecture)
Assessment test (working on the “meat and flesh” of the UI)
Validation test (does it meet the objectives?)
Comparison test (compare two competing designs)
Rubin, J. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley: New York, NY, (Superseded by Rubin and Chisnell.)

12. Set up Create environment (ambient setting, HW, SW)
Identify participants
Establish test roles (test monitor/administrator, data logger, timer, video operator, product experts (SMEs), other observers)
Create test plan
Prepare observers
Prepare test materials

13. What materials? Instructions Informed consent form NDA
Permission to videotape
Test scenarios
Questionnaire(s)

14. Conduct the Test Welcome the test participant
Communicate that this is not a test of THEM, and they can leave any time
The scenarios should match the real world setting
Ask the test participants to “think aloud” to better understand intent
Offer post-test questionnaire, and debrief

15. After the Test Quick data to product team
Assign severities and build recommendations
Build archival report
Serve as change agents!

16. Ah, but REMOTE Saves tons of travel money
Allows you to get otherwise hard-to-get test participants.
Allows them to be in their own environments.
Might allow product designers/developers to watch from their own office.
But lose some fidelity of the test environment (video?)
Some added set-up cost (time)

17. What is a Heuristic Evaluation?
Evaluators systematically inspect the application interface to check for compliance with recognized usability guidelines (heuristics). (Thus, an INSPECTION method.)
Identifies major and minor usability problems
- Conducted by three to five experienced usability engineers (or one!)
- Problems are reported along with the heuristic it violates

18. Problems Identified The probability of one evaluator finding . . .
A major usability problem - 42% *
A minor usability problem - 32%
More evaluators = more problems identified
*from

19. Strengths of an HE
Done by people experienced in usability not just “dumb” users
Can identify both major and minor usability problems
Can be done relatively quickly and inexpensively
UNlike EUT, can sometimes cover every corner of a UI or web site

20. Weaknesses of an HE
If done at end of design, designers may be resistant to changes
Some designers/developers may be unmoved by “just opinions”
Experienced usability evaluators may miss content problems that actual users would find
Can HELP address this issue by using SMEs

21. Typical Methodology Interface is exercised
-1st pass to develop the big picture
-2nd pass to accomplish typical tasks
Each problem is reported along with the heuristic it violates
Comments are consolidated
Severity levels – Critical, Major, Moderate, Minor

22. Nielsen’s Usability Heuristics
Visibility of system status
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
Match between system and the real world
The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
User control and freedom
Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
Consistency and standards
Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
Error prevention
Even better than good error messages is a careful design which prevents a problem from occurring in the first place.

23. Nielsen’s Heuristics (cont’d.)
Recognition rather than recall
Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
Flexibility and efficiency of use
Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
Aesthetic and minimalist design
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
Help users recognize, diagnose, and recover from errors
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
Help and documentation
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

24. Severity Levels Used Critical Usability Issue: Loss of user data
System shutdown
Abandoned task
Major Usability Issue –
Completed task but considerable frustration or extra steps
Moderate Usability Issue –
- Moderate work around or multiple attempts
* Usability Suggestion

25. Sample Summary of Results38 4
Totals 2
Help
Icons 7
Main IDE 1
Debug
Run Project
Code Editor 16 3
Visual Layout 5
Project Creation
Installation
MODERATE
MAJOR
CRITICAL
TYPE

26. How to . . .

27. Other Heuristics

28. Next Week IX Lab demo Due dates: Book review – up on the wiki today
White paper – 2 weeks
Project test plan – 4 weeks