1. Online, Remote Usability Testing  Use web to carry out usability evaluations  Two main approaches agent-based evaluation (e.g., WebCritera)  model automatically evaluates UI (web site)  WebCritiera uses a modified GOMS model remote usability testing (e.g., NetRaker & Vividence)  combines usability testing + market research techniques  automatic logging & some analysis of usage

2. Online Useability Testing

3. Max – WebCriteria’s Agent  Predicts how long information seeking tasks would take on a web site  Automated procedure: seed with start page & goal page procedure 1) reads page 2) model predicts how long to find & click proper link load time, scan time, & mouse movement time 3) repeat until find goal page  Claim: time is directly related to usability

4. Sample of Max’s Reports

5. Tradeoffs of Max-style Model  Advantages very inexpensive (no participants needed) fast turnaround (hours) can run on many sites & compare -> benchmarks  Disadvantages focus on time (much of it download time)  only 3 rd in important factors driving repeat web visits  can’t tell you anything about your content  doesn’t say anything directly about usability problems robots aren’t humans  doesn’t make mistakes remember, GOMS assumes expert behavior!  doesn’t account for understanding text  only tries the best path – users will use many major flaw is the lack of real users in the process

6. Remote Usability Testing  Move usability testing online research participants access “lab” via web answer questions & complete tasks in “survey” system records actions or screens for playback can test many users & tasks -> good coverage  Analyze data in aggregate or individually find general problem areas  use average task times or completion rates playback individual sessions focus on problems w/ traditional usability testing

7. Warning I am a founder of the following company – watch for bias!

8. NetRaker: Web Experience Evaluation  NetRaker Index short pop-up survey shown to 1 in n visitors on-going tracking & evaluation data  Market Research & Usability Templates surveys & task testing records clickstreams as well invite delivered through email, links, or pop-ups  NetRaker Experience Recording captures “video” of remote participants screen indexed by survey data or task performance

9.  Small number of rotated questions increases response rate NetRaker Index: On-going customer intelligence gathering

10.  Small number of rotated questions increases response rate NetRaker Index: On-going customer intelligence gathering

11.  Increasing these indices (e.g., retention) moderately (5%) leads to a large increase in revenue growth NetRaker Index: On-going customer intelligence gathering

12. NetRaker Usability Research: See how customers accomplish real tasks on site

15. Advantages of Remote Usability Testing  Fast can set up research in 3-4 hours get results in 24 hours  More accurate can run with large sample sizes  50-200 users -> reliable bottom-line data (stat. sig.) uses real people (customers) performing tasks natural environment (home/work/machine)  Easy-to-use templates make setting up easy for non-specialists  Can compare with competitors indexed to national norms

16. Disadvantages of Remote Usability  Miss observational feedback facial expressions verbal feedback (critical incidents)  Need to involve human participants costs money (typically $20- $50/person)  People often do not like pop-ups need to be careful when using them