1. 05/18/2004 Verizon Labs 1 Techniques for Understanding the User Experience Measuring Web Search Effectiveness: The User Perspective WWW 2004 Conference Workshop May 18, 2004 Demetrios Karis (demetrios.karis@verizon.com) Verizon Labs

2. 05/18/2004 Verizon Labs 2 Multiple Techniques  Laboratory-based studies  Remote usability studies  Direct User Feedback  Log Analyses  Ethnographic Studies  Field Studies  Surveys  Interviews and Focus Groups  Techniques for experiencing what the user experiences

3. 05/18/2004 Verizon Labs 3 Laboratory-based Techniques  Usability Studies  Qualitative (formative)  Quantitative (summative)  Comparative Studies  Longitudinal Studies  Experimental Studies

4. 05/18/2004 Verizon Labs 4 Remote Usability Evaluations  Experimenter involved, real-time; e.g., via WebEx, Live Meeting  No experimenter, instrumented browsers; e.g., Vividence, Keynote’s WebEffective™ Intelligence Platform (was NetRaker)  No experimenter, normal browser; program opens 2 windows, one with tasks, one with website  Also, fully automated techniques, no users, no experimenters (e.g., Bobbi - accessibility, many research systems)

5. 05/18/2004 Verizon Labs 5 Direct User Feedback  Email feedback from web site  Very useful  Very insensitive to problems  Calls to customer care reps

6. 05/18/2004 Verizon Labs 6 Log Analysis  Incredibly valuable  Incredibly difficult to do well and to interpret meaning correctly  Multiple tools and packages available, but custom scripts often required

7. 05/18/2004 Verizon Labs 7 Measuring Web Search Effectiveness (1)  Create meaningful segments of users (& perhaps personas) to help choose study participants (e.g., male/female, age, occupation, computer & web experience)  Lab-based Usability Studies  6 people a week, using both self-generated and standard tasks  Insights into problems, confusions, the search process, people’s different thought processes  Ideas for improvements & new features  Automated Remote Testing  100 to 1000 people/month  Self-generated tasks (relevant & important to individual)  Each individual rates success and provides comments

8. 05/18/2004 Verizon Labs 8 Measuring Web Search Effectiveness (2)  Direct User Feedback  At least once a month, increase the visibility of the feedback link on select pages (e.g., results)  This will increase amount of feedback, plus increase diversity and make them more representative  Analyze and categorize the resulting email feedback  Field Study using a Panel of Experienced Users  Create a panel of users with characteristics that match those of the user base (N = 50 to 100)  They report, via email, on all their searches over the course of a week: what search engine they used, what they were searching for, success/failure, comments/suggestions  Could partially automate