Introduction to System Evaluation IS 588 Dania Bilal & Lorraine Normore Spring 2007

Design Philosophies System-centered design philosophy –Software works well; why evaluate from users’ perspectives? User-centered design philosophy –Software should be usable: based on users’ needs, tasks, and cognitive and affective behaviors.

Why Evaluate Following design guidelines does not guarantee usability Need to assess the user’s experience with system, cognitively and affectively –Allow identification of level of usability Saves money –Problems can be fixed before system is released

What Users Need? Class Discussion AS users of various information systems (Web engines, OPAC, databases, etc.) what do you need a system that you use to do for you, in general?

What Users Need from a System? A system that is –Pleasing; satisfying –Easy to learn; friendly –Effective; challenging –Efficient; saves user’s time and efforts –Supported by positive, corrective feedback mechanisms –Predictable; consistent

Investing in User Testing Tognazzini: –Problem can be fixed before product is released –Designers can identify “real” problems with system –Provide objective assessment for redesigning system –Save money and marketing time –Robust design to sell High confidence in product

Why NOT Evaluate? Class Discussion Why most designers are against involving users in the design and evaluation of products/systems?

Why NOT Evaluate? Save time Save money Philosophy: users don’t know much about system capabilities and some may not articulate well their needs. Other???

When to Evaluate? During product development –After market research, mock-ups, low-fidelity prototypes, etc. are designed and users’ reactions are elicited –Formative evaluation –Summative evaluation Before release of final product –After prototypes and actual designs have been tested and problems have been fixed –Summative evaluation After release of final product –To measure the usability and success of the product with users in a variety of settings.

What to Evaluate? The user’s experience with the product –User needs and requirements –Success, ease of use, friendliness, and other basic usability features –Affective states Likes, dislikes, satisfaction, frustration, attitude –Support of use patterns –Other interaction behaviors based on purpose of product

HutchWorld Case Study Virtual community chat system Developed by Microsoft Virtual Worlds Research Group and Fred Hutchinson Cancer Research Center Purpose –To enable cancer patients, caregivers, family, and friends to chat and gain emotional and information support from one another.

HutchWorld Case Study Early design ideas –Learn about the patients’ experiences… How was this done? –Students to reply

HutchWorld Case Study: First Test Informal on-site test –Computers were set up in hospital and a scaled-back prototype of V-chat was installed. –A sample of patients and their families interacted with prototype

First Test: Usability Issues Identified User preference for asynchronous communication Addition of websites about cancer information Addition of games and entertainment features

First Test: Redesign Based on Issues Identified Software was redesigned based on user needs HutchWorld became a portal containing a variety of information, communication, and entertainment areas –Bulletin boards, , text chat, web page creation wizard, and other features

Second Test: Usability Study Seven participants (male & female) Prior experience was assessed Interaction with system Performed at Microsoft usability labs –Structured tasks (see Text, p. 329) –Verbalization (think aloud protocol) –Participants activities were captured online –Observer/evaluator interacted with participant in lab –Evaluator took observational notes

Second Test: Usability Study Development team member interacted with system as a participant Short questionnaire was completed by each participant after completing tasks Participants rated difficulties in completing each task (see Text, p. 331)

Second Test: Lessons Learned from Usability Study Problems with software and their severity were identified –Rating of problems: low, medium, high –Problems can be seen in Text, pp –Fixes were made

After the Second Test Additional observations and testing were done Six new participants –Simultaneous interaction in labs –More detailed and focused –Structured similar to previous test –Fewer problems were identified –Fixes were made

Third Test: Focus Group Setting Focus group interacted with system in Cancer Research Center Patients and caregivers observed interaction Feedback on final version was obtained

Final Test: Natural Setting System was used in a residential building with an Internet access Building housed patients and their families Informal observation of patterns of use Assess how HutchWorld integrated with patients’ lives (medical care routines; access to social support, etc.)

When to Stop System Testing? Class Discussion 1. When do you think testing a system or a product should stop and why? 2. What are the various methods employed in HutchWorld Study? Are these multiple methods useful and why?