Chapter 10: Introducing Evaluation Group 4: Tony Masi, Sam Esswein, Brian Rood, Chris Troisi

Chapter Goals Explain the concepts and terms used to discuss evaluation. HutchWorld case study. Examine how different techniques are used at different stages of development. Discuss how developers cope with real- world constraints.

Definition of Evaluation Dictionary Definition – “To ascertain or fix the value or worth of.” Book’s Definition – “The process of systematically collecting data that informs us about what it is like for a particular user or group of users to use a product for a particular task in a certain type of environment.”

Some Evaluation Companies: META Group, Inc. – Canadian Innovation Centere Hitachi Data Systems How To Evaluate Your Software -

Iterative design & evaluation is a continuous process that examines: Early ideas for conceptual model Early prototypes of the new system Later, more complete prototypes Designers need to check that they understand users’ requirements. Designers need to check that they understand users’ requirements. What to evaluate

Why to evaluate Designers should not presume everyone is like them or that following set guidelines guarantees usability. Evaluation is needed to check that users can use the product and like it.

Why to evaluate –“Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process you are going to throw buckets of money down the drain.” – Bruce Tognazzini

Why to evaluate Tognazzini’s 5 reasons to evaluate: –Problems are fixed before the product is shipped, not after. –The team can concentrate on real problems, not imaginary ones. –Engineers code instead of debating –Time to market is sharply reduced –Upon first release, sales department has a rock-solid design it can sell without having to pepper pitches with how well the NEXT release will work.

Why to evaluate USABILITY TESTING involves measuring the performance of typical users on typical tasks. SATISFACTION can be evaluated through questionnaires and interviews. Trends are towards evaluating more subjective user-experience goals, like emotionally satisfying, motivating, fun, etc.

When to evaluate Throughout design New product –Use mockups, sketches, and other low-fidelity prototyping techniques to elicit users’ opinions –Goal is to asses how well a design fulfills users’ needs and whether the users like it Upgrade existing product –Evaluations to compare user performance and attitudes toward new design with those of the previous versions Evaluation is a key ingredient for a successful design.

Two main types of evaluation Formative evaluation is done at different stages of development to check that the product meets users’ needs. –Design proceeds through iterative cycles of ‘design-test- redesign’ –Helps ensure products success upon first arrival in market Summative evaluation assesses the quality of a finished product. –Satisfy sponsoring agency –Check that standard is being upheld Chapter 10 focuses on Formative Evaluation.

Story of the 1984 OMS (Box 10.1) Background – –Voice mail system for Olympic Games contestants and their families could send and receive messages –Developed by IBM –Could be used from almost any push-button phone system around the world Reasons for intense evaluation - –IBM’s reputation at stake –Olympics a high-profile event

Story of the 1984 OMS (Box 10.1) Evaluation activities – –Use of printed scenarios –Iterative testing of user guides –Development of early simulations KeypadsReactions –Olympian on design team –Interviews with other Olympians –Overseas tests –Free coffee and donut tests –‘Try-and-destroy-it’ with Computer Science students –Pre-Olympic field tests –Heavy traffic tests

HutchWorld Case Study A Virtual Community Collaboration: –Microsoft’s Virtual Worlds Research Group –Fred Hutchinson Cancer Research Center Uses: –chatting –storytelling –discussions –emotional support Why? –cancer patient isolation issues

How to design HutchWorld? Needs: –useful –engaging –easy-to-use –emotional satisfaction Early ideas: –what is typical cancer treatment? –what resources are available to patients? –what are the specific needs of the users? –what kind of “world” should be the model? –how will users interact within the virtual community? –what should it look like?

No stone left unturned Interviews with patients, caregivers, family, friends, clinicians, social support groups, former patients, and experts Reading of latest research literature, and HutchWorld web pages Visiting Fred Hutch research facilities, and the Hutch school for pediatric patients and juvenile patient family members

Problem Inadequate non-verbal feedback –potential for misunderstanding –no: facial expressions body language tone of voice

Research Studies indicates social support helps cancer patients cope psychologically with their disease. Patients also benefit in their overall physical wellbeing. Example: women with breast cancer receiving therapy lived on average twice as long as those that did not.

Features of HutchWorld Availability: –anytime, day or night –regardless of geographic location Designed to resemble the outpatient facility –This real-world metaphor helped users infer the functionality. Synchronous chat environment was selected for realism (vs. asynchronous) 3D photographic avatars (p. 326)

Before testing Logistical issues: –Who would provide training for patients and testers? –How many systems were needed for testing? –Where should these systems be placed?

Testing HutchWorld Test 1: –six computers –scaled-back prototype –Microsoft specialists trained Hutch volunteers –events were hosted in the prototype Test 1 observations: –general usage of the prototype –usage of the space during unscheduled times

Testing HutchWorld Test 1 results: –small user community –critical mass concept – not enough participants to fill the chat room for successful conversation –lack of interest –patient availability –patients preferred asynchronous communication (via , journals, etc.) –prototype did not include original computer uses patients played games and searched the internet

Redesigning HutchWorld a more “unified” product was desired that included a variety of communication, information, and entertainment tasks new support: –more asynchronous communication –information-retrieval tools – , a bulletin board, text-chat –games and other entertainment tasks –a web page creation tool –a way to check if anyone is around to chat with

Usability Tests Seven participants –four had used chat rooms –all had browsed the web –given five minutes to get familiar with software A running commentary was given by each during exploration (what each was looking at, thinking, or confused by) After five minutes, a series of structured tasks were given focusing on how the participants (p. 329): –dealt with their virtual identity –communicated with others –retrieved desired information –found entertainment

Questionnaire After the test participants were asked to fill out a questionnaire about their experience with HutchWorld –What did you like about HutchWorld? –What did you not like about HutchWorld? –What did you find confusing or difficult to use in HutchWorld? –How would you suggest improving HutchWorld?

Usability Findings The back button did not always work. Users ignored navigation buttons –more prominent buttons were needed Users expected that objects in 3D would do something when clicked on –provide links to web pages when objects are clicked Users did not realize other real people were interacting with them in the world –wording was changed in the overview description Users did not notice the chat window and instead chatted with people on the participation list –instructions on where to chat were clarified

Follow-up more rounds of observation and testing were conducted with new subjects HutchWorld was installed at the Fred Hutchinson Center observations continued on the users –which parts of the system are being used? –when are they being used? –why are they being used?

Future of HutchWorld evaluation of the effects of the software at the Fred Hutchinson Center investigation will include: –How the computers and software impact the social wellbeing of the patients and their caregivers? –What type of computer-based communication best supports this patient community? –What are the general usage patterns of the system? –How might any medical facility use computers and software like HutchWorld to provide social support for its patients and caregivers?