CHAPTER 10 Introducing Evaluation Doosup Baek Sungmin Cho Syed Omer Jan

Chapter Goals Explain the concepts and terms used to discuss evaluation Discuss HutchWorld Case Study Examine different techniques used at different stages of development Show and discuss how developers deal with real world problems and constraints

Definition of Evaluation ?? It is the process of systematically collecting date that inform us about what it is like for a particular user or group of users to use a product for a particular task in a certain type of environment

Two main types of Evaluation Formative Evaluation is done at different stages of development to check that the product meets users needs Summative Evaluation assesses the quality of a finished product –Our focus is going to be on formative evaluation

What to Evaluate It’s a continuous process which examines: Focus on users and their tasks Observe, measure, and analyze their performance with the system Design iteratively Important for designer to check and make sure that they understand user requirements

Why you need to Evaluate Designers should not presume that everyone is like them or that the following of set guidelines would guarantee them good usability Evaluation is needed to check that users can use the product and like it

Why you need to Evaluate (cont’d) Bruce Tognazzini – Usability Consultant “Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process you are going to throw buckets of money down the drain”

Why you need to Evaluate (cont’d) Toganazzini’s 5 resons to evaluate: –Problems are fixed before the product is shipped, not after –The team can concentrate on real problems, not imaginary ones –Engineers code is sharply reduced –Time to market is sharply reduced –Upon 1 st release, your sales department has a rock solid design it can sell without having to pepper their pitches with how well the next release will work

Why you need to Evaluate (cont’d) Usability testing involves measuring the performance of typical users on typical tasks Satisfaction, can be evaluated through questionnaires and interviews Trends are towards evaluating more subjective user-experience goals, like emotionally satisfying, motivating fun etc.

When to Evaluate New product –Use mockups, sketches, and other low fidelity prototyping techniques are used to represent design ideas Upgrading existing products –Compare user performance and attitudes and contrast new products with the previous versions Evaluation is a key ingredient for a successful design

1984 OMS Background –Voice mail system for Olympic Games Contestants and their families could send and receive messages –Developed by IBM –Reason for intense evaluation IBM’s reputation at stake Olympics a high profile event

1984 OMS (cont’d) Evaluation activates during development –Use of printed scenarios for feedback –Iterative testing of user guides –Early simulations –Early demonstrations –Overseas test –“Try-and-Destroy-it” with CS students –Heavy traffic tests –Pre-Olympic field tests

HutchWorld Case Study Virtual community Collaboration –Microsoft’s Virtual Worlds Research Group –Fred Hutchinson Cancer Research Center Uses –Chatting, storytelling, discussions and emotional support Why ?? –Cancer patient isolation issues

HutchWorld Design Question ?? Needs –Useful –Engaging –Easy to use –Emotional satisfaction Early Ideas –What resource are available to patients? –What are the specific needs of the user’s? –What should it look like? –How will users interact within the Virtual community

Design Team Efforts !!! Interviews with patients, caregivers, family, friends, clinicians, social support groups, former friends, and experts Reading of latest research literature and HutchWorld web pages Visiting Fred Hutch research facilities, and the Hutch school for pediatric patients and juvenile patient family members

Problems Inadequate non-verbal feedback –Potential for mis-understanding –No : Facial expressions Body language Tone of voice

Features of HutchWorld Availability –Anytime, day or night –Regardless of geographic location Design to resemble the outpatient facility –This real-world metaphor helped users infer the functionality Synchronous chat environment was selected for realism 3D photographic avatars

Testing HutchWorld Test 1: –six computers –scaled-back prototype –Microsoft specialists trained Hutch volunteers –events were hosted in the prototype Test 1 observations: –general usage of the prototype –usage of the space during unscheduled times

Testing HutchWorld (cont’d) Test 1 results: –small user community –critical mass concept – not enough participants to fill the chat room for successful conversation –lack of interest –patient availability –patients preferred asynchronous communication (via , journals, etc.) –prototype did not include original computer uses patients played games and searched the internet

Redesigning HutchWorld a more “unified” product was desired that included a variety of communication, information, and entertainment tasks new support: –more asynchronous communication –information-retrieval tools – , a bulletin board, text-chat –games and other entertainment tasks –a web page creation tool –a way to check if anyone is around to chat with

Usability Tests Seven participants –four had used chat rooms –all had browsed the web –given five minutes to get familiar with software A running commentary was given by each during exploration (what each was looking at, thinking, or confused by) After five minutes, a series of structured tasks were given focusing on how the participants: –dealt with their virtual identity –communicated with others –retrieved desired information –found entertainment

Questionnaire After the test participants were asked to fill out a questionnaire about their experience with HutchWorld –What did you like about HutchWorld? –What did you not like about HutchWorld? –What did you find confusing or difficult to use in HutchWorld? –How would you suggest improving HutchWorld?

Usability Findings The back button did not always work. Users ignored navigation buttons –more prominent buttons were needed Users expected that objects in 3D would do something when clicked on –provide links to web pages when objects are clicked Users did not realize other real people were interacting with them in the world –wording was changed in the overview description Users did not notice the chat window and instead chatted with people on the participation list –instructions on where to chat were clarified

Future of HutchWorld Evaluation of the effects of the software at the Fred Hutchinson Center Investigation will include: –How the computers and software impact the social wellbeing of the patients and their caregivers? –What type of computer-based communication best supports this patient community? –What are the general usage patterns of the system? –How might any medical facility use computers and software like HutchWorld to provide social support for its patients and caregivers?

Discussion

Reference

The “Star Fire” Video Prototype Project CHESS (Comprehensive Health Enhancement Support System) Module DemosCHESS (Comprehensive Health Enhancement Support System) Module Demos –AsthmaAsthma Microsoft Social Computing Group Olympic Message Service The Schaffer Method of user-centered design –ChartChart HutchWorld video clip