1. Presentation: Techniques for user involvement ITAPC1.

2. Motivation How to evaluate pervasive computing projects? Performance, reliability, feasibility, usability Increasing data validity levels 1.Basic feasibility lab testing 2.Feasibility testing in situ Work, home, bus … 3.Testing with real users a.Controlled and observed b.Uncontrolled c.Longitudinal

3. Main Guidelines 1.Remember that you are NOT your typical user “The designer” is not a good user 2.Know your users “Everyone” is not a user “The Vice President” is rarely the user “Marketing” is rarely the user If you can’t find a user – then you’re in big trouble! 3.Users aren’t designers It is your job to generate visions (prototypes, future workshops) and to collect their feedback 4.Users aren’t all-knowing Beware of users giving you wrong information – use & study several

4. The Context Tradition (cognitive) HCI is not concerned with the context where the work is situated Focus is on the mapping between the user & the computer in an isolated setting This approach has various flaws

5. The “new wave” of HCI Social Computing (born in the late 80’s) Users are not isolated beings They are part of a context – social & cultural Not possible to isolate the work context completely in a lab Users act differently when not in their work context Solution Study the users at the work setting (field studies & interviews) Integrate users in the development team

6. The “new wave” of HCI Concepts Distributed Cognition Situated Action Theory Activty Theory Participatory Design (field studies & interviews) Prototyping XP: eXtreme Programming (and other Agile Methods)

7. Warning Not all user involvement needs to be in the work context as this might be expensive As always - cost/benefit Therefore – come well prepared ! Different methods to apply involving users

8. Techniques Field studies or lab testing Think aloud: talk while doing the job Talk right after : debriefing after the job Role playing Cueing recall with videotape (Focus Shift Analysis) Focus groups & interviews Mailed surveys Context recording Others: Retrospective testing, Coaching method, Constructive interaction, Contextual inquiry

9. Field studies or lab testing Field studies are undertaken to understand the users work context and enable the designer to discover possible user needs Video recording and/or note taking Video not always acceptable (hospital ward) Evaluation methods – studying the use of prototypes or the finished product – can either be performed in the users work context or in a lab Depending on the prototypes level of completion

10. Focus group Requires a skilled and experienced leader One or two people may dominate Some people may get into side conversations May work if very well done

11. Surveys and Questionnaires Very difficult to write good questions Choice of respondents list can bias results A return rate of 10% is extremely good; 1-2% is typical Consider finding a patient organization or similar May report on what people think they do, not what they actually do May be inexpensive

12. Think aloud: the Gold Standard Both lab & in situ (Tognazzini, 1992): might be big controlled lab experiments with video cameras – or cheap no nonsense studies “May be the single most valuable … method” (Nielsen, 1993) Makes the users mental model of the system clear to us – to close the gap between the model of the system

13. Think aloud: the Gold Standard Samples of facilitating remarks: “Could you tell me what options you are considering?” “Is anything in particular puzzling you?” “What might you do next?” BUT: No feedback, positive or negative No answers or hints If users get stuck, have them go on Neutral demeanor at all times ”I’m not the designer”

14. Talk right after Sometimes talking while doing the work is not feasible

15. Talk right after Talking right after may be best alternative to thinking aloud

16. Context recording Avoid observer bias Understand “real use” Video recording and analysis expensive Record context (contextual tagging) Combine with qualitative follow-up interviews May uncover “hidden” patterns

17. Role playing Some things happen so infrequently that there is no way to observe the user Some one may be able to play the role of a user in that situation

18. Recruiting Users If product is in use, test with real current users: Employees Customers If it is a new product – test with potential users Provide incentives: Coffee mugs, T-shirts, wine, cake Not family or friends: they won’t be critical enough Ethical aspects Conduct all test with the deepest respect for their well- being Users are only “novice” users the first time! Do not waste your usability testing resources on basic problems – make sure your product is ready for testing

19. Common Pitfalls Reliability Huge differences between test users Best user up to 10x efficient 25% users twice as fast as the slowest 25% May use standard statistical tests Number of test users will determine reliability Validity Are we measuring something useful? Statistics won’t help here – only common sense and experience