Presented by Arseniy Mstislavskiy

Debriefing & Performance Performance is «What?» Debriefing is «Why?» D. is for reviewing participant's actions at performance part of test done after every test session to understand each error, difficulty, omission occured during session D. is the last chance for it Participants explain things you can't see (thoughts, preferences, rationale behind actions) Observers give diff. point of view, fill gaps in memory, may feel engaged and help with resolution later, the moderator can take from interactions with observers some notion of the priorities of the stakeholders on the design team, a useful thing to know when it comes time to report results.

I. Debriefing with participants Don't make p. feel defensive about actions, opinions Don't react to p.'s answers If that happens p.'s answers become suspect «Thanks for answering that. I appreciate your feedback» - and go further

1. Place Same room: reference to product needed, p. became comfortable in a room Another room: p. is shy, distracted or tired (testing in a large observed room) Move if there is no choice (observers can sill be involved)

2. Basic guidelines Right after the test «We're getting to the end» P. fills any post-test questionnaires You collect thoughts (simultaneously) How the session went (in general) Unresolved issues left Unnoticed things Missed crucial instructions Possible reasons for unclear actions Review debriefing topics guide (sort and remove issues)

2. Basic guidelines Continuation Review questionnaire (p. rests) Unexpected interesting answers Behaviour and answer don't match Or skim through notes gather questions from observers

2. Basic guidelines Let p. express any ideas «What do you think?» Start with high-level issues (use debr. topics guide) general research questions Move to specific issues (found) Review marked places of the post-test questionnaire — understand the basis of a problem D. is not for resolving problems (e.g. asking p. what he/she thinks) Finish questioning before letting observers get in O. write their questions down (even in same room) O. ask themselves (if p. is comfortable)

3. Advanced guidelines Replay the test Manual method (reexperiencing emotions, reactions) p. can visualise what he/she experienced at specific point You ask questions you had Video method Requires more time Is better then memorising Seeing themselves may help p. Removes the need to interrupt p. during the test performance

3. Advanced guidelines audio record d. Transcribe it later Good for pulling remarks Helps in content analysis Review alternate designs Collecting p.'s preferences during early stages of developement Comparing prototypes is easier for p. than generating ideas

3. Advanced guidelines «What did you remember» Can't be asked during the test («Do you see this …?» - «Of course I do!») Cover/remove particular element ! Be aware of dangerous situations «Devil's advocate» P. doesn't want to be critical Very important feature (need to be 100% sure) Be careful Say you're surprised LIE! Say that others say the opposite. Doesn't work on Very young/old persons Ingratiating persons Testing => possible job Warn observers and team beforehand (!) Is only used to get real feelings, not to bias

II Debriefing with observers Make o. co-researchers O. are other people to talk about what happened O. become more engaged later You see important things to include in report You make team agree on issues => remedies

II Debriefing with observers No break after test session «How did that go?» Gather ideas in the rolling issues list (will help for top 10 type reports) «What surprised you?» Group ideas Go through task by task Let everybody understand the issue agree on the reasons Document all items (in a way accessible to o.) Stop when all o.'s insights voiced

III At the end Make last debriefing session with o. Step throught the rolling issues list Helps organising findings in the final report Be sure noone is surprised with report Ask o. Any revisions Sort items Voting Call out top 10 Card sorts Don't talk about remedies Remember you will have to review collected issues Those are just accumulated impressons and personally biased recalls You then see subtle differencie possibly conflicting with debriefing with o. ==> ==> Ch. 11 «Analyze Data and Observations»