1. Observation Watch, listen, and learn…

2. Agenda  Observation exercise Come back at 3:40.  Questions?  Observation

3. Observing Users  Not as easy as you think  One of the best ways to gather feedback about your interface  Watch, listen and learn as a person interacts with your system  Qualitative & quantitative, end users, experimental or naturalistic

4. Observation  Direct In same room Can be intrusive Users aware of your presence Only see it one time May use 1-way mirror to reduce intrusiveness  Indirect Video or app recording Reduces intrusiveness, but doesn’t eliminate it Cameras focused on screen, face & keyboard Gives archival record, but can spend a lot of time reviewing it

5. Location  Observations may be In lab - Maybe a specially built usability lab  Easier to control  Can have user complete set of tasks In field  Watch their everyday actions  More realistic  Harder to control other factors

6. Usability Lab http://www.surgeworks.com/services/observ ation_room2.htm Large viewing area in this one- way mirror which includes an angled sheet of glass the improves light capture and prevents sound transmission between rooms. Doors for participant room and observation rooms are located such that participants are unaware of observers movements in and out of the observation room.

7. Observation Room  State-of-the-art observation room equipped with three monitors to view participant, participant's monitor, and composite picture in picture.  One-way mirror plus angled glass captures light and isolates sound between rooms.  Comfortable and spacious for three people, but room enough for six seated observers.  Digital mixer for unlimited mixing of input images and recording to VHS, SVHS, or MiniDV recorders.

8. Task Selection  What tasks are people performing? Representative and realistic? Tasks dealing with specific parts of the interface you want to test? Problematic tasks?  Don’t forget to pilot your entire evaluation!!

9. Engaging Users in Evaluation  What’s going on in the user’s head?  Use verbal protocol where users describe their thoughts  Qualitative techniques Think-aloud - can be very helpful Post-hoc verbal protocol - review video Critical incident logging - positive & negative Structured interviews - good questions  “What did you like best/least?”  “How would you change..?”

10. Think Aloud  User describes verbally what s/he is thinking and doing What they believe is happening Why they take an action What they are trying to do  Widely used, popular protocol  Potential problems: Can be awkward for participant Thinking aloud can modify way user performs task

11. Cooperative approach  Another technique: Co-discovery learning (Constructive iteration) Join pairs of participants to work together Use think aloud Perhaps have one person be semi-expert (coach) and one be novice More natural (like conversation) so removes some awkwardness of individual think aloud  Variant: let coach be from design team (cooperative evaluation)

12. Alternative  What if thinking aloud during session will be too disruptive?  Can use post-event protocol User performs session, then watches video afterwards and describes what s/he was thinking Sometimes difficult to recall Opens up door of interpretation

13. What if a user gets stuck?  Decide ahead of time what you will do. Offer assistance or not? What kind of assistance?  You can ask (in cooperative evaluation) “What are you trying to do..?” “What made you think..?” “How would you like to perform..?” “What would make this easier to accomplish..?” Maybe offer hints This is why cooperative approaches are used

14. Inputs / Outcomes  Need operational prototype could use Wizard of Oz simulation  What you get out “process” or “how-to” information Errors, problems with the interface compare user’s (verbalized) mental model to designer’s intended model

15. Capturing a Session  1. Paper & pencil Can be slow May miss things Is definitely cheap and easy Time 10:00 10:03 10:08 10:22 Task 1 Task 2 Task 3 … SeSe SeSe

16. Capturing a Session  2. Recording (audio and/or video) Good for think-aloud Hard to tie to interface Multiple cameras may be needed Good, rich record of session Can be intrusive Can be painful to transcribe and analyze

17. Capturing a Session  3. Software logging Modify software to log user actions Can give time-stamped key press or mouse event Two problems:  Too low-level, want higher level events  Massive amount of data, need analysis tools

18. Example logs 2303761098721869683|hrichter|1098722080134|MV|START|566 2303761098721869683|hrichter|1098722122205|MV|QUESTION|false|false|false|false|false|false| 2303761098721869683|hrichter|1098724978982|MV|TAB|AGENDA 2303761098721869683|hrichter|1098724981146|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098724985161|MV|SLIDECHANGE|5 2303761098721869683|hrichter|1098724986904|MV|SEEK|PRESENTATION-A|566|604189|0 2303761098721869683|hrichter|1098724996257|MV|SEEK|PRESENTATION-A|566|604189|604189 2303761098721869683|hrichter|1098724998791|MV|SEEK|PRESENTATION-A|566|604189|604189 2303761098721869683|hrichter|1098725002506|MV|TAB|AGENDA 2303761098721869683|hrichter|1098725003848|MV|SEEK|AGENDA|566|149613|604189 2303761098721869683|hrichter|1098725005981|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098725007133|MV|SLIDECHANGE|3 2303761098721869683|hrichter|1098725009326|MV|SEEK|PRESENTATION|566|315796|149613 2303761098721869683|hrichter|1098725011569|MV|PLAY|566|315796 2303761098721869683|hrichter|1098725039850|MV|TAB|AV 2303761098721869683|hrichter|1098725054241|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098725056053|MV|SLIDECHANGE|2 2303761098721869683|hrichter|1098725057365|MV|SEEK|PRESENTATION|566|271191|315796 2303761098721869683|hrichter|1098725064986|MV|TAB|AV 2303761098721869683|hrichter|1098725083373|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098725084534|MV|TAB|AGENDA 2303761098721869683|hrichter|1098725085255|MV|TAB|PRESENTATION 2303761098721869683|hrichter|1098725088690|MV|TAB|AV 2303761098721869683|hrichter|1098725130500|MV|TAB|AGENDA 2303761098721869683|hrichter|1098725139643|MV|TAB|AV 2303761098721869683|hrichter|1098726430039|MV|STOP|566|271191 2303761098721869683|hrichter|1098726432482|MV|END

19. Analysis  Many approaches  Task based How do users approach the problem What problems do users have Need not be exhaustive, look for interesting cases  Performance based Frequency and timing of actions, errors, task completion, etc.  Very time consuming!!

20. Example: Heather’s study  Software: MeetingViewer interface fully functional  Criteria – learnability, efficiency, see what aspects of interface get used, what might be missing  Resources – subjects were students in a research group, just me as evaluator, plenty of time  Wanted completely authentic experience

21. Heather’s evaluation  Task: answer questions from a recorded meeting, use my software as desired  Think-aloud  Video taped, software logs  Also had post questionnaire  Wrote my own code for log analysis  Watched video and matched behavior to software logs

22. Example: Analysis  Printed logs  Watched video and marked actions, thoughts, and video timing info on the logs  Wrote down interesting quotes from the think aloud  Took 2 to 3 times the session time just to do this analysis alone!