1. Chapter 12 Observing Users Li, Jia Li, Wei

2. Outline What and when to observe Approaches to observation How to observe How to collect data Indirect observation Analyzing, interpreting and presenting data

3. Goals and Questions Provide a focus for observation  Example: watch a group of people  First watch: what are they doing? …  Second watch: more focused questions Necessary for all evaluation studies  Balance between being guided by goals and being open-minded

4. What and When to Observe During early design of a product Help understand users ’ needs During later refinement of a product Find out whether it meets users ’ needs

5. Type of Observation Participant observation (e.g., in ethnography) (Not applicable)Insider “Quick and dirty” In field studies “Quick and dirty” In usability testing Outsider looking on Field environment (i.e., natural) Controlled environment (i.e., lab-like) Observation

6. Approaches to Observation “ Quick and dirty ” observation find out what is happening quickly and with little formality Observation in usability testing controlled environment: Video, interaction logs Watch through one-way mirror or via remote TV screen Observation in field studies

7. Observation in Field Studies Levels of participation (By Colin Robson, 1993) Complete participants More marginal participants Observers who also participate Outside observer

8. Observation in Field Studies - Outside Observer When an observer is interested only in the presence of certain types of behavior Example: Observe when a computer is used by boys and when by girls, and how long

9. Observation in Field Studies - Inside Observer Participant observers Participate in the lives of the people under study Maintain a professional distance that allows adequate observation and recording of data Ethnographers Controversy Participant observation is virtually synonymous with ethnography Participant observation is a technique used in ethnography

10. How to Observe - In Controlled Environments Issues before observation Decide laboratory to do the tests where appropriate equipment is set up Test equipment to make sure it works Provide an informed consent form for users to read and sign at the beginning of the study Problems Observers do not know what users are thinking  Think-aloud technique

11. How to Observe - In the Field Issues before observation State initial study goal and questions clearly Select a framework to guide activity in the field Decide how to record events Go through notes and other records as soon as possible Highlight and separate personal opinion from what really happens

12. How to Observe - In the Field (Cont ’ d) Issues before observation Refine goals and questions while observation Gain the acceptance and trust of those to be observed Think about ways to handle sensitive issues Consider working as a team Check notes with an informant or members of the group Plan to look at situation from different perspectives

13. Frameworks Why use frameworks Provide focus Organize/structure observation What framework to select A simple framework:  The person  The place  The thing

14. Frameworks (Cont ’ d) What framework to select Goetz and LeCompte framework  Who is present?  What is happening?  When does the activity occur?  Where is it happening?  How is the activity organized?

15. Frameworks (Cont ’ d) What framework to select Colin Robson framework  Space: appearance and lay-out  Actors: names and relevant details  Activities: what are they doing and why  Objects: present physical objects  Acts: what are specific individuals doing  Goals: what to accomplish for actors  Feelings: mood of the group and individuals

16. How to Observe – Participant Observation and Ethnography Issues before observation Identify goal and questions with the help of observation framework Get immersed in the group to observe Collect a variety of data Get balance between broad picture and specific questions Analyze data using a holistic approach

17. Dilemmas When should I stop observing When see similar patterns of behavior being repeated When finish listening to all the main stakeholder groups and understand their perspectives How can I adapt ethnography to fit the development process Preparation Field study Analysis Reporting

18. Data collection Notes plus still camera Audio recording plus still camera Video

19. Notes plus still camera Advantages Less technical. Flexible. Equipment easy to get and not expensive. Unobtrusive. Easy to transcribe. Disadvantages Difficult and tiring to write and observe at the same time. Writing speed limited. Less reliable, rely on the note-takers.

20. Audio plus still camera Advantages Inexpensive Flexible Relatively unobtrusive Low disturbance Disadvantages Lack of visual record Difficult to transcribing the data. Have to change tapes and position microphone.

21. Video Advantages Capture both visual and audio High reliability Permanent original record that can be revisited. Disadvantages Equipment more expensive Intrusive Time-consuming to analyze the video data Need positioning and focusing camera lens.

22. Indirect observation: tracking users ’ s activities Diaries Interaction logging

23. Diaries (1) What users did When they did it What they thought about their interactions with the technology Useful when users are scattered and unreachable such as: Internet and web evaluations

24. Diaries (2) Advantages Inexpensive No special equipment or expertise Suitable for long-term studies Disadvantages Rely on participants being reliable and remembering to complete the diaries. Participants often remember events as being better or worse than they really were, or taking more or less time than they actually did.

25. Interaction logging Advantages Unobtrusive Large volumes of data can be logged automatically Disadvantages Ethical concerns. Powerful tools are needed. Webtrends

26. Dilemma --- observing without being seen They do not know we are watching. Shall we tell them? Tell: users may react or change their behavior. Depends on the context:  How much personal information is collected?  How the information will be used?

27. Analyzing, interpreting, and presenting the data Qualitative data that is interpreted and used to tell “ the story ” about what was observed. Qualitative data that is categorized using techniques such as content analysis. Quantitative data that is collected from interaction and video logs and presented as values, tables, charts and graphs and is treated statistically.

28. Qualitative analysis to tell a story Review the data Record the themes Record the date and time Check your understanding Iterate this process Report your findings

29. Qualitative analysis for categorization Looking for incidents or patterns Analyzing data into categories Analyzing discourse

30. Looking for incidents or patterns Look for critical incidents, such as times when users were obviously stuck. Theory may also be used to guide the study. A variety of tools are available to record, manipulate and search the data. NIDIST Observer Video-Pro tool

31. Analyzing data into categories Content analysis provides another fine grain way of analyzing video data. It is a systematic, reliable way of coding content into a meaningful set of mutually exclusive categories.

32. Analyzing discourse Another approach to video and audio analysis is to focus on the dialog. Analyzing discourse on the Internet has started to influence designers ’ understanding about users ’ needs in these environments.

33. Quantitative data analysis Typically this data is further analyzed using simple statistics such as means, standard deviations, etc.

34. Feeding the findings back into design Clearly written reports with Overview at the beginning Detailed content list Verbal presentation