1. Chapter 12 Observing Users
Youlan Hu
Xiaoqian Hu
Nov 15, 2002

2. Outline What and when to observe Approaches to observation
How to observe(in lab and in field)
Data collection techniques
Indirect observation
Analyzing, interpreting and presenting data

3. What and When to Observe
Observation is valuable any time during design stages
Help understand users’ need in early design
Find out whether it meets users’ needs during later refinement of a product
Goals and questions determine the paradigm and techniques used
Provide a focus for observation
Should guide all evaluation studies

4. Approaches to Observation
“Quick and dirty” observation
Any where, any time
Find out what’s happening quickly and with little formality
Observation in usability test
Prototype/products
In controlled setting
Observation in field studies
Early on
Real-world test of prototype/products

5. Type of Observation Observation Controlled environment(i.e.lab-like)
Field environment(i.e.natural)
Outsider looking on
“Quick and dirty” in usability testing
“Quick and dirty” in field studies
Insider
(Not applicable)
Participant observation(e.g.,in ethnography )

6. Observers Outsiders Participant observation In lab or field studies
Recording is continuous
Data analysis and observation almost simultaneous
More objective than participant observation
Participant observation

7. Participant Observation
Field studies
Must get co-operation of people observed
Informants are useful
Recording might be interrupted
Data analysis is continuous
Interpretive technique
Questions get refined as understanding grows
Reports usually contain examples

8. In Lab Versus In Field? In field In lab
– Focus on context in which individuals perform their work
– More realistic
– Harder to control other factors
In lab
– Focus on the details of what individuals do
– Easier to control
– Easier to replicate
– more objective

9. How to Observe Direct observation Indirect observation In same room
Can be intrusive
Only see it one time
Notes usually incomplete
May use 1-way mirror
Help get informal and direct information
Indirect observation
More distance
Reduces intrusiveness, but doesn’t eliminate it
Gives archival record
Data has to be synchronized and managed
Need time to analyze data

10. Direct Observation – in controlled environments
Issues before observation
Decide what laboratory to do the tests
Set up appropriate equipment
Test equipment to make sure it works
Provide an informed consent form for users to read and sign at the beginning of the study

11. Simple Observation Method
User is given the task, and the evaluator just watches the user
Problem:does not give insight into the user’s decision and attitude

12. Think-aloud Method
Subjects are asked to say what they are thinking/doing
Gives insight into what the user is thinking
Very widely used in industry
Potential problems:
Can be awkward
for subject
Think aloud
may modify the
way users perform
their task.

13. Constructive Interaction Method
Join pairs of participants to work together
Perhaps have one person be semi-expert and one be novice
Remove awkwardness of individual think-aloud
Provides insight
into thinking
process of both
beginner and
intermediate.

14. How to observe – in the field
Need to consider in advance:
State initial study goal and question clearly
Select a framework to guide activity in the field
Decide how to record events
Be prepared to go through notes and other records as soon as possible after each evaluation session
Highlight and separate personal opinion from what really happens

15. How to observe – in the field(cont’d)
Need to consider in advance:
Refine goals and questions while observing
How to gain acceptance and trust of those subjects
How to handle sensitive issues
Consider working as a team
Whether and how to involve informants
How to analyze the data
Plan to look at situation from different perspectives

16. Frameworks to Guide Field Observation
Why use frameworks
Organize/structure and focus observation
Simple framework to capture context
The person. Who?
The place. Where?
The thing. What?
The Goetz and Lecompte(1984) framework
Who is present?
What is happening?
When does the activity occur?
Where is it happening?
Why is it happening?
How is the activity organized?

17. The Robinson Framework
Space. What is the physical space like?
Actors. Who is involved?
Activities . What are they doing and why?
Objects. What objects are present?
Acts. What are specific individuals doing?
Events . What kind of event is it?
Goals. What are they trying to accomplish?
Feelings. What is the mood of the group and individuals?

18. Dilemmas When should I stop observing
When see similar patterns of behavior being repeated
When finish listening to all the main stakeholder groups and understand their perspectives
How can I adapt ethnography to fit the development process
Preparation
Field study
Analysis
Reporting

19. Indirect Observation When direct observation is not possible
Tracking Users’ Activities without intruding
Diaries
Interaction Logging

20. Diaries
Track what users did, when they did it , and what they thought about their interactions with the technology
Useful when users are scattered and unreachable
internet and web evaluations
Templates can also be created online to standardize entry format

21. Diaries Advantages Inexpensive
No special equipment or expertise needed
Suitable for long-term studies
Disadvantages
Rely on participants being reliable and remembering to complete the diaries
Often incentives are needed and the process has to be straightforward and quick

22. Interaction Logging
For example, you want to track how long people stayed at a site, which areas they visited, where they came from, and where they went next
Web page (visitors) counter
Server logs
can also raises ethical concerns that need careful consideration

23. Interaction Logging Advantages Unobtrusive
Large volumes of data can be logged automatically
Disadvantages
Ethical concerns
Powerful tools are needed to analyze the logs (WebLog)

24. Data Collection
Notes plus still camera
Audio plus still camera
Video

25. Notes + Camera Advantages Unobtrusive Less technical and flexible
inexpensive
Easy to transcribe
Disadvantages
Writing speed limited
Difficult to write and observe at the same time
Rely on the note-takers

26. Audio + Camera Advantages less obtrusive than video Less expensive
Flexible
Good for recording “thinking aloud” activities
Disadvantages
Lack of visual record (user actions missing)
Difficult to transcribe the data
Have to change tapes and position microphone

27. Video Advantages Capture both visual and audio High reliability
Permanent original record can be revisited
Disadvantages
More expensive
Can be intrusive
Easy to miss other things going on outside of the camera view
need more than one camera (subject + screen)
Need positioning and focusing camera lens
Analysis of video data can be time-consuming

28. Dilemma – Observing Without Being Seen
Shall we tell users they are being observed?
Tell: users may react or change their behaviors
Not tell: raising ethical issues
Depends on the context
How much personal information to be collected
How the information will be used

29. Analyzing, Interpreting, and Presenting the Data
Qualitative analysis to tell “the story”
interpret what was observed
Qualitative analysis for categorization
content analysis
Quantitative data analysis

30. Qualitative Analysis to Tell a Story
Review the data – identify key themes, make collections
Record the themes – with examples
Record the date and time – for each session
Check your understanding – with people you observed
Iterate this process
Report your findings – to development team

31. Qualitative Analysis for Categorization
Looking for incidents or patterns
Analyzing data into categories
Analyzing discourse

32. Looking For Incidents Or Patterns
Look for critical incidents, such as times when users were obviously stuck
comments, silence, or looks of puzzlement
Theory may also be used to guide the study
help to focus on relevant incidents
Use Tools to record, manipulate and search the data
NIDIST
Observer video-pro tool

33. Analyzing Data Into Categories
Content analysis – fine grain way of analyzing video data
determine a meaningful set of mutually exclusive categories
decide on the appropriate granularity
train a second person, both will analyze
labor-intensive and time consuming

34. Analyzing Discourse
Focus on the dialog – meaning of what is said, rather than the content
Strongly interpretive, pay great attention to context
E.g. Analyzing discourse on the internet (Chat rooms, bulletin boards) has started to influence designers’ understanding about users’ need in these environments
how to start a conversation
how turn-taking is structured

35. Quantitative Data Analysis
Data from video and interaction logs are annotated
Typically this data is analyzed and treated statistically
Usually presented as values, tables, charts and graphs

36. Summary of data analysis
Flag events in real time and examine them in more detail later
Fine-grained analyses can be very time-consuming
Identifying key events is an effective approach

37. Feeding the Findings Back Into design
Written reports (overview + details)
Verbal presentation
Companied by anecdotes, quotations, pictures and video clips
Both qualitative and quantitative analysis useful
quantitative analysis is used less often and depends on goal