IN4MATX 231 Human-Computer Interaction Presenters: Ritesh Subramanian Tanmay Goel Observing Users

Outline Benefits and challenges of different types of observations. Benefits and challenges of different types of observations. How to observe as an onlooker, a participant, and an ethnographer. How to observe as an onlooker, a participant, and an ethnographer. How to collect, analyze and present observational data. How to collect, analyze and present observational data. Indirect observation using diary studies and logging. Indirect observation using diary studies and logging. Qualitative and Quantitative analysis. Qualitative and Quantitative analysis.

What and when to observe Observation is usable at all stages during product development. Observation is usable at all stages during product development. Goals and questions determine the paradigms and techniques used. Goals and questions determine the paradigms and techniques used. Observers can be: Observers can be: –Onlookers –Participants –Ethnographers The degree of immersion that evaluators adopt varies across a broad outsider-insider spectrum. The degree of immersion that evaluators adopt varies across a broad outsider-insider spectrum.

Types of Observation Observation Controlled Environment (e.g. Lab) Field Environment (e.g. Natural) Outsider looking on “Quick and Dirty” In usability testing “Quick and Dirty” In field testing Insider N.A. Participant observation (e.g. in ethnography)

Approaches to Observation “Quick and Dirty” observation “Quick and Dirty” observation –It can occur anywhere, anytime –Least formality involved Observation in usability testing Observation in usability testing –Video and interaction logs capture all the user operations –One-way mirrors or remote TV screen –Data is used to analyze what users are doing and provide insight into users’ affective reactions

Approaches to Observation Observation in field studies Observation in field studies –The observer may be a passive observer, participant observer or an ethnographer. –Goal is to cause as little disruption as possible. –Passive or outside observer – An observer in a class of boys and girls whose primary job to keep track of how much time a particular technology is used by each gender. –Participant observer – Participates in social conventions of a group, combines participation with maintenance of professional distance for unbiased observation. –Ethnography – Takes weeks, months or longer. Inside information of community activity is obtained.

How to observe: In controlled environment Practical issues that need to thought of in advance: Practical issues that need to thought of in advance: –Decide where users are located so that equipment can be setup in advance –Decide about modes of data capture e.g. video, interaction logs –Equipment testing to get expected performance (audio and video) –Get legal user consent –Prepare a script to guide users through the set of questions –It is important to make users feel comfortable –Problem with this approach: Observers do not know what users are thinking.

Controlled Environment – Think-Aloud Technique The user is trying to perform a certain task. Observer wonders, what is going on, what is he thinking, why did he do that, etc. The user is trying to perform a certain task. Observer wonders, what is going on, what is he thinking, why did he do that, etc. Eternalize thought process. Eternalize thought process. A big problem is occurrences of silence during think-aloud process. A big problem is occurrences of silence during think-aloud process. Maybe have two people work together and talk to each other. Maybe have two people work together and talk to each other.

Controlled Environment – Think Aloud Technique

Checklist of things to plan before going into the field State the initial goal and questions clearly State the initial goal and questions clearly Select a framework to guide activities in the field Select a framework to guide activities in the field Decide how to record events Decide how to record events Think about how to gain acceptance and trust of users under observation Think about how to gain acceptance and trust of users under observation Be prepared to go through these notes to weed out ambiguities Be prepared to go through these notes to weed out ambiguities Highlight and separate personal opinion from what happens Highlight and separate personal opinion from what happens Be prepared to refine and refocus the study for same or new user groups Be prepared to refine and refocus the study for same or new user groups Think about how to handle sensitive issues Think about how to handle sensitive issues Consider working as a team; conform notes with another member Consider working as a team; conform notes with another member Consider checking notes with an informant and plan to look at the situation from a different perspective Consider checking notes with an informant and plan to look at the situation from a different perspective

How to observe (contd.): In the field Frameworks to guide observation Frameworks to guide observation –The person. Who? –The place. Where? –The thing. What? The Goetz and LeCompte (1984) framework: The Goetz and LeCompte (1984) framework: –Who is present? –What is happening? –When does the activity occur? –Where is it happening? –Why is it happening? –How is the activity organized?

In the field The Robinson (1993) framework The Robinson (1993) framework –Space. What is the physical space like? –Actors. Who is involved? –Activities. What are they doing? –Objects. What objects are present? –Acts. What are individuals doing? –Events. What kind of event is it? –Goals. What do they to accomplish? –Feelings. What is the mood of the group and of individuals?

How to observe (contd.): Participant observer or ethnography Checklist for doing ethnography Checklist for doing ethnography –Identify the problem or goal and ask good questions to be answered by the study –The most important part of fieldwork is being there to observe, ask questions and record what is seen and heard –Collect variety of data. E.g. Notes, still pictures, audio and video –Be prepared to move backwards and forwards between the broad picture and specific questions –Analyze the data using a holistic approach in which observations are contextualized

Data Collection Techniques Notes plus still camera Notes plus still camera –The least technical way of collecting data –Transcription the first step in data analysis –Photographs, sketches, etc. can be easily collected Audio recording plus still camera Audio recording plus still camera –Less intrusive than video –More flexible and mobile –One drawback is transcribing the data which can be onerous if many hours have to be transcribed

Data Collection Techniques Video Video –Advantage of both audio and video data; but more intrusive –Attention becomes focused on what is seen through the lens –Analysis of video can be long and time consuming –Sound may get muffled when recording in noisy conditions

Comparison Table source: Preece, Rogers, Sharp: Interaction Design: Beyond Human-Computer Interaction. Wiley

Comparison (contd.) Table source: Preece, Rogers, Sharp: Interaction Design: Beyond Human-Computer Interaction. Wiley

Indirect observation: tracking user activities Diaries Diaries –Diaries provide a record of what users did, when they did it and what they thought about their interactions with technology –Useful when users and scattered and unreachable –Inexpensive and require no special equipment or expertise –They rely on participants being reliable and remembering to complete them –Participants usually remember incidents for better or for worse

Indirect observation: tracking user activities Interaction logging Interaction logging –Includes analysing data from key presses, mouse movements etc. –Usually synchronized with video and audio logs –Time stamped to calculate how much time a user spends on a particular task –E.g. Explicit counters that record visits to a website. If you want to find out if adding a bulletin board increases the number of visits, comparison of traffic before and after is useful –Unobtrusive and large volumes of data can be collected

Interaction Logging- Webtrends Log Analyzer

Interaction Logging

Ethical Concerns: They don't know we are watching. Shall we tell them? If we tell users about logging they may react or change their behavior If we tell users about logging they may react or change their behavior It depends on the context, and how much personal information is collected It depends on the context, and how much personal information is collected e.g. Companies tell about data logged for quality assurance purposes e.g. Companies tell about data logged for quality assurance purposes Concerns arise when personal information is used to infer financial or health information Concerns arise when personal information is used to infer financial or health information

Analyzing, interpreting and presenting the data Three main types of data are explored Three main types of data are explored  Qualitative data - interpreted & used to tell the ‘story’ about what was observed.  Qualitative data - categorized using techniques such as content analysis.  Quantitative data - collected from interaction & video logs. Presented as values, tables, charts, graphs and treated statistically.

Qualitative analysis to tell a story Review the data after each observation to identify key themes Review the data after each observation to identify key themes Record themes in a coherent yet flexible form Record themes in a coherent yet flexible form –E.g. Laptops, audio recordings Record the date and time of each data session Record the date and time of each data session As themes emerge, check your understanding with informants or people you observe As themes emerge, check your understanding with informants or people you observe Iterate until the story reflects observation Iterate until the story reflects observation Report findings to a group in the form of a presentation or written report Report findings to a group in the form of a presentation or written report

Analyzing and reporting ethnographic data Look for key events within a group that drive a group’s activity Look for key events within a group that drive a group’s activity Look for patterns or behavior among different players Look for patterns or behavior among different players Compare sources of data against each other Compare sources of data against each other Report findings in a convincing and honest way Report findings in a convincing and honest way e.g. Software tools such as Ethnograph allow ethnographers to code their notes so that they can be searched, sorted and retrieved. The information can be printed out as a tree showing relationships of occurrences. e.g. Software tools such as Ethnograph allow ethnographers to code their notes so that they can be searched, sorted and retrieved. The information can be printed out as a tree showing relationships of occurrences.

Ethnograph

Ethnograph

Qualitative analysis for categorization Looking for incidents or patterns Looking for incidents or patterns –Look for critical incidents, such as times when the user was obviously stuck. (marked by silence, looks of confusion)‏ –E.g. Jurgen Koenemann-Belliveau et al(1994) used this approach to compare the efficacy of two versions of a Smalltalk programming manual for novice programmers. –They were able to identify specific problems by tracing through sequence of incidents and thereby achieving a holistic understanding.

Qualitative analysis for categorization Analyzing data into categories Analyzing data into categories –Content analysis provides a reliable and systematic way of coding content into meaningful sets of mutually exclusive categories –Categories should be orthogonal i.e. no overlap –Appropriate granularity must be selected –e.g. Training two people to use categories and letting them analyze the same data sample. If huge discrepancy in the analyses is observed, then either the training was insufficient or categories need to be redefined –Inter-research reliability rating- Percentage of agreement between two researchers, defined as ratio of number of items categorized in the same way to the total number of items observed

Qualitative analysis for categorization Analyzing Discourse Analyzing Discourse –Focus on the dialog, i.e. meaning of what was said rather than content –Discourse analysis is strongly interpretive –Language is viewed as a constructive tool and discourse analysis provides a way of focusing upon people’s use of language to construct versions of their worlds –E.g. Analyzing discourse on the Internet (e.g. in chat rooms, bulletin boards) has started to influence designers’ understanding about user needs in these environments

Quantitative data analysis and feedback Quantitative data analysis Quantitative data analysis –Video data collected is usually annotated to calculate performance times –The data stream from interaction logs is used to calculate performance times –Data is analyzed using means, standard deviations, t-tests etc. Feeding the findings back into the design Feeding the findings back into the design –Written reports make for easy reading and a good reference document –Include anecdotes, pictures, quotations and video clips to bring study to life and stimulate interest –Verbal presentations can be a powerful mode of feedback

Summary Observation in usability study tends to be objective Observation in usability study tends to be objective In contrast, in participant observation the evaluator works with the user to understand their activities and beliefs In contrast, in participant observation the evaluator works with the user to understand their activities and beliefs Ethnography uses a technique wherein the ethnographer immerse themselves in the culture they study Ethnography uses a technique wherein the ethnographer immerse themselves in the culture they study Observational data collection depends on the paradigm used Observational data collection depends on the paradigm used Combination of video, audio, diaries and logs can be used to collect observational data Combination of video, audio, diaries and logs can be used to collect observational data Analyzing video and data logs can be tedious due to the sheer volume of data Analyzing video and data logs can be tedious due to the sheer volume of data Evaluators flag events in real time and return to examine them later. Evaluators flag events in real time and return to examine them later. Identifying key events is an effective approach Identifying key events is an effective approach