Ch. 12 Observing Users Reece, Rogers, Sharp. Beyond human computer interaction. Team 1:Andy, Nikhil, Vladimir, Sajay

12.1 Introduction Learning Goals  Benefits and challenges of different methods  How to observe  How to deal with data  Key issues in think-aloud, diary studies, and interaction logging  Selecting and doing observations Additional Content  Critical Incident Reporting: A type of diary observation  CIF: common industry format Summative vs Formative  Interaction Logging for the Web  Uzilla: A tool for web usability testing by Andy

12.2 Observing in a controlled environment i.e., usability testing Plan user and equipment location. Test the setup. Provide participants with a written consent form and script of the study. Use think-aloud technique (p. 365) to find out what participants are thinking.

Observing as an outsider More objective than participant observation Lab equipment is in place Recording is continuous Analysis & observation almost simultaneous Care needed to avoid drowning in data Analysis can be coarse or fine grained Video clips can be powerful for telling story

Frameworks to guide observation The Goetz and LeCompte (1984) framework: - Who is present? - What is their role? - What is happening? - When does the activity occur? - Where is it happening? - Why is it happening? - How is the activity organized?

The Robinson (1993) framework  Space. What is the physical space like?  Actors. Who is involved?  Activities. What are they doing?  Objects. What objects are present?  Acts. What are individuals doing?  Events. What kind of event is it?  Goals. What do they to accomplish?  Feelings. What is the mood of the group and of individuals? Frameworks to guide observation

Participant observation & ethnography Must get co-operation of people observed Informants are useful Data analysis is continuous Move backwards and forwards from broad to specific questions Collect variety of data Holistic approach – broad analysis

You need to consider Goals & questions Which framework & techniques How to collect data Which equipment to use How to gain acceptance How to handle sensitive issues Whether and how to involve informants How to analyze the data Whether to triangulate

12.4 Data collection techniques Notes & still camera Audio & still camera Video Tracking users: - diaries - interaction logging

Notes plus still camera Taking notes  Least technical way  Most feasible way  Could prove a challenge as one has to observe and listen while taking notes.  Writing skills such as writing speed, use of short hand could prove useful  One needs to translate and transcribe the written content into useful information Other alternative are include another person in the data collection process; use a laptop Use a camera to record images

Audio recording plus still camera Audio recording  Light and easy to carry  Less intrusive than a video camera  Transcribing long interview hours could be a challenge  Lack of a visual record  Cannot be used in a noisy environment

Video Captures both visual and audio data Extremely intrusive Instruments could be expensive but digicams/handheld camera are a cheaper option One may tend to focus only on what is seen through the lens and ignore what has been missed out. Sound may get muffled when recording under noisy conditions Analysis of video data could be very time consuming. Lets compare the Data-collection techniques – Table 12.2 page 376

12.5 Indirect Observation Done when direct observation is not possible (obtrusive or evaluators cannot be present) Tracking users  Diaries  Critical Incidents  Interaction logging

Diaries Classical paper based or using online forms linked to database etc. Participant records what she did and experience she had while doing the task Diaries collected from all the participants are then analyzed Barry Brown’s research using diarys’ for HP design framework for devices used to capture or record information. chi%20paper.pdf

Advantages Inexpensive Subjects could be tested location independent Experimenter not required to be present Large amount of data could be gathered

Disadvantages Relies on the reliability of the participant Problem of participant not remembering actual experience of event (better or worse) Subject is the experimenter and hence variability in data – not much control other than structures templates to collect information.

Interaction logging Record all the interactions of the subject with the system (key presses, mouse movements, eye tracking, etc.) combined with audio and video logs. Log is time stamped. Data can be analyzed to understand cognitive aspects, user behaviors and reactions to variations in task etc.

Interaction logging - Uses Useful for web studies. Testing Web layouts, displays, GUI interfaces etc. Understanding the users’ reactions to variations in tasks and layout combinations. Testing the efficiency of the interface and its ease of use.

Interaction Logging on the Web Server log file miss cached pages To circumvent  Web bugs – useful for visitors at large  Proxies – ala WebQuilt /projects/webquilt/ /projects/webquilt/  Instrumented Browsers Andy’s Uzilla.net PARC’s WebLogger Challenges  Inferring user goals  Analyzing paths Triangulation  Establishing the magnitude of problems observed in testing from log files

12.6 Analyzing, Interpreting, and Presenting Data Qualitative analysis to tell a story  Hilight tapes, effective communication Qualitative analysis for categorization  Advanced linguistic analysis tools Quantitative Data Analysis  Measure learning, efficiency Feeding the findings back into design  Picture in a picture highlight tapes

Common Industry Format (CIF) For summative testing  (later stages in software development) Subjective and Quantitative Measures  On key tasks Intended for vendors to communicate usability  Boeing has adopted for all products and suppliers

Uzilla: A product suite for web usability testing Commercial venture by Andy Edmonds  Overview talk SCIP 2002 Overview talk  Edmonds, K. A. (in press). Uzilla: A new tool for web usability testing. Behavior Research Methods, Instrumentation and Computers. May, Special Issue of Soc. For Computers in Psychology. Automates test design Instruments data collection Aggregates results Free for student projects

Sarah Bly Interview Unfinished book on non-speech audio in UI design Unfinished book on non-speech audio in UI design At Chi ’03Chi ’03  Visual Support for Conversations: People's Choice Tutorial, Steve Harrison (Dyxsis D/R Consulting), Sara Bly (Sara Bly Consulting) To familiarize the participant with the underlying ideas and subtle design considerations that make shared drawing systems essential in many conversational settings. Visual Support for Conversations