Chapter 12 Observing Users

Slides:



Advertisements
Similar presentations
Chapter 14: Usability testing and field studies
Advertisements

REVIEW OF QUALITATIVE RESEARCH AND PRINCIPLES OF QUALITATIVE ANALYSIS SCWK 242 – SESSION 2 SLIDES.
Chapter 7 Data Gathering 1.
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
Observing users Chapter 12. Observation ● Why? Get information on.. – Context, technology, interaction ● Where? – Controlled environments – In the field.
1 User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept. Cal Poly San Luis Obispo FJK 2005.
CHAPTER 10, qualitative field research
Observing Users Paul Bogen, LaShon Johnson, Jehoon Park.
Data gathering.
Observation Watch, listen, and learn…. Agenda  Observation exercise Come back at 3:40.  Questions?  Observation.
Observing users.
Observing users. The aims Discuss the benefits & challenges of different types of observation. Describe how to observe as an on-looker, a participant,
An evaluation framework
An evaluation framework
Evaluation How do we test the interaction design? Several Dimensions
From Controlled to Natural Settings
ICS 463, Intro to Human Computer Interaction Design: 8. Evaluation and Data Dan Suthers.
Damian Gordon.  Summary and Relevance of topic paper  Definition of Usability Testing ◦ Formal vs. Informal methods of testing  Testing Basics ◦ Five.
IN4MATX 231 Human-Computer Interaction Presenters: Ritesh Subramanian Tanmay Goel Observing Users.
CSCI 4163 / CSCI 6904 – Winter Housekeeping  Write a question/comment about today’s reading on the whiteboard (chocolate!)  Make sure to sign.
Evaluation Framework Prevention vs. Intervention CHONG POH WAN 21 JUNE 2011.
CSCI 4163/6904, summer Quiz  Multiple choice  Answer individually - pass in  Then class discussion.
Online, Remote Usability Testing  Use web to carry out usability evaluations  Two main approaches agent-based evaluation (e.g., WebCritera)  model automatically.
Ch 14. Testing & modeling users
Data Collection Methods
Chapter 12 Observing Users Li, Jia Li, Wei. Outline What and when to observe Approaches to observation How to observe How to collect data Indirect observation.
Ch. 12 Observing Users Reece, Rogers, Sharp. Beyond human computer interaction. Team 1:Andy, Nikhil, Vladimir, Sajay.
Human Computer Interaction
Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal.
©2010 John Wiley and Sons Chapter 6 Research Methods in Human-Computer Interaction Chapter 6- Diaries.
CS2003 Usability Engineering Usability Evaluation Dr Steve Love.
Chapter 15 Qualitative Data Collection Gay, Mills, and Airasian
Virtual University - Human Computer Interaction 1 © Imran Hussain | UMT Imran Hussain University of Management and Technology (UMT) Lecture 40 Observing.
User Interface Design & Usability for the Web Card Sorting You should now have a basic idea as to content requirements, functional requirements and user.
CSCI 4163 / CSCI 6904 – Winter Housekeeping  Clarification about due date for reading comments/questions  Skills sheet  Active listening handout.
Observing users. The aims Discuss the benefits & challenges of different types of observation. Describe how to observe as an on-looker, a participant,
EVALUATION PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS ( TEMPUS FR-TEMPUS-JPCR)
Key principles Everything is strange –Question why? –Stop and reflect Members’ point of view –Developer themselves – why do that? No a priori expectations.
Qualitative Research Chapter Four. Chapter Four Objectives Define qualitative research Explore the popularity of qualitative research Understand the limitations.
Observing users. What and when to observe Goals & questions determine the paradigms and techniques used. Observation is valuable any time during design.
Data gathering (Chapter 7 Interaction Design Text)
LIS 570 Qualitative Research. Definition A process of enquiry that draws from the context in which events occur, in an attempt to describe these occurrences,
Observation Direct observation in the field –Structuring frameworks –Degree of participation (insider or outsider) –Ethnography Direct observation in controlled.
1 ITM 734 Introduction to Human Factors in Information Systems Cindy Corritore Testing the UI – part 2.
© 2009 Pearson Prentice Hall, Salkind. Chapter 10 Nonexperimental Research: Qualitative Methods.
Lecture 4 Supplement – Data Gathering Sampath Jayarathna Cal Poly Pomona Based on slides created by Ian Sommerville & Gary Kimura 1.
CHAPTER 10, qualitative field research
Data Gathering and Analysis
CHAPTER OVERVIEW The Case Study Ethnographic Research
Chapter 7 GATHERING DATA.
Presented by your 6th grade Language Arts Teachers 
Lecture3 Data Gathering 1.
Qualitative research: an overview
Chapter 7 Data Gathering 1.
Observations contextual inquiry
Chapter 18 Qualitative Research: Specific Methods
GATHERING DATA.
Data Collection Strategies
From Controlled to Natural Settings
CHAPTER 10, qualitative field research
Chapter 7 GATHERING DATA.
Observing users.
User interface design.
Data gathering.
Evaluation.
Dr Amina Rashad and Dr Nahed Kandeel
Empirical Evaluation Data Collection: Techniques, methods, tricks Objective data IRB Clarification All research done outside the class (i.e., with non-class.
COMP444 Human Computer Interaction Evaluation
CHAPTER OVERVIEW The Case Study Ethnographic Research
Presentation transcript:

Chapter 12 Observing Users Youlan Hu Xiaoqian Hu Nov 15, 2002

Outline What and when to observe Approaches to observation How to observe(in lab and in field) Data collection techniques Indirect observation Analyzing, interpreting and presenting data

What and When to Observe Observation is valuable any time during design stages Help understand users’ need in early design Find out whether it meets users’ needs during later refinement of a product Goals and questions determine the paradigm and techniques used Provide a focus for observation Should guide all evaluation studies

Approaches to Observation “Quick and dirty” observation Any where, any time Find out what’s happening quickly and with little formality Observation in usability test Prototype/products In controlled setting Observation in field studies Early on Real-world test of prototype/products

Type of Observation Observation Controlled environment(i.e.lab-like) Field environment(i.e.natural) Outsider looking on “Quick and dirty” in usability testing “Quick and dirty” in field studies Insider (Not applicable) Participant observation(e.g.,in ethnography )

Observers Outsiders Participant observation In lab or field studies Recording is continuous Data analysis and observation almost simultaneous More objective than participant observation Participant observation

Participant Observation Field studies Must get co-operation of people observed Informants are useful Recording might be interrupted Data analysis is continuous Interpretive technique Questions get refined as understanding grows Reports usually contain examples

In Lab Versus In Field? In field In lab – Focus on context in which individuals perform their work – More realistic – Harder to control other factors In lab – Focus on the details of what individuals do – Easier to control – Easier to replicate – more objective

How to Observe Direct observation Indirect observation In same room Can be intrusive Only see it one time Notes usually incomplete May use 1-way mirror Help get informal and direct information Indirect observation More distance Reduces intrusiveness, but doesn’t eliminate it Gives archival record Data has to be synchronized and managed Need time to analyze data

Direct Observation – in controlled environments Issues before observation Decide what laboratory to do the tests Set up appropriate equipment Test equipment to make sure it works Provide an informed consent form for users to read and sign at the beginning of the study

Simple Observation Method User is given the task, and the evaluator just watches the user Problem:does not give insight into the user’s decision and attitude

Think-aloud Method Subjects are asked to say what they are thinking/doing Gives insight into what the user is thinking Very widely used in industry Potential problems: Can be awkward for subject Think aloud may modify the way users perform their task.

Constructive Interaction Method Join pairs of participants to work together Perhaps have one person be semi-expert and one be novice Remove awkwardness of individual think-aloud Provides insight into thinking process of both beginner and intermediate.

How to observe – in the field Need to consider in advance: State initial study goal and question clearly Select a framework to guide activity in the field Decide how to record events Be prepared to go through notes and other records as soon as possible after each evaluation session Highlight and separate personal opinion from what really happens

How to observe – in the field(cont’d) Need to consider in advance: Refine goals and questions while observing How to gain acceptance and trust of those subjects How to handle sensitive issues Consider working as a team Whether and how to involve informants How to analyze the data Plan to look at situation from different perspectives

Frameworks to Guide Field Observation Why use frameworks Organize/structure and focus observation Simple framework to capture context The person. Who? The place. Where? The thing. What? The Goetz and Lecompte(1984) framework Who is present? What is happening? When does the activity occur? Where is it happening? Why is it happening? How is the activity organized?

The Robinson Framework Space. What is the physical space like? Actors. Who is involved? Activities . What are they doing and why? Objects. What objects are present? Acts. What are specific individuals doing? Events . What kind of event is it? Goals. What are they trying to accomplish? Feelings. What is the mood of the group and individuals?

Dilemmas When should I stop observing When see similar patterns of behavior being repeated When finish listening to all the main stakeholder groups and understand their perspectives How can I adapt ethnography to fit the development process Preparation Field study Analysis Reporting

Indirect Observation When direct observation is not possible Tracking Users’ Activities without intruding Diaries Interaction Logging

Diaries Track what users did, when they did it , and what they thought about their interactions with the technology Useful when users are scattered and unreachable internet and web evaluations Templates can also be created online to standardize entry format

Diaries Advantages Inexpensive No special equipment or expertise needed Suitable for long-term studies Disadvantages Rely on participants being reliable and remembering to complete the diaries Often incentives are needed and the process has to be straightforward and quick

Interaction Logging For example, you want to track how long people stayed at a site, which areas they visited, where they came from, and where they went next Web page (visitors) counter Server logs can also raises ethical concerns that need careful consideration

Interaction Logging Advantages Unobtrusive Large volumes of data can be logged automatically Disadvantages Ethical concerns Powerful tools are needed to analyze the logs (WebLog)

Data Collection Notes plus still camera Audio plus still camera Video

Notes + Camera Advantages Unobtrusive Less technical and flexible inexpensive Easy to transcribe Disadvantages Writing speed limited Difficult to write and observe at the same time Rely on the note-takers

Audio + Camera Advantages less obtrusive than video Less expensive Flexible Good for recording “thinking aloud” activities Disadvantages Lack of visual record (user actions missing) Difficult to transcribe the data Have to change tapes and position microphone

Video Advantages Capture both visual and audio High reliability Permanent original record can be revisited Disadvantages More expensive Can be intrusive Easy to miss other things going on outside of the camera view need more than one camera (subject + screen) Need positioning and focusing camera lens Analysis of video data can be time-consuming

Dilemma – Observing Without Being Seen Shall we tell users they are being observed? Tell: users may react or change their behaviors Not tell: raising ethical issues Depends on the context How much personal information to be collected How the information will be used

Analyzing, Interpreting, and Presenting the Data Qualitative analysis to tell “the story” interpret what was observed Qualitative analysis for categorization content analysis Quantitative data analysis

Qualitative Analysis to Tell a Story Review the data – identify key themes, make collections Record the themes – with examples Record the date and time – for each session Check your understanding – with people you observed Iterate this process Report your findings – to development team

Qualitative Analysis for Categorization Looking for incidents or patterns Analyzing data into categories Analyzing discourse

Looking For Incidents Or Patterns Look for critical incidents, such as times when users were obviously stuck comments, silence, or looks of puzzlement Theory may also be used to guide the study help to focus on relevant incidents Use Tools to record, manipulate and search the data NIDIST Observer video-pro tool

Analyzing Data Into Categories Content analysis – fine grain way of analyzing video data determine a meaningful set of mutually exclusive categories decide on the appropriate granularity train a second person, both will analyze labor-intensive and time consuming

Analyzing Discourse Focus on the dialog – meaning of what is said, rather than the content Strongly interpretive, pay great attention to context E.g. Analyzing discourse on the internet (Chat rooms, bulletin boards) has started to influence designers’ understanding about users’ need in these environments how to start a conversation how turn-taking is structured

Quantitative Data Analysis Data from video and interaction logs are annotated Typically this data is analyzed and treated statistically Usually presented as values, tables, charts and graphs

Summary of data analysis Flag events in real time and examine them in more detail later Fine-grained analyses can be very time-consuming Identifying key events is an effective approach

Feeding the Findings Back Into design Written reports (overview + details) Verbal presentation Companied by anecdotes, quotations, pictures and video clips Both qualitative and quantitative analysis useful quantitative analysis is used less often and depends on goal