Gathering Usability Data

Slides:



Advertisements
Similar presentations
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
Advertisements

Interaksi Manusia Komputer Evaluasi Empiris1/68 EVALUASI EMPIRIS Pengenalan Evaluasi Empiris Perancangan Eksperimen Partisipasi, IRB dan Etika Pengumpulan.
6.811 / PPAT: Principles and Practice of Assistive Technology Wednesday, 16 October 2013 Prof. Rob Miller Today: User Testing.
CS305: HCI in SW Development Evaluation (Return to…)
Data analysis and interpretation. Agenda Part 2 comments – Average score: 87 Part 3: due in 2 weeks Data analysis.
ACTIVELY ENGAGING THE STAKEHOLDER IN DEFINING REQUIREMENTS FOR THE BUSINESS, THE STAKEHOLDER, SOLUTION OR TRANSITION Requirements Elicitation.
Evaluation 3: Approaches and Data Collection Slides adapted from Saul Greenberg’s “Evaluating Interfaces with Users”.
IAT 334 Experimental Evaluation ______________________________________________________________________________________ SCHOOL OF INTERACTIVE ARTS + TECHNOLOGY.
Data gathering.
©N. Hari Narayanan Computer Science & Software Engineering Auburn University 1 COMP 7620 Evaluation Chapter 9.
Observation Watch, listen, and learn…. Agenda  Observation exercise Come back at 3:40.  Questions?  Observation.
Focus Groups. Contents What is a focus group and why use it Methods When to use Focus Groups Advantages and Disadvantages Example.
Evaluation. formative 4 There are many times throughout the lifecycle of a software development that a designer needs answers to questions that check.
Think-aloud usability experiments or concurrent verbal accounts Judy Kay CHAI: Computer human adapted interaction research group School of Information.
User Interface Testing. Hall of Fame or Hall of Shame?  java.sun.com.
Evaluation Methodologies
An evaluation framework
Evaluation Methods April 20, 2005 Tara Matthews CS 160.
Data collection methods Questionnaires Interviews Focus groups Observation –Incl. automatic data collection User journals –Arbitron –Random alarm mechanisms.
Human Computer Interaction Design Evaluation and Content Presentation
ICS 463, Intro to Human Computer Interaction Design: 8. Evaluation and Data Dan Suthers.
Intro to Evaluation See how (un)usable your software really is…
HCI revision lecture. Main points Understanding Applying knowledge Knowing key points Knowing relationship between things If you’ve done the group project.
Quantitative Research
‘Hints for Designing Effective Questionnaires ’
Interviews and Questionnaires a.k.a. How to talk to your users.
Chapter 14: Usability testing and field studies
User Interface Evaluation Usability Inquiry Methods
Evaluation Framework Prevention vs. Intervention CHONG POH WAN 21 JUNE 2011.
1 Testing the UI This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley,
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
Gathering User Data IS 588 Dr. Dania Bilal Spring 2008.
Evaluating a Research Report
Data Collection Methods
Chapter 12 Observing Users Li, Jia Li, Wei. Outline What and when to observe Approaches to observation How to observe How to collect data Indirect observation.
Fall 2002CS/PSY Empirical Evaluation Analyzing data, Informing design, Usability Specifications Inspecting your data Analyzing & interpreting results.
Human Computer Interaction
©2010 John Wiley and Sons Chapter 6 Research Methods in Human-Computer Interaction Chapter 6- Diaries.
CS2003 Usability Engineering Usability Evaluation Dr Steve Love.
COMP5047 Pervasive Computing: 2012 Think-aloud usability experiments or concurrent verbal accounts Judy Kay CHAI: Computer human adapted interaction research.
Observation, Interviews, and Questionnaires a.k.a. How to watch and talk to your users.
Data analysis and interpretation. Project part 3 Watch for comments on your evaluation plans Finish your plan – Finalize questions, tasks – Prepare scripts.
Chapter 15 Qualitative Data Collection Gay, Mills, and Airasian
Ways of Collecting Information Interviews Questionnaires Ethnography Books and leaflets in the organization Joint Application Design Prototyping.
Chapter 8 Usability Specification Techniques Hix & Hartson.
CS5714 Usability Engineering Formative Evaluation of User Interaction: During Evaluation Session Copyright © 2003 H. Rex Hartson and Deborah Hix.
The product of evaluation is knowledge. This could be knowledge about a design, knowledge about the user or knowledge about the task.
AMSc Research Methods Research approach IV: Experimental [1] Jane Reid
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Requirements Gathering CS 561. Where do Requirements Come From? Handed to you (?) Dialogue with – Customer – User Are these always the same? Are these.
Observation & Experiments Watch, listen, and learn…
©2011 1www.id-book.com Data Gathering Chapter 7. ©2011 Data Gathering What is data gathering? –The act of gathering data through a study The data can.
Fall 2002CS/PSY Empirical Evaluation Data collection: Subjective data Questionnaires, interviews Gathering data, cont’d Subjective Data Quantitative.
EVALUATION PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS ( TEMPUS FR-TEMPUS-JPCR)
Research Methods Observations Interviews Case Studies Surveys Quasi Experiments.
Data gathering (Chapter 7 Interaction Design Text)
Observation & Experiments Watch, listen, and learn…
Usability Engineering Dr. Dania Bilal IS 582 Spring 2007.
Evaluation / Usability. ImplementDesignAnalysisEvaluateDevelop ADDIE.
Fact Finding (Capturing Requirements) Systems Development.
1 ITM 734 Introduction to Human Factors in Information Systems Cindy Corritore Testing the UI – part 2.
Day 8 Usability testing.
Evaluation through user participation
Interviews and Questionnaires
Data analysis and interpretation
Chapter 23 Deciding how to collect data
Observation & Experiments
Gathering data, cont’d Subjective Data Quantitative
Experimental Evaluation
Empirical Evaluation Data Collection: Techniques, methods, tricks Objective data IRB Clarification All research done outside the class (i.e., with non-class.
Presentation transcript:

Gathering Usability Data Observing users & subjective data

Directing Sessions Issues: Are you in same room or not? Single person session or pairs of people Objective data -- stay detached Fall 2006 PSYCH / CS 6750

Collecting Data Data gathering Note-taking Audio and video tape Instrumented user interface Post-experiment questions and interviews Fall 2006 PSYCH / CS 6750

Collecting Data Identifying errors can be difficult Qualitative techniques Think-aloud - can be very helpful Post-hoc verbal protocol - review video Critical incident logging - positive & negative Structured interviews - good questions “What did you like best/least?” “How would you change..?” Fall 2006 PSYCH / CS 6750

Observing Users Not as easy as you think One of the best ways to gather feedback about your interface Watch, listen and learn as a person interacts with your system Fall 2006 PSYCH / CS 6750

Observation Direct Indirect In same room Can be intrusive Users aware of your presence Only see it one time May use 1-way mirror to reduce intrusiveness Indirect Video recording Reduces intrusiveness, but doesn’t eliminate it Cameras focused on screen, face & keyboard Gives archival record, but can spend a lot of time reviewing it Fall 2006 PSYCH / CS 6750

Location Observations may be In lab - Maybe a specially built usability lab Easier to control Can have user complete set of tasks In field Watch their everyday actions More realistic Harder to control other factors Fall 2006 PSYCH / CS 6750

Challenge In simple observation, you observe actions but don’t know what’s going on in their head Often utilize some form of verbal protocol where users describe their thoughts Fall 2006 PSYCH / CS 6750

Verbal Protocol One technique: Think-aloud User describes verbally what s/he is thinking and doing What they believe is happening Why they take an action What they are trying to do Fall 2006 PSYCH / CS 6750

Think Aloud Very widely used, useful technique Allows you to understand user’s thought processes better Potential problems: Can be awkward for participant Thinking aloud can modify way user performs task Fall 2006 PSYCH / CS 6750

Teams Another technique: Co-discovery learning (Constructive interation) Join pairs of participants to work together Use think aloud Perhaps have one person be semi-expert (coach) and one be novice More natural (like conversation) so removes some awkwardness of individual think aloud Fall 2006 PSYCH / CS 6750

Alternative What if thinking aloud during session will be too disruptive? Can use post-event protocol User performs session, then watches video afterwards and describes what s/he was thinking Sometimes difficult to recall Opens up door of interpretation Fall 2006 PSYCH / CS 6750

Historical Record In observing users, how do you capture events in the session for later analysis? ? Fall 2006 PSYCH / CS 6750

Capturing a Session 1. Paper & pencil Can be slow May miss things Is definitely cheap and easy Task 1 Task 2 Task 3 … Time 10:00 10:03 10:08 10:22 S e S e Fall 2006 PSYCH / CS 6750

Capturing a Session 2. Recording (audio and/or video) Good for talk-aloud Hard to tie to interface Multiple cameras probably needed Good, rich record of session Can be intrusive Can be painful to transcribe and analyze Fall 2006 PSYCH / CS 6750

Capturing a Session 3. Software logging Modify software to log user actions Can give time-stamped key press or mouse event Two problems: Too low-level, want higher level events Massive amount of data, need analysis tools Fall 2006 PSYCH / CS 6750

Issues What if user gets stuck on a task? You can ask “What are you trying to do..?” “What made you think..?” “How would you like to perform..?” “What would make this easier to accomplish..?” Maybe offer hints Can provide design ideas Fall 2006 PSYCH / CS 6750

Subjective Data Satisfaction is an important factor in performance over time Learning what people prefer is valuable data to gather Fall 2006 PSYCH / CS 6750

Methods Ways of gathering subjective data (Focus on first two) Questionnaires Interviews Booths (eg, trade show) Call-in product hot-line Field support workers (Focus on first two) Fall 2006 PSYCH / CS 6750

Questionnaires Preparation is expensive, but administration is cheap Oral vs. written Oral advs: Can ask follow-up questions Oral disadvs: Costly, time-consuming Forms can provide better quantitative data Fall 2006 PSYCH / CS 6750

Questionnaires Issues Only as good as questions you ask Establish purpose of questionnaire Don’t ask things that you will not use Who is your audience? How do you deliver and collect questionnaire? Fall 2006 PSYCH / CS 6750

Questionnaire Topic Can gather demographic data and data about the interface being studied Demographic data: Age, gender Task expertise Motivation Frequency of use Education/literacy Fall 2006 PSYCH / CS 6750

Interface Data Can gather data about screen graphic design terminology capabilities learning overall impression ... Fall 2006 PSYCH / CS 6750

Question Format Closed format Answer restricted to a set of choices Typically very quantifiable Variety of styles Fall 2006 PSYCH / CS 6750

Closed Format Likert Scale Typical scale uses 5, 7 or 9 choices Above that is hard to discern Doing an odd number gives the neutral choice in the middle Characters on screen hard to read easy to read 1 2 3 4 5 6 7 Fall 2006 PSYCH / CS 6750

Other Styles Rank from 1 - Very helpful 2 - Ambivalent 3 - Not helpful 0 - Unused Which word processing systems do you use? LaTeX Word ___ Tutorial ___ On-line help ___ Documentation FrameMaker WordPerfect Fall 2006 PSYCH / CS 6750

Closed Format Advantages Disadvantages Clarify alternatives Easily quantifiable Eliminate useless answer Disadvantages Must cover whole range All should be equally likely Don’t get interesting, “different” reactions Fall 2006 PSYCH / CS 6750

Open Format Asks for unprompted opinions Good for general, subjective information, but difficult to analyze rigorously May help with design ideas “Can you suggest improvements to this interface?” Fall 2006 PSYCH / CS 6750

Questionnaire Issues Question specificity Language Clarity “Do you have a computer?” Language Beware of terminology, jargon Clarity Leading questions Can be phrased either positive or negative Fall 2006 PSYCH / CS 6750

Questionnaire Issues Prestige bias Embarrassing questions People answer a certain way because they want you to think that way about them Embarrassing questions Hypothetical questions “Halo effect” When estimate of one feature affects estimate of another (eg, intelligence/looks) Fall 2006 PSYCH / CS 6750

Deployment Steps Discuss questions among team Administer verbally/written to a few people (pilot). Verbally query about thoughts on questions Administer final test Fall 2006 PSYCH / CS 6750

Interviews Get user’s viewpoint directly, but certainly a subjective view Advantages: Can vary level of detail as issue arises Good for more exploratory type questions which may lead to helpful, constructive suggestions Fall 2006 PSYCH / CS 6750

Interviews Disadvantages Subjective view Interviewer can bias the interview User may not appropriately characterize usage Time-consuming Fall 2006 PSYCH / CS 6750

Interview Process How to Can be done in groups Plan a set of questions (provides for some consistency) Don’t ask leading questions “Did you think the use of an icon there was really good?” Can be done in groups Get consensus, get lively discussion going Fall 2006 PSYCH / CS 6750

Data Analysis Simple analysis Determine Determine the means (time, # of errors, etc.) and compare with goal values (coming up…) Determine Why did the problems occur? What were their causes? Fall 2006 PSYCH / CS 6750

Experimental Results How does one know if an experiment’s results mean anything or confirm any beliefs? Example: 20 people participated, 11 preferred interface A, 9 preferred interface B What do you conclude? Fall 2006 PSYCH / CS 6750

Hypothesis Testing In experiment, we set up a “null hypothesis” to check Basically, it says that what occurred was simply because of chance For example, any participant has an equal chance of preferring interface A over interface B Fall 2006 PSYCH / CS 6750

Hypothesis Testing If probability result happened by chance is low, then your results are said to be “significant” Statistical measures of significance levels 0.05 often used Less than 5% possibility it occurred by chance Fall 2006 PSYCH / CS 6750

Presentation Techniques Middle 50% low high Age Mean 20 Time in secs. Fall 2006 PSYCH / CS 6750

Upcoming Audio Web Fall 2006 PSYCH / CS 6750

Using the Results How do you use the results of your evaluation? How can you make your design better with this knowledge? Fall 2006 PSYCH / CS 6750