From Controlled to Natural Settings

Slides:



Advertisements
Similar presentations
©2011 1www.id-book.com Evaluation studies: From controlled to natural settings Chapter 14.
Advertisements

Chapter 13: An evaluation framework
Chapter 14: Usability testing and field studies
Methodology and Explanation XX50125 Lecture 2: Experiments Dr. Danaë Stanton Fraser.
Methodology and Explanation XX50125 Lecture 1: Part I. Introduction to Evaluation Methods Part 2. Experiments Dr. Danaë Stanton Fraser.
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
CS305: HCI in SW Development Evaluation (Return to…)
WHAT IS INTERACTION DESIGN?
Chapter 14: Usability testing and field studies. 2 FJK User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept.
User Testing & Experiments. Objectives Explain the process of running a user testing or experiment session. Describe evaluation scripts and pilot tests.
Data gathering.
Chapter 14: Usability testing and field studies. Usability Testing Emphasizes the property of being usable Key Components –User Pre-Test –User Test –User.
Evaluation. formative 4 There are many times throughout the lifecycle of a software development that a designer needs answers to questions that check.
Copyright © 2009 Pearson Education Canada2-1 Chapter 2: Child Development 2.1 Doing Child-Development Research 2.2 Child-Development Research and Family.
User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept. Cal Poly San Luis Obispo FJK 2005.
An evaluation framework
An evaluation framework
Chapter 3 Preparing and Evaluating a Research Plan Gay and Airasian
Usability Evaluation 1
From Controlled to Natural Settings
Design in the World of Business
Chapter 14: Usability testing and field studies
Computer Science Department California Polytechnic State University San Luis Obispo, CA, U.S.A. Franz J. Kurfess CPE/CSC 484: User-Centered Design and.
Usability testing and field studies
+ Controlled User studies HCI /6610 Winter 2013.
Evaluation Framework Prevention vs. Intervention CHONG POH WAN 21 JUNE 2011.
Chapter 11: An Evaluation Framework Group 4: Tony Masi, Sam Esswein, Brian Rood, & Chris Troisi.
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
Ch 14. Testing & modeling users
Gathering User Data IS 588 Dr. Dania Bilal Spring 2008.
©2011 1www.id-book.com Introducing Evaluation Chapter 12 adapted by Wan C. Yoon
Human Computer Interaction
Usability testing. Goals & questions focus on how well users perform tasks with the product. – typical users – doing typical tasks. Comparison of products.
Usability Evaluation June 8, Why do we need to do usability evaluation?
Evaluation approaches Text p Text p
Testing & modeling users. The aims Describe how to do user testing. Discuss the differences between user testing, usability testing and research experiments.
Level 2 Prepared by: RHR First Prepared on: Nov 23, 2006 Last Modified on: Quality checked by: MOH Copyright 2004 Asia Pacific Institute of Information.
CS2003 Usability Engineering Usability Evaluation Dr Steve Love.
© An Evaluation Framework Chapter 13.
Review of Research Methods. Overview of the Research Process I. Develop a research question II. Develop a hypothesis III. Choose a research design IV.
AMSc Research Methods Research approach IV: Experimental [1] Jane Reid
AVI/Psych 358/IE 340: Human Factors Data Gathering October 3, 2008.
AVI/Psych 358/IE 340: Human Factors Evaluation October 31, 2008.
Um ambiente para avaliação. Objetivos Explicar conceitos e termos da avaliação Descrever paradigmas de avaliação e técnicas utilizadas no design de.
Data gathering (Chapter 7 Interaction Design Text)
Chapter 13: An evaluation framework. The aims are: To discuss the conceptual, practical and ethical issues involved in evaluation. To introduce and explain.
Chapter 7 GATHERING DATA.
Evaluation through user participation
Lecture3 Data Gathering 1.
Chapter 7 Data Gathering 1.
Usability Evaluation, part 2
Introducing Evaluation
Chapter 7 GATHERING DATA.
GATHERING DATA.
From Controlled to Natural Settings
WHAT IS INTERACTION DESIGN?
Chapter 7 GATHERING DATA.
Scientific Method Rabab Surani UGS 303 April 12, 2010.
Experimental methods Worksheet:
Introduction to Experiments
Evaluation.
Introducing Evaluation
Testing & modeling users
Human-Computer Interaction: Overview of User Studies
COMP444 Human Computer Interaction Evaluation
Chapter 2: Research in Child Development
AS Psychology Research Methods
Chapter 14 INTRODUCING EVALUATION
From Controlled to Natural Settings
Presentation transcript:

From Controlled to Natural Settings Chapter 14 Evaluation Studies: From Controlled to Natural Settings

Usability testing Involves recording performance of typical users doing typical tasks. Controlled settings. Users are observed and timed. Data is recorded on video & key presses are logged. The data is used to calculate performance times, and to identify & explain errors. User satisfaction is evaluated using questionnaires & interviews. www.id-book.com 2

Experiments & usability testing Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more variables. Usability testing is applied experimentation. Developers check that the system is usable by the intended user population for their tasks. www.id-book.com 3

Usability testing & research Improve products Few participants Results inform design Results reported to developers Experiments for research Discover knowledge Many participants Results validated statistically Scientific report to scientific community www.id-book.com 4

Usability testing Goals & questions focus on how well users perform tasks with the product. Comparison of products or prototypes is common. Focus is on time to complete task & number & type of errors. Data collected by video & interaction logging. User satisfaction questionnaires & interviews provide data about users’ opinions. www.id-book.com 5

Testing conditions Usability lab or other controlled space. Emphasis on: selecting representative users; developing representative tasks. 5-10 users typically selected. Tasks usually around 30 minutes Test conditions are the same for every participant. Informed consent form explains procedures and deals with ethical issues. www.id-book.com 6

Types of data Time to complete a task. Number and type of errors per task. Number of errors per unit of time. Number of times online help and manuals accessed. Number of users making an error. Number of users successfully completing a task. www.id-book.com 7

How many participants is enough for user testing? The number is a practical issue. Depends on: schedule for testing; availability of participants; cost of running tests. Typically 5-10 participants. www.id-book.com 8

Usability lab with observers watching a user & assistant www.id-book.com

Portable equipment for use in the field www.id-book.com

Portable equipment for use in the field www.id-book.com

Usability testing the iPad 7 participants with 3+ months experience with iPhones Signed an informed consent form explaining: what the participant would be asked to do; the length of time needed for the study; the compensation that would be offered for participating; participants’ right to withdraw from the study at any time; a promise that the person’s identity would not be disclosed; and an agreement that the data collected would be confidential and would be available to only the evaluators Then they were asked to explore the iPad Next they were asked to perform randomly assigned specified tasks www.id-book.com

Examples of the tasks www.id-book.com

Example of the equipment www.id-book.com

Experimental designs Different participants - single group of participants is allocated randomly to the experimental conditions. Same participants - all participants appear in both conditions. Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc. www.id-book.com 15

Field studies Field studies are done in natural settings. “In the wild” is a term for prototypes being used freely in natural settings. Aim to understand what users do naturally and how technology impacts them. Field studies are used in product design to: identify opportunities for new technology; determine design requirements; decide how best to introduce new technology; evaluate technology in use. www.id-book.com 16

Data collection & analysis Observation & interviews Notes, pictures, recordings Video Logging www.id-book.com 17

Data presentation The aim is to show how the products are being appropriated and integrated into their surroundings. www.id-book.com

Key points Usability testing takes place in controlled usability labs or temporary labs. Usability testing focuses on performance measures, eg. how long and how many errors are made when completing a set of predefined tasks. Indirect observation (video and keystroke logging), user satisfaction questionnaires and interviews are also collected. Affordable, remote testing systems are more portable than usability labs. Many also contain mobile eye-tracking and other devices. Experiments test a hypothesis by manipulating certain variables while keeping others constant. The experimenter controls independent variable(s) in order to measure dependent variable(s). Field studies are evaluation studies that are carried out in natural settings to discover how people interact with technology in the real world. Field studies that involve the deployment of prototypes or technologies in natural settings may also be referred to as ‘in the wild’. Sometimes the findings of a field study are unexpected, especially for in the wild studies in which explore how novel technologies are used by participants in their own homes, places of work, or outside. www.id-book.com 19