+ Controlled User studies HCI - 4163/6610 Winter 2013.

Slides:

Advertisements

Similar presentations

©2011 1www.id-book.com Evaluation studies: From controlled to natural settings Chapter 14.

Advertisements

Chapter 14: Usability testing and field studies

Experimental Design True Experimental Designs n Random assignment n Two comparison groups n Controls threats to internal validity n Strongest evidence.

Experimental and Quasi-Experimental Research

Randomized Experimental Design

GROUP-LEVEL DESIGNS Chapter 9.

Control Any means used to rule out threats to validity Example –Hypothesis: Rats learned to press a bar when a light was turned on. –Data for 10 rats bar.

Research Variables.

Quasi Experiments Non-Experimental Research

Correlation AND EXPERIMENTAL DESIGN

Chapter 14: Usability testing and field studies. 2 FJK User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept.

User Testing & Experiments. Objectives Explain the process of running a user testing or experiment session. Describe evaluation scripts and pilot tests.

EXPERIMENTAL DESIGNS Criteria for Experiments

Tuesday, March 17, 2015 Pop Quiz – Controlled Experiments in HCI 1.

Quasi-Experimental Designs

Basics of Experimentation (1) Experimental Design: Which to Choose and Why?

Chapter 14: Usability testing and field studies. Usability Testing Emphasizes the property of being usable Key Components –User Pre-Test –User Test –User.

Single-Factor Experiments What is a true experiment? Between-subjects designs Within-subjects designs.

Lect 10a1 Experimental Research Experimental research is conducted to demonstrate functional (cause-and-effect) relationships An experiment must demonstrate.

Single-Factor Experiments What is a true experiment? Between-subjects designs Within-subjects designs Designs to avoid (not true experiments)

Chapter 28 Design of Experiments (DOE). Objectives Define basic design of experiments (DOE) terminology. Apply DOE principles. Plan, organize, and evaluate.

Questions What is the best way to avoid order effects while doing within subjects design? We talked about people becoming more depressed during a treatment.

© 2001 Dr. Laura Snodgrass, Ph.D.1 Basic Experimental Design Common Problems Assigning Participants to Groups Single variable experiments –bivalent –multivalent.

From Controlled to Natural Settings

Factorial Experiments Factorial Design = experiment in which more than one IV (factor) at a time is manipulated Uses all possible combinations of the levels.

Psychology 242 Research Methods II Dr. David Allbritton

Chapter 9 Experimental Research Gay, Mills, and Airasian

McGraw-Hill © 2006 The McGraw-Hill Companies, Inc. All rights reserved. Experimental Research Chapter Thirteen.

Experimental Research

Smith/Davis (c) 2005 Prentice Hall Chapter Ten Designing and Conducting, Experiments with Two Groups PowerPoint Presentation created by Dr. Susan R. Burns.

Chapter 8 Experimental Research

Experimental Design The Gold Standard?.

Chapter 8 Experimental Design.

Chapter 14: Usability testing and field studies

Research Methods in Psychology

Consumer Preference Test Level 1- “h” potato chip vs Level 2 - “g” potato chip 1. How would you rate chip “h” from 1 - 7? Don’t Delicious like.

Chapter 11 Experimental Designs

Design Experimental Control. Experimental control allows causal inference (IV caused observed change in DV) Experiment has internal validity when it fulfills.

Single-Factor Experimental Designs

Chapter 5, Suter. Constructs Abstract, an idea Presumed to exist Can’t be seen or even directly measured (it doesn’t really exist – it’s a presumption!)

Today: Our process Assignment 3 Q&A Concept of Control Reading: Framework for Hybrid Experiments Sampling If time, get a start on True Experiments: Single-Factor.

Chapter Seven Causal Research Design: Experimentation.

Chapter Four Experimental & Quasi-experimental Designs.

Selecting and Recruiting Subjects One Independent Variable: Two Group Designs Two Independent Groups Two Matched Groups Multiple Groups.

Experimental Design. Threats to Internal Validity 1.No Control Group Known as a “one-shot case study” XOXO (IV)(DV)

Types of Research and Designs This week and next week… Covering –Research classifications –Variables –Steps in Experimental Research –Validity –Research.

Today: Assignment 2 misconceptions Our process True Experiments: Single-Factor Design Assignment 3 Q&A Mid-term: format, coverage.

Testing & modeling users. The aims Describe how to do user testing. Discuss the differences between user testing, usability testing and research experiments.

Introduction section of article

1 MP2 Experimental Design Review HCI W2014 Acknowledgement: Much of the material in this lecture is based on material prepared for similar courses by Saul.

McGraw-Hill/Irwin Copyright © 2011 by The McGraw-Hill Companies, Inc. All rights reserved. Using Between-Subjects and Within- Subjects Experimental Designs.

1.) *Experiment* 2.) Quasi-Experiment 3.) Correlation 4.) Naturalistic Observation 5.) Case Study 6.) Survey Research.

 Descriptive Methods ◦ Observation ◦ Survey Research  Experimental Methods ◦ Independent Groups Designs ◦ Repeated Measures Designs ◦ Complex Designs.

Research Design. “The best way to escape a problem is to solve it.” -- Brendan Francis.

Chapter 10 Experimental Research Gay, Mills, and Airasian 10th Edition

Experimental Research Methods in Language Learning Chapter 4 Experimental Research Designs.

Chapter 11.  The general plan for carrying out a study where the independent variable is changed  Determines the internal validity  Should provide.

Chapter 8: Between Subjects Designs

Aim: What factors must we consider to make an experimental design?

Today: Assignment 2 back on Friday

 Allows researchers to detect cause and effect relationships  Researchers manipulate a variable and observe whether any changes occur in a second variable.

Chapter 11: The Nuts and Bolts of one-factor experiments.

Review This template can be used as a starter file for presenting training materials in a group setting. Sections Right-click on a slide to add sections.

Experimental Design-Chapter 8

Experimental Design.

Experimental Design.

Evaluating research Is this valid research?.

Experiments: Part 2.

Reminder for next week CUELT Conference.

Presentation transcript:

+ Controlled User studies HCI /6610 Winter 2013

+ Usability Experiments Predict the relationship between two or more variables. Independent variable is manipulated by the researcher. Dependent variable depends on the independent variable. Typical experimental designs have one or two independent variable. Validated statistically & replicable.

+ True Experiment Experimental control Control as many potential threats to validity as possible Random assignment of participants/data to conditions Could be within-subjects or between-subjects

+ Control True experiment = complete control over the subject assignment to conditions and the presentation of conditions to subjects Control over the who, what, when, where, how Control of the who => random assignment to conditions Only by chance can other variables be confounded with IV Control of the what/when/where/how => control over the way the experiment is conducted

+ Quasi-Experiment When you can’t achieve complete control Lack of complete control over conditions Subjects for different conditions come from potentially non-random pre-existing groups (smokers vs nonsmokers)

+ It’s a matter of control True Experiment Random assignment of subjects to condition Manipulate the IV Control allows ruling out of alternative hypotheses Quasi Experiment Selection of subjects for the conditions Observe categories of subjects If the subject variable is the IV, it’s a quasi experiment Don’t know whether differences are caused by the IV or differences in the subjects

+ Other features In some instances cannot completely control the what, when, where, and how Need to collect data at a certain time or not at all Practical limitations to data collection, experimental protocol

+ Validity Internal validity is reduced due to the presence of controlled/confounded variables But not necessarily invalid It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences Need to convince self and audience of the validity

+ External validity If the experimental setting more closely replicates the setting of interest, external validity can be higher than a true experiment run in a controlled lab setting Often comes down to what is most important for the research question Control or ecological validity?

+ Terminology Factors: Independent Variables (Ivs) of an experiment Level: particular value of an IV Condition: a group or treatment (technique) e.g., Condition 1: old system, Condition 2: new system Treatment: a condition of an experiment Subject: participant (can also think more broadly of data sets that are ‘subjected’ to a treatment)

+ Factors to Treatments At least 1 Factor (IV) has to vary to have an experiment Effect of screen size and input technique on performance (speed, accuracy) An IV must always have at least 2 levels Condition refers to a particular way that subjects are treated Between subject: experimental conditions are the same as the groups Within subjects: only 1 group, that experiences every condition (can be many conditions in an experiment)

+ Good Experimental Design Two-Group, Post-Test Design Two conditions Two groups: Between subjects: random allocation Treatment Post-test: measure the DV What’s really important?

+ Experimental designs  Between subjects: Different participants - single group of participants is allocated randomly to the experimental conditions.  Within subjects: Same participants - all participants appear in both conditions.  Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc.

+ Within-subjects Similar to the one-group pre-test-post-test design It solves the individual differences issues But raises other problems: Need to look at the impact of experiencing the two conditions Will they get tired? Gain practice? Learn what is expected? Need to control for order and sequence effects?

+ Order Effects Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?) Arises from warm-up, learning, fatigue, etc. Effect can be averaged and removed if all possible orders are presented in the experiment and there has been random assignment to orders

+ Sequence effects Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2) Effects viewed may not be main effects of the IV, but interaction effects Can be controlled by arranging each condition to follow every other condition equally often

+ Counterbalancing Controlling order and sequence effects by arranging subjects to experience the various conditions (levels of the IV) in different orders Self-directed learning: investigate the different counterbalancing methods Randomization Block Randomization Reverse counter-balancing Latin squares and Greco squares (when you can’t fully counterbalance) design.html design.html

+ Between, within, matched participant design

+ Key points 1  Usability testing is done in controlled conditions.  Usability testing is an adapted form of experimentation.  Experiments aim to test hypotheses by manipulating certain variables while keeping others constant.  The experimenter controls the independent variable(s) but not the dependent variable(s).