Today: Assignment 2 back on Friday True Experiments: Single-Factor Design Today’s readings: The base paper What did you find in your domain of interest Research paper

It’s a matter of control Quasi Experiment True Experiment Selection of subjects for the conditions Observe categories of subjects If the subject variable is the IV, it’s a quasi experiment Don’t know whether differences are caused by the IV or differences in the subjects Random assignment of subjects to condition Manipulate the IV Control allows ruling out of alternative hypotheses

Other features In some instances cannot completely control the what, when, where, and how Need to collect data at a certain time or not at all Practical limitations to data collection, experimental protocol

http://www.socialresearchmethods.net/kb/destypes.php

Validity Internal validity is reduced due to the presence of controlled/confounded variables But not necessarily invalid It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences Need to convince self and audience of the validity

External validity If the experimental setting more closely replicates the setting of interest, external validity can be higher than a true experiment run in a controlled lab setting Often comes down to what is most important for the research question Control or ecological validity?

Terminology Factors: Independent Variables (IVs) of an experiment Level: particular value of an IV Condition: a group or treatment (technique) e.g., Condition 1: old system, Condition 2: new system Treatment: a condition of an experiment Subject: participant (can also think more broadly of data sets that are ‘subjected’ to a treatment)

Factors to Treatments At least 1 Factor (IV) has to vary to have an experiment Effect of screen size and input technique on performance (speed, accuracy) An IV must always have at least 2 levels Condition refers to a particular way that subjects are treated Between subject: experimental conditions are the same as the groups Within subjects: only 1 group, that experiences every condition (can be many conditions in an experiment) Mixed: some variables are between, some within

Experimental designs Between subjects: Different participants - single group of participants is allocated randomly to the experimental conditions. Within subjects: Same participants - all participants appear in both conditions. Takes care of individual differences Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc. Compromise – groups not likely to be equal, but can match on the factors you think might most impact results www.id-book.com 9 9

Within-subjects It solves the individual differences issues But raises other problems: Need to look at the impact of experiencing the two conditions Will they get tired? Gain practice? Learn what is expected? Need to control for order and sequence effects?

Order Effects Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?) Arises from warm-up, learning, fatigue, etc. Effect can be averaged and removed if all possible orders are presented in the experiment and there has been random assignment to orders

Sequence effects Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2) Effects viewed may not be main effects of the IV, but interaction effects Can be controlled by arranging each condition to follow every other condition equally often

Counterbalancing Controlling order and sequence effects by arranging subjects to experience the various conditions (levels of the IV) in different orders Self-directed learning: investigate the different counterbalancing methods Randomization Block Randomization Reverse counter-balancing Latin squares and Greco squares (when you can’t fully counterbalance) http://www.experiment-resources.com/counterbalanced-measures-design.html

Between, within, matched participant design www.id-book.com 14 14

True Experiment – Single Factor Design Images & additional notes text from: http://www.nationaltechcenter.org/index.php/products/at-research-matters/quasi-experimental-study/ True Experiment – Single Factor Design

Experimental Design: spot the flaw One-Group Post-Test-Only Design Group of subjects are given a treatment (x) Single factor – only one IV Then tested on the dependent variables (observation – o) What’s the problem? Failure of internal validity: Cause and effect - don’t know what the DV measurements were before the treatment (was there a change?), don’t know that the treatment caused the DV score (was it something else that also occurred?) Survey of motivational program in which people engage in several activities, some of which are humiliating and/or exhausting Survey partiparticipants to evaluate effect of training Most report experience was worthwile and feel better about themselves than before training But survey is worthless because we don’t know how they felt before training – so we don’t know if they really changed No assurance that the training caused the change and not something else (a break in their normal routine)

Experimental Design: spot the flaw Post-Test-Only, non-equivalent control groups Non-random (N) allocation of subjects into groups One group is given the treatment, one doesn’t receive it (different levels to each group) Post-test: measure the DV What’s the problem? Addition of control condition is an attempt to get that baseline against which to compare the DVs after treatment (compare with/without treatment) Attempting to improve the study by adding a control condition Can match the subjects of the control condition on as many variables as possible But the control group is NOT equivalent in every way The non-randomness of the allocation can cause inequalities (self-selection may reveal different biases) Consider as a quasi-experiment N X O

Experimental Design: spot the flaw One-Group Pre-Test-Post-Test Design Single group (within subjects) Pre-test: measure the DVs Give the treatment Post-test: Re-measure the DVs What’s the problem? Another way of improving upon the one-group post-test only design Individual differences are controlled for But possibly threats to internal validity But still don’t know if there are other explanations for any difference involved other than the treatment (would need to think hard about confounding variables that are not related to the individual), also issues related to the temporal nature introduced (learning, changes over time, etc.) May not be clear what aspects of the treatment caused the change A pre-post test design requires that you collect data on study participants’ level of performance before the intervention took place (pre-), and that you collect the same data after the intervention took place (post-). This study design only looks at one group of individuals who receive the intervention, which is called the treatment group. The pre-post test design allows you to make inferences on the effect of your intervention by looking at the difference in the pre-test and post-test results. However, interpreting the pre-test and post-test difference should be done with caution since you cannot be sure that the differences in the pre-test and the post-test are causally related to the intervention. While the pre-post test design will allow you to measure the potential effects of an intervention by examining the difference in the pre-test and post-test results, it does not allow you to test whether this difference would have occurred in the absence of your intervention. For example, perhaps the effect of improved academic achievement is due to the students getting used to taking a test rather than the use of educational software.

Two-Group, Pre &Post-Test Design Two groups: Between subjects: random allocation Treatment Pre-test and Post-test: measure the DV Trying to achieve control over threats to validity (but can never rule out all) Need to have enough subjects (many users, many data runs) to achieve acceptable power To get the true effects of the program or intervention, it is necessary to have both a treatment group and a control group. As the names suggest, the treatment group receives the intervention. The control group, however, gets the business-as-usual conditions, meaning they only receive interventions that they would have gotten if they had not participated in the study. By having both a group that received the intervention and another group that did not, researchers control for the possibility that other factors not related to the intervention (e.g., students getting accustomed to a test, or simple maturation over the intervening time) are responsible for the difference between the pre-test and post-test results. It is also important that both the treatment group and the control group are of adequate size to be able to determine whether an effect took place or not. While the size of the sample ought to be determined by specific scientific methods, a general rule of thumb is that each group ought to have at least 30 participants.

Within-subjects (repeated measures) Similar to the one-group pre-test-post-test design It solves the individual differences issues But raises other problems: Need to look at the impact of experiencing the two conditions Will they get tired? Gain practice? Learn what is expected? Need to control for order and sequence effects

Counter balanced

Determining effect of IV on the DV Advantage of single factor design: Easy to analyze (only one IV) Fewer conditions (as many as number of levels) Simple experimental design BUT Do not know how results would change for other levels of the controlled variables Only measuring at one level of the controlled variable Will the results hold for other levels? Or are there interactions between the IV and other variables?

Solutions? Is repeating the experiment for another level of the controlled variable a valid solution? Why or why not?