1. Introduction to Design8
Introduction to Design

2. 8.1 Foundations of Design
Research design is the glue that holds pieces of the research project together
The sample
The measures
The treatments or programs
The method of assignment

3. 8.2 Research Design and Causality
Pertaining to a cause-effect question, hypothesis, or relationship
Something is causal if it leads to an outcome or makes an outcome happen
Don’t confuse this word with casual!

4. 8.2a Establishing Cause and Effect in Research Design
Causal relationship
A cause-effect relationship: for example, when you evaluate whether your treatment or program causes an outcome to occur, you are examining a causal relationship
Three criteria:
Temporal precedence
Covariation of the cause and effect
No plausible alternative explanations

5. 8.2a The Third-Variable Problem
An unobserved variable that accounts for a correlation between two variables
Correlation does not equal causation!

6. 8.2a Control Groups
Figure 8.3 Control groups are important in research designs for helping address alternative explanations for a causal relationship. Okay, so researchers do have a sense of humor! No, there really aren’t “out-of-control” groups in research designs, or at least we hope not.
Courtesy of McCracken, Theresa/CartoonStock
Control group: A group, comparable to the program group, that didn’t receive the program

7. 8.2b Internal Validity
Figure 8.4 A schematic for understanding internal validity.
Internal validity: The approximate truth about inferences regarding cause-effect or causal relationships.

8. 8.2b Internal Validity The three main types of threats
Single-group threats
Multi-group threats
Social-interaction threats

9. 8.2b Internal Validity – Single Group Threats
Occurs in single group post-test or pretest- post-test designs
Types
History
Maturation
Testing
Instrumentation
Mortality
Regression
History threat: A threat to internal validity that occurs when some historical event affects your study outcome
Maturation threat: A threat to validity that occurs as a result of natural maturation that occurs between pre- and post measurement.
Testing threat: A threat to internal validity that occurs when taking the pretest affects how participants do on the posttest.
Instrumentation threat: A threat to internal validity that arises when the instruments (or observers) used on the posttest and the pretest differ.
Mortality threat: A threat to validity that occurs because a significant number of participants drop out.
Regression threat or regression artifact or regression to the mean: A statistical phenomenon that causes a group’s average performance on one measure to regress toward or appear closer to the mean of that measure, more than anticipated or predicted. Regression occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated. A regression threat will bias your estimate of the group’s posttest performance and can lead to incorrect causal inferences.

10. 8.2b Internal Validity – Single Group Threats (cont’d.)
Deal with single-group threats by adding a second group!
Called a control group
Once you’ve added a control group, you need to address the multi-group threats to internal validity

11. 8.2b Internal Validity – Multi-Group Threats
There really is only one multiple-group threat to internal validity: that the groups were not comparable before the study
See Fine et al. (2003) – Section 3.3c: including prisoners as researchers provided critical insights

12. 8.2b Internal Validity – Types of Selection Bias
Selection-history threat
Selection-maturation threat
Selection-testing Threat
Selection-instrumentation threat
Selection-mortality threat
Selection-regression threat
Selection-history threat: A threat to internal validity that results from any other event that occurs between pretest and posttest that the groups experience differently.
Selection-maturation threat: A threat to internal validity that arises from any differential rates of normal growth between pretest and posttest for the groups.
Selection-testing threat: A threat to internal validity that occurs when a differential effect of taking the pretest exists between groups on the posttest
Selection instrumentation threat: A threat to internal validity that results from differential changes in the test used for each group from pretest to posttest.
Selection-mortality threat: A threat to internal validity that arises when there is differential nonrandom dropout between pretest and posttest.
Selection-regression threat: A threat to internal validity that occurs when there are different rates of regression to the mean in the two groups.

13. 8.2b Internal Validity – Multi-Group Threats
When you do not use randomization to groups, you have a quasi-experimental design
Quasi-experimental designs have weaker internal validity

14. 8.2b Internal Validity – Social-Interaction Threats
Diffusion or imitation of treatment
Compensatory rivalry
Resentful demoralization
Compensatory equalization of treatment
Diffusion or imitation of treatment: A social threat to internal validity that occurs because a comparison group learns about the program either directlyor indirectly from program group participants.
Compensatory rivalry: A social threat to internal validity that occurs when one group knows the program another group is getting and, because of that, develops a competitive attitude with the other group.
Resentful demoralization: A social threat to internal validity that occurs when the comparison group knows what the program group is getting, and instead of developing a rivalry, control group members become discouraged or angry and give up.
Compensatory equalization of treatment: A social threat to internal validity that occurs when the control group is given a program or treatment (usually
by a well-meaning third party) designed to make up for or “compensate” for the treatment the program group gets. By equalizing the group’s experiences, this threat diminishes the researcher’s ability to evaluate the program effect.

15. 8.2b Internal Validity – Other Methods
Other ways to rule out threats to internal validity
By argument
By measurement or observation
By analysis
Main Effect
Covariance Analysis
By preventative action
Main effect: An outcome that shows consistent differences between all levels of a factor.
Covariance analysis: A statistical analysis that adjusts for one or more variables when looking at the relationship between two variables of interest.

16. 8.3 Developing a Research Design
Four elements to any research design:
Time
Treatments or programs (IV)
Measures or observations (DV)
Groups or individuals

17. 8.3 Design Notation Figure 8.7 A detailed example of design notation.
Figure 8.8 An example of a design notation that includes subscripts

18. 8.4 Types of Designs
Figure 8.9 Basic questions that distinguish the major types of designs.
Random selection: Process or procedure that assures that the different units in your population are selected by chance.

19. 8.4 Notational Examples
Figure 8.10 Notational examples of each of the three major classes of research design.
Posttest-only randomized experiment. An experiment in which the groups are randomly assigned and receive only a posttest.
Pre-post nonequivalent groups quasi-experiment: A two-group quasi-experimental design structured like a randomized experiment, but lacking random assignment to group.
Posttest-only non-experimental design: A research design in which only a posttest is given. It is referred to as nonexperimental because no control group exists.

20. 8.4a Expanding on Basic Designs
Expanding across time
Expanding across programs
Expanding across observations
Expanding across groups
The classic pretest-posttest control group design
Figure 8.14 The basic pre-post randomized experimental design

21. Discuss and Debate
Why is internal validity so important in research design?
What is the purpose of randomization in research design?
Internal validity is the truthfulness of the conclusions reached in a research study. Without taking steps to ensure internal validity, the researcher cannot be certain of causality.
Randomization creates equivalent groups in a research study. This helps the study to be more internally valid, because if both groups are equivalent, it is unlikely that any other factor than the independent variable caused the change in the dependent variable.