1. William M. Trochim James P. Donnelly Kanika Arora 8 Introduction to Design

2. Research design is the glue that holds pieces of the research project together –The sample –The measures –The treatments or programs –The method of assignment 8.1 Foundations of Design

3. Causal –Pertaining to a cause-effect question, hypothesis, or relationship –Something is causal if it leads to an outcome or makes an outcome happen –Don’t confuse this word with casual! 8.2 Research Design and Causality

4. Causal relationship –A cause-effect relationship: for example, when you evaluate whether your treatment or program causes an outcome to occur, you are examining a causal relationship Three criteria: –Temporal precedence –Covariation of the cause and effect –No plausible alternative explanations 8.2a Establishing Cause and Effect in Research Design

5. An unobserved variable that accounts for a correlation between two variables Correlation does not equal causation! 8.2a The Third-Variable Problem

6. 8.2a Control Groups

7. 8.2b Internal Validity

8. The three main types of threats –Single-group threats –Multi-group threats –Social-interaction threats 8.2b Internal Validity

9. Single Group Considered non-experimental, or, pre- experimental. Design Notation - Sketch 8.2b Internal Validity – Single Group Design

10. Occurs in single group post-test or pretest- post-test designs –Types History Maturation Testing Instrumentation Mortality Regression 8.2b Internal Validity – Single Group Threats

11. Deal with single-group threats by adding a second group! –Called a control group Once you’ve added a control group, you need to address the multi-group threats to internal validity 8.2b Internal Validity – Single Group Threats (cont’d.)

12. 8.2b Multi-group Design Notation – Two-group Post-test Only Design Sketch

13. There really is only one multiple-group threat to internal validity: that the groups were not comparable before the study –See Fine et al. (2003) – Section 3.3c: including prisoners as researchers provided critical insights 8.2b Internal Validity – Multi-Group Threats

14. Selection-history threat Selection-maturation threat Selection-testing Threat Selection-instrumentation threat Selection-mortality threat Selection-regression threat 8.2b Internal Validity – Types of Selection Bias

15. When you do not use randomization to groups, you have a quasi-experimental design Quasi-experimental designs have weaker internal validity Example Sketch – Two-group Pre-Post Design 8.2b Internal Validity – Multi-Group Threats

16. Diffusion or imitation of treatment Compensatory rivalry Resentful demoralization Compensatory equalization of treatment 8.2b Internal Validity – Social-Interaction Threats

17. Other ways to rule out threats to internal validity –By argument –By measurement or observation –By analysis Main Effect Covariance Analysis –By preventative action 8.2b Internal Validity – Other Methods

18. Four elements to any research design: –Time –Treatments or programs (IV) –Measures or observations (DV) –Groups or individuals 8.3 Developing a Research Design

19. 8.3 Design Notation

20. 8.4 Types of Designs

21. 8.4 Notational Examples

22. Why is internal validity so important in research design? What is the purpose of randomization in research design? Discuss and Debate

23. Program group Treatment group Comparison group Control group Probabilistically equivalent 9.2a Distinguishing Features of Experimental Design

24. 9.2b Experimental Design and Threats to Internal Validity

25. Two-group Post-test Only Design – Sketch Design Notation Post-Test Only Examples

26. Two-group Post-test Only Design - Sketch Switching Replications Design – Sketch Design Notation Post-Test Only Examples

27. Two-group Post-test Only Design - Sketch Switching Replications Design - Sketch 2 x 2 Factorial Design – see next slides –Allows one to examine “levels” of the variables, and, most importantly, interactions between variable levels. –Let’s use the example from the book: Time in Instruction (1 hour vs. 4 hour) and Setting (In- Class vs. Pull-out) Randomized Block Design – see next slides Design Notation Post-Test Only Examples

28. 9.4a The Basic 2 x 2 Factorial Design

29. 9.4a The Basic 2 x 2 Factorial Design: Possible Outcome

30. 9.4a A Main Effect of Time in Instruction in a 2 x 2 Factorial design

31. 9.4a A Main Effect of Setting in a 2 3 2 Factorial Design

32. 9.4a Main Effects of Both Time and Setting in a 2 x 2 Factorial Design

33. 9.4a An Interaction in a 2 x 2 Factorial Design

34. 9.4a A Crossover Interaction in a 2 x 2 Factorial Design

35. Benefits –Enhances the signal –Efficient design –Only design that allows you to examine interactions Limitations –Complex –More participants required 9.4b Benefits and Limitations of Factorial Designs

36. 9.4c Factorial Design Variations: 2 x 3

37. Helps minimize noise through the grouping of units (e.g., participants) into one or more classifications (blocks) that account for some of the variability in the outcome 9.5 Noise-Reducing Designs: Randomized Block Designs

38. Differential drop out (mortality threat) Ethical problems Social threats to internal validity Difficult to generalize to the real world 9.8 Limitations of Experimental Design

39. What is the difference between random selection and random assignment? What are some strengths and weaknesses of experimental designs? Can you think of some research topics for which a factorial design may be a good approach? –Need at least two variables with multiple levels. Discuss and Debate

40. “Quasi” means “sort of”, Quasi- experiments have: –A control group –A treatment (or program) group –Variables Quasi-experiments do not have: –Random assignment to groups 10.1 Foundations of Quasi-Experimental Design

41. One of the most frequently used quasi-experimental designs –Looks just like a pretest-posttest design –Lacks random assignment to groups –As a result, the treatment and control groups may be different at the study’s start –Raises a selection threat to internal validity 10-2 The Nonequivalent Groups Design

42. 10.2a Plot of Pretest and Posttest Means for Possible Outcome 1

43. 10.2a Plot of Pretest and Posttest Means for Possible Outcome 2

44. 10.2a Plot of Pretest and Posttest Means for Possible Outcome 3

45. 10.2a Plot of Pretest and Posttest Means for Possible Outcome 4

46. 10.2a Plot of Pretest and Posttest Means for Possible Outcome 5

47. A pretest- posttest program comparison- group quasi-experimental design in which a cutoff criterion on the preprogram measure is the method of assignment to a group 10.3 The Regression-Discontinuity Design

48. Notation –C indicates that groups are assigned by means of a cutoff score on the premeasure –An O stands for the administration of a measure to a group. –An X depicts the implementation of a program Each group is described on a single line 10.3a The Basic RD Design

49. A line that describes the relationship between two or more variables 10.3a Regression Line

50. A post-only design in which, after the fact, a pretest measure is constructed from preexisting data –Usually done to make up for the fact that the research did not include a true pretest 10.4a The Proxy Pretest Design

51. A design in which the people who receive the pretest are not the same as the people who take the posttest 10.4b The Separate Pre-Post Samples Design

52. A design that includes two waves of measurement prior to the program –Addresses selection-maturation threats 10.4c The Double-Pretest Design

53. A two-group design in two phases defined by three waves of measurement –In the repetition of the treatment, the two groups switch roles: The original control group in phase 1 becomes the treatment group in phase 2, whereas the original treatment group acts as the control 10.4d The Switching-Replications Design

54. At first, looks like a weak design But pattern matching gives researchers a powerful tool for assessing causality –The degree of correspondence between two data items 10.4e The Nonequivalent Dependent Variables (NEDV) Design

55. A pre-post quasi-experimental research design where the treatment is given to only one unit in the sample, with all remaining units acting as controls –This design is particularly useful to study the effects of community level interventions 10.4f The Regression Point Displacement (RPD) Design

56. Why can quasi-experiments be more ethical than randomized experiments? What are the strengths and the weaknesses of quasi-experimental designs? Discuss and Debate