1. Research Design: Causal Studies l Quick Review: Three general forms of quantitative research studies –Descriptive: Describes a situation –Relational : Relationship between variables –Causal: Seeks to show a cause and effect. l Internal Validity: The approximate truth of inferences regarding cause-effect or causal relationships. –Establishing Cause & Effect Establishing Cause & EffectEstablishing Cause & Effect –Single-Group Threats Single-Group ThreatsSingle-Group Threats –Multiple-Group Threats Multiple-Group ThreatsMultiple-Group Threats –“Social” Interaction Threats “Social” Interaction Threats“Social” Interaction Threats l Research Design & Design Notations Research Design & Design Notations Research Design & Design Notations

3. Establishing a Cause-Effect Relationship

4. Internal Validity Is the relationship causal between...Is the relationship causal between... What you did and what you sawWhat you did and what you saw Program and ObservationsProgram and Observations Need to consider “alternative” causes for the observation that might not be your program / treatment.Need to consider “alternative” causes for the observation that might not be your program / treatment. Observation ProgramObservations What you do What you see Alternativecause Alternativecause Alternativecause Alternativecause In this study program-outcome relationship

5. Establishing Cause and Effect Temporal precedence

6. Establishing Cause and Effect Temporal precedence CauseEffect then Time

7. Establishing Cause and Effect Temporal precedence CauseEffect then Time Why is this important?

8. Establishing Cause and Effect Temporal Precedence CauseEffect then Time Not always straight forward – cyclical example: Does inflation lead to unemployment? Or does unemployment lead to inflation? Cyclical – ongoing relationship. Both may cause and be affected by the other. Difficult to establish temporal precedence (egg before chicken?) Cyclical functions Inflation Unemployment

9. Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time

10. Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen time if X, then Y if not X, then not Y

11. Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time if X, then Y if not X, then not Y if program given, then outcome observed if program not given, then outcome not observed

12. Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time if X, then Y if not X, then not Y if program given, then outcome observed if program not given, then outcome not observed if more of program, then more of outcome observed if less of program given, then less of outcome observed

13. Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations CauseEffectthen Time if X, then Y if not X, then not Y

14. Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations CauseEffectthen Time if X, then Y if not X, then not Y ProgramOutcome Causes?

15. Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations CauseEffectthen Time if X, then Y if not X, then not Y ProgramOutcome Causes? Alternativecause Alternativecause Alternativecause Alternativecause

16. In Typical Outcome Evaluation... l Is taken care of because you intervene before you measure outcome l Is taken care of because you control the intervention l Is the central issue of internal validity -- usually taken care of through the research design Temporal precedence Covariation of cause and effect No alternative explanations

18. Single-Group Threats to Internal Validity

19. The Single Group Case l l In a single group study there is only one group examined. – –No control group. l l The use of a single group creates uncertainties in any observed influence of the program / treatment. – –The reason is that you don’t know if the observation would have occurred without the intervention. l l These uncertainties are referred to as “single group threats to validity.” The following slides will address different kinds of “single group threats to validity.”

20. The Single Group Case AdministerprogramMeasureoutcomes XO Two single group designs examples: AdministerprogramMeasureoutcomes XO Measurebaseline O Posttest -only Pretest- posttest

21. The Single Group Case: Alternate Explanations – “Threats to Validity” AdministerprogramMeasureoutcomes XO Two designs: AdministerprogramMeasureoutcomes XO Measurebaseline O AlternativeExplanations? AlternativeExplanations? AlternativeExplanations?

22. Single Group Example Threats to Internal Validity l Compensatory education in math for 1st graders l Pretest-posttest single group design –O X O (observation – program – observation) l Measures (O) are standardized achievement tests (at start of grade 1 and start of grade 2; Forms A & B)

23. History Threat l Any other event (BESIDES PROGRAM) that occurs between pretest and posttest –For example, kids pick up math concepts watching Sesame Street. ProgramPosttest XO Pretest O

24. Maturation Threat l l Normal growth between pretest and posttest. – –They would have learned these concepts anyway, even without program. ProgramPosttest XO Pretest O

25. Testing Threat l The effect on the posttest of taking the pretest l May have “primed” the kids or they may have learned from the test, not the program. The act of completing the pretest serves as practice and thus inflates the results of the posttest. ProgramPosttest XO Pretest O

26. Instrumentation Threat l l Any change in the test from pretest and posttest – –For example, change due to different forms of test, not to program. ProgramPosttest XO Pretest O

27. Mortality Threat l l Nonrandom dropout between pre & post – –Example, kids “challenged” out of program by parents or teachers » »The problem then is that the posttest group is no longer equivalent to the pretest group. ProgramPosttest XO Pretest O

28. Regression Threat l l Group is a nonrandom subgroup of population. l l Regression Threat: Statistical phenomenon – –Nonrandom group score will tend to move towards the population mean (up or down) upon repeated measures. Thus, the posttest scores are not accurately representing the intervention (program). l l Example – a group that is composed of participants who tend to score low in math, may appear to improve in math ability on posttest because of regression to the mean, and not the implemented math program. ProgramPosttest XO Pretest O

30. Multiple-Group Threats to Internal Validity

31. The Central Issue for Multiple Groups l Multiple group designs address many threats to validity that exist for single group designs. l When you move from single to multiple group research, the big concern is whether the groups are & remain comparable. l Usually this has to do with how you assign or select units (e.g., persons) into groups. –We call this issue selection or selection bias. l Threats to internal validity are similar to the single group threats, except now we focus on how they may effect the comparability of the different groups.

32. The Multiple Group Pretest-Posttest Design AlternativeExplanations? AlternativeExplanations? AdministerprogramMeasureoutcomesMeasurebaseline XOO OO Do not administer program MeasureoutcomesMeasurebaseline

33. Multiple Group Example Threats to Internal Validity l l Compensatory education in math for 1st graders l l Pre-post program-comparison group design – –One group gets math program – –Second group taught with traditional methods l l Measures (O) are standardized achievement tests (at start of grade 1 and start of grade 2; forms A & B) Before moving to the next series of slides, consider what might be threats to validity with this two-group example. What might affect the groups in such as way as to make them different from each other besides the program / treatment?

34. Selection Threats to Internal Validity l l Any factor other than the program that leads to posttest differences between groups. l l For example, because of group differences, kids in one group watch Sesame Street more frequently and pick up more math concepts. Selection Threats: This is a broad category that includes various specific forms of selection threats – see next slides.XOOOO

35. Selection-History Threat l Any other event that occurs between pretest and posttest that the groups experience differently. –For example, kids in one group pick up more math concepts because they watch more Sesame Street. XOOOO

36. Selection-Maturation Threat l l Differential rates of normal growth between pretest and posttest for the groups. – –They are learning at different rates, even without program.XOOOO

37. Selection-Testing Threat l l Differential effect on the posttest of taking the pretest. – –The test may have “primed” the kids differently in each group or they may have learned differentially from the test, not the program.XOOOO

38. Selection-Instrumentation Threat l l Any differential change in the test used for each group from pretest and posttest – –For example, change due to different forms of test being given differentially to each group, not due to programXOOOO

39. Selection-Mortality Threat l l Differential nonrandom dropout between pretest and posttest. – –For example, kids drop out of the study at different rates for each group.XOOOO

40. Selection-Regression Threat l l Different rates of regression to the mean because groups differ in extremity. – –For example, program kids are disproportionately lower math scorers and consequently have greater regression to the mean.XOOOO

42. “Social Interaction” Threats to Internal Validity

43. What Are “Social” Threats to Internal Validity? All are related to social pressures in the research context Can lead to posttest differences that are not directly caused by the treatment itself. Most of these can be minimized by isolating the two groups from each other, but this leads to other problems (for example, hard to randomly assign and then isolate, or may reduce generalizability).

44. Diffusion or Imitation of Treatment Controls might learn about the treatment from treated people (for example, kids in the school cafeteria).

45. Compensatory Rivalry Controls compete to keep up with treatment group.

46. Resentful Demoralization Controls "give up" or get discouraged. The “screw you” effect.

47. Compensatory Equalization of Treatment Administrators give a compensating treatment to controls. Thus, the results may show no difference when in fact there was. Only Social Threat directly dealing with those who manage the research. =

49. Research Design & Design Notations

50. What Is Research Design? The structure of research

51. Elements of a Design l l Observations or measures l l Treatments or programs l l Groups l l Assignment to group l l Time Each of the above is addressed in the subsequent slides.

52. Observations or Measures l l Symbolized with an "O". – –Subscripts are used to distinguish different combinations of measures only if this is necessary.

53. Treatments or Programs Symbolized with an "X". Subscripts are used to indicate different programs or combinations of programs.

54. Groups. When diagramming the research design, each group of interest is placed on a separate line.

55. Assignment to Groups R = Random assignment N = Nonequivalent groups C = Assignment by cutoff

56. Time Left-to-right movement denotes the passage of time.

57. Design Notation Example Pretest-Posttest Random Design ROXOROOROXOROO

58. Design Notation Example ROXOROOROXOROO Time

59. ROXOROOROXOROO Os indicate different waves of measurement.

60. Design Notation Example ROXOROXOROOROOROXOROXOROOROO Vertical alignment of observations (“O”) shows that the pretest of groups were measured at the same time. It also shows posttest measurements to be measured at same time.

61. Design Notation Example ROXOROOROXOROO X is the treatment.

62. Design Notation Example ROXOROOROXOROO There are two lines, one for each group.

63. Design Notation Example ROXOROOROXOROO R indicates the groups arerandomlyassigned.

64. Design Notation Example RO 1 XO 1, 2 RO 1 O 1, 2 Subscriptsindicate subsets of measures.

65. Design Notation Example Pretest-posttest (before-after) Treatment versus comparison group This is a “randomized experimental design” More details about specific designs will be addressed in subsequent lessons. ROXOROOROXOROO