1 Exploring Quasi-Experiments Lab 5: May 9, 2008 Guthrie, J.T., Wigfield, A., & VonSecker, C. (2000). Effects of integrated instruction on motivation and strategy use in reading. Journal of Educational Psychology, 92,

2 Purpose of Research Causal relationship –To identify whether the Concept-Oriented Reading instruction (CORI) intervention produces greater intrinsic motivation and strategy-use than traditional instruction

3 Variables of interest The Independent Variables (causes) –Instructional intervention: CORI intervention traditional comparison group The Dependent Variables (outcomes) –Intrinsic motivation (operationalized: curiosity, involvement, and preference for challenge) –Extrinsic motivation (operationalized: recognition and competition) –Strategy use (operationalized: self-report of cognitive strategies) The Covariates –Past achievement (operationalized: standardized reading achievement scores— CBST/MAT)

4 Participant assignment Non-random assignment into intervention and comparison groups (i.e., into classrooms) Classrooms assigned based on comparable “subjective matching” of teachers, students, and school settings

5 Cook and Campbell’s UTOS Units: 3 rd and 5 th grade low- achieving students. Treatment: Concept- oriented reading instruction (CORI) Observations: intrinsic and extrinsic motivation, and strategy use (also past achievement). Setting: three different mid- Atlantic grade schools.

6 Threats to Internal Validity Specific to Quasi-Experiments History –Some event related to the outcome could occur to one group but not the other Maturation –Groups may differ in rate of change on the outcomes prior to treatment Instrumentation –Measurement might change from pre to post test in only one group Statistical Regression –Variable used to determine group selection may be unreliable or unstable

7 Threats to Internal Validity Specific to Quasi-experiments History: A celebrity may visit the schools to discuss the importance of reading; CORI students may be more susceptible to the message and therefore show greater gains in motivation (due to the celebrity, not the intervention). Maturation: Children in the traditional classrooms could be losing motivation at a faster rate than those in the CORI classrooms prior to treatment. Instrumentation: Likely not an issue given the lack of pretest design. Statistical Regression: The choosing of similar classrooms based on teacher, student, and school make-up could have been based on inaccurate, unreliable, or temporarily skewed subjective judgments.

8 Additional Threats to Internal Validity Lack of pretest makes it difficult to say whether differences are due to CORI or whether the differences existed at the onset of the research Attrition: 11% for grade 3 and 17% for grade 5 due to moving. Resentful Demoralization: teachers or students could have shown less motivation knowing that they were not getting the treatment (p.334/ p. 47) Compensatory Rivalry: comparison teachers may have tried harder because of the study (mentioned on p. 334 as the John Henry effect). Third variables: Teacher expectations could have produced results Authors did enhance their design by including extrinsic motivation (p. 332) as a nonequivalent dependent variable (p. 74).

9 Threats to Construct Validity Diffusion of treatment: Traditional teachers frequently visited the CORI classrooms and adopted some texts used in the CORI condition. –The authors did collect video, interview, and questionnaire data regarding the use of the treatment integrity in the CORI classrooms. Low internal consistency: Several internal consistencies were low; because reliability puts a lid on validity, this is cause for concern. Mono-operation bias: Motivation is captured using only aspects of the Motivation for Reading Questionnaire (MRQ) when other methods may be possible. (e.g., Teacher-reports on individual students). –The authors did address more than one aspect of motivation (intrinsic and extrinsic)

10 Threats to External Validity Schools are chosen based on need, therefore results may not generalize to less needy students/schools. Results may not generalize to other grades Results may not generalize to interventions that are less intensive. Results may not generalize to areas outside of the mid-Atlantic. Results may not generalize to participants from dissimilar schools.

11 Threats to Statistical Conclusion Validity Violated statistical assumptions: Group administration of treatment violates assumption of independent observations (p. 49). –Addressed by analyzing the data using HLM

12 Improving the research Proximal Similarity: The traditional instruction comparison group could have maintained integrity and differentiation from CORI instruction group. The measures could have more closely approximated intrinsic motivation. –Researchers did a good job employing similar settings and units to whom they wished to generalize. –Researchers did a good job ensuring that CORI instruction was being implemented in the CORI classrooms Heterogeneous Irrelevancies: Triangulating measures of motivation could have improved construct validity. –The authors did use two different Units and three different Settings. Causal explanation: Rule out third variables and include pretest.