1 Exploring Quasi-Experiments Lab 5: May 9, 2008 Guthrie, J.T., Wigfield, A., & VonSecker, C. (2000). Effects of integrated instruction on motivation and.

Slides:



Advertisements
Similar presentations
PhD Research Seminar Series: Valid Research Designs
Advertisements

Agenda Group Hypotheses Validity of Inferences from Research Inferences and Errors Types of Validity Threats to Validity.
A Guide to Education Research in the Era of NCLB Brian Jacob University of Michigan December 5, 2007.
Validity and Reliability
Defining Characteristics
Validity of Quantitative Research Conclusions. Internal Validity External Validity Issues of Cause and Effect Issues of Generalizability Validity of Quantitative.
Experimental Research Designs
Correlation AND EXPERIMENTAL DESIGN
Introduction to Research Design Threats to Internal Validity Two or More Groups Social Threats.
Educational Action Research Todd Twyman Summer 2011 Week 1.
Experimental Design Week 9 Lecture 1.
Research Problems.
Who are the participants? Creating a Quality Sample 47:269: Research Methods I Dr. Leonard March 22, 2010.
MSc Applied Psychology PYM403 Research Methods Validity and Reliability in Research.
1 Exploring Experiments Lab #4: May 2, 2008 Today’s Article: Goodrick, G.K., Poston, S.C., Kimball, K.T., Reeves, R.S., & Foreyt, J.P. (1998). Nondieting.
Educational Research by John W. Creswell. Copyright © 2002 by Pearson Education. All rights reserved. Slide 1 Chapter 11 Experimental and Quasi-experimental.
1 Exploring Quasi-experiments Lab Assignment #5 Due May 16, 2008 Locate an empirical journal article that employs a quasi-experimental design. Closely.
Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.
Chapter 9 Experimental Research Gay, Mills, and Airasian
L1 Chapter 11 Experimental and Quasi- experimental Designs Dr. Bill Bauer.
EVALUATING YOUR RESEARCH DESIGN EDRS 5305 EDUCATIONAL RESEARCH & STATISTICS.
Experimental Research Take some action and observe its effects Take some action and observe its effects Extension of natural science to social science.
Chapter 8 Experimental Research
Experimental Design The Gold Standard?.
Learning Objectives 1 Copyright © 2002 South-Western/Thomson Learning Primary Data Collection: Experimentation CHAPTER eight.
Quantitative Research Designs
Group Discussion Explain the difference between assignment bias and selection bias. Which one is a threat to internal validity and which is a threat to.
Copyright © 2008 by Pearson Education, Inc. Upper Saddle River, New Jersey All rights reserved. John W. Creswell Educational Research: Planning,
Evaluating Research Articles From Start to Finish By Ellen R. Girden (Sage Publications, Inc. Thousand Oaks, CA) 2001 Chapter One PAF 502 Terrie Ekin.
Day 6: Non-Experimental & Experimental Design
Chapter 11 Experimental Designs
Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 11 Experimental Designs.
Between- Subjects Design Chapter 8. Review Two types of Ex research Two basic research designs are used to obtain the groups of scores that are compared.
 Internal Validity  Construct Validity  External Validity * In the context of a research study, i.e., not measurement validity.
Learning Objectives Copyright © 2002 South-Western/Thomson Learning Primary Data Collection: Experimentation CHAPTER eight.
Techniques of research control: -Extraneous variables (confounding) are: The variables which could have an unwanted effect on the dependent variable under.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
INTERNAL VALIDITY AND BASIC RESEARCH DESIGN. Internal Validity  the approximate truth about inferences regarding cause-effect or causal relationships.
Experimental Design Chapter 1 Research Strategies and the Control of Nuisance Variables.
Research methods and statistics.  Internal validity is concerned about the causal-effect relationship in a study ◦ Can observed changes be attributed.
Internal Validity. All about whether the research design (and data analysis) warrants the conclusions. Concerned with: – Causal relationships – Various.
Chapter 10 Experimental Research Gay, Mills, and Airasian 10th Edition
Research Design ED 592A Fall Research Concepts 1. Quantitative vs. Qualitative & Mixed Methods 2. Sampling 3. Instrumentation 4. Validity and Reliability.
Experimental Research Methods in Language Learning Chapter 5 Validity in Experimental Research.
Experimental & Quasi-Experimental Designs Dr. Guerette.
Chapter 11.  The general plan for carrying out a study where the independent variable is changed  Determines the internal validity  Should provide.
 The basic components of experiments are: 1) taking action 2) observing the consequence of that action  Experimental model is most closely linked to.
Chapter Eight: Quantitative Methods
Experiments.  Labs (update and questions)  STATA Introduction  Intro to Experiments and Experimental Design 2.
Research Design: Causal Studies l Quick Review: Three general forms of quantitative research studies –Descriptive: Describes a situation –Relational :
CJ490: Research Methods in Criminal Justice UNIT #4 SEMINAR Professor Jeffrey Hauck.
Quantitative and Mixed Research Designs V. Darleen Opfer.
Can you hear me now? Keeping threats to validity from muffling assessment messages Maureen Donohue-Smith, Ph.D., RN Elmira College.
School of Public Administration & Policy Dr. Kaifeng Yang 研究设计 : 实验研究的基本问题.
William M. Trochim James P. Donnelly Kanika Arora 8 Introduction to Design.
Chapter 9 Scrutinizing Quantitative Research Design.
CHAPTER 4: HOW PSYCHOLOGISTS USE THE SCIENTIFIC METHOD.
CHAPTER 13: THE NUTS AND BOLTS OF QUASI- EXPERIMENTS.
Issues in Evaluating Educational Research
Experiments Why would a double-blind experiment be used?
Ron Sterr Kim Sims Heather Cruz aka “The Carpool”
Chapter Eight: Quantitative Methods
Making Causal Inferences and Ruling out Rival Explanations
Introduction to Design
Experiments and Quasi-Experiments
Quasi-Experimental Design
Experiments and Quasi-Experiments
Group Experimental Design
Chapter 11 EDPR 7521 Dr. Kakali Bhattacharya
Reminder for next week CUELT Conference.
Presentation transcript:

1 Exploring Quasi-Experiments Lab 5: May 9, 2008 Guthrie, J.T., Wigfield, A., & VonSecker, C. (2000). Effects of integrated instruction on motivation and strategy use in reading. Journal of Educational Psychology, 92,

2 Purpose of Research Causal relationship –To identify whether the Concept-Oriented Reading instruction (CORI) intervention produces greater intrinsic motivation and strategy-use than traditional instruction

3 Variables of interest The Independent Variables (causes) –Instructional intervention: CORI intervention traditional comparison group The Dependent Variables (outcomes) –Intrinsic motivation (operationalized: curiosity, involvement, and preference for challenge) –Extrinsic motivation (operationalized: recognition and competition) –Strategy use (operationalized: self-report of cognitive strategies) The Covariates –Past achievement (operationalized: standardized reading achievement scores— CBST/MAT)

4 Participant assignment Non-random assignment into intervention and comparison groups (i.e., into classrooms) Classrooms assigned based on comparable “subjective matching” of teachers, students, and school settings

5 Cook and Campbell’s UTOS Units: 3 rd and 5 th grade low- achieving students. Treatment: Concept- oriented reading instruction (CORI) Observations: intrinsic and extrinsic motivation, and strategy use (also past achievement). Setting: three different mid- Atlantic grade schools.

6 Threats to Internal Validity Specific to Quasi-Experiments History –Some event related to the outcome could occur to one group but not the other Maturation –Groups may differ in rate of change on the outcomes prior to treatment Instrumentation –Measurement might change from pre to post test in only one group Statistical Regression –Variable used to determine group selection may be unreliable or unstable

7 Threats to Internal Validity Specific to Quasi-experiments History: A celebrity may visit the schools to discuss the importance of reading; CORI students may be more susceptible to the message and therefore show greater gains in motivation (due to the celebrity, not the intervention). Maturation: Children in the traditional classrooms could be losing motivation at a faster rate than those in the CORI classrooms prior to treatment. Instrumentation: Likely not an issue given the lack of pretest design. Statistical Regression: The choosing of similar classrooms based on teacher, student, and school make-up could have been based on inaccurate, unreliable, or temporarily skewed subjective judgments.

8 Additional Threats to Internal Validity Lack of pretest makes it difficult to say whether differences are due to CORI or whether the differences existed at the onset of the research Attrition: 11% for grade 3 and 17% for grade 5 due to moving. Resentful Demoralization: teachers or students could have shown less motivation knowing that they were not getting the treatment (p.334/ p. 47) Compensatory Rivalry: comparison teachers may have tried harder because of the study (mentioned on p. 334 as the John Henry effect). Third variables: Teacher expectations could have produced results Authors did enhance their design by including extrinsic motivation (p. 332) as a nonequivalent dependent variable (p. 74).

9 Threats to Construct Validity Diffusion of treatment: Traditional teachers frequently visited the CORI classrooms and adopted some texts used in the CORI condition. –The authors did collect video, interview, and questionnaire data regarding the use of the treatment integrity in the CORI classrooms. Low internal consistency: Several internal consistencies were low; because reliability puts a lid on validity, this is cause for concern. Mono-operation bias: Motivation is captured using only aspects of the Motivation for Reading Questionnaire (MRQ) when other methods may be possible. (e.g., Teacher-reports on individual students). –The authors did address more than one aspect of motivation (intrinsic and extrinsic)

10 Threats to External Validity Schools are chosen based on need, therefore results may not generalize to less needy students/schools. Results may not generalize to other grades Results may not generalize to interventions that are less intensive. Results may not generalize to areas outside of the mid-Atlantic. Results may not generalize to participants from dissimilar schools.

11 Threats to Statistical Conclusion Validity Violated statistical assumptions: Group administration of treatment violates assumption of independent observations (p. 49). –Addressed by analyzing the data using HLM

12 Improving the research Proximal Similarity: The traditional instruction comparison group could have maintained integrity and differentiation from CORI instruction group. The measures could have more closely approximated intrinsic motivation. –Researchers did a good job employing similar settings and units to whom they wished to generalize. –Researchers did a good job ensuring that CORI instruction was being implemented in the CORI classrooms Heterogeneous Irrelevancies: Triangulating measures of motivation could have improved construct validity. –The authors did use two different Units and three different Settings. Causal explanation: Rule out third variables and include pretest.