Change Score Analysis versus ANCOVA in Pretest/Posttest Designs:

Slides:



Advertisements
Similar presentations
Agenda Group Hypotheses Validity of Inferences from Research Inferences and Errors Types of Validity Threats to Validity.
Advertisements

EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013.
Experimental and Quasi-Experimental Research
ANCOVA Workings of ANOVA & ANCOVA ANCOVA, Semi-Partial correlations, statistical control Using model plotting to think about ANCOVA & Statistical control.
1. 2 Interesting Statistical Phenomenon VA San Diego Addictions Seminar 4/16/08 Kevin Cummins.
Factorial and Mixed Factor ANOVA and ANCOVA
Correlation and regression
Chapter 6: Correlational Research Examine whether variables are related to one another (whether they vary together). Correlation coefficient: statistic.
Who are the participants? Creating a Quality Sample 47:269: Research Methods I Dr. Leonard March 22, 2010.
ANCOVA Workings of ANOVA & ANCOVA ANCOVA, Semi-Partial correlations, statistical control Using model plotting to think about ANCOVA & Statistical control.
Intro to Statistics for the Behavioral Sciences PSYC 1900 Lecture 10: Hypothesis Tests for Two Means: Related & Independent Samples.
Matching level of measurement to statistical procedures
Correlations and T-tests
Today Concepts underlying inferential statistics
Richard M. Jacobs, OSA, Ph.D.
Chapter 9 Experimental Research Gay, Mills, and Airasian
McGraw-Hill © 2006 The McGraw-Hill Companies, Inc. All rights reserved. Experimental Research Chapter Thirteen.
Experimental Research
Chapter 12 Inferential Statistics Gay, Mills, and Airasian
Chapter 8 Experimental Research
Christopher Dougherty EC220 - Introduction to econometrics (chapter 7) Slideshow: weighted least squares and logarithmic regressions Original citation:
ANCOVA Lecture 9 Andrew Ainsworth. What is ANCOVA?
Inferential Statistics & Test of Significance
Hypothesis Testing in Linear Regression Analysis
1 Research Method Lecture 6 (Ch7) Multiple regression with qualitative variables ©
CORRELATION & REGRESSION
T-Tests and Chi2 Does your sample data reflect the population from which it is drawn from?
Povertyactionlab.org Planning Sample Size for Randomized Evaluations Esther Duflo MIT and Poverty Action Lab.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Analysis of Covariance adjusting for potential confounds.
Educational Research Chapter 13 Inferential Statistics Gay, Mills, and Airasian 10 th Edition.
Mixed-Design ANOVA 5 Nov 2010 CPSY501 Dr. Sean Ho Trinity Western University Please download: treatment5.sav.
Welcome to Econ 420 Applied Regression Analysis Study Guide Week Seven.
Evaluating Impacts of MSP Grants Ellen Bobronnikov Hilary Rhodes January 11, 2010 Common Issues and Recommendations.
Testing the Differences between Means Statistics for Political Science Levin and Fox Chapter Seven 1.
Chapter 11.  The general plan for carrying out a study where the independent variable is changed  Determines the internal validity  Should provide.
T tests comparing two means t tests comparing two means.
IMPORTANCE OF STATISTICS MR.CHITHRAVEL.V ASST.PROFESSOR ACN.
Handout Twelve: Design & Analysis of Covariance
1 HETEROSCEDASTICITY: WEIGHTED AND LOGARITHMIC REGRESSIONS This sequence presents two methods for dealing with the problem of heteroscedasticity. We will.
Psych 230 Psychological Measurement and Statistics Pedro Wolf October 21, 2009.
Producing Data: Experiments BPS - 5th Ed. Chapter 9 1.
Educational Research Inferential Statistics Chapter th Chapter 12- 8th Gay and Airasian.
VARIABLE MISSPECIFICATION I: OMISSION OF A RELEVANT VARIABLE In this sequence and the next we will investigate the consequences of misspecifying the regression.
Copyright © 2009 Pearson Education, Inc t LEARNING GOAL Understand when it is appropriate to use the Student t distribution rather than the normal.
Can you hear me now? Keeping threats to validity from muffling assessment messages Maureen Donohue-Smith, Ph.D., RN Elmira College.
CHAPTER 13: THE NUTS AND BOLTS OF QUASI- EXPERIMENTS.
Section Testing a Proportion
EXPERIMENTAL RESEARCH
Inference and Tests of Hypotheses
Experimental Research Designs
Experimental Design-Chapter 8
Design (3): quasi-experimental and non-experimental designs
Analysis of Covariance (ANCOVA)
12 Inferential Analysis.
Unit 7 Today we will look at: Normal distributions
More about Tests and Intervals
Designing Experiments
Reasoning in Psychology Using Statistics
Impact Evaluation Methods
The Nonequivalent Groups Design
Reasoning in Psychology Using Statistics
12 Inferential Analysis.
Regression To The Mean 林 建 甫 C.F. Jeff Lin, MD. PhD.
OMGT LECTURE 10: Elements of Hypothesis Testing
Inferential Statistics
Inferential Statistics & Test of Significance
Power analysis Chong-ho Yu, Ph.Ds..
Understanding Statistical Inferences
Reminder for next week CUELT Conference.
Presentation transcript:

Change Score Analysis versus ANCOVA in Pretest/Posttest Designs: The Assignment Mechanism Matters Justin C. Mary, Dale E. Berger, Giovanni Sosa, & Christopher Pentoney Claremont Graduate University Scenario 1: ANCOVA is Biased Gain Scores VS. ANCOVA ANCOVA is Often Biased when Intact Groups are Studied ANCOVA assumes that any group differences on the covariate are due to sampling error1; posttest means are adjusted to account for regression towards the mean. The test of significance in ANCOVA is performed on these adjusted means2. When groups are assigned non-randomly, the assumption that groups have the same population mean on the covariate may be invalid and any adjustments are suspect. Gain scores analysis does not assume that pretest scores are equivalent across groups. Gain score analysis treats any differences between groups as a real effect. Thus, when pretest differences are real, gain scores are unbiased and ANCOVA is biased. These findings have been confirmed by several simulation studies4.5 . Two common methods for comparing change across paired observations for two groups are the analysis of change or gain scores (Posttest – Pretest) and Analysis of Covariance (ANCOVA). For example, suppose we wish to assess gender differences in the effects of an intervention on test performance. Participants were tested both before and after engaging in the intervention. Should we use ANCOVA to covary out pretest scores, or should we conduct a t-test on change scores? For over 40 years gain score analysis (GSA) has been criticized for possible bias, low reliability, and low power. These criticisms have been successfully challenged as being too broad. Gain scores are often biased when assignment to treatment groups is based on pretest scores. However, ANCOVA is often biased when intact groups are studied. Graphs of these two testing situations are shown here to demonstrate the bias inherent in both methods. ANCOVA adjusts the means for men and women to be equal on the covariate (50 is the overall mean on the covariate). Imagine sliding the points along the regression line and lining them up at the covariate mean. Women Men Pre = 70 Pre = 30 Post = 70 Post = 30 Difference = 0 Difference = 0 GSA Unbiased. Difference scores both zero Although neither group improved from pretest to posttest, ANCOVA shows the test of the group effect on the posttest is significant after covarying out pretest scores. Women Men Adj Pre = 50 Adj Pre = 50 Adj Post = 60 Adj Post = 40 ANCOVA Adjusted Posttest means are not equal. ANCOVA Interpretation: After equating men and women on pretest score, the effect of the intervention was larger for women than for men. Gain Score interpretation: There was no gain for either men or women. Post – Pre = 0 for both men and women. This discrepancy in findings between gain score analysis and ANCOVA is known as Lord’s Paradox3 Using the ANCOVA approach, a researcher may falsely conclude that there are gender differences in gains, when in fact there were none. This is called a “regression artifact” or pseudo effect3 because ANCOVA under-corrects for pretest effects. Lord’s paradox can be attributed to measurement error4 and trait instability2; this paradox can be eliminated only when the slope of the within-group regression lines = 1.2 Scenario 2: GSA is Biased When participants are assigned to groups based upon the pretest, regression towards the mean is likely to occur and ANCOVA will be unbiased, while GSA will be biased1 For example, assume students (N = 16) were randomly assigned to an intervention group or a control group based upon pretest scores. That is, any student scoring above the mean score of 10 was placed in the control group and everyone below the mean into the intervention group. Regression towards the mean suggests that students with extremely low scores are likely to have higher posttest scores and students with extremely high scores on the pretest are likely to have lower scores at posttest. Consequently, even in the absence of a treatment effect, students in the treatment group will likely have increased performance and students in the control group will likely have worse performance on the posttest. ANCOVA corrects for regression to the mean, while GSA does not. Gain Scores are Biased when Assignment is Based on Pretest Scores There is no overall treatment effect (overall pre and posttest means are both 10). Control Intervention Pre = 14.5 Pre = 5.5 Post = 13.0 Post = 7.0 Difference= -1.5 Difference = 1.5 GSA is biased. GSA mistakes regression towards the mean for an actual treatment effect. ANCOVA appropriately adjusts for regression towards the mean Control Intervention Adj Pre = 10.0 Adj Pre = 10.0 Adj Post = 9.6 Adj Post = 10.4 Difference = -.4 Difference = +.4 ANCOVA is unbiased. The posttest means are not reliably different (p = .778). ANCOVA Interpretation: When the posttest scores are adjusted to the predicted values if the two groups had the same average pretest score, there is no reliable difference between the control group and the intervention group on posttest performance Gain Score interpretation: Participants in the treatment group had larger gain scores than the control group. In this example, assignment was based on pretest scores and regression towards the mean artificially increased scores in the Intervention condition and decreased scores in the Control condition. ANCOVA assumes that any pretest differences are due to sampling error and corrects for regression towards the mean by adjusting scores for the group with the highest mean downward and adjusting scores from the group with the lower mean upwards. GSA does not adjust scores to account for regression towards; therefore, it is biased.