Issues of Quantitative Research: Part 1 V. Darleen Opfer

You can prove anything with statistics Versus There are three kinds of lies: lies, damned lies, and statistics Competing Notions of the Use of Quantitative Data

Statistical Pitfalls Sources of bias Sources of bias Errors in method Errors in method Problems with interpretation Problems with interpretation

Sources of bias Representative sampling Representative sampling Statistical assumptions Statistical assumptions

Gender & Achievement on GCSEs GCSE Grades Difference in Percentage Points between years (not between genders) Percent Change BoysGirlsBoysGirlsBoysGirlsBoysGirls A*-C

Girls performance has increased more rapidly than that of boys. Speed, 1998 Girls performance has increased more rapidly than that of boys. Speed, 1998 Female students have improved their performance markedly, whereas male students have not shown a similar improvement. Arnot, 1996 Female students have improved their performance markedly, whereas male students have not shown a similar improvement. Arnot, 1996 There are increasing gender differences in performance. Stobart, 1992 There are increasing gender differences in performance. Stobart, 1992

Errors in Method Multiple comparisons Multiple comparisons Measurement error Measurement error

Problems with interpretation Confusion over significance Confusion over significance Precision and accuracy Precision and accuracy Causality Causality Unequal comparisons Unequal comparisons

Definitions of School Effects School effects as the absolute effects of schooling, using naturally occurring “control” groups of students who receive no schooling School effects as the absolute effects of schooling, using naturally occurring “control” groups of students who receive no schooling School effects as the unadjusted average achievement of all students in a school School effects as the unadjusted average achievement of all students in a school School effects as the impact of schooling on the average achievement of all students in a school, adjusted for family background and/or prior achievement School effects as the impact of schooling on the average achievement of all students in a school, adjusted for family background and/or prior achievement School effects as measuring the extent of ‘between schools’ variation in the total variation of their students’ individual test scores School effects as measuring the extent of ‘between schools’ variation in the total variation of their students’ individual test scores School effects as measuring the unique effect of each school on their students’ outcomes School effects as measuring the unique effect of each school on their students’ outcomes School effects as measuring the impact of schools on student performance over time School effects as measuring the impact of schools on student performance over time

What is Good Statistical Practice? Be sure your sample is representative of the population in which you’re interested Be sure your sample is representative of the population in which you’re interested Be sure you understand the assumptions of your statistical procedures and be sure they are satisfied Be sure you understand the assumptions of your statistical procedures and be sure they are satisfied Be sure the sample size is not misleading you about the amount of significance Be sure the sample size is not misleading you about the amount of significance Be sure to use the best measurement/collection tools Be sure to use the best measurement/collection tools Be aware of impact of multiple comparisons and the lack of an a priori analysis plan Be aware of impact of multiple comparisons and the lack of an a priori analysis plan Be clear about what you’re trying to discover Be clear about what you’re trying to discover Use numerical notation in a rational way Use numerical notation in a rational way Be sure you understand the conditions for causal inference Be sure you understand the conditions for causal inference Be explicit about the measures you are comparing Be explicit about the measures you are comparing

The task: Read the articles you’ve been given for next week Read the articles you’ve been given for next week The articles represent a disagreement among researchers and each group has one side of the argument The articles represent a disagreement among researchers and each group has one side of the argument Identify for your article the possible quantitative issues are involved Identify for your article the possible quantitative issues are involved –Sources of bias –Errors in method –Problems with interpretation