Using Statistics Effectively in Statistics Education Research Sterling C. Hilton Brigham Young University

OUTLINE  Background on SMER workshops  Summary of SMER final report  Connections to statistics education research  Discussion

BACKGROUND  Richard Scheaffer was PI on ASA sponsored study supported by NSF grant  Three workshops –Jan –Sept –March 2006  11 statisticians and 9 math education researchers

PURPOSE/MOTIVATION  No Child Left Behind Act –“scientifically based research” –“…involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge…”  Call to improve mathematics education in K-12 grades (e.g. TIMMS 3)  Widespread use of qualitative research methods in mathematics education research

REPORT SUMMARY  Reporting guidelines for research program not research study  Goal to help build a useful and informative body of cumulative research  Help establish consistency in –Interventions –Measurement tools –Data collection –Data analysis –Reporting

REPORT SUMMARY  Influential documents –Scientific Research in Education (NRC, 2002) –Advancing Scientific Research in Education (NRC, 2005) –Mathematical Proficiency for All Students (RAND, 2003)  Influential perspective –Medical research model (clinical trials)

REPORT SUMMARY  Research program components (with characteristic activities and reporting guidelines) –Generate –Frame  Goals and Constructs  Measurement  Logistics and Feasibility –Examine –Generalize –Extend

REPORT SUMMARY  Generate ideas –Develop a conceptual framework –Consider own values, beliefs, biases  Characteristic activities –Identify interesting issues/questions –Search research literature –Build case for importance of issue  No reporting guidelines given

REPORT SUMMARY  Framing: Goals and Constructs  Characteristic activities –Formulate central ideas and underlying constructs –Formulate a research question –Identify relevant measures or need for new measures –Select and define research methods  Reporting guidelines –Define variables and measures used –State conjectures rather than causal statements –Exploratory and descriptive statistics

REPORT SUMMARY  Framing: Measurement –Validity, reliability, fairness  Characteristic activities –Examine previously used measures –Develop new measures –Test new measures  Reporting guidelines –Describe key details of development process –State validity, reliability and bias of measures

REPORT SUMMARY  Framing: Logistics and Feasibility  Characteristic activities –Formulate study design –Develop data collection protocols –Conduct pilot study  Reporting guidelines –Describe study design –Describe variables of interest –Describe the pilot tests of instruments / interventions

REPORT SUMMARY  Examine –If warranted, expansion of the pilot study to a larger community (single institution) –Main goal is to establish efficacy  Characteristic activities –Explore and define setting –Refine measures and protocols –Identify extraneous sources of variability  Reporting guidelines –Provide enough information to allow replication –Give estimates of parameters –Report formal statistical inference

REPORT SUMMARY  Generalize –If warranted, expansion of the study to larger community (multiple institutions) –Generally requires interdisciplinary work –Main goal is to generalize findings to larger population  Characteristic activities –Assess portability of measures and protocols –Design a multi-institutional study –Conduct rigorous statistical analysis  Reporting guidelines –Describe implementation –Describe efforts to minimize bias –Specify a statistical model

REPORT SUMMARY  Extend –If warranted, extend the study over time and to multiple communities –Generally requires interdisciplinary work –Main goal is to understand long-term effects  Characteristic activities –Design and conduct longitudinal study –Document the need for program improvements  Reporting guidelines –Describe implementation of longitudinal study –Describe dropout rate and its handling in the analysis

REPORT SUMMARY  Additional issues of interest –Stable unit treatment value assumption: the treatment is not dependent on the person administering it –The use of pre-post scores or gain scores is potentially problematic –Measurement development takes time and effort

REPORT SUMMARY  Connection to medical clinical trial model GeneratePre-clinical FramePre-clinical, Phase I ExaminePhase I, Phase II GeneralizePhase III ExtendPhase IV

CONNECTIONS  Different research methods answer distinct research questions  Statistics education research is young (primarily in pre-clinical, phase I stages)  Researchers in statistics education need to: –talk the language of statisticians –allow the discipline to develop –not overstate their findings –have patience –persevere in doing quality research