Designs to Estimate Impacts of MSP Projects with Confidence. Ellen Bobronnikov March 29, 2010

2 According to the 2007 Report of the Academic Competitive Council “Successful, large-scale interventions to improve STEM education are unlikely to arise without serious study and trial and error. There is a critical pathway for the development of successful educational interventions and activities, starting generally with small-scale studies to test new ideas and generate hypotheses, leading to increasingly larger and more rigorous studies to test the effect of a given intervention or activity on a variety of students in a variety of settings. Different research methodologies are used along the development pathway, and corresponding evaluation strategies must be used to assess their progress.”

Hierarchy of Study Designs for Evaluating Effectiveness

Overview Criteria for Classifying Designs of MSP Evaluations (“the rubric”) created through the Data Quality Initiative (DQI) Rubric’s key criteria for a rigorous design Common issues with evaluation reports Recommendations for better reporting Discussion of your evaluation challenges and solutions

5 Evaluations Reviewed Using the Rubric All final year evaluations that report using an experimental or quasi-experimental design are considered for review Evaluations need to include a comparison group to ultimately be reviewed with the rubric Within each project, we review evaluations of teacher content knowledge, classroom practices, and student achievement

6 6 criteria used in rubric Rubric comprises six criterion: 1.Equivalence of groups at baseline 2.Adequate sample size 3.Use of valid & reliable measurement instruments 4.Use of consistent data collection methods 5.Sufficient response and retention rates 6.Reporting of relevant statistics

7 Criterion 1 – Baseline Equivalence Requirement Study demonstrates no significant differences in key characteristics between treatment and comparison groups at baseline (for the analytic sample) OR Adequate steps were taken to address the lack of baseline equivalence in the statistical analysis Purpose – Helps rule out alternative explanations for differences between groups

8 Criterion 2 – Sample Size Requirement Sample size is adequate to detect a difference, based on a power analysis using: – Significance level = 0.05, – Power = 0.8 – Minimum detectable effect informed by the literature or otherwise justified Alternatively, meets or exceeds “rule of thumb” sample sizes: – School/district-level interventions: 12 schools – Teacher-level interventions: 60 teachers (teacher outcomes) or 18 teachers (student outcomes) Purpose – Increases the likelihood of finding an impact

9 Criterion 3 – Measurement Instruments Requirement – Data collection instruments used were shown to be valid and reliable to measure key outcomes Use existing instruments that have already been deemed valid and reliable Refer to TCK instrument database developed by MSP Knowledge Management and Dissemination Project at OR Create new instruments that have either been: – Sufficiently tested with subjects comparable to the study sample and found to be valid and reliable, OR – Created using scales and items from pre-existing data collection instruments that have been validated and found to be reliable  Resulting instrument needs to include at least 10 items, and at least 70 percent of the items are from the validated and reliable instrument(s) Purpose – By testing for validity and reliability, you ensure that instruments used accurately capture the intended outcomes

10 Criterion 4 – Data Collection Methods Requirement - Methods, procedures, and timeframes used to collect the key outcome data from treatment and comparison groups are comparable Purpose – Limits possibility that observed differences can be attributed to factors besides the program, such as passage of time and differences in testing conditions

11 Criterion 5 – Attrition Requirement Need to measure key outcomes for at least 70% of original sample (both treatment and control groups), or evidence that attrition is unrelated to treatment If the attrition rates between groups equal or exceed 15 percentage points, difference should be accounted for in the statistical analysis Purpose – Helps ensure that sample attrition does not bias results as participants/control group members who drop out may systematically differ from those who remain

12 Criterion 6 – Relevant Statistics Reported Requirement Include treatment and comparison group post-test means and tests of significance for key outcomes OR, Provide sufficient information for calculation of statistical significance (e.g., mean, sample size, standard deviation/standard error) Purpose – Provides context for interpreting results, indicating where observed differences between groups are most likely larger than what chance alone might cause

13 Common Issues Found in Evaluation Reports Information critical for complete assessment of all criteria is often not reported, inconsistently reported, or only reported for the treatment group – Pre & post sample sizes for both groups, means, standard deviations/ errors are frequently missing – these are needed for statistical testing and to calculate attrition rates – Varying sample sizes throughout report without explanations for changes – Validity and reliability testing not reported for locally – developed instruments or cited for pre-existing instruments – Data collection methods are not discussed

Key Recommendation – Report the Details Report pre & post sample sizes for both groups and explain changes in samples sizes; if reporting sub-groups, indicate their sample sizes as well Report key characteristics associated with outcomes at baseline (e.g., pretest scores, teaching experience) Document and describe the data collection procedures Report means, standard deviations/errors, for both groups on key outcomes; if using a regression model, describe it Report results from appropriate significance testing of differences observed between groups (e.g., t-statistics or p-values)

Discussion Questions What challenges have you encountered in your efforts to evaluate the MSP project? How have you/might you overcome these obstacles? What has enabled you to increase the rigor of your evaluations? If you could start your evaluation anew, what would you do differently?

Mathematics and Science Partnership (MSP) Programs U.S. Department of Education New Orleans Regional Meeting March 29, 2010