Measuring the impact of education interventions Stephen Taylor STELLENBOSCH, AUGUST 2015

PLAN Locating impact evaluation Menu of methods Case Study: A randomised experiment Challenges with RCTs in education Opportunities for research in government

Locating impact evaluation Qualitative work – Hoadley (2003,2007); Ensor et al (2009) Systemic analysis with mixed methods – Taylor, Vinjevold, Muller (2003); Fleisch (2008) Descriptive quantitative work – Reddy (2006); Taylor & Yu (2008); Spaull (2011) Correlational analysis – Crouch & Mabogoane (1998); Van der Berg (2008); Gustafsson (2007); Spaull (2012); Shepherd (2011) Moving toward causal quantitative analysis

The evaluation problem: knowing a counterfactual The evaluation problem: – We cannot observe the counterfactual: – 2 alternative scenarios for the same person or group. So we have to identify or construct comparison groups as a “pseudo-counterfactual”. – Or an estimate of the counterfactual The big question is: when is a comparison group a valid estimate of the counterfactual? – Internal validity Selection bias (endogeneity): – Years of Schooling and IQ – Libraries and learning outcomes

A menu of methods Pre & Post Simple Difference Difference-in-differences Regression & matching Fixed effects RCT RDD IV } } } Non- experimental (observed data) Experimental Quasi- Experimental

Case Study: The impact of study guides on matric performance: Evidence from a randomised experiment

Background to the “Mind The Gap” study Mind the Gap study guides developed during 2012 Aimed at acquiring the basic knowledge and skills necessary to pass the matric exam Distributed to schools in some parts of the country – Mainly underperforming districts in EC, NC, a bit in Gauteng and elsewhere, but not in Mpumalanga Impact evaluation using 4 subjects in MP – ACCN, ECON, GEOG, LFSC

The Sampling Frame National list of schools that were enrolled for the matric 2012 examination. The list was then restricted to only schools in Mpumalanga. Further restricted to schools registered to write the matric 2012 exam in English. The final sampling frame consists of 318 schools. Randomly allocated guides to 79 schools (books were couriered – delivery reliable) Leaves 239 control schools Books delivered late in Year: September

Main Results: OLS regressions with baseline To summarise: No significant impact in Accounting & Economics; Impacts of roughly 2 percentage points in Geography & Life Sciences

Heterogeneous effects

Did impact vary by school functionality? GeographyLife Sciences

Matric 2010 simulation Roughly a 1 percentage point increase in matric pass rate 5609 The number of children who did not pass matric in 2010 but would have passed had Mind The Gap been nationally available Geography and Life Sciences.

Interpreting the size of the impact Very rough rule of thumb: 1 year of learning = 0.4 to 0.5 standard deviations of test scores Geography: 13.5% SD Life Sciences: 14.4% SD Roughly a third of a year of learning The unit cost per study guide (reflecting material development, printing and distribution) is estimated to be R41,82

Kremer, Brannen & Glennerster, 2013 MTG: 3.04 SD per $100

Interpretation of results 2 guides had no impact: Interventions do not always impact on desired outcomes Interventions are not uniform in effectiveness The quality of the ACCN & ECON material? – Or of the GEOG & LFSC materials? Contextual factors pre-disposing LFSC & GEOG to have an impact but not ACCN & ECON? A certain level of school functionality / managerial capacity needed in order for resources to be effective Timing of the delivery of guides External validity – We are more certain about delivery in MP than if this were taken to scale – Awareness campaigns could increase the impact at scale

Critiques of RCTs External validity Necessary and sufficient conditions for impact evaluations (internal and external validity) Internal validity = causal inference External validity = transferability to population – Context: geography, time, etc...? E.g. Private schools, class size – Special experimental conditions Hawthorne effects Implementation agent System support

External validity: Recommendations Choose a representative & relevant study population Investigate heterogeneous impacts Investigate intermediate outcomes Use a realistic (scaleable) model of implementation and cost structure Work with government... But be careful No pre-test...? Or use administrative data (ANA & NSC provide opportunity here for DBE collaboration)

RCTs in Education: Practical challenges Fund raising Stakeholder engagement Test development Fieldwork quality Project management

Evaluations with Government: Advantages Accountability Shifts the focus from inputs (e.g. number of teachers trained) to outcomes; From form to function (mimicry). Cooperation between government and other actors (researchers, NGOs, etc) Encourages policy-makers to interact with research and evidence Thinking about theories of change – Shifts the focus from did government programme X succeed or fail, to why? The agency of programme recipients to change behaviour. Benefits for research: reduces publication bias

Evaluations in Government: Opportunities Low hanging RCTs – 1000 libraries; EGRA RDD Encouragement designs – Online tools; winter schools Good analysis of existing data – Grade R evaluation – LOLT paper

Concluding thought Broader benefits of an evaluation culture – Not all programmes/policies can be subjected to a quantitative impact evaluation – Theories of change – Accountability – Binding constraints – Interaction btw government and researchers