Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank 1 October 2010

Outline 1.Hypotheses 2.Detour: Evaluation of Programs 3.Empirical strategy 4.The data 5.Results 2

Why concessions (a PPP) may increase quality of education? An application of an already proven pedagogic model Concession schools are financially stable, ensuring the stability of the pedagogic model. Freedom to choose (and fire) teachers and management staff Better infrastructure Concession schools work actively with the families and the community – “Affection deficit” and family problems are important issues in these communities Nutrition is an important variable in educational outcomes 3

Three hypothesis Dropout rates are lower in concessions schools than in similar, public schools. Nearby schools have lower dropout rates than public schools outside the influence of concessions. Test scores are higher in concessions schools than in similar, public schools. 4

Detour: Evaluating Programs 5

What is Impact Evaluation? Impact evaluation is a set of methods to identify and quantify the causal impact of programs For example, What is the effect of a concession schools on drop-out rates and test scores?

Why are these effects difficult to estimate? The basic question: What would it happen in the absence of the program? (e.g., what is the contrafactual?) We need the same individual with and without the program – For example, we need to observe John attending one concession school, and (the same) John attending a public school – …but it is impossible to observe the same individual in both states! Solution: “build” the correct contrafactual – Find individuals who do not have the benefits of the program, but are very similar to the ones that have the program – For example, find David who is very similar in all characteristics to John…

What is the problem? In short, the problem is to find the correct comparison (control) group – The control group and the treatment group should have the same characteristics, observable and non-observable, before the beginning of the program – External factors will affect in the same way control and treatment group An, usually, people self-select into programs, and therefore, the beneficiaries are (with high probability), different that the ones that did not enter into the program

The basic intuition: only data before and after the program Y: Test scores Time t = 0 before program t = 1 after program Impact of the program? NO! We need a contrafactual Intervention

We need the right comparison group Y Time t = 0 before program t = 1 after program Impact of the program Intervention Control Treatment Y: Test scores

The basic intuition 2: only data after the program…. Y Time t = 0 before program t = 1 after program Impact of the Program? Intervention Control Treatment Y: Test scores

We need the right comparison group! Y Time t = 0 before program t = 1 after program Impact of the Program? NO! At t=0, two groups were very different… Intervention Control Treatment Y: Test scores

Four possibilities to find or construct the right control group Prospective evaluation – Randomization of benefits – Randomization of entry (phase-in approach) – Randomization of information (encouragement design) Retrospective evaluation – Regression discontinuity analysis – Instrumental variables – Differences in differences – Propensity and matching estimators

Randomization Lottery among individuals will separate the sample between winners and losers – Model of “over-subscription” A lottery will create homogenous control and treatment group: they will be very similar in all characteristics, observable and non- observable.

Regression discontinuity When to use this method? – The beneficiaries/non-beneficiaries can be ordered along a quantifiable dimension. – This dimension can be used to compute a well-defined index or parameter. – The index/parameter has a cut-off point for eligibility. – The index value is what drives the assignment of a potential beneficiary to the treatment. (or to non- treatment)

Intuition The potential beneficiaries just above the cut- off point are very similar to the potential beneficiaries just below the cut-off point. We compare outcomes for individuals just above and below the cutoff point.

Non-poor Poor

Treatment effect

Difference in differences 1.Estimating the impact of a “past” program 2.We can try to find a “natural experiment” that allows us to identify the impact of a policy. For example,  An unexpected change in policy could be seen as “natural experiment”  A policy that only affects 16 year olds but not 15 year olds 3.We need to identify which is the group affected by the policy change (“treatment”) and which is the group that is not affected (“control”). 4.The quality of the control group determines the quality of the evaluation.

Intuition Find a group that did not receive the program …with the same pattern of growth in the outcome variable before the intervention – The two groups, treatment and control, have the same profile before the intervention

Intuition: when it is right to use DD? Time t = -1t = 1 after program Impact of the Program Intervention Control t = 0 Slopes, before intervention (between -1 and 0) were equal Y: Test scores

Intuition: when it is right to use DD? Time t = -1t = 1 after program Impact of the Program? NO: wrong comparison group Intervention Control t = 0 Slopes, before intervention (between -1 and 0) were different Y: Test scores

Propensity and matching estimation Find the comparison group from a large survey Each treatment will have a comparison that has observable characteristics as similar as possible to the treated individuals This method assumes that there is not self-selection based on unobservable characteristics: – Selection is based on observable characteristics

How is this procedure done? Two steps Estimation of the propensity score: The propensity score is the conditional probability of receiving the treatment given the pre- treatment variables. – Estimate the probability of been treated based on the observable characteristics Estimation of the average effect of treatment given the propensity score – match cases and controls with exactly the same (estimated) propensity score; – compute the effect of treatment for each value of the (estimated) propensity score – obtain the average of these conditional effects

…Back to Concessions 25

Empirical Strategy for Concessions: propensity and matching estimation Propensity score estimation: – “Find” controls that have similar characteristics as the treatment one Matching estimation: – Once the controls are found, estimate impact: 26

The data 27 1.C100 and C600: Rich information about schools For 1999 and 2003: Administrative personnel, number and level of education of teachers, physicians and psychologists, students by grade and by age, students who failed a grade and dropped out. For 1999: infrastructure variables like computers, rooms, labs, etc. 2.ICFEX 2003: Standardized test scores Test scores at the individual level Some information at the individual level

28

29

Flavor of the (strong) result 30 Concessions reduced the dropout rate significantely Concessions reduce the jump in the desertion in grade 6

Test scores: 31 Public schools have lower test scores Concessions and public non-concessions are “similar”

32

33

Dropout Results: Probit estimates 34

Dropout results: impact 35 Matching: 10 nearest estimators, common support, balance groups Direct Effect: reduction in 1.7 points dropout rates Indirect Effect: reduction in 0.82 points

Test Results: Probit estimates 36

Test results: impact 37 Matching: 10 nearest estimators, common support, balance groups Effect over math test scores: improvement of 2.4% Effect over language test scores: improvement of 4%

38 Some conclusions (and some questions) Concessions delivers: – Strong evidence of differences in dropout rates – Some evidence of impact over test scores Colombian government is implementing the II stage of Concessions What makes concessions different from public? – Is it the work with families and community? – Freedom of hiring teachers? – Pedagogic method?

39 A discussion about the data Limits of the data: – Not a randomized experiment – Still some source of bias Ideally, we want to have more hard- evidence – Randomization of students who applied to concessions

Why is this important? Impact evaluation can provide reliable estimates of the causal effects of programs Impact evaluation can potentially help improve the efficacy of programs by influencing design and implementation Impact evaluation can broaden political support for programs Impact evaluation can help in the sustainability of successful programs and the termination of programs that are a failure Impact evaluation can help expand our understanding of how social programs produce the effects that they do