Background, achievements and challenges in the external evaluation process of Oportunidades November 19 th, 2013

Background  The external evaluation process of Oportunidades (formerly PROGRESA) started at the same time the programme was implemented in rural and, eventually, urban areas.  In both areas: ­ The main objective of the evaluation process is to assess the fulfillment of its objectives and goals, as well as to determine its efficiency, effectiveness, operation and impact. ­ The external evaluation used panel data samples of households. ­ The external evaluation considered the full package of interventions among different age groups within the households, and did not take them separately (components). ­ Quantitative as well as qualitative approaches were considered in the evaluation process.

Background: Rural evaluation 1997: Original evaluation sample (randomized) Treatment (320 localities / 16,651 households) New control group 152 localities/ 6,728 selected households by matching Control – Late treatment (186 localities / 10,834 households)

Up to now 8 follow-up surveys: ENCEL 1998: March and October. ENCEL 1999: March and November. ENCEL 2000: March and November. ENCEL 2003: October – New control group. ENCEL 2007: October – Differences due to exposition time. More than 80 research papers: Increases in schooling and school attendance. Positive effects in income and increase in quality and variety of the diet in children and adults. Positive effects in children height (differences of more than 1 cm between beneficiaries and no-beneficiaries), reduction of anemin in children. Increase in health services usage. Reduction in disease days and disability. Reduction in youth risk behavior. Background: Rural evaluation

Background: Urban evaluation  Quasi-experimental methods: Propensity Score Matching (blocks).  17 states, 179 localities y 746 blocks.  States: Campeche, Chiapas, Colima, Guerrero, Guanajuato, Hidalgo, México, Michoacán, Morelos, Querétaro, Puebla, San Luis Potosí, Veracruz, Sinaloa, Sonora, Tabasco, Tamaulipas and Tlaxcala.  Methodology challenge : ­ The inclusion of households into the programme was made by self-selection. ­ Block boundaries are less clear than localities in rural areas.

Recent External Evaluation Framework The Performance Evaluation System (or SED in Spanish), is leaded by the Ministry of Finance, CONEVAL and the Ministry of Public Function. SED’s main objective is that budgetary decisions are taken considering sound and reliable information about the design, results and performance of social programmes, which stems from (i) external evaluation process and (ii) monitoring & supervision activities.. Results Based Budget (PbR) Performance Evaluation System (SED) Public expenditure quality management Programmes and Public Policy Evaluation Indicators Logical Framework Approach New management of the budget process Resources assigned by results Performance Evaluation System Components Source: SHCP. SED. Annual Evaluation Program (PAE)

Monitoring Today: Indicators Matrix Goal Contribute to brake the intergenerational transmission of poverty, by favoring the development of capacities associated to feeding, health and education of the beneficiary families of the Programme Purpose The beneficiary families of Oportunidades increase their capacities of feeding, heath and education Components School attendance certification of the scholarship holders in primary and secondary education School continuity certification of scholarship holders in high school education Promotion of the incorporation of schools with Oportunidades scholarship holders into the Quality Schools Programme Provision of health services Compliance certification in health responsibilities of the elderly beneficiaries Compliance certification in health responsibilities of beneficiary families EDUCATION Beneficiary families with children and adolescents that complied with their responsibilities in primary, secondary and high school education received the school grants HEALTH Beneficiary families that complied with their health responsibilities, received the Basic Health Kit and the in-kind grants NUTRITION Beneficiary families that complied with their responsibilities, received the cash transfer Activities Transversal Activities Beneficiary families coverage Good assistance to the beneficiary female head of family in the Assistance Centers and Oportunidades Services Resources transferred

Achievements and challenges in the external evaluation process Documentation of Oportunidades’ impacts and results led to:  It’s continuity through three different presidential administrations.  It´s expansion to urban areas & the increase in its coverage  More transparency and accountability in public expenditure  Support several improvements in design and operative issues  Better identification of the external elements that hinder the impact of the Programme Also, with the information provided by external evaluation, supervision & monitoring it is possible to identify:  Methodology challenges: (i) attrition in panel data samples (rural and urban areas), (ii) loss of control groups, (iii) combination of quantitative as well as qualitative methodologies in the evaluation process, (iv) external validity in the results  Need to monitor the program’s results using data from national surveys  Improve the coordination of the increasing actors involved in the evaluation process  Increase the opportuneness of the results in order to be taken into account into the decision making process

External evaluation  Advice from government experts, academics, researchers in technical and methodology topics  Coordination among the actors involved in the evaluation process → Improve planning activities  Search for new approaches and methodologies  Increase the participation of institutions and researchers in the evaluation activities: (i) design and interventions analysis, (ii) surveys, (iii) look over the studies.  Continuity of the panel data samples and implementation of specific surveys to evaluate new interventions.  Samples in national surveys: ENSANUT, ENIGH, ENOE.

Thank you