February 2009 The Process Evaluation for Federal Social Programs Mexico's Evaluation Strategy for Social Policy and Programs

1.Background/Preliminaries –The Mexican Evaluation Experience –CONEVAL and the 2007 evaluations 2. Process Evaluation –Objectives –Methodology –Testing 2008/ Agenda

Important facts regarding evaluation in Mexico: –Progresa 1997 –Federal Budget Bill for 2001 and External Evaluations for Federal Programs –Mexican Social Development Law (2003) : –CONEVAL (2006) –Results based Budget and the Performance Evaluation System (SHCP) –Guidelines for the Evaluation of Federal Programs (2007) –Annual Evaluation Program –In 2007, for the first time, a single type of evaluation was applied to more than 100 federal programs Preliminaries

The Mexican Council for the Evaluation of Social Policy (CONEVAL), has the following objectives: 1.Establish guidelines and criteria for the definition and measurement of multidimensional poverty 2.The regulation and coordination of the evaluation process for social policy and programs CONEVAL is a decentralized public federal agency with special characteristics in the Mexican federal administration –Secretary of Social Development (SEDESOL) – A board of six academic researchers democratically elected by the Social Development Commission –An Executive Secretary (with Evaluation and Poverty Areas) Preliminaries

Evaluate the links and consistency between federal objectives (national and policy sector plans) and the objectives of federal programs Federal Planning and Strategic Objectives To determine and disseminate the methodology for its elaboration by all federal programs (logical framework) Indicators Matrix Define the types of evaluation to be applied to federal programs and formulate the Annual Evaluation Plan Evaluation Types and Coordination Preliminaries General Guidelines for the Evaluation of Federal Programs

On the basis of the Indicators Matrix, the analysis of the design of federal programs, in terms of its logical framing Design Analysis of design and of the general performance of the programs in terms of its targeting, coverage, planning, operation, results and perception Consistency and Results Analysis of indicators design in terms of consistency with objectives, pertinence and usefulness Indicators The systematic analysis of the programs performance and progress through the realization of the indicators of its main objectives and purpose Performance Analysis of the programs operation and the consistency of processes with objectives and results Process The estimation of the change in variables of interest on beneficiaries that are due strictly to the programs implementation Impact Preliminaries Types of Evaluations

- Aplica para los programas que se propongan incluir en el PPEF y aquellos programas presupuestarios ya existentes que propongan ampliación o modificación sustantiva Impact Evaluation -Aplica para los programas nuevos que están en su primer año de operación -Aplica para los programas con al menos dos años de operación -Aplica para los programas con al menos dos años de operación y que tengan ECR previamente Impact Evaluation Performance Evaluation Year 0 Year 3 Year 2Year 1 New ProgramDesign Evaluation Consistency and Results Evaluation (every 3 years) Process Evaluation Impact Evaluation Preliminaries Types of Evaluations

Inputs Financial Resources Human Resources Physical Resources Infrastructure Outputs Patients attended Food baskets delivered Scholarships Legal Services Outcomes Health Conditions Access to services Implementation and Management Impact Evaluation General Framework

ThemeFindings Compliance with norms 84% of programs are compliant with established norms on the distribution of benefits. 89% have documental evidence on established processes for federal programs Operational Effectiveness 76% of programs have adequate procedures to process applications for beneficiaries 72% of programs have standardized and adequate procedures for the selection process of projects and beneficiaries Physical and organizational infrastructure and resources 80% of programs have an organizational structure that allows them to reach their objectives 80% of programs have systematic information to allow for an adequate follow up of beneficiaries and projects 84% of programs have efficacy indicators of their operation 61% of programs have systematic information for their administration and operation Consistency and Results Evaluation General Findings on the Operation section of the Consistency and Results Evaluation

The Process Evaluation analyses the implementation and management of social programs using information obtained through documental and fieldwork evidence The purpose of the Process Evaluation is to investigate if and how the implementation of a program contributes adequately to the achievement of the objectives formulated in its design Two important attributes for the Process Evaluation Methodology: Some degree of comparability between programs Comparability over time within programs Process Evaluation

Elaborating TORs Research on concepts, methodology and approaches Previous experiences of Evaluation of Processes in Mexico Review of International Experiences WB input Research on concepts, methodology and approaches Previous experiences of Evaluation of Processes in Mexico Review of International Experiences WB input Review and Proposal of Evaluation of Processes Objective Methodology Indicators Types of instruments Review and Proposal of Evaluation of Processes Objective Methodology Indicators Types of instruments Internal discussion and comments Evaluation of Processes Workshop Elaboration of the 1 st Terms of Reference Draft Delivery of the ToR draft for comments Integration of Final Version of ToR for triaI Hiring of a consultant for the first trial of the ToR Analysis of results and integration of the final version of ToR for its application by mid 2009

Elaboration of TORs Some evaluations reviewed Microrregiones 2004 y 2006 Evaluation of Results of the Microfinance Fund to Rural Women (Evaluación de Resultados del Fondo de Microfinanciamiento a Mujeres Rurales-FOMMUR) Evaluation of Processes of Seguro Popular (Evaluación de Procesos del Seguro Popular ) National Evaluation of Processes of PROGAN (Evaluación Nacional de Procesos del PROGAN ) Evaluation of the PYME Fund 2006 (Evaluación del Fondo PYME 2006) Evaluation of the Fulfillment of Goals, Unit costs and addition of the Program of Human Development OPORTUNIDADES to the Rules of Operation 2005 (Evaluación del Cumplimiento de Metas, Costos Unitarios y Apego del Programa de Desarrollo Humano Oportunidades a las Reglas de Operación 2005) External evaluation of the Program of Support to Farming and Fishing Social Organizations 2005 (Evaluación Externa del Programa de Apoyo a las Organizaciones Sociales Agropecuarias y Pesqueras PROSAP)

Process Evaluation Objectives Overall: To analyze, with fieldwork and documental evidence, the implementation of the federal social programs to identify the characteristics of its operational processes, specifying problems areas and best practices. Specific Objectives: Describe programs implementation by means of explaining its operative processes in the different (government) levels in which these are carried out; Investigate programs operative processes to qualify them in terms effectiveness, opportunity, sufficiency and pertinence for the achievement of programs objectives; Identify and describe the problems that limit implementation, as well as the good practices that fortify their capacity; Assess whether the processes established in the Programs legal framework (Reglas de Operación) are adequate, applicable, deficient or insufficient, be it in their internal consistency and on their application.

Methodology Detailed description of the programs implementation scheme through the description of its composing operational processes and their characteristics in terms of their attributes Description and analysis of the legal framework for its operation Description and analysis of the observational (true) processes and comparison to the norm Formulating indicators to measure the attributes of processes Detailed description of problem areas and best practices Information Documental sources Field work: in depth interviews, semi structured interviews, case studies, surveys (flexible) PROCES EVALUATION Methodology and information sources

Dissemination Request of affiliation Selection of beneficiaries Production of goods or services Distribution of goods or services EVALUATION OF PROCESSES Evaluation General scheme Delivery of goods or services Processes/Attributes Effectiveness Opportunity Sufficiency Relevance Monitoring/ Follow up on beneficiaries

EVALUATION OF PROCESSES Instruments for collecting information and types of analysis

EVALUATION OF PROCESSES Expected Results

TERMS OF REFERENCE TESTING