Methodological Proposal for the Evaluation of the Relevance and Scope of Indicators of Social Programs Gilberto Moncada Consultant Banco Mundial Noviembre 2008

1. Background Methodological proposal developed in the framework of Fees For Services with the National Evaluation Council of Mexico. The proposal has to be piloted with the indicators of different programs in order to contrast its utility in different situations and make refinements. The proposal can serve as a base for the development and application of a methodology for the evaluation of indicators in different contexts.

2. Objetive of the methodology To facilitate the measurement and evaluation of the relevance and scope of inidcators used by social programs, with the objective of refining/improving them. Are the selected indicatoirs the most adequate to measure the advances of social programs? How can we examine if these indicators are the best?

3. Focus of evaluation Indicators of social programs created through a logic framework process. Indicators at the level of: outcome, objective and component.

4. Area of evaluation Dimensions of evaluation Categories to be evaluatedVariables General Variables related to basic capacities of indicators Consensus in the formualtion: information capacity, dissimination and access. Specific aspects of basic capacities of indicators as defined by a guide of questions Quality of the indicator CREAM criteria: clear, relevant, economic, adequate, monitorable. Specific aspects of the quality of an indicator as defined by a guide of questions. Use of the indicator Specific aspects of the use of an indicator as defined by a guide of questions.

5. Method 5.1 Guide of Questions 5.2 Information Collection 5.3 Question Scoring 5.4 Weights and Points 5.5 Metrics of evaluation

5.1 Method: Guide of Questions The evaluation should follow a guide of questions related to the variables. It is proposed that the variables can relate to a group of indicators and/or single indicators. Each dimension of evaluation will be evaluated on the basis of the results of the questions.

5.2 Method: Collection of Information Desk review of key documents( matrix of indicators, evaluation reports, registers, normative documents, benficiary information, databases etc.) In depth interviews with operators of the program, superior authorities, external and internal users of the indicators of a program. The guide of questions should be answered by the evaluation team based on the information obtained.

5.3 Method: Question Scores Every question has response alternatives which represent a scale:  Maximum score is 1;  Second is 0.5;  Third is 0.25;  Lowest score is 0. Some question are binomial: yes/no (value of 1 or 0). Each response must be justified clearly and concisely, with evidence. If evaluating a group of indicators rather than an individual indicator (in for example, basic capacities) is indicator is given the corresponding score.

5.4 Method: Weights and Points

Calculation of weights It is proposed to determine the relative weights of the different dimensions of evaluation (general variables, quality and use), as will as the crtieria in each dimension (CREAM etc), through a process of consulting national experts. The experts should use a scoring table, using a scale of 1 to 10 to indicate the value they would attach to each dimension and criteria given their experience. The results obtained through this consultation can serve as the basis of determining the average weight of each dimension and criteria.

5.5 Method: Metric to Evaluate Indicator Example of the calculation for the criteria: quality

Score of indicators It is proposed to consider three rankings for each dimension using three equal ranges.  Low quality/use/general capacity0 – 33.3  Medium quality/use/general capacity33.4 – 66.7  High quality/use/general capacity66.8 – 100 The total score of the indicators in terms of its relevance and scope is found estimating the average weight of each dimension.  Low relevance and scope0 – 33.3  Medium relevance and scope 33.4 – 66.7  High relevance and scope 66.8 – 100

6. Methodology to evaluate the quality of indicators at the level of activitiy Proposal is to use a guide of 10 questions that explore the basic characteristics of an indicator. Binomial reponses (Yes/No) which represent a values of 0 and 1. The points recievd by each indicator are determined by the following ranking:  Low Quality0 – 3  Medium Quality4 – 6  High Quality7 – 10

Muchas gracias