Performance Management Planning Development Grants Program (DGP) Detailed Implementation and Management Planning (DIMP) Workshop PART 2 Dakar, Senegal January 2010

Session V What is an Indicator?

What is a Performance Indicator ? (ADS Def. p 67) 3 A particular characteristic or dimension used to measure intended changes defined by a Result Framework. … A phenomenon we observe that tells us there is a problem and whether it is being solved. The measurement that shows how and by how much a project is achieving its intended results.

An example of an Indicator 4 Body temperature - an indicator that may show that illness is present. In order to use this indicator to diagnose illness, it is necessary to know, at a minimum, how far above the normal level the temperature is.

Role of Indicators in a Results Framework 5 Indicators apply at all levels of the project hierarchy of results. Indicators are used to measure change over time. The result statement will show the direction of change desired (increase or decrease) NOT the indicator. Example: Percentage of children ages 12 to 24 months who are fully vaccinated (against the five vaccine-preventable diseases) in the target area. This indicator tells us who, what, and where to observe, not which way the change might move.

Context Indicators Reality surrounding a development problem is usually greater than USAID’s manageable interest. Example: Jobs for victims of trafficking or domestic violence are more difficult to find when unemployment is increasing. Tracking unemployment as a context indicator informs the analysis of USAID efforts to reinsert victims into society. 6

If a Result is the amount of change achieved or a problem solved: What is a Problem? A discrepancy in somebody’s head between a Perception (based on current data) and an Ideal (based on ideal data) floating on an Emotion supported by Values 7 P(cd) / I(id) E V Reality Data

To formulate Indicators for a Results Framework, ask these key questions 1.How do we know it is a problem? = Data  What data generated the problem through which of the 5 senses? 2.What phenomenon are we observing? = Indicator  What will we observe to see whether the problem is solved? 8

Table Exercise 3 9 Each table receives an envelope with cards. On each card is a problem statement or an indicator. Instructions: Match the indicators to the appropriate problem statement and post the matches on a flipchart page. 10 min.

10 The loss of tropical glaciers in the Andes impacts the hydrologic cycle and raises grave concerns over future water supplies throughout the region (Barnett et al., 2005). The chronic lack of planning capacity, linkages, and communication among and between stakeholder groups is resulting in an increase in conflict between lowland and highland populations in Peru and other Andean countries (Oré 2005). Tropical glaciers loss rate Number of years of glacial water supply remaining Number of water resources management-related conflicts Problem Statements…Matched with Indicators

11 Coastal regions are home to nearly 70 per cent of the country’s population and that produce 60 per cent of its agricultural GDP Water from the Santa watershed also powers three hydroelectric stations with a total capacity of 271 MW, or 5.5 per cent of the total national hydroelectric output for Peru, that directly and indirectly serves 1,697,221 people throughout the region percentage of country population in coastal regions percentage of agricultural GDP of coastal regions Ratio of current capacity to original capacity of three hydroelectric stations

12 Water resources management in the watershed basins of Santa/Piura Chira improved Indicators : 1.Tropical glaciers loss rate 2.Number of years of glacial water supply remaining 3.Number of water resources management-related conflicts 4.Percentage of country population in coastal regions 5.Percentage of agricultural GDP of coastal regions 6.Ratio of current capacity to original capacity of three hydroelectric stations

13 Decision-making processes function effectively # of active highland/lowland decision making processes Politically balanced decisions made # of consensus decisions reached Conflicts over water avoided # of water related conflicts Investment in improved water resources management increased $ value of qualified project indicators Political will to change water management decision making generated # of working groups formed Good representation at the conference secured % of targeted stakeholders who attend Peru Case: Indicators with Associated IRs

NGO Task V: Creating Indicators for the Results Framework Using your Problem Description/Analysis in your Cooperative Agreement, create one or more indicators for your Project Result and each IR. Write each indicator on a card and place it under the corresponding result in your Results Framework. Ask these questions: How do we know it is a problem? Ask this question until you get to the data that generated the problem in your heads. What phenomenon are we observing that generated the data? Ask this question to define the indicator(s) Add any informative Context Indicators. 45 min 14

Session VI What are criteria for good Indicators ?

Indicator Job Description and Criteria Objective Practical Useful for Management Direct Attributable to USAID efforts Timely Adequate (ADS – p 16-17) 16 Job Description: To tell the most truth about whether a result is being achieved at the least cost when you need to hear it. USAID’s characteristics of good performance indicators:

Objective? Unambiguous and operationally precise about What is being measured What data are being collected Uni-dimensional—measures only one thing Consistent over time 17 Result: Performance of CSOs active in local governance improved Imprecise Indicator: Number of successful democracy-building CSOs Precise Indicator: Number of CSOs that achieve at least 1 measurable objective in increased government transparency.

Practical? Data are available when required for decision making. Data can be obtained at reasonable cost. 18

Useful? Is the indicator useful for management? Which indicators are most meaningful at a given point in time? 19

Direct? The indicator closely tracks the result it is intended to measure. 20 Result: Citizens’ knowledge of their rights expanded Direct Indicator: % survey respondents able to identify 3 or more key civil rights

Proxy Indicators Indirect measures of the intended result. Use if data for direct indicator is unavailable or not feasible to collect. 21 Result: Citizens’ knowledge of their rights expanded Direct Indicator: % survey respondents able to identify 3 or more key civil rights Proxy Indicator: Number of civil rights cases brought to court by targeted community organizations

Attributable? Indicators selected for the PMP should measure changes that are clearly and reasonably attributable, at least in part, to USAID effort. Attribution exists when the links between USAID outputs and the results being measured are clear and significant. A simple way to assess attribution: 22 If there had been no USAID activity, would the measured change have been different?

Timely? Indicators should be available when they are needed for decision making. Are the data available frequently enough? Are the data current? 23

Adequate? Taken as a group, the indicator and its companion indicators should be the minimum necessary to capture progress towards the desired result. How many indicators? As many as are necessary and cost effective for management purposes. Too many indicators results in information overload Too few indicators could be misleading Use the “Necessary and Sufficient Rule” to select the minimum number per result 24 RULE OF THUMB 2-3 PER RESULT

Indicators 25 Quantitative: Number, amount, ratio, percentage, proportion, average scores, rating, weighted or non-weighted index, etc. Qualitative: Description of the status of an intended achievement, documented observations, representative case descriptions, perceptions, opinions of value, photos, etc.

Indicators: Quantitative, Qualitative or Both? 26 QUALITATIVE Expert opinion on comprehensiveness of a law QUANTITATIVE Dollar value of plantains exported BOTH  Country score on “Corruption Perceptions Index”

Quantifying Qualitative Indicators (ADS p 14) Milestone Scales Rating Scales Indexes Examples? 27

28 Example of Indicators at Different Levels  Reduced incidence of diphtheria, pertussis and tetanus in young children  No. of children vaccinated against diphtheria (D)  No. of children vaccinated against pertussis (P)  No. of children vaccinated against tetanus (T)  No. of vaccines provided to clinics  Decline in the under-five mortality rate Input/Activity Output Outcome Impact

29 Disaggregation and Gender  ADS (p 17-18): Performance management systems and evaluations at the AO and project or activity levels must include gender-sensitive indicators and sex- disaggregated data when the technical analysis supporting the AO, project or activity to be undertaken demonstrate that: The activities or their anticipated results involve or affect women and men differently; and If so, this difference would be an important factor in managing for sustainable program impact.

30 Table quiz: Selecting indicators (“x” rejection criteria) Result: Citizens’ awareness increased DirectObjectiveUsefulPracticalAttrib.TimelyAdequate 1. No. of people trained 2. Ratio of people trained/target population 3. % of people trained to target population 4. Ratio of cumulative No. of people trained to target population 5. No. of courses delivered 6. Average pre- and post-test scores of citizen awareness 7. Random focus group pre- and post-test scores of citizen awareness 8. Key informant interviews 9. No. 4 and No. 7

31 Water resources management in the watershed basins of Santa/Piura Chira improved 1.Tropical glaciers loss rate 2.Number of years of glacial water supply remaining 3.Number of water resources management-related conflicts 4.% of country population in coastal regions 5.% of agricultural GDP of coastal regions 6.Ratio of current capacity to original capacity of three hydroelectric stations Water resources management improved # of WRM investments validated both scientifically and politically Politically balanced decisions made # of consensus decisions reached Investment in improved water resources management increased $ value of qualified projects Conflicts over water avoided # of water related conflicts Political will to change water management decision making generated # of working groups formed Effective decision making processes function # of active highland/lowland decision-making processes Stakeholder capacity increased # of shared planning tools used in decision making Good representation at the conference secured % of targeted stakeholders who attend ?

Indic/CriteriaObjectivePract.UsefulDirectAttribTimelyAdeq. Assessing Performance Indicators Against Criteria Result: Water resources management in the watershed basins of Santa/Piura Chira improved 32 Tropical glaciers loss rate Number of years of glacial water supply remaining Number of water resources management-related conflicts % of country population in coastal regions % of agricultural GDP of coastal regions Ratio of current capacity to original capacity of three hydroelectric stations YES? NO ? YESNO ? YES NOYES? NO ? YESNO ? YES NO YES?

33 Indicator/CriteriaObjectivePract.UsefulDirectAttrib.TimelyAdeq. Assessing Performance Indicators Against Criteria Result: Water resources management in the watershed basins of Santa/Piura Chira improved Number of water resource management investments validated both scientifically and politically Number of water resources management-related conflicts (context indicator) Number of Hectares (HA) classified as “conserved” Number of Hectares (HA) classified as “restored” YES NOYES

34 Context Indicators 1. # of water resources management -related conflicts 2. Rate of glacial melt Result: Water resources management in the watershed basins of Santa/Piura Chira improved 1. Number of Hectares (HA) classified as “restored” 2. Number of HA classified as “conserved” Critical Assumptions 1. National policy supports watershed basin regional planning IR1: Models for restoration/conservation validated 1. Number of models replicated 2. Number of replications IR2: Water management decision making restructured 1. Cases of highland/lowland decision making processes functioning IR3: Investment in improved water resources managem. increased 1. $ value of qualified projects Peru Case Indicators

How do the Indicators fit in? 35

NGO TASK VI: Selecting Good Performance Indicators Using a Performance Indicator worksheet: 1.Select one Result with multiple Indicators 2.Use the table to put your indicators on the left with the 7 standards across the top. Do your indicators meet all the criteria? 3.Fill in the boxes with “Yes” or “No”. 4.Anything need to be changed? 45 min. 36

Session VII What is included in the resume of an Indicator?

Indicator Resume Sections Description (What is it made of?) Plan for data acquisition (How are data born and who are the midwives?) Data quality issues (Does it have any defects?) Plan for data analysis, review and reporting (What is its job in life? How does it do it?) Baselines, targets and actuals (Expected/actual performance on the job?) 38

ADS Requirements met by a completed PIRS (p. 11) Full set of performance indicators (1 per result in RF) Base line and targeted values (Disaggregated by sex if indicated) Source and method for data collection, schedule, data limitations, DQA procedures, estimated cost 39

Indicator Description Tell the most truth about the related result when we need to hear it at the least possible cost. Let’s review the Indicator description (PIRS) in the Peru case to see what is included. Does it do the job? 40

41 NGO TASK VII: Completing the Sections of a PIRS 1.Select one indicator from your previous Task VI final list 2.Review the PIRS instructions and apply them to completing the following sections on a flip chart or electronic form Indicator Precise Definition Unit of Measure Disaggregated by Justification y Management Utility Data Collection Method Data Source *The test is: Your colleagues will imagine themselves taking your PIRS out the door to take the measure: Where to go? What to observe? What tools to use? How to record? Working Time: 50 minutes

Session VIII How do I set a Baseline and Targets for an Indicator?

Performance Baseline The value of the performance indicator at the beginning of the planning period. Baselines can/should be: Set just prior to the implementation of USAID-supported activities that contribute to the achievement of the relevant Project Result(s) Measured using the same data collection method that the NGO will use to assess progress (PIRS) Changed if the data collection method changes Zero if measuring direct Project Result(s) 43

Performance Target(s) Commitments made by the NGO about the level and timing of results to be achieved in a specified time period. Targets: Can be expressed in quantity, quality or efficiency May be determined by setting final target first, then interim targets May need to be set after activities or sites are selected Can be adjusted over time Should be realistic! Should be outside the margin of error of historical trend “ If you don’t know where you’re going, you’ll end up somewhere else. ” - Yogi Berra 44

Role of Baselines and Targets in Performance Management Pre-selected indicators must report data against targets compared to a baseline. The data are produced by implementation of a PMP based on a Development Hypothesis expressed by a Results Framework with corresponding indicators. 45

46 When setting targets, review: Baseline (condition before intervention) Historical trends Expert judgments Research findings Achievements of similar programs elsewhere Stakeholder expectations Objectives and Results Frameworks Prospective budgets WORKPLAN ACTIVITIES: Review Case Example Matrix and Summary Data Table How Do We Set Targets?

47 NGO TASK VIII: Defining a Baseline and Targets for at least one Indicator Use Result and Indicators from Task VI “Select good performance Indicators” Fill in the form, writing Result at the top and Indicators across the table List the tasks or activities needed to achieve the Result from Work Plan Fill in the time table for those tasks or activities indicating quarters when they occur Set targets for 2-3 years and fill in the Summary Data Table (don’t forget to identify the person responsible) Working Time: 45 min

Session IX What are the next steps to complete and use the PMP?

49 Contents of a PMP Development Hypothesis Results Framework Performance Indicator Reference Sheets Performance Data Table Calendar of Performance Management Tasks

Performance Data Table 50 PIRSs Context? Assumptions? Risks? Critical Assumptions Calendar of Performance Management Tasks

Calendar: Performance Management Tasks (Peru case) 51 Calendar of Performance Management Tasks Vet draft PMP with key Stakeholders and achieve staff consensus on final x 2. Submit PMP to USAID – Renegotiate Quarterly Report Formatx 3. Negotiate with USAID on “F” indicators to be incorporatedx 4. Train staff in data collection methods and reporting responsibilities x 5. Design and train staff in Excel trend graph generation – common drive x 6. Establish all baselines and enter in Excel common drivex 7. Do a data quality assessment on baseline datax 8. Collect data for Quarterly Reportsxxxxxxxxxxx 9. Submit Quarterly Reportsxxxxxxxxxxxx 10. Review data with key Stakeholders for feedbackxxxxxx 11. Provide data plus additional information for USAID Portfolio Review xxxxxx 12. Survey Municipalities: fugitive dataxxx 12. Review/revise PMPxx 13. Conduct midterm evaluationx 14. Conduct final evaluationx

52 NGO TASK IX: Creating a Calendar of Performance Management Tasks 1.Identify tasks needed to complete the PMP or refine it 2.Identify additional tasks needed to implement the PMP Measure baselines Measure results Analyze data Report 3.Consult your AOTR on USAID requirements that could change your PMP, including its Calendar Performance Management Tasks 4.List the Tasks generated from the above three steps as your Calendar Performance Management Tasks and complete the time table

Session X Workshop Review and Summary

54 PMP workshop design road map Cooperative Agreement/ Work Plan Project Causal Logic Results Framework Performance Indicator Reference Sheet (PIRS) Calendar Performance Management Tasks Baseline(s), Targets, Data Table Results Framework & Indicators Results Causal Logic Why are we doing that? Necessary + Sufficient Rule PIRS Instructions What is needed to complete or refine & implement PMP? Restatement + Criteria How do you know? + Indicator criteria Indicator/Workplan Matrix

55 Contents of a PMP Development Hypothesis Results Framework Performance Indicator Reference Sheets Performance Data Table Calendar of Performance Management Tasks ► ► ► ► ►

Summary Thoughts Reality doesn’t have any problems, we do. Reality doesn’t lie, we do. “The Lord God [reality] is very complicated; but not downright mean.” – Albert Einstein “No amount of data can prove me right; any amount can prove me wrong.” – Albert Einstein Blessed are those who know what they are doing; for they shall know whether they have done it. 56

Thank you for your attention and hard work!