1. Results-based Monitoring and Evaluation in Bank projects
HD Learning Week. November 6, 2006
Jody Zall Kusek & Mohamed Khatouri
World Bank

2. Traditional Management Approach
Session I- Overview of Results-Based Management Concepts
Managing for results. Use the information to improve decision making and steer development interventions towards clearly defined goals.
Results-Based Management Approach
Focus on the results obtained rather than just on the inputs used or the activities conducted.
Traditional Management Approach
Focused mainly on inputs and activities.
Better indicators + Monitoring = Better results
Which statement is accurate ?
2. High quality Project design (specific focus on outcomes) + High quality implementation = Better results
Results agenda. Focus on development outcomes to be achieved by Bank operations, which requires high quality project design and supervision.

3. Critical factors for defining results to be achieved
Session I- Overview of Results-Based Management Concepts
Critical factors for defining results to be achieved
Socio- economic context:
Results statement should reflect local needs and priorities
Local Capacity: Existing skills, leadership, and management capacity will impact on what can be implemented to achieve expected results
Resources: Level of resources will impact on what can realistically be achieved
Timetable: Results framework must identify the results (changes) to be achieved in the life of the program

4. Principles of Management for Development Results*
Session I- Overview of Results-Based Management Concepts
Principles of Management for Development Results*
1. Focusing the dialogue on results at all phases of the development process
5. Using results information for learning and decision making
2. Aligning programming, M&E with results
4. Managing for, not by, results
3. Keeping measurement & reporting simple & cost-effective
Tools for performance measurement to increase the effectiveness of development interventions
Results-based M&E
* Roundtable on Managing for Results in 2004

5. Why is Results-Based M&E Important?
Session I- Overview of Results-Based Management Concepts
Why is Results-Based M&E Important?
Improving internal management
Focusing interventions/reallocating inputs to achieve results. Programs focused are better able to demonstrate results over time.
Better reporting: Programs have effective M&E system develop results-oriented reporting
Marketing successes
Capturing lessons-learned: Showing general program progress & sharing practices and innovations.
Secure political & public support: “If you can demonstrate results, you can win public support “
Ensuring accountability
Preoccupation with results is a global issue : demonstrate results against money spent
Results are more important than processes: provide evidence that the program is producing longer term benefits.

6. Monitoring Session I- Overview of Results-Based Management Concepts
Clarifies program objectives
Monitoring
Link inputs and activities to results to be achieved
Translates results to be achieved into performance indicators
Periodically collect data on the indicators and compare actual results with target
Regular collection and reporting of information to track whether actual results are being achieved as planned
Reports progress and alerts management to problems in implementation
Indicator
Year
30%
40%
50% 1 2 3 4 5
20 %

7. Evaluation Session I- Overview of Results-Based Management Concepts
Analyses why intended results were or were not achieved
Evaluation
Explores unintended results
Assesses causal contributions of activities to results and validate hypothesis
Examines implementation processes
Analytical efforts to answer specific questions about performance of a program activities.
Oriented to answering WHY? And HOW?
Provides lessons learned and recommendations for improvement
With project
With out project
Indicator
Year
30%
40%
50% 1 2 3 4 5
15 %

8. Long-term Goal (Impact)
Session I- Overview of Results-Based Management Concepts
Results Building Blocs
Attribution Gap
PLANING FOR RESULTS
Long-term Goal (Impact)
Outcomes
Outputs
Activities
Inputs
Effects or behavior changes resulting from program outputs
Long-term, widespread improveme nt in society
Products and services to be used to simulate the achievement of results
Utilization of resources to generate products and services
Resources committed to program activities
Results
Implementation
Traditional M&E
Results-based M&E

9. Examples of Results Chain
Session I- Overview of Results-Based Management Concepts
Examples of Results Chain
Long-Term Goal (Impact)
Outcomes
Outputs
Increase literacy rates
Teachers trained
Text Books provided
Increased student completion rates
Education
Improved maternal mortality
Increased use of health clinics
Doctors hired
Health workers trained
Health
Increased population is food secure
Increased % of vulnerable population using saftey net programs
Increased coverage of crop-insurance programs
Social Protection and labor

10. Session I- Overview of Results-Based Management Concepts
Identify the sequence of inputs, activities, outputs, outcomes, and impacts
Example of Results Chain (Health Sector)
Information is available for parents about the importance of breast feeding
Children in Local Community healthier
Fewer children are having diarrhea diseases
Mothers Parents breast feeding rather than using formula
New funds available to implement a health project to reduce babies mortality rates
Implement information campaigns on the importance of breast feeding

11. Session I- Overview of Results-Based Management Concepts
Critical Elements of an Effective M&E System
Active involvement of customers and partners (planning, conducting, reviewing, & interpreting performance information)
Fully integrated into the government’s existing management systems at the central and local level
Responsive to information management and policy making of government institutions at the central and local level
Built around reporting requirements
M&E capacity in place for collecting, analyzing, and reporting performance information

12. Design Logic of Bank Operations
Session I- Overview of Results-Based Management Concepts
Design Logic of Bank Operations
Country’s National Poverty Reduction Strategy (PRSP) Provides a central framework for donor coordination
PRSP
Strategic Goals
Outcomes
Align Projects and Programs Results with
CAS & PRSP Results
Country Assistance Strategy (CAS) to support the national development strategy
CAS
AAA
Policy Dialogue
Operations Projects/Programs
Operationalize the CAS

13. Exercise 1. Develop Results Building Blocs
Session I- Overview of Results-Based Management Concepts
Exercise 1. Develop Results Building Blocs
Assignment
Break out into groups of 4-5 persons
Each group should develop a results building bloc (inputs, outputs, outcomes, & impacts) for at least 2 programs. Spend no more than 15 minutes per program
Select one case to report on in the plenary session.

14. Session II- Development of Results Framework
Program’s strategy for achieving specific objective
Identifies the Program Development Objective (PDO) and Intermediate Outcomes which are critical to achieve it
Illustrates the development hypothesis and the cause & effect relationships linking all levels of a program’s objectives
(IF a and b happen THEN c should happen)
An effective tool for:
Strategic clarity and planning & managing for results
Building consensus and ownership
Effective communication

15. Session II- Development of Results Framework
Causal Chain
Hypothesis. Results are caused by some interventions IF… THEN
IF we build health centers
THEN local population will use them to for health services
IF we train teachers on new pedagogic skills
THEN teachers will use these skills to improve the quality of education
Design with clear logic
Reduction in the child mortality
1. Determine the key development hypothesis & its contribution to higher order impacts
THEN incidence of preventable disease will be reduced
IF expand child immunizations

16. Session II- Development of Results Framework
Design with clear logic (Cont.)
2. Develop a hierarchy of objectives showing the causal linkages
3. Determine what is possible in the operation (program or project)
Reduction in Child Mortality
Expanded health care system
Improved nutrition
Incidence of preventable disease reduced
Increased family health practices
Increased immunizations
DESIGN. Project success depends on the development of the right activities & assumptions
Vaccines availability
Public awareness
Enhance regulations
IMPLEMENTATION. Each part of the results chain has a role to play in achieving the Project Development Objective
Expand facilities
Provide Funds

17. Session II- Development of Results Framework
Program Development Objective (PDO)
The PDO describes the effect that the program’s outputs will have on the beneficiaries in terms of changed behavior or improved performance. It defines the program’s success.
Focus on the outcome that the project can directly influence, given its duration, resources, and approach
Focus on the expected outcome for the primary beneficiaries
At the close of the program, what problem has been solved for the key recipient of the program outputs ?
The PDO outcomes are one step beyond the products of the activities.
Keep the PDO statement clear, precise, and stated as realistic results— not overly broad or too ambitious
PDO example: “To achieve improved, expanded, sustainable utilization of basic social and economic services and to support a governance system where local government and communities can gradually become mutually accountable.
Is the PDO precise ?

18. Session II- Development of Results Framework
Program Development Objective (Cont.)
Focus on what is going to be transformed (not what is going to be done)
Which outcome example describe what is going to be transformed?
Example 1: To provide grant financing to educate “at risk” groups in the risks of HIV/AIDS.
Example 2: To increase the awareness of the dangers of HIV/AIDS of “at risk” groups.
PDO Outcomes. Capture single outcome by outcome statement
Example: PDO objective
To expand access and improve quality of primary education throughout the country.
Outcomes to be achieved ?
……………… ………………

19. Session II- Development of Results Framework
Performance Indicators
“What gets measured gets done”
“If you don’t measure results, you can’t tell success from failure”
Indicator: A variable that provides accurate and reliable evidence about the achievement of a specific result
Work with stakeholders and technical experts to determine which indicators are most important
Avoid selection of too many indicators Less is Better!
Information about indicators should be easy to gather and useful for management decisions
Performance indicators should be identified at all levels of results chain. 7

20. Specific: measure as closely as possible what is intended to measure
Session II- Development of Results Framework
Criteria for Selecting Good Indicators
Performance indicators should be SMART:
Specific; Measurable; Attributable; Realistic; Targeted
Specific: measure as closely as possible what is intended to measure
Which indicator is more specific ?
Outcome Indicator
Parents insure that Increased utilization of clinics
Children get treatment for Increased use of malaria
Malaria drugs
Measurable: clear and unambiguous about what is measured
% of health centers without stock out of drugs x, y & z for more than a week at a time
Which indicator is measurable ?
2. % of health centers with availability of drugs 8

21. Attributable: attributable to the project’s or program’s efforts
Session II- Development of Results Framework
Criteria for Selecting Good Indicators (Cont.)
Attributable: attributable to the project’s or program’s efforts
Which indicator is attributable ?
1. Life expectancy
2. % of children fully immunized at 1 year
Realistic: data obtained at reasonable cost with enough frequency
1. HIV prevalence among year old pregnant women
Which indicator is more realistic?
2. HIV prevalence among the total population
Targeted: it should be specific about the targeted population/area
Which indicator is targeted ?
1. Percent increase in employment
2. Percent increase in employment of graduate of technical training center X in the first year after completion of training. 8

22. Intermediate Outcomes by Component Intermediate Outcome Indicators
Session II- Development of Results Framework
EXAMPLE
PDO Outcomes
Outcomes Indicators
PDO: Increased use of health facilities
% of under 5 yr covered by DPT immunization
Increased coverage of clinics providing immuniatio
% of health deliveries carried out in public health facilities (%)
Improved quality of doctors/nurses
Intermediate Outcomes by Component
Intermediate Outcome Indicators
Component I- Extension of Quality Health Services
% of facilities without 7-day stock outs of essential drugs (List of essential drugs defined)
Sustained availability of essential drugs
Component II. Development of Human resources
Reduced shortage of human resources
% of facilities with minimum staffing norms (List of minimum staffing defined)

23. Session II- Development of Results Framework
Each Indicator must have a M&E plan
Definition. Clear and precise definition of what will be measured and unity of measure.
Baseline & Targets
Baseline. Value of the indicator at the beginning of the program. Used as a point of comparison when measuring progress toward a specific result
Target values. The intended value of the indicator at the end of a specified point in time, against which actual results will be measured
Data Acquisition Method. Data source, frequency/schedule, and responsibility for its collection
Data Analysis & Reporting Method. Frequency/schedule, Analysis method, and responsibility of reporting.
Critical Assumptions. External factors that could significantly affect the achievement of the program results targets 7

24. Session II- Development of Results Framework
Example of a M&E plan for one indicator
Indicators
(Definition % unit)
Base Line & Target Values (/Year) BL
Data Collection & Reporting
Frequency/Schedule
Instrument (Method)
Responsible Party
% of health deliveries carried out in public health facilities (%): 25 28 30 35 40
Annually (June)
Routine administrative records
Ministry of Health 10

25. Session II- Development of Results Framework
Ways to Improve the development of Results Frameworks
High level outcomes
Impacts
Example:
Reduced maternal mortality
Intermediate outcomes which the program is expect to influence directly through its defined interventions during the program period.
Intermediate Outcomes
Define
Missing Middle
Outcomes
Example:
Birth attended by skilled staff
Processes, Policies, & Outputs
Interventions
Example:
Provide essential obstetrical health services

26. Example: Health Sector
Session II- Development of Results Framework
Example: Health Sector
Reduced Maternal Mortality
Hygiene and sanitation practices
Increased use of maternal health services
Improved access to family planning
Increased adoption of prevention methods
Availability of ITNs
Provision of contracep-tives and counseling to women and youth
Increased coverage of Antenatal, new born, emergency obstetric, & post-natal care
Improved Communi-cation networks & ambulances
Information on family planning available
Health services provided ( Heath workers trained, Health facilities built & rehabilitated, Pharmaceutical drugs purchased)

27. Session II- Development of Results Framework
Data Collection Methods
Rigor/Difficulty (Cost/Time/skills)
Formal Methods
Census
Impact Evaluation (Experimental Design)
Panel
Surveys
Direct Observation
Rapid Appraisal Methods
One-Time Survey
Community Interviews
Review of official records
Focus Group
Interviews
Validity Reliability Credibility
Key Informant Interviews
1. Issue to be examined
Statistical Emphasis
Choice depends
2. Quality of the information needed
3. Time frame in which information is needed
4. Cost

28. Session II- Development of Results Framework
Key Steps for planning an Evaluation
1. Clarify the evaluation purpose and audience
2. Identify the evaluation questions
3. Select appropriate methods based on the questions to answer
4. Prepare data collection and analysis plan
Example. Which data collection method is more appropriate in providing an answer to each of the following questions?
What % of women using health facilities?
Why didn’t more parents send their kids to school?
Did the CDD project contribute to the increase in income of local communities?

29. Session II- Development of Results Framework
Focus on outcomes all phase of the project cycle
PCN
Develop causal chain
Define appropriate PDO and outcomes to be achieved
Completion (ICR)
Review and validate the causal chain
Report on the Project outcomes
Design/PAD
Develop the results framework with SMART indicators
Develop the M&E plan with baselines & targets
Implementation (ISR)
Report on outputs & outcomes
Adjust project design (as necessary to achieve outcomes) 7

30. Session II- Development of Results Framework
Exercise 2. Assess the quality of the Project Development Objectives (PDO) & outcome indicators of selected projects
Assignment
Break out into groups of 4-5 person
Review the quality of the PDO statement and its corresponding outcome indicators for 2 projects
Report on one case in the plenary session