1. SELECTED SLIDES FOR M&E GROUP

2. Evaluation is part of the
Management Cycle
Planning
Implementation
Evaluation 2

3. Complementary Functions
Without monitoring, evaluation cannot be done well.
Monitoring is necessary, but not sufficient for evaluation.
Monitoring facilitates evaluation; evaluation generates additional data collection and information.
An evaluation of an intervention will often lead to improvement of plans.
An evaluability exercise often results in modifying data collection for monitoring purposes. 3

4. Evaluation is answering
Whether we are Doing the Right Things
Rationale/justification
Client satisfaction
Whether we are Doing it Right
Effectiveness/coherence
Efficiency: optimizing resources
Whether there are Better Ways of Doing it
Alternatives
Good practices
Lessons learned 4

5. Monitoring Definition
A process of tracking or measuring what is happening.
It includes:
Performance monitoring - measuring progress in relation to an implementation plan;
Situation monitoring - measuring change in a condition or a set of conditions or lack thereof (e.g., changes in the situation of women or changes in the broader socio-economic country context).
Purpose
Measure progress according to plan.
Trigger timely adaptation of response, particularly in crises or unstable contexts;
Generate critical information for the evaluation;
Provide accountability for implementation; 5

6. Evaluation Purpose Supports decision-making
Improve programme performance;
Learning by doing;
Provide accountability for programme results.
Definition
Assess the contribution and worth of an intervention.
Judgment is based on criteria such as relevance, efficiency, effectiveness, impact and sustainability.
Evaluation may be conducted during an intervention (at end of phase or mid-point) and/or at the end of the intervention (ex post).
Generate lessons learned. 6

7. A result is a measurable or describable change arising from a cause and effect relationship.
What is a result?
Sustainable
We no longer have goals and objectives – but results. 7 7 7

8. A result is NOT the completion of activities!!
Two major elements:
1. Change
Measurable/ describable transformation
Of rights holders’ status
Of beliefs, behaviours and practices
In institutional performance
In quality & availability of Services and other development tools (products) – knowledge, skills
Notes:
As an introduction to RBM, emphasise that there are no absolutes in RBM – rather this is a rough typology to help them navigate – a guide – not a straightjacket.
Each of the columns of information appear in order, from left to right. The presenter introduces the results terminology from top to bottom (impact down to activity)
Next the arrows come in. The main message here is that there is a vertical logic of causality flowing between the levels.
Next the yellow column of examples emerge. These examples match the previous slide, so it is an opportunity to clarify again the levels and the causality
Next the “If – Then” flags appear. Again, emphasise the vertical, causal logic of results.
The white column emerges to provide the overall focus of change for each level of result.
The green column then emerges, emphasising the rough timeframe for levels of results.
Next the red triangle emerges. Message: that collective accountability for results increases as they move up the chain of results. At the outcome level, no single agency is likely to be able to guarantee a result. Rather, the collective efforts of several agencies and partners will be needed.
Last – the second level of outcome appears. This is the time to introduce the UN’s approach to RBM which has two levels of Outcome – UNDAF Outcome and contributing CP Outcomes. Point out that the second level of outcome is still an outcome level result (ie institutional/ behavioural; 5 years) BUT that it has a higher level of ambition.
Now, move to the next slide to show them how these results fit into the Results Matrix.
A result is NOT the completion of activities!! 8 8 8

9. 2. Causality
Cause and effect relationship between an action and the results achieved. “If-then” logic.
then
Strategic Result if
then
Programme Result if
Results hierarchies can be applied to any programme or project design. Sometimes the creation of hierarchies for different projects or programmes needs correction in order to ensure there is one coherent overall hierarchy for the entire country programme, and for the UNDAF.
Result
then
Result if 9 9 9

10. Hierarchy of Results Desired changes in:
Rights holders’ status - Impact
Beliefs, behaviour and practices – Outcome
Institutional performance – Outcome
Quality & availability of Services and other development tools (products) (knowledge, skills) – resulting from completion of activities of development interventions - Output
Development
bottlenecks 10

11. Changes in the lives of people: realisation of their rights
Types of Change
Impact
Changes in the lives of people: realisation of their rights
Institutional Change: values, laws – associated with institutional performance, new institutions
Behavioural change: new attitudes, practices
Outcomes
Outputs
Operational Change: products and services, knowledge, skills resulting from completion of activities of development interventions. 11 11

12. Definitions Impact Outcome Output
Increased Primary education completion;
Pupils with enhanced communication & life skills
BUT overloaded secondary education system
Positive and negative long-term effects on identifiable population groups produced by a development intervention, directly or indirectly, intended or unintended. .
Impact
e.g. Long lasting Changes in the situation (life) of
Claim Holders (Children, women, other vulnerable
groups) or of their environment
The intended or achieved short-term and medium-term effects of an intervention’s outputs, usually requiring the collective effort of partners. Outcomes represent changes in development conditions which occur between the completion of outputs and the achievement of impact.
Institutional Change: values, ethic, rules, laws, practices, performance
Or Cognitive & Behaviour change: resulting from knowledge, skills acquisition, practices (individual, society)
Increased retention (particularly of girls)
Increased enrolment (girls and boys)
Improved quality of teaching/learning
Outcome
Operational Change: products and services,
Knowledge, skills resulting from completion of
activities of development interventions
Two Primary schools in each of the focus districts
meet Child-Friendly School standard
Products & services resulting from completion of activities of development interventions.
Output 12

13. Definitions Impact Outcome Output
More children likely to be living in a protective, nurturing environment;
Positive and negative long-term effects on identifiable population groups produced by a development intervention, directly or indirectly, intended or unintended. .
Impact
e.g. Long lasting Changes in the situation (life) of
Claim Holders (Children, women, other vulnerable
groups) or of their environment
The intended or achieved short-term and medium-term effects of an intervention’s outputs, usually requiring the collective effort of partners. Outcomes represent changes in development conditions which occur between the completion of outputs and the achievement of impact.
Institutional Change: values, ethic, rules, laws, practices, performance
Or Cognitive & Behaviour change: resulting from knowledge, skills acquisition, practices (individual, society)
New protection policy adopted by government
Implementation institutions at national and
sub-national levels created/ strengthened
National protective environment increasingly in
line with recommendations of CRC
Outcome
Operational Change: products and services,
knowledge, skills resulting from completion of
activities of development interventions
Child protection policy drafted
Key decision makers convinced and committed
Products & services resulting from completion of activities of development interventions.
Output 13

14. Outputs: definition & key features
Outputs are deliverables
Operational changes: new skills or abilities, the availability of new products and services
Must be achieved within the programme period
Managers have a high degree of control
 If the result is mostly beyond the control or influence of the programme or project, it cannot be an output
Failure to deliver is failure of the programme or project
4 to 6 outputs per CP outcome
Unless under a joint programme, outputs are NOT collective results
Outputs is where the specific comparative advantages of different agencies emerge
Outputs are not completed activities – the represent the changes resulting from the completion of a set of activities
(eg. Training conducted; Workshop completed; Survey implemented – these are activities) 14

15. Is it mostly (>80%) in your control?
Test for “Outputs”:
Is it mostly (>80%) in your control?
Will you take singular accountability for it?
Will your funds be used to achieve this result fully?
If achieved, will you take the credit for it?
If not achieved, will you take the blame for it?
If the answer is yes to all five questions above, then it is an output. If not, think again. 15

16. CP Outcomes: Definition & key features
The 2nd LEVEL RESULTS of UN system cooperation
Represent changes in institutional performance or behaviours
A strategic contribution to the achievement of UNDAF outcomes
6 to 8 only per UNDAF outcome
Accountability may be single agency, but is increasingly shared
Produced by the combined effects of the CP outputs 16

17. Assumption Assumption Often quite unpredictable/ even more risk
Probable/ less risk
Less Probable/ more risk
Inputs Activities
Outputs
Outcomes
Impact
Change in realisation
of rights
Implementation partners
Plugging capacity gaps of
DB/ CH instts
Improved performance of
DB and/ or behavioural
change in DB/CH
Decreasing Management Control..
Increasing external influences
Increasing difficulty in demonstrating attribution
Attribution
Contribution 17 17

18. Joint or individual agency result at the outcome level
An UNDAF programme result at the outcome level. Achievement depends on the contribution of more than one partner
A strategic UNDAF result at the impact level. Achievement depends on joint UNDAF outcomes plus inputs from other partners
Result
Joint or individual agency result at the outcome level
Joint or individual agency results at the output level 18 18

19.      Partnership agreements needed? Advocacy needed?
Achievement of UNDAF results depends on
the successful delivery of all partners’ contributions
Partnership agreements needed?
Advocacy needed? 
Result X 
Result
Result 
Result X X  
Result
Result
Result
Result
Result X X
Result
In this example, the “light blue” agency doesn’t deliver (or the respective activities don’t take place). All the higher-level results are put into jeopardy.
Result
Result
Result
Result
Result
Result X X
Not delivered
Result
Result
Result
Result X X
Result
Result
Result
Result
Result 19 19

20. A hypothetical UNDAF Results Framework
UNDAF outcome
25% reduction in new HIV infections by 2018
Existence of reliable HIV surveillance system
HIV surveillance system designed
(WHO)
Enabling environment exists that promotes risk free behaviour
National and sub-national bodies exist that oversee implementation of AIDS prevention activities.
(UNDP)
All large employers have policy and campaigns on HIV in workplace
(ILO)
Voluntary and confidential testing and advice available
Health workers able to conduct VCT
(UNICEF)
Policy on VCT agreed among major stakeholders
(WHO)
90% of people practicing safe sexual behaviour
Universal access to condoms
Regular and adequate supply of condoms guaranteed
(UNFPA)
Policy on condoms agreed among major stakeholders
High-risk population groups know how to prevent HIV infection
(Europ.Union)
90% of young people know how to protect themselves
(UNICEF)
A UNDAF (United Nations Development Assistance Framework) can be visualized through a results framework. Initially, such a framework would be created without regard to the interests, mandates, or expertise of the participating organisations.
Once all the main results and next-level results have been identified, the work can be “distributed” to the participating agencies – including other donors and of course government agencies and any other development partners.
The Framework thus illustrates, how the different contributions of all partners help to achieve the UNDAF outcome (which again contributes to the achievement of a national goal or priority). The UNDAF outcome should, as much as possible, relate to a change in the lives of people.
In order to correctly identify the necessary “contributing” results, one has to relate to the causal analysis. Without a complete or sound causal analysis, the subsequent results framework is likely to remain incomplete, too.
It is not necessary, to develop the UNDAF Results Framework down to the activity level. This can be left to agencies when they develop their requisite country programme and projects.
An important question that ahs to be asked throughout is: Are the lower-level results sufficient to achieve the medium-level results? Are the medium-level results sufficient to achieve the higher level results? If, for instance, the higher-level results are insufficient to achieve the UNDAF results, we either have to add the “missing” higher-level results to the framework, or we have to “lower” (adjust) the UNDAF outcome.
CP-level outcomes
Multi-media campaign messages reach 90% of all people
(UNESCO)
CP-level outputs 20 20

21. Key Principle # 1: Language Action Vs Results Language
Action Language
services provider’s Perspectives:
Unclear success Criteria
Emphasis on Processes (activities)
Change Language
Rights holders/ Stakeholders’ perspective: They can internalize and own it
success Criteria: clearer
Emphasis on change (of capacities and status) 21

22. Refining Results Action Language Change Language
To strengthen the capacity of teachers to teach life skills through training on gender sensitive, child friendly, learner directed learning approaches
The capacity of teachers to teach life skills is strengthened
By 2010, the capacity of all life skills teachers in the 10 districts with the highest adolescent HIV incidence rates is strengthened
All teachers in the 10 districts with the highest adolescent HIV incidence rates are able to teach life skills to school children by 2010
Change Language
Results language to emphasise future condition and take out information relating to strategy or activities
2. How many? Everywhere? By the year 4000? Be more specific
Bring the subject of change to the front and shift from passive to active language 22 22

23. by undertaking a workshop
To strengthen the capacity of UNICEF staff to do RBP/M by undertaking a workshop…
Let’s use results language to emphasise the future condition we want to achieve.
The capacity of UNICEF staff is strengthened to do RBPM by undertaking a workshop…
All UNICEF staff everywhere? Can you be more specific? Are there particular staff we should emphasise?
The capacity of select UNICEF staff in ICO is strengthened to do RBPM by undertaking a workshop
We can take out information that relates to either strategy or activities.
The capacity of select UNICEF staff in in ICO is strengthened to do RBP/M
by undertaking a workshop
Now, let’s try bringing the subject of change to the front, and shifting from passive to active language and add time frame.
Select UNICEF staff in in ICO have the capacity to do RBPM by December 2009. 23 23

24. Quality Criteria for a Results statement
Is it stated using change language?
Is the scale/scope realistically within the control of you and your partners?
Is there an If-then causality between output and outcome and are sum of outputs sufficient to achieve higher level outcome?
Make sure output does not repeat the outcome statement and is not a completed activity (training conducted, workshop completed; survey implemented; law reviewed)
Make sure it is SMART – Specific, Measurable, Achievable, Relevant, Time bound
Take reference to strategy out of sentence
To what extent will the most vulnerable/disadvantaged participate in and benefit from the output;
Has the gender dimension been considered? 24 24

25. Checklist for INDICATORS
SMART – Are they Specific, Measurable, Affordable, Relevant and Time bound
Validity - Does it measure the result?
Reliability - Is it a consistent measure over time and, if supplied externally, will it continue to be available?
Sensitivity - When a change occurs will it be sensitive to those changes?
Simplicity - Will it be easy to collect and analyze the information?
Equality – Is the status or situation of women and men compared? Comparisons between ethnicity…geography?
Utility - Will the information be useful for decision-making and learning?
Affordability – Do we have the resources to collect the information? What baseline do we have? 25 25

26. Output in the context of human rights
How?
What do we want to change? Why?
Inputs Activities
Outputs Outcomes Impact
Change in realisation
of rights
Implementation partners
Plugging capacity gaps of
DB/ CH instts
Improved performance of
DB and/ or behavioural
change in DB/CH 26

27. CAPACITY ANALYSIS Components Authority
Responsibility/ motivation/ will to do
Resources
Triple A decision-making capability
Communication capability 27

28. The challenge of capacity development:
Working towards good practice (OECD paper )
Individual level
(experience, knowledge & technical skills)
Systemic
factors, i.e.,
relationships
between the
enabling
environment,
organizations
and individuals
Organizational level
(systems, procedures & rules)
Influences by
means of
incentives it
creates
Enabling environment
(institutional framework, power structure & influence)
Successful capacity development requires not only skills &
organizational procedures, but also incentives & good governance 28

29. UNDG Capacity Assessment Framework – a Three-Dimensional Cube (2008)29

30. The UN System and Capacity Development Background Paper (2006)
Don’t rush - long-term process.
Respect the value systems and foster self-esteem - self-esteem is at the root of capacity and empowerment.
Scan locally and globally; reinvent locally - there are no blueprints.
Challenge mindsets and power differentials - capacity development is not power neutral.
Think and act in terms of sustainable capacity outcomes. 30

31. The UN System and Capacity Development Background Paper (2006)…
Establish positive incentives.
Integrate external inputs into national priorities, processes and systems - external inputs need to correspond to real demand
Build on existing capacities rather than creating new ones - this implies the use of national expertise as prime option.
Stay engaged under difficult circumstances - the weaker the capacity the greater the need.
Remain accountable to ultimate claim holders - any responsible government is answerable to its people, and should foster transparency as the foremost instrument of public accountability. 31

32. Examples of Indicators Project Logical Framework
Goal
Impact indicators
Infant mortality rate,
under 5 mortality rate
Objectives
results
Results monitoring
Purpose
Outcome indicators
Number & % of child diarrhea
cases treated with ORT
Number & % of mothers knowing
about & having access to ORT services
Outputs
Output indicators
Activities
Process indicators
Number of health personnel trained in
ORT, number of media ads produced, etc
Strategies
means
Implementation monitoring
Funding amounts ($), trainers
(person-months), ORT supplies (numbers),
etc
Inputs
Input indicators
Project Logical Framework

33. University of Wisconsin-Extension, Program Development and Evaluation
Business Counseling Example
Improved business performance
These owners gain knowledge and change practices resulting in
A variety of educational activities are provided to business owners who participate
Let’s apply this to a typical Extension example
Agency invests time and resources 33
University of Wisconsin-Extension, Program Development and Evaluation 33 33

34. University of Wisconsin-Extension, Program Development and Evaluation
Example: Water quality 34
University of Wisconsin-Extension, Program Development and Evaluation 34 34 34

35. PROGRAM LEVEL RESULTS CHAIN
ATTRIBUTION
Macro Long term Low
Millennium Development Goals
Impacts
Country Program Results
Outcomes
Financial, Knowledge, Sector Results
Outputs
5. Evaluation challenges: methodological and institutional
This revisits several points so far and summarizes their methodological challenge.
Aggregation low to high
Attribution degrades as aggregation increases
Self evaluation less frequent at higher evaluation levels
(Bank just started at country level, naught at sector level)
Results chain—associated with short term, long term.
Many determinants.
This makes attribution more difficult.
Budget Resources
& Projects
Inputs
Micro Short term High 35 35

36. 36 36 36

37. Definitions Impact Outcome Output
Positive and negative, primary and secondary
long-term effects produced by a
development intervention, directly or indirectly,
intended or unintended.
Impact
Outcome
The likely or achieved short-term and
medium-term effects of an intervention’s
outputs.
Output
The products and services which result from the completion of activities within a development intervention.

38. Result Chain: Definitions
the overall and long-term effect of an intervention; it is the ultimate improvement/ changes in people’s lives; intended/ unintended e.g. Improved access to justicernance
Institutional and behavioral changes in the real world triggered by the outputs; outcomes are not under the control of the project any more (in contrast to outputs) e.g. Adherence to international human rights instruments and treaties.
Tangible, time-bound products resulting from completion of activities; are under the control of the project e.g. Policy makers, CSOs and judiciary officials trained and able to apply human rights treaties and instruments
activities are the various steps taken to carry out a project e.g. Material on raising awareness on human rights developed and translated into local languages
Organize training workshops
Example: workshop
inputs are means mobilized to carry out activities e.g. funds, staff time and other resources

39. Outcome Formulation: An example…
Action Statements:
1. Increase participation of vulnerable groups & their access to basic health
2. Enhance safe and equal access to quality education, health
3. Increase the safe and equal access to primary and secondary education
Basic Services
There is increased and equal access to quality basic health and education services
Action Statements:
1. Strengthen capacity to comply with EU/UN standards and environmental conventions
2. Support government/CSOs to develop and implement sound agricultural and environ. policies
3. Ensure sustainable use of natural resources
Environment
National capacities for sustainable management and protection of natural resources are strengthened
National and local capacities for transparent and accountable provision of decentralized services are strengthened
Action Statements:
1. Support people-oriented governance through decentralization and local capacity building
2. Build the capacity of government structures to be transparent and accountable
Good Governance

40. Changes in the lives of people
Types of Change
Changes in the lives of people
Impact
Outcome
Institutional Change: values, ethic, rules, laws – associated with to institutional performance, access..
Behavioural change: knowledge, skills, practices (individual level)
Output
Operational Change: provision of goods and services

41. To achieve results you need to..
Set strategic intents which provide a focus for action
Formulate “SMART” results
Identify baselines
Set intermediary targets
Select performance indicators
Be determined to do it. 41 41

42. Selection Criteria for Results
Specific M
Measurable A
Achievable
S: Specific (it should specify the nature of the change, the target group, the target region, etc)
M: Measurable (it can be measured by using indicators)
A: Achievable (it is realistic)
R: Relevant (it is an answer to the identified problem)
T: Time bound (it can be achieved in the time frame of the country programme) R
Relevant T
Time bound & Trackable

43. A Typology for RBM: Poverty Reduction (2)
Results
Like…
Focus @
Timeframe
more
less
Collective Accountability
then
Outcome
Impact
Output
Activity
Poverty reduced
Institutional/ Behavioural
Operational/ skills, abilities, products & services
Human!
<1 yr
<5 yrs
5 yrs
5-10 yrs if
then
Outcome
Employment and income generation increased
Institutional/ Behavioural
5 yrs if
then
3000 new small enterprises developed in poorest provinces
Notes:
As an introduction to RBM, emphasise that there are no absolutes in RBM – rather this is a rough typology to help them navigate – a guide – not a straightjacket.
Each of the columns of information appear in order, from left to right. The presenter introduces the results terminology from top to bottom (impact down to activity)
Next the arrows come in. The main message here is that there is a vertical logic of causality flowing between the levels.
Next the yellow column of examples emerge. These examples match the previous slide, so it is an opportunity to clarify again the levels and the causality
Next the “If – Then” flags appear. Again, emphasise the vertical, causal logic of results.
The white column emerges to provide the overall focus of change for each level of result.
The green column then emerges, emphasising the rough timeframe for levels of results.
Next the red triangle emerges. Message: that collective accountability for results increases as they move up the chain of results. At the outcome level, no single agency is likely to be able to guarantee a result. Rather, the collective efforts of several agencies and partners will be needed.
Last – the second level of outcome appears. This is the time to introduce the UN’s approach to RBM which has two levels of Outcome – UNDAF Outcome and contributing CP Outcomes. Point out that the second level of outcome is still an outcome level result (ie institutional/ behavioural; 5 years) BUT that it has a higher level of ambition.
Now, move to the next slide to show them how these results fit into the Results Matrix. if
then
7 model business incubators operational in poorest provinces if
Acquire facilities
Staff training
Micro-credit provision..

44. Annual Workplan Output 1 Output 2 Output 3 TOTAL Baseline: Indicators:
EXPECTED OUTPUTS
And baseline, indicators including annual targets
PLANNED ACTIVITIES
List activity results and associated actions
TIMEFRAME
RESPONSIBLE PARTY
PLANNED BUDGET Q1 Q2 Q3 Q4
Funding Source
Budget Description
Amount
Output 1
Baseline:
Indicators:
Targets:
Related CP outcome:
1. Activity Result
-Activity action
2. Activity Result
3. Activity Result
- Activity action
4. Activity Result
Output 2
Output 3
TOTAL

45. Harmonised AWP Monitoring Tool
Regional (or Country) Programme Component: _________________
Implementing Partners: _________________Year: ____
RP Outputs, indicators, Annual Targets
Planned Activity
(UNDP:
Delivera-bles)
Expenditures
(from FACE form or DEX expenditure)
Results (UNDP: Delivera-bles) of Activities
Progress towards achieving RP Outputs
Output 1
Output 2
Updating Risks, Issues & Lessons Learned

46. Result Chain Development hypothesis Control/no control
Attribution problem: many different activities contribute to one impact; no scientific way to prove effectiveness of UNDP’s intervention; but: logical, causal argumentation; EXAMPLE: UNDP’s support to Employment generation – because/despite UNDP’s intervention

47. What are Good Indicators?
SPECIFIC
In terms of quantity, quality, time, location, target groups, baseline and target for the indicator
MEASURABLE
Will the indicator show desirable change?
Is it a reliable and clear measure of results?
Is it sensitive to changes in policies & programmes?
Do stakeholders agree on exactly what to measure?
ACHIEVABLE
Are the result (s) realistic and based on risk assessment, partnership strategy and other factors contributing to the underlying result
RELEVANT
Is it relevant to the intended result?
Does it reflect the expectations and success criteria for change in the target groups?
TRACKABLE/TIMEBOUND
Are data actually available at reasonable cost & effort?
Can proxy indicators be used?
Are data sources known? M A R
OBJECTIVE:
- Wall painted in white T

48. (UNDG Harmonized Terminology, 2003)
What are Indicators?
A quantitative or qualitative variable that allows the verification of changes produced by a development intervention relative to what was planned
(UNDG Harmonized Terminology, 2003)
In other words ….
Means of measuring what actually happens against what has been planned in terms of quantity, quality and timeliness, for every level of result.

49. Indicators ‘indicate’ that change is happening or not happening.
Purpose of Indicators
Indicators ‘indicate’ that change is happening or not happening.
They can….
Clarify the scale and scope of a result in the results framework
Demonstrate progress when things go right
Provide early warning when things go wrong
Assist in identifying changes that need to be made in strategy and practice
Inform decision making
Facilitate effective evaluation

50. Baseline
A baseline is the situation just before, or at the outset of a new program, project, service or operation against which progress can be measure or comparisons can be made as part of monitoring and evaluation.
Effective monitoring is nearly impossible without an established baseline.
OBJECTIVE:
- Wall painted in white

51. Targets
A target is an explicit statement or result derived for an indicator over any specified time period (to be provided at the level of output, outcomes and impact).
OBJECTIVE:
- Wall painted in white