1. Programme Cycle Management
Managing for Results
Programme Cycle Management
Trine Eriksen and Inger K. Stoll
Financial Mechanism Office
November 2012

2. RESULTS BASED MANAGEMENT
What is Results Based Management (RBM) RBM is a management strategy by which all actors, contributing directly or indirectly to achieving a set of results, ensure that their processes, products and services contribute to the achievement of desired results (outputs, outcomes and impact)
What is a result and a risk?
In the context of the Grants: A result is the output, outcome or impact of a development intervention A risk is an event that may occur and impede the objective

3. Results achievement KEY CONCEPTS Beneficiary oriented
Managing for results and reduced risks

4. RESULTS BASED MANAGEMENT IN PRACTICE
1. Clearly identifying program beneficiaries and designing programs to meet their needs.
2. Defining realistic expected results, based on appropriate analysis.
3. Identifying and managing risks.
4. Monitoring progress towards results with the use of appropriate indicators.
5. Increasing knowledge by learning lessons and integrating them into decisions, and
6. Reporting on results achieved and the resources involved.

5. Part I Quality at entry - Basis for good management

6. The Programming Cycle and Continuous Feedback
Planning
Programming
Budgeting
Implementation
Monitoring
Evaluation
Reporting
Based on RBM’s principle of continuous feedback, the lessons learned during the implementation, monitoring, evaluation and reporting stages should inform and influence the planning, programming, and budgeting stages in the next cycle. The learning continues from cycle to cycle. This is an example of how RBM is not a one-time event, but a progression, and the implementation of RBM is an ongoing effort

7. The Programme and the Context
National/ sector area
National/ sector area
Impact
Outcome
Programme area
Programme area
Inputs
Outputs
Activities
Present situation
Future situation
Targets
Baseline

8. RESULTS CHAIN
Impact Widespread improvement in society /sector Long term
Outcomes Intermediate effects on target groups and systems Medium term
Outputs Products and services delivered Short term
Sometimes also called a "results chain," a results logic is a depiction of the causal or logical relationships between inputs, activities, outputs, and the outcomes of a given policy, program or investment. The aim is to:
Improve planning
To link interventions to results
For more systematic performance monitoring The results logic is divided into five levels: inputs, activities, outputs, outcomes, and impact. Each of these represents a distinct step in the causal logic of a policy, program, or investment. The bottom three levels (inputs, activities, and outputs) address the how of an investment, whereas the top two levels constitute the actual changes that take place: the results.
Activities Tasks transforming inputs to outputs Ongoing
Inputs Financial, human & material resources Ongoing

9. Identify main programme elements of EEA/N Grants
Objective (impact)
Widespread improvements in societ or a sector
32 predefined objectives
Outcome(s)
Intermediate effects on
target groups and systems
160 predefined outcomes
In DoRIS
data base
Outputs
Products and services
delivered
Self-defined (at least two outputs per outcome)
Activities
Tasks transforming inputs to outputs
Self-defined
Not in DoRIS
Inputs
Financial, human and material resources
Budget +
Budget in DoRIS

10. WHAT IS AN OUTCOME? Outcomes can occur on multiple levels:
Individual level;
Family or household;
Community or population group;
Organization;
System (involving multiple institutions or agencies); or
State.
Within a RBM framework, outcomes play a large role in expressing why organizations exist and do what they do. Through RBM’s focus on outcomes, we can direct our focus outwards on external changes, rather than focus inwards on internal factors, such as inputs, activities and outputs.

11. DIFFERENT TYPES OF OUTCOMES
The nature of the changes can vary as well. Outcomes can involve different types of changes:
Changes in perceptions, attitudes, feelings, beliefs, or values;
Knowledge;
Skills or level of functioning;
Behaviours or practices;
Conditions;
Status (such as socio-economic status, and status can reflect formal or informal status); or
Policies.
There are also different dimensions of outcomes:
An outcome related to improving the skills of young women enrolled in a training program reflects changes for individuals, involves changes in skill level, and reflects short-term changes;
An outcome related to increasing the employment of people in the lowest income bracket reflects changes for a population group, involves changes in employment conditions, and reflects medium-term changes; and
An outcome related to expanding the scope of national social security coverage reflects changes for a state, involves changes in policies, and reflects long-term changes.

12. Defining outcomes and outputs
Express an outcome and an output
as a positive condition.
A measurable, desired result: Wait list for a cancer operation has been reduced from 16 moths to 10 months at hospital x in y province
Avoid «double»objectives as these blur what will be measured.
in order to through

13. Cross-Cutting issues as outputs and targets
Good governance
Gender equality
Sustainable development: Environmental, Economic and Social
Cross-cutting issues are either reflected in the outputs or the indicators (targets), or assumptions.
If the programme substantially addresses cross-cutting issues, it is advisable to define outputs and targets that reflect the cross-cutting issues. This strengthens follow-up.

14. Programme logic which will contribute to the two overall objectives
which will contribute to the fulfilment of the objective
Contribute to
Objective
if the outputs are produced, then the expected outcome is likely to be achieved
Expected
outcomes
Likely to happen
if the activities take place, then the outputs will be produced
Should be able to guarantee that this will happen
A development programme is seen as a causally linked sequence of events.
These are described as above.
Outputs
Activities
if the inputs are available, then the activities will take place
Inputs

15. ASSUMPTIONS
The intervention logic is an aim-means chain and a cause-effect chain.
It is assumed that “certain events” need to occur at one level in order to achieve the desired effect at the next level.
It is not certain that these events happen; they represent uncertainties.

16. Outputs (deliverables)
Impact/Objective
There are
Assumptions
between each
level:
Identify
Assess
Monitor
Manage and Mitigate
Outcome
Outputs (deliverables)
Activities
Inputs

17. Identify important assumptions
Assumptions are worded as positive conditions (see objectives)
Assumptions are linked to the different levels of the Programme Matrix. Each level must contain the necessary and sufficient conditions for the next level above.
Start from bottom of matrix and work upwards
Are weighted according to importance and probability
Example: Scholarship recipients return to assigned jobs

18. Check the significance of the assumptions
1. Eliminate the obvious factors which are:
Not important for the outcome
Very likely to occur
2. Assess the probability of occurrence for the remaining factors:
a) Quite likely but not certain:
b) Not likely to occur (killing factor!)
Include the factor into the PM and make sure to:
Monitor it
Report changes
If possible influence it
Redesign the programme
c) If this is not possible:
Reject the programme proposal

19. RISKS
If important assumptions do no occur, they might represent a risk.
A risk is any event that could occur and adversely impact the achievement of results at all levels
Risks are an expression of uncertainty
Assumptions are expressed as a positive condition (a necessary event), whilst risks are expressed as negative conditions (un-wanted events)

20. Identify main programme elements
Objective (impact)
Widespread improvements in societ or a sector
Assumptions and risks
Outcome(s)
Intermediate effects on
target groups and systems
Outputs
Products and services
Delivered
Activities
Tasks transforming inputs to outputs
Inputs
Financial, human and material resources

21. Cross-Cutting issues as assumptions
Good governance
Gender equality
Sustainable development: Environmental, Economic and Social
Cross-cutting issues are either reflected in the outputs or the indicators (targets), or assumptions. If not taken care of they can represent a risk to the sustainability of the programme results. Therefore cross-cutting issues should be monitored in the same way as assumptions and risks.

22. The basic elements of the Programme Matrix
Impact (Objective)
Outcome
Outputs (deliverables)
Activities
Inputs
Assumptions describe the necessary conditions that must exist for the cause and effect relationships between the levels to occur.
If an assumption does not occur, it might represent a risk

23. The Matrix is the point of reference throughout the life of the programme
National/ sector area
National/ sector area
Programme implementation
Programme area
Programme area
Programme
Present situation
Future situation
Baseline

24. PART II Risk Management

25. Programmes and projects
PART III
Programmes and projects
Sometimes also called a "results chain," a results logic is a depiction of the causal or logical relationships between inputs, activities, outputs, and the outcomes of a given policy, program or investment. The aim is to:
Improve planning
To link interventions to results
For more systematic performance monitoring The results logic is divided into five levels: inputs, activities, outputs, outcomes, and impact. Each of these represents a distinct step in the causal logic of a policy, program, or investment. The bottom three levels (inputs, activities, and outputs) address the how of an investment, whereas the top two levels constitute the actual changes that take place: the results.

26. Programmes and projects Strategic approach Programme proposal
Pre-defined projects
Strategic
approach
Programme
proposal
Projects
Pre-defined projects

27. Programme design PROGRAMME OUTCOME PROGRAMME OUTPUTS Programme
OBJECTIVE
(INDICATORS)
(ASSUMPTIONS)
OUTCOME
INDICATORS
ASSUMPTIONS
OUTPUTS
PROGRAMME OUTCOME
Programme
PROGRAMME OUTPUTS
OBJECTIVE
(INDICATORS)
(ASSUMPTIONS)
OUTCOME
INDICATORS
ASSUMPTIONS
OUTPUTS
ACTIVITIES
INPUTS
OBJECTIVE
(INDICATORS)
(ASSUMPTIONS)
OUTCOME
INDICATORS
ASSUMPTIONS
OUTPUTS
ACTIVITIES
INPUTS
OBJECTIVE
(INDICATORS)
(ASSUMPTIONS)
OUTCOME
INDICATORS
ASSUMPTIONS
OUTPUTS
ACTIVITIES
INPUTS
Projects

28. Example – Renewable Energy Objective (pre-defined)
Expected outcome(s) (pre-defined)
Outputs
Types of projects
Modernised RES infrastructure
R&D on RES
Feasibility of RES mapped out in relation to local conditions
Training in RES planning competence
Plans/policy development
Windmills
Solar systems
Hydropower
Bioenergy
Awareness raising campaigns at local level
Train the trainers
Training courses for officials at regional level

29. EXAMPLE Gender-based violence
Objective (pre-defined)
Expected outcome(s) (pre-defined)
Outputs
Support to existing centers
Support to start new centers
Support to provide services for women at risk such as counseling, phone-lines
Training
Types of projects
New coordination center
Support for coordination activities
Training
Methodologies and guidelines development
Studies
Media campaign
Education activities
Curriculum development
Training of police and medical personnel at local and regional levels
Training for counselors
Support to start new counseling centers
Support to develop new standards and methodologies
Training
Manual developed

30. Call for proposals
Be clear about which outcomes and outputs you want the projects to contribute to. Which projects are necessary and sufficient? Project template in DoRIS:

31. Logical relationship

32. Basis for results measurement
PART IV
Basis for results measurement
Sometimes also called a "results chain," a results logic is a depiction of the causal or logical relationships between inputs, activities, outputs, and the outcomes of a given policy, program or investment. The aim is to:
Improve planning
To link interventions to results
For more systematic performance monitoring The results logic is divided into five levels: inputs, activities, outputs, outcomes, and impact. Each of these represents a distinct step in the causal logic of a policy, program, or investment. The bottom three levels (inputs, activities, and outputs) address the how of an investment, whereas the top two levels constitute the actual changes that take place: the results.

33. BASIS FOR RESULTS MEASUREMENT
Result based management

34. "The starting point for a results-oriented approach is ex ante setting of clear and measurable targets and outcome indicators. Indicators must be clearly interpretable, statistically validated, truly responsive and directly linked to policy intervention, and promptly collected and publicised."
From the conclusions of the 5th Cohesion Report, adopted by the European Commission in November 2010
QUALITY AT ENTRY!

35. A means to measure achievement, or to help assess a development
INDICATOR
A means to measure achievement, or to help assess a development

36. HOW SHOULD INDICATORS BE FORMULATED?
Once indicators have been identified, they should be formulated to reflect measurable change. Indicators frequently utilize the following metrics to depict measurable amounts:
Number;
Ratio;
Percentage; or
Rate;
Time
Indicators shall whenever relevant, be broken down per gender
In DoRIS, always specify unit of measurement!
LESS IS MORE

37. EXAMPLES OF INDICATORS
Example 1: Outcome: Improved health status of young children in region/district X
Indicator: Percentage of children under 6 years who
are immunized against x y z diseases in district A
Example 2: Outcome: Increased participation of worker’s
organisations in policy development
Indicator: Number of cases in which the submitted
comments of worker’s organisations on draft legislation
are reflected in final legislation  

38. INDICATORS
To describe how results will be measured, indicators must be specific, observable and come from reliable data.
Does the indicator provide the most direct evidence of the condition or result to be measured?
Is the indicator important, understood, and accepted by important stakeholders?
Is the indicator defined in the same way over time?

39. Are data for the indicator collected in the same way over time?
Will data be available for the indicator?
Will data be collected frequently enough?
Are data currently being collected?
If data are not currently collected, can cost effective instruments for data collection be developed quickly?

40. Indicators related to Bilateral Relations
All programmes should contribute to strengthened Bilateral Relations
Use the Bilateral guideline.
Four types of bilateral results expected:
Extent of cooperation
Shared results
Wider effects
Knowledge and mutual understanding

41. Indicators related to «Horizontal issues»
A human-rights based approach and concerns for horizontal issues like fighting discriminiation, racism, hate speech etc. should guide defining the outputs and the indicators.
Example: X% of members of Parliament are minorities.
Example: Incidences of hate speech has decreased in Y school

42. Programme matrix - indicators

43. Sources of Verification
When defining the indicators, consider the data sources.
Where can you easily get data that are reliable and timely?
Specify your sources of information
Assess where you can get easy access to data
Example
Ministry of Health’s statistics on cancer (health status of
population) and various statistics on cancer detection and
treatment services

44. Programme matrix – Sources of Verification

45. WHAT ARE BASELINES?
Baselines provide context for the setting of targets and capture the situation before a development intervention begins, or at the beginning of a time period that will be monitored and assessed.
Baselines describe the conditions prior to programming efforts. The present situation!
Essential to make credible and meaningful assessments on progress towards outcomes.
Baseline data form a set of conditions existing at the outset of a program/investment-the quantitative and qualitative data collected to establish a profile. Baseline data is collected at one point in time, and is used as a point of reference against which results will be measured or assessed. A baseline is needed for each performance indicator that will be used to measure results during the investment.
Depending on the nature of the indicator, baseline data can be gathered in a variety of ways, including:
analysing secondary data (such as national census data, household survey data, and other publicly available datasets);
reviewing records or information (from multilateral agencies, national administrative bodies, or internal records); and
administering surveys, questionnaires, assessments or interviews (conducted directly with target groups and constituents, or through intermediaries).

46. Programme matrix – Baseline data

47. WHAT ARE TARGETS?
Targets set the level for the magnitude of change expected by the end of a predetermined time period. They reflect what organizations commit themselves to achieve by the end of some duration of time.
Targets provide tangible and meaningful points of discussion with beneficiaries, stakeholders, and partners, and allow us to add further specificity to the outcomes from the results logic.
Note that the time period associated with targets may not necessarily match the entire duration of a development intervention or program, since the intervention or program may be ongoing. This is perfectly acceptable. Targets simply specify how much progress organizations anticipate making towards outcomes. In some cases, the time period for targets may be two years, as many organizations frequently use a biennium period for budgeting.
In setting targets, organizations should apply knowledge of trends or previous experiences related to the indicator. Targets need to be simultaneously ambitious enough and realistic enough; organizations need to strike a balance in wanting to improve performance over time and grounding projections in knowledge of previous performance. A thorough understanding of previous challenges and successes in implementation should inform the process of setting targets. This represents one small example of how the principle of continuous learning in RBM comes into play.

48. TARGETS
When you have defined the indicators and know the baseline situation, you set the targets.
What do you want the situation for the beneficiaries to be in 3 or 5 years?
Examples:
1. 60% of W 40+ called in for cancer test in Southern Region by year 2016.
We want to increase the % of W 40+ called in from 30% in 2010 to 60%
in 2016.
2. Time lag between detection of cancer to operation is not more than 3
months in 2016.

49. BASELINES AND TARGETS SUMMARY
A baseline is the value of the indicator before the implementation of the project starts
Targets orients stakeholders to the tasks to be accomplished
Targets help establish clear expectations
Targets serve as the guideposts for monitoring whether progress is being made
Targets promote transparency

50. Programme matrix - Targets43

51. Experience: The EU
In 2010 the EU evaluated the 4,131 impact indicators under the ERDF programmes. Among the programmes;
94% had final achievements;
58% had targets;
6% had baselines;
55% had targets and achievements;
5% had baselines, targets and achievement; and finally
0.5% had baselines but no targets and no achievements.
How can we assess impacts if we have no idea of the starting point?
Conclusion: Much of the data collected are not meaningful and do not represent the impact of the Structural Funds.

52. Part IV Quality at implementation
Monitoring
Reporting on results and risks
Evaluation

53. The Programming Cycle and Continuous Feedback
Planning
Programming
Budgeting
Implementation
Monitoring
Evaluation
Reporting
Based on RBM’s principle of continuous feedback, the lessons learned during the implementation, monitoring, evaluation and reporting stages should inform and influence the planning, programming, and budgeting stages in the next cycle. The learning continues from cycle to cycle. This is an example of how RBM is not a one-time event, but a progression, and the implementation of RBM is an ongoing effort

54. PERFORMANCE MONITORING
Define processes and systems for performance measurement;
Decide the intervals at which measurement will occur;
Assign roles and responsibilities;
Set standard requirements for project promoters
Build in adequate time and resources for analysis of performance measurement data so that progress is reviewed and assessed.
Use web site for communication with project promoters and
other stakeholders
All the programme proposals have a monitoring plan, this is where the PO should describe how they intend to monitor the results framework

55. Performance monitoring
Status regarding the output and outcome indicators is measured and reported towards the baseline situation (“where we were”), and also towards the future, desired situation (“the targets”).

56. The Matrix is the point of reference throughout the life of the programme
National/ sector area
National/ sector area
Programme implementation
Programme area
Programme area
Programme
Present situation
Future situation
Baseline

57. Programme performance framework

58. Reporting requirements in annual report from Programme Operators
Programme outcome
Report and analyse status of the selected indicators for the expected programme outcome(s)
Analyse how the projects and the programme outputs have contributed to the expected programme outcomes.
Programme outputs
Report and analyse status of the selected indicators
Summerize and analyse how projects have contributed to the planned programme outputs
Outcome and outputs of bilateral relations
Report and assess achievemenet of outcome and outputs of bilateral relations
Assess risks and risk mitigation, cross cutting issues, sustainability, and outreach to target groups
Results-based reporting is one of the key challenges to reporting on results achieved. All too often, reports do not adequately tell
the story of the effects that their development interventions are having.
Results-based reporting seeks to shift attention away from activities to communicating important results that your programme or project has achieved at the output and outcome levels. An important aid is the results matrix, which clearly articulates the results at the output and outcome level and the indicators, baselines and targets.

59. EVALUATIONS Evaluations have three key functions:
Learning: As an input to provide decision-makers
with knowledge and evidence about performance and good practices;
(2) Accountability: To donors, funders, political authorities, stakeholders and the general public, and
(3) Communication: Results for the beneficiaries
THANK YOU!
So, in essence, evaluations have three key functions.