1. Results-Based Programme Management Indicators and Evaluation
Trine Eriksen
Evaluation and Reporting, FMO
27 April 2012
First I will speak about the key elements of results based management, then I will talk a little bit about the use of evaluations and finally I will mention some of the key challenges when it comes to reporting on results in the new mechanism.

2. RESULTS BASED MANAGEMENT (RBM)
RBM is a management strategy focusing on performance and on achievement of outputs, outcomes and impact.
The concept of results management is a well established strategy. Alice in Wonderland was told that “If you don’t know where you are going, any road will take you there”. This lack of direction is what results management aims to fix. It’s about choosing a destination first, then deciding on the route, checking progress against a map and making adjustments, as required to achieve the desired results.

3. BRIEF HISTORY OF RBM
1960s to 1970s in private sector management by objectives
1980s to 2010 public sector reforms in many OECD countries in response to
budget deficits, value for money
lack of public confidence in government; and
demands for greater transparency and accountability
The concept has been applied in the private sector, starting in the 60s and 70s and in the public sector since the 80s. So this is not new, however during a financial crisis it becomes even more important. Where competition for limited resources is high, both national governments and donors want to get maximum value for money.
Historically, government departments-and implementing organizations focused their attention on inputs (what they spent), activities (what they did), and outputs (what they produced). Although accurate information at this level is important, they discovered it did not tell them whether or not they were making progress toward solving the problem they had set out to resolve, and that the problems often remained once projects were completed.
Along with results-based management, the issue of accountability has assumed increasing importance over the last decade.

4. PUBLIC SECTOR REFORM
Focus on performance issues and on achieving results;
Emphasis on participation and teamwork; and
Budget processes and financial systems focused on “value-for-money.”
Modern management requires that we look beyond activities and outputs to focus on actual results: the changes created, and contributed to, by our programming. By establishing clearly defined expected results, collecting information to assess progress toward them on a regular basis, and taking timely corrective action, practitioners can manage their programmes and projects in order to maximize achievement of development results
Results-based management (RBM) is a life-cycle approach to management that integrates strategy, people, resources, processes, and measurements to improve decision making, transparency, and accountability. The approach focuses on achieving outcomes, implementing performance measurement, learning, and adapting, as well as reporting performance.

5. RBM QUESTIONS What do we wish to achieve?
What will we do to achieve the results?
How will we know whether we have achieved the results?

6. KEY ELEMENTS OF RBM Results based Management Planning
Identifying clear expected results;
Selecting indicators to measure progress toward results;
Setting explicit targets for each indicator; and
Analysing assumptions and risks.
Results based Management
Developing performance monitoring systems;
Reviewing, analysing and reporting on results; and
Using evaluations for additional analysis of performance.
Planning
Performance measurement
Using performance information for internal management, accountability, learning and decision-making processes
There are certain elements that need to be in place in order to be able to report on results. If these elements have not been part of the programme planning phase it will be very challenging to after 2-3 years of programme implementation provide the evidence needed to prove that the results have been achieved.
These are the key elements of RBM in the planning phase and for performance measurement. I will go through each of them more in details.

7. DEFINING RESULTS
Managers need to define clear, measurable results at the project level (outputs, outcomes and impact);
Results need to be defined through a joint planning process; and
Stakeholder and beneficiaries should ideally be consulted so that it is clear that the project meets their needs.
We define results as an output, outcome or impact of an intervention. Outputs are normally quantitative results, products or services which are relevant for the achievements of outcomes. Outcomes represent the most important results-level in results management as these represents the intended effects on the target groups or systems. Impact are the long-term improvements in society and measuring impact represents major methodological challenges. I will get back to this later on.

8. Difference between output and outcome
Output = quantitative results, products or services which are relevant for the achievement of outcomes
Outcomes = the intended effects on the target groups/systems
The purpose of defining clear outcomes is for managers to stay focused on what ultimately matters: the effects of the intervention on people and systems
Output are normally quantitative results, products or services which are relevant for the achievement of outcomes
Output are possible to attribute to the activities performed
Outcomes are the intended effects on the target groups/systems
Outcomes may stem from factors both within and beyond the control of the programme
The purpose of defining clear outcomes is for managers to stay focused on what ultimately matters: the effects of the intervention on people and systems

9. KEY ELEMENTS OF RBM Results based Management Planning
Identifying clear expected results;
Selecting indicators to measure progress toward results;
Setting explicit targets for each indicator; and
Analysing assumptions and risks.
Results based Management
Developing performance monitoring systems;
Reviewing, analysing and reporting on results; and
Using evaluations for additional analysis of performance.
Planning
Performance measurement
Using performance information for internal management, accountability, learning and decision-making processes

10. INDICATORS TO MEASURE PROGRESS TOWARDS RESULTS
Performance indicators define a measure of change for the results identified;
Track the indicators to measure achievement of planned results over time; and
Use the indicators to assess the degree of results achieved and make programme/project adjustments.
Based on the results to be achieved, think of ways to measure progress;
Establish performance indicators and targets for each result;
Performance indicators define a measure of change for the results identified;
Track the indicators to measure achievement of planned results over time; and
Use the indicators to assess the degree of results achieved and make project
adjustments.

11. Performance indicators
A result identifies what we hope to accomplish while an indicator tell us by what standard that result will be measured.
Indicators are an indispensable management tool for making evidence based decisions.
Indicators define the data to be collected and enable actual results achieved to be compared with planned results.
Indicators are an indispensable management tool for making evidence based decisions.

12. Performance indicators
Within the EEA and Norway grants we have 2 types of outcome indicators that POs can use:
1. Optional indicators that should reflect progress within each unique programme, they are useful for the POs to report on programme results
2. Standard indicators (pre-defined) that are used primarily for Donor reporting purposes and can be aggregated across many similar programmes in different countries

13. KEY ELEMENTS OF RBM Results based Management Planning
Identifying clear expected results;
Selecting indicators to measure progress toward results;
Setting explicit targets for each indicator; and
Analysing assumptions and risks.
Results based Management
Developing performance monitoring systems;
Reviewing, analysing and reporting on results; and
Using evaluations for additional analysis of performance.
Planning
Performance measurement
Using performance information for internal management, accountability, learning and decision-making processes

14. BASELINES AND TARGETS
A baseline is the value of the indicator before the implementation of the project starts
Baseline helps identify the extent to which change has happened
Targets orients stakeholders to the tasks to be accomplished
Targets help establish clear expectations
Targets serve as the guideposts for monitoring whether progress is being made
Targets promote transparency
Baseline data form a set of conditions existing at the outset of a program/investment-the quantitative and qualitative data collected to establish a profile. Baseline data is collected at one point in time, and is used as a point of reference against which results will be measured or assessed. A baseline is needed for each performance indicator that will be used to measure results during the investment.
A target specifies a particular value for a performance indicator to be accomplished by a specific date in the future. It is what the investment would like to achieve within a certain period of time in relation to one of its expected results. Targets provide tangible and meaningful points of discussion with beneficiaries, stakeholders, and partners, and allow us to add further specificity to the outcomes from the results logic.

15. KEY ELEMENTS OF RBM Results based Management Planning
Identifying clear expected results;
Selecting indicators to measure progress toward results;
Setting explicit targets for each indicator; and
Analysing assumptions and risks.
Results based Management
Developing performance monitoring systems;
Reviewing, analysing and reporting on results; and
Using evaluations for additional analysis of performance.
Planning
Performance measurement
Using performance information for internal management, accountability, learning and decision-making processes

17. Analysing assumptions and risk
Risks are an expression of uncertainty
Assumptions and risks need to be analysed at each level of results during the project planning process;
Risks can be mitigated or their effects reduced through careful management and planning; and
Managers of implementation agencies usually have less control over risks at the outcome and impact levels.

18. KEY ELEMENTS OF RBM Results based Management Planning
Identifying clear expected results;
Selecting indicators to measure progress toward results;
Setting explicit targets for each indicator; and
Analysing assumptions and risks.
Results based Management
Developing performance monitoring systems;
Reviewing, analysing and reporting on results; and
Using evaluations for additional analysis of performance.
Planning
Performance measurement
Using performance information for internal management, accountability, learning and decision-making processes

19. Performance monitoring
Define processes and systems for performance measurement;
Decide the intervals at which measurement will occur;
Assign roles and responsibilities; and
Build in adequate time and resources for analysis of performance measurement data so that progress is reviewed and assessed.
All the programme proposals have a monitoring plan, this is where the PO should describe how they intend to monitor the results framework

20. KEY ELEMENTS OF RBM Results based Management Planning
Identifying clear expected results;
Selecting indicators to measure progress toward results;
Setting explicit targets for each indicator; and
Analysing assumptions and risks.
Results based Management
Developing performance monitoring systems;
Reviewing, analysing and reporting on results; and
Using evaluations for additional analysis of performance.
Planning
Performance measurement
Using performance information for internal management, accountability, learning and decision-making processes

21. Reporting requirements in annual report from Programme Operators
Programme outcome
Report and analyse status of the selected indicators for the expected programme outcome(s)
Analyse how the projects and the programme outputs have contributed to the expected programme outcomes.
Programme outputs
Report and analyse status of the selected indicators
Summerize and analyse how projects have contributed to the planned programme outputs
Outcome and outputs of bilateral relations
Report and assess achievemenet of outcome and outputs of bilateral relations
Assess risks and risk mitigation, cross cutting issues, sustainability, and outreach to target groups
Results-based reporting is one of the key challenges to reporting on results achieved. All too often, reports do not adequately tell
the story of the effects that their development interventions are having.
Results-based reporting seeks to shift attention away from activities to communicating important results that your programme or project has achieved at the output and outcome levels. An important aid is the results matrix, which clearly articulates the results at the output and outcome level and the indicators, baselines and targets.

22. Reporting requirements in annual strategic report from National Focal Points
The National Focal Point reports on results in the annual Strategic Report, which shall provide, ref Regulation Article 2.2:
An assessment of the contribution of the Financial Mechanism (FM) towards the reduction of social and economic disparities in the Beneficiary State.
Assess relevance of FMs in a national socio-economic context
An assessment of the contribution of the FM towards strengthening of bilateral relations with the Donor States
A summary of actual outputs and an assessment of progress towards expected outcomes for each programme.
The donors through the FMO will annually report on the same.
A challenge will be to report on results, rather than activities
The National Focal Points are depending on the POs to report on outcomes in order for them to be able to report on progress towards the programmes contribution to the overall objectives.

23. KEY ELEMENTS OF RBM Results based Management Planning
Identifying clear expected results;
Selecting indicators to measure progress toward results;
Setting explicit targets for each indicator; and
Analysing assumptions and risks.
Results based Management
Developing performance monitoring systems;
Reviewing, analysing and reporting on results; and
Using evaluations for additional analysis of performance.
Planning
Performance measurement
Using performance information for internal management, accountability, learning and decision-making processes

24. Evaluations
Clarifying results during planning and internal monitoring prepares projects for effective evaluations.
Linking the results framework to the evaluation plan
In the evaluation context, RBM reflects the move away from focusing on evaluating the input side of a project (e.g., financial efficiency, use of human resources) to evaluating the output side, giving more weight to how a project affects the lives of its primary stakeholders.
Once the indicators, baselines and targets have been established it should be linked to a plan that supplies programme managers and other stakeholders with reliable information about what works and what doesn’t.
Within the FM, the donors conducted 13 evaluations of various sectors. There were some mid-term evaluations and some end evaluations. These were used primarily for learning purposes, so-called formative evaluations. Lessons learned were brought into the FM. Common for most of the project evaluations, were that the projects had been implemented in an efficient and effective manner and most activities went according to plan. Therefore one could assume that the effects on the final benefiscaries were positive. However, this could not be said with certainty because there was no evidence.
If 5000 people have been trained, we can assume that their capacities have improved and that they are able to do their jobs better, but how do we know? If 5000 students have benefitted from mobility, we can assume that this had positive effects that will make him/or her more employable, but how we do we know?

25. Evaluations Evaluations have three key functions:
(1) Utilization. As an input to provide decision-makers with knowledge and evidence about performance and good practices;
(2) Accountability. To donors, funders, political authorities, stakeholders and the general public, and
(3) Contribution. To institutional policymaking, development effectiveness and organizational effectiveness.
So, in essence, evaluations have three key functions.

26. KEY ELEMENTS OF RBM Results based Management Planning
Identifying clear expected results;
Selecting indicators to measure progress toward results;
Setting explicit targets for each indicator; and
Analysing assumptions and risks.
Results based Management
Developing performance monitoring systems;
Reviewing, analysing and reporting on results; and
Using evaluations for additional analysis of performance.
Planning
Performance measurement
Using performance information for internal management, accountability, learning and decision-making processes

27. Internal management Management commitment
Organisational culture conducive to learning
Stakeholder involvement
KISS!
A major precondition for successful results management is management commitment. The focus on outcomes must be supported from top political and or management level. Parliaments play an increasingly important role in demanding results and result information.
A results oriented institution is structured so that knowledge and learning from reporting and evaluations are used systematically in order to improve policy making. The focus on learning implies that flexibility in responding to the demands and needs of the target group/systems is encouraged.
Participation by all stakeholders is vital to set relevant and realistic goals.
Keep it simple and smart!

28. Regulation Annex 9, figure 1
This is the results framework in the Regulation of the EEA and Norway grants

29. RESULTS LOGIC Impact Widespread improvement in society Long term
Outcomes Intermediate effects on target groups and systems Medium term
Outputs Products and services delivered Short term
Activities Tasks transforming inputs to outputs Ongoing
Sometimes also called a "results chain," a results logic is a depiction of the causal or logical relationships between inputs, activities, outputs, and the outcomes of a given policy, program or investment. The results logic is divided into five levels: inputs, activities, outputs, outcomes, and impact. Each of these represents a distinct step in the causal logic of a policy, program, or investment. The bottom three levels (inputs, activities, and outputs) address the how of an investment, whereas the top two levels constitute the actual changes that take place: the results.
Inputs Financial, human & material resources Ongoing

30. Programme design which will contribute to the two overall objectives
which will contribute to the fulfilment of the objective
Contribute to
Objective
if the outputs are produced, then the expected outcome is likely to be achieved
Expected
outcomes
Likely to happen
if the activities take place, then the outputs will be produced
Should be able to guarantee that this will happen
Outputs
if the inputs are available, then the activities will take place
Activities
Inputs

31. The objectives hierarchy - an example from Scholarship
Enhanced human capital and knowledge base in the Beneficiary States (predefined)
Increased higher education students and staff mobility between Beneficiary and EEA states (=one predefined outcome)
1. Higher education students mobility increased with 20% in country X
2. Higher education staff mobility increased with 10% in country x (at least two self-defined outputs per outcome)
1. Agreement for mobility signed
2. Higher education students and staff identified 3.
Funds, personnel, expertise, exchange(budget)
Objective (impact)
Outcome(s)
Outputs
Activities
Inputs 31 31

32. Assumptions
The intervention logic is a cause-effect chain and an aim-means chain.
The chain implies certain assumptions between each level.
It is assumed that “certain events” need to occur at one level in order to achieve the desired effect at the next level.
Since it is not certain that these events happen, they represent uncertainties. Must be monitored and managed as part of the results management process.
NB: Especially important to check “outputs to outcome”; i.e. to assess the risks that outputs do not deliver the expected outcome!

33. Example of results logic from one programme

34. Another example

35. Achieving impact Reduction of economic and social disparities
in the European Economic Area
Strengthened bilateral relations
between the Donor States and the beneficiary States

36. Challenges
Reporting on results at the outcome level, rather than input and output
Aggregate results across Scholarship programmes in different countries
Evidence of results – choosing good performance indicators that says something about the effects of scholarship and student/staff mobility

37. Suggested EU response to these challenges
The LLP should have fewer objectives (more focus) that are related to indicators
There will be a “more result-oriented approach”
Assessments that include qualitative outcomes aimed at measuring effects on individual beneficiaries (young people, teachers, staff, youth workers), organisations and systems
Formal evaluations
Better communication on the achievements of the programmes

38. Tentative list of EU indicators
% of participants who have increased their key competences and/or their skills relevant for employability
% of organisations that have participated in the Programme and that have developed/adopted innovative methods
% of participants who have increased their language skills
Clearly, these are more qualitative indicators that requires surveys and interviews to be able to report on.

39. Thank you!

40. Workshop What do we wish to achieve?
- what results/effects would we like to see on:
- individuals,
- organisations
- systems
II. How will we know whether we have achieved the desired results/effects?
- what kind of information would be needed
- how do we collect it?
- how often?
The Blue book has pre-defined objectives and outcomes and the FMO has developed a list of standard indicators in consultation with various stakeholders, however neither of them are perfect and we need your brains, your input to think of a few good results that we want the scholarship programmes to contribute to and we need your input to think of ways to measure them. You are a group of well educated people and this is a challenge that you, more than anyone else are equipped to solve.
After the coffee break we will divide you into groups where you will discuss desired effects and suggestions for how to measure these.