1. Title Abel Schumann, Economist
Public Governance and Territorial Development Directorate
OECD

2. Why is policy monitoring important?
Obtain a better understanding of current conditions
Learn about the effectiveness of policies
Create transparency and build a factual basis for policy debates
Create accountability by setting objective targets
Foster learning and capacity building by obtaining regular feedback
Enable incentive-based performance frameworks

3. Three main types of Indicators
Outcome/Result Indicators (Monitor Effectiveness)
Measure what results are achieved by the outputs
Output Indicators (Monitor Efficiency)
Measure what policies produce by using inputs
Input Indicators (Monitor Effort)
Measure resources spent on policies (money, staff, time,…)
Key messages:
Crucial to distinguish indicators according to type
All 3 types of indicators have their justification
But outcome indicators are arguably the most important ones because they measure if a policy achieves its objectives
The graph shows that:
Input indicators measure the effort in terms of resources that is expended by a policy
Output indicators monitor the efficiency with which inputs are converted into desired outputs
Outcomes monitor whether the outputs are effective in achieving the motivation behind policy

4. What are the main differences between Outputs and Outcomes/Results?
What a policy produces to achieve its objectives
Outcomes:
The conditions that are supposed to be altered by a policy
Can be directly determined by a policy
No uncertainty what outputs are produced
Can be affected only by producing outputs
Never perfectly certain in advance what outcomes are caused by a policy
Key messages:
Outcomes can only indirectly be influenced by policy
I.e. it is possible to hire teachers to improve learning outcomes, but there is no policy that can decree better learning outcomes directly
It is usually certain what outputs will be created by a policy (although not their quality and their quantity)
It is almost never certain if an outcome will be caused by a policy  Therefore the need for outcome indicators
Additionally hired teachers
Test scores of students
New buses for public transport
Share of people using public transport

5. Clear policy objectives are a prerequisite for outcome indicators
Without objectives it is unclear what indicators should measure
The more precise objectives are about outputs and outcomes, the easier the development of indicators! 
“Increase employment among single mothers by extending childcare provision” 
Key messages:
Explicit policy objectives that clearly specify desired outcomes and outputs of policies make it easier to develop indicators. If outcomes are quantifiable, the respective indicator follows directly from the objective.
Notes:
The example serves to distinguish precisely formulated objectives from imprecisely formulated ones.
The first two exemplary objectives clearly specify both desired outputs and desired outcomes
The third exemplary objective is unclearly formulated because “a more productive relationship” describes the outcome ambiguously and no output of the policy is specified
“Protect biodiversity by reducing the number of species threatened by extinction” 
“Ensure a more productive relationship across government branches”

6. Structuring Policy Objectives
Objectives can be ordered in a hierarchy that goes from broad strategic objectives to specific outcomes of individual policies.
Key messages:
Policy objectives can be structured according to the higher level objectives they contribute to
Policies at all levels of the hierarchy can be monitored by outcome indicators
Indicators at different levels of the hierarchy have a different importance and a different meaning
Indicators monitoring high-level objectives tend to monitor strategic goals that tend to be influenced by sets of policies
Indicators monitoring low-level objectives tend to monitor the effectiveness of individual policies
Notes:
The graph presents a sample hierarchy of policy objectives:
Indicators at the top measure progress towards a high-level policy objective and are most likely influenced by many different policies
Indicators at the bottom measure more closely the effectiveness of individual policies, but are of course less important
The hierarchy is stylistic and does not consider that a policy can contribute to many high-level objectives
For example, increasing the share of electric cars improves air quality and increases the resilience of the economy by lowering its dependency on oil imports

7. Using indicators
Ideally, one would like to know the effects that are directly caused by a policy
But indicators cannot distinguish the effects of policies from confounding factors
Policy complementarities are difficult to identify … …
Policy … …
Key messages:
Indicators cannot identify causal effects of policies
Note:
The figure illustrates that an outcome is influenced by many other factors besides the policy that is monitored. Indicators cannot separate these influences from each other and therefore cannot identify causal effects. … …
Outcome

8. Quality criteria for performance indicators
High Quality
Low Quality
Indicators aligned to objectives
Responsive to policies
Suitably normalised
Available with short time lags
Measuring outputs when outcomes are targeted (and vice versa)
Frequently changing definitions
Costly or bureaucratic data collection
Unreliable data

9. Indicator based performance frameworks
For agents, the easiest ways to achieve good indicator performance are not always those that are desirable
Make sure the right incentives are provided
Better not to use any performance incentives than to provide bad incentives
First principle: Do no harm
Prepare people to find easily available, but temporary positions
Train people for jobs below their qualification levels
Accept only people who are likely to find jobs anyway
Offer training only in areas where people are likely to find jobs
Example: Consequences of misaligned incentives for providers of job training for the unemployed

10. What indicators for performance frameworks
Trade-0ffs between using high level indicators and low level indicators
The more important the incentives related to an indicator, the greater the need to choose a low level indicator under close control of the agent
Indicators by relevance
Outcome indicators>Output indicators>Input indicators
High level indicators
Measure important outcomes, but difficult to influence by a single policy; many confounding factors
Low level indicators
Less important outcomes, but easier to influence by a single policy; fewer confounding factors

11. One size fits all solutions desirable?
Key message from the work of this year’s winner of the Economics Nobel Prize:
Intuition: Incentives are useful, but agents should not rewarded for good luck or punished for bad luck.
Necessary to adapt performance frameworks to specific cases? Feasible?
The more influence an agent has on the outcome, the more strongly should incentives be used.

12. How to assess macro-economic impacts of individual policies
Impossible to do by monitoring
Policy evaluation can go further
Costly and has long time lags
Small changes to implementation process of policies go a long way to make evaluation easier
Even then, assessing impacts of individual policies on macro-economic outcomes very difficult
Theoretical model necessary of how policies have an effect

13. Contact: abel.schumann@oecd.org
Thank you
Contact: