RESULTS BASED MANAGEMENT UNEP Brown Bag Lunch September 2006 Kabell Konsulting ApS Managing for results

Agenda Presentation Dialogue What is RBM? The Evolution of RBM Key Concepts Results Reporting Behavioural Issues Dialogue RBM in UNEP Feed back

Introduction Objectives To provide input to UNEP’s discussions on RBM To share some international lessons with using RBM To discuss where strengthening is needed in UNEP

What is RBM? Definition: “RBM is a management strategy focusing on performance and achievement of outputs, outcomes and impacts” (DAC OECD 2002) Facilitates the systematic thinking about three basic questions: What is our goal: “Are we doing the right thing”? How will we reach that goal: “Are we doing it right?” How do we know whether we have achieved our goal and that we are doing it right: “How do we know?” Men Hvad er Results Based Management mere præcist. Den præcise definition ifølge DAC kan i jo selv læse. Overordnet set, så angiver Results-Based Management en metode til systematisk at vurdere: Gør vi det rigtige Gør vi det på den rigtige måde Og hvordan ved vi om vi gør det rigtige på den rigtige måde (M&E systemerne) Synonymous with Performance Management or Outcome Based Management -

What is RBM? RBM is not: “A management strategy focusing on inputs and activities” It may include, but should not be limited to, the following elements: Performance Measurement Performance Reporting Performance Based Budgeting Vigtigt at gøre sig klart at Results-Based Management som helhed er en omfattende tilgang til management som indeholder mere end måling og rapportering af resultater. Disse elementer er nødvendig elementer, men det er en stor fejltagelse at tro at implementeringen af disse elementer isoleret set er udtryk for RBM.

The Evolution of RBM In the 1980s: compliance and control. In the 1990s: project based, introduction of log frames, evaluations focus on input and results and is made for accountability. After 2000: more systemic approach with results frameworks, focus on programmes not projects and on outcomes and impact, evaluation for learning, often jointly with partners

Focus on performance not compliance The Evolution of RBM Focus on performance not compliance “Did projects spend their budget and comply with rules and procedures?” “Did projects achieve their objectives and deliver results?” Focus on outcomes and impact not input, activities (and output) “How many trees were planted and people trained?” Overordnet set kan man sige at der gennem de sidste 20 år er sket et paradigmeskift i den offentlige sektor fra et fokus på compliance til resultater og fra fokus på input og outputs til outcomes og impact. Realiteten er dog ofte den at mange organisationer i dag fortsat fokusere på deres output. Det kan vi vende tilbage til senere. “What was the development result e.g. reduction in poverty and improved living conditions”

The Evolution of RBM Focus on management, not administration Client-centered and citizen focused Emphasis upon Outputs versus Inputs; reduction in ex-ante controls Extensive use of outsourcing, competition and private service providers Next Steps Program in the United Kingdom Reinventing Government in the United States Australian and New Zealand PEM Reforms Alternative Service Delivery in Canada

The Evolution of RBM RBM has gained considerable following because: It improves transparency It provides the basis for stronger external and internal accountability It rewards achievement of results It encourages evidence based policymaking and management …and as a result improves the quality of policy and decision making RBM forventes at bidrage til mere gennemsigtighed . Bestræbelserne på at kunne træffe beslutninger på baggrund af dokumenterede erfaringer kender vi især for uddannelses sektoren. Afhængigt af hvor man stiller sig i den debat kan man til tider stille spørgsmål til validiteten af den erfaringsopsamling der foretages og heraf til de beslutninger der træffes på baggrund af erfaringsopsamling.

RBM in Development Cooperation 2000 MDGs 2002 Monterrey 2003 Rome 2004 Marrakesh 2005 Paris HLF-2 Mutual responsibility Coherence Partnership Harmonization Alignment Results for shared results Precedent sector publique en general, domaine du developpement nous retrouvos ces memes pre-ocupations et principes, mais avec un element de plus - le partenariat 2000 – meilleur coherence a travers les MDG – resultats et indicateurs 2002 partenariat - Bonne gouvernance dans les pays en vois de developpement, ressources et efficacite dans la gestion de l’aide – transaction costs - aussi First International Roundtable on Better Measuring, Monitoring, and Managing for Development Results Rome - harmonization - action plans - nouvelle structure mises en place: “l´architecture globale” Marrakesh – manaaging for results – Paris 2005 The second High-Level Forum on Harmonization and Alignment for Aid Effectiveness

Lessons Indicators and evaluations should go beyond compliance and accountability M&E should contribute to build bridge between individual projects and sectorwide or global programmes Indicators should measure contribution to MDGs and intermediary outcomes Evaluation of the performance of partners should be based on results achieved towards joint objectives

Glossary of Key Terms in Evaluation & Results Based Management Key Concepts Glossary of Key Terms in Evaluation & Results Based Management www.oecd.org/dac

Key Concepts The Results Chain Impact Outcome Assumptions and Risks Indicators Outputs Consider limitations of the Logframe approach Assumes that necessary resources will be provided The assumed cause-effect relations may not evolve as expected Absorbing inputs from other actors Static Capturing unintended consequences Med henblik på at undgå misforståelser i forhold til brugen f begreberne i the results chain har OECD DAC i 2002 lavet en lille ordbog med definitioner på nøgle begreberne. Med udgangspunkt i disse definitioner vil jeg kort gennemgå begreberne i the results chain. Activities Inputs

Key Concepts Input Activities Output The financial, human and material resources for the development intervention Activities The coordination, technical assistance or training tasks delivered Output The products, capital goods and services which results from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes

Key Concepts Outcome (- a consequence of outputs) Impact The likely or short term and medium term effect of an intervention’s outputs Impact Positive and negative, primary and secondary long term effects produced by a development intervention, directly or indirectly, intended or unintended DAC and UNFPA measurable change in behaviour, attitudes or socio-cultural values of groups; measurable change in legal, institutional and society practices

Key Concepts Result The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention Results may: Appear within a short time or take years to be fully realized Be planned or unforseen; Be either positive or negative; Be reflected at the level of individuals, groups, institutions or society A result is a describable or measurable change resulting from a cause and effect relationship. It is a change that can be observed, described and measured in some way, for which the cause can be identified

Key Concepts Indicator Quality-test of indicators: A quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the the changes connected to an intervention or to help asses the performance of a development actor Quality-test of indicators: S Specific M Measurable A Available R Relevant T Trackable Imparative to include stakeholders in this proces: those that are going to collect the data and those that are going to use the data

Key Concepts Identify indicators for output, outcome and process: Specify exactly what is to be measured Use a mix of qualitative and quantiative indicators Use project rating systems, where aggregation is needed Consider proxy indicators where direct indicators are not available Make sure to have baselines and target value Remember: ”An imprecise indicator about an important result is better than a precise indicator of an insignificant result” Provide example of project rating system Qualitative indicators: Peoples judgement and perceptions about a subject e.g. such as the confidence people have in sewing machines as instruments of financial independence Quantitative indicators: Objective or idenpendently verifiable numbers or rations. Measures quantity, such as the number of people who own sewing machines in a village Perhaps there is too much focus on indicators Measuring an indicator only provides an indication of development not evidence Danger of goal displacement – managing for indicators An indicator is normatively defined Limitations - Not everything is measurable Does not capture unintended effects

Example Indicators in UNDP Strategic Framework Do they say something significant? Mix of qualitative or quantitative? Can they be aggregated? Are there baselines or target values? Use of process indicators? Can you actually find data to support them? Also No change theory: How we move from input to impact No defined outcome

Elements of an RBM System A Results Based Management System Situation and problem analysis Formulating objectives, intended results and strategies Identifying indicators Setting baseline and targets Collect data on performance Analyse and report performance Integrating monitoring and evaluation findings Using performance information for future planning and decision making Strategic Planning Performance Measurement Performance Management Performance measurement = Monitoring

Example of results measurement system RBM in UNDP 5 Goals 30 services lines 6 cross-cutting drivers Reporting on: rate of achievement of annual targets towards expected outcomes extent to which UNDP promote drivers of development effectiveness Difficulties identified in review by Dalberg  Do not demonstrate the actual achievements targeted. Indicators are not currently standardized across offices, and thus there is variability in the measurability and ability to attribute output and outcomes. The distinction between output and outcomes is in some cases not clear Goals Achieving the MDGs and reducing human poverty; Fostering democratic governance; Managing energy and environment for sustainable development[1] Supporting crisis prevention and recovery; and Responding to HIV/AIDS [1] Service lines for this strategic goal are: 1)Frameworks and strategies for sustainable development 2) Effective water governance 3) Access to sustainable energy services 4) Sustainable land management to combat desertification and land degradation 5) Conservation and sustainable use of biodiversity 6) National/sectoral policy and planning to control emissions of ozone-depleting substances and persistent organic pollutants. The six drivers are: (i) developing national capacities; (ii) enhancing national ownership; (iii) advocating and fostering an enabling policy environment; (iv) seeking South-South solutions; (v) promoting gender equality; and (vi) forging partnerships for results. (UNDP/DP/2005/16, 17)

Results Reporting Having determined expected results, identified indicators, and set baselines and targets as described, it is time to: Collect data on performance Analyse performance Report on performance – for external and internal use Three important issues to consider: Purpose of results reporting Aggregation Attribution/contribution Purpose: Internal vs External or both? Budgetting purposes Accountability or learning

Difficult to aggregate due to diversity in outcomes Results Reporting Aggregating results: output or outcome? Project/programme outcome Easier to establish link between outputs and project outcome than national development changes More relevant for internal learning purposes but Difficult to aggregate due to diversity in outcomes One possibility is to establish standard outcomes for common program approaches (groupings of similar projects) Establish rating system measuring succes in meeting outcomes (Consider issue of validity/reliability) Establish a solid validation process

Results Reporting Attribution vs. Contribution The move from projects and outputs to programmes and outcomes/higher order goals makes it difficult to attribute specific outcomes to individual donors In Denmark, even National Audit Office has accepted this in terms of performance measurement However, in-depth evaluations are still sometimes carried out with the specific objective of demonstrating attribution (counterfactual methodology)

Results Reporting Lessons learned in regards to results reporting Qualitative assessments / statements are more interesting for internal learning than quantitative assessments Ratings provide a good indication of status in projects but should be supplemented with qualitative assessment and validation Supplement performance measurement (monitoring) with evaluations Establish a solid validation process Examples of performance monitoring systems at agency level: Ratings: Self-assessments at project/programme level. Needs to be assisted by some sort of validation process e.g. review of assessments and field visits (field-based auditing)

Behavioural Issues Different approaches to RBM are likely to effect equally different on behaviour of staff Managing By Results Performance measurement is used for accountability/reporting to internal and external stakeholders Sanctions if targets not met (creates push for staff to focus on outputs) Fixed targets Ressource allocation Managing For Results Performance measurement is used with the aim of achieving better results Stretch targets and assessments for learning to modify targets Consider both external and internal demands and needs for accountability and learning when establishing a performance measurement system

Behavioural Issues Experiences from implementing RBM ”What gets measured gets done” ”An imprecise indicator of a significant result is better than a precise indicator of an insignificant result” ”You become what you measure” Organizations typically focus more attention on the methodological and technical aspects of results-oriented M&E systems than on changing mental models (World Bank 2004) It is pivotal to create an organisation culture conducive to learning and innovation and establishing incentives that promote managing for results (World Bank 2004) Strong emphasis on formal systems of tight specification and measurement. Failure to recognizing the fact that for complex activities (..) a highly formalised approach to management has severe limitations (OECD 2002). The shift to a results-based approach has been largely characterized by a focus on the budgeting and programming aspects of management without realizing or emphasizing at the outset the scope of changes required in other areas of management for an effective implementation of an RBM system (JIU 2004)

RBM in UNEP? Conducive Conditions Problematic Conditions Enablers and constraints for introducing RBM: Conducive Conditions An organisation has products and products that are simple Products are uniform, isolated and the production is autonomous An organisation is product oriented Causalities are known Quality definable in performance indicators Environment is stable Problematic Conditions An organisation has obligations and is highly value-oriented Products are multiple and generated together with others An organisation is process oriented Products are interwoven and causalities are unknown Quality not definable in performance indicators Environment is dynamic Follow up on exercise

Self assessment in UNEP Conditions? Experiences? Incentives? Capacity? Plans?

What next?