Risk Management Indicators – Verifying Change 24.06.2018 Risk Management Indicators – Verifying Change NORHED seminar in Addis Ababa 13 – 14 March 2014 Lill-Ann Bjaarstad Medina

Today’s mission Where are we? 24.06.2018 Today’s mission Where are we? Wen have formulated some generic Outcomes of strengthening capacity We have identified some essential Outputs to reach these Outcomes We need indicators to monitor progress in delivering these essential Outputs and to whether they have the intended effect.

24.06.2018 Risk Management Risk management as a method to address the risks of not reaching planned or expected results. A project risk management plan should include management risks Does not include risks for own staff

What are risks? What is risks management? 24.06.2018 What are risks? What is risks management? Risk Risk Management “Factors or events that may occur and influence the results achievements in a negative way». Risks surge from uncertainties that have negative impacts A step-wise method to be able to handle risks Identification Analysis Mitigation Tolerance It is about danger and the unsecurtey of the unknown

Impact Risks Resources 1.0 Outcome ( Strengthened Institutions) 24.06.2018 Impact 1.0 Outcome ( Strengthened Institutions) 1.1 Outcome 1.2 Outcome Risks E 1.1.1. Output 1.1.2 Output 1.1.3 Output 1.1.4 Output Activities Resources

24.06.2018 Types of risks «Internal” vs. “external” risk factors Interne risiko Eksterne risiko

Main Principles in Risks Management 24.06.2018 Main Principles in Risks Management Risks must include both aspects; uncertainties and negative impact on achievement of results Risk management divides into “phases” (identification, assessment, mitigation and tolerance) of which all are necessary steps to take The assessment of probability and consequences helps identify the most significant risks to be prioritized when planning mitigating measures and how much risk you may tolerate

Risk Assessment Probability and Consequence 24.06.2018 Risk Assessment Probability and Consequence

Mitigation and Tolerance 24.06.2018 Mitigation and Tolerance Implementation of preventive measures to reduce likelihood (ref. probability) Preventive measures to reduce the degree of negative impact (ref. consequence) Early intervention Contingencies plans Mitigation measures may reduce risks but not remove it. Mitigation measures reduces, but not necessarily removes the risk

Some common flaws in risk management 24.06.2018 Some common flaws in risk management Risk factors are identified, but the consequences for the intervention are not identified nor assessed External risks are mistaken as not being possible to mitigate “Jumping to conclusions”; Risk factors are identified, and mitigation measures are discussed, without a sound assessment of probability and consequence Risk assessments needs to be updated . Problems can be removed and should be addressed.

We need indicators at each results level 24.06.2018 Indicators Impact We need indicators at each results level Outcomes Outputs Let os turn to the terminology and the actual approcah. I am using OECD DAC teminology The abstaction that is the basis for the entier method. Some of thsi may be basic for you – and you have heard it before. It is a chain of cause ena effect. Where you distingush between what you do and what you acheiye or come out of what you do . Input – resurces, money, human resources, technlogy that make you able to undertake som kind of activies This is the intervetnion – the aid intervention that we want to transform into products ans services that have some effects – these are results. What is that product and services. Products – sees the production line terminology. The quality of that product is just as important as the quantety to be able to get an effct on the next result level We distingush between interventions and results – but with we have come to the results level – there are three, we disinguish between Outputs ans effect – that are defned at two differnt levels, Outcomes and Impact. Here there are som divergencies in how you define the tow highest levels. OECD/Dac says Oucomes are short term effect, Impact are long term. We say that Outcomes are effect on your target group – while Impact is effect on Society. So again, it is how you defien your unit or your populations so to say. Very often Impacat has normative aspect – we don’t want any change – we want a developmet change. That is way Impact often is substituted with the term Development Goal. Society – it can be global, regional, national. Outcome is alos about effects on defined populations – UN call it behavioral outcomes – your tareget gropu act diffetnly or get access to a service etc. Usually we are happy with this – but here I want to caomplicate it, the reason for that is that Insitutions can also be Outcomes. UN refer insitutional outcome – I am saying this because capacity maight be a kind of insitutional Outcome. A law can be an outcome. Guatemala – Hag convention of adoption - you need to see an effect You have to get your results chain right. The leogic – what leads to what – needs to be clear I am no so conserned about what is what – becaus often you gat confused. It is a chain, ans since there are infenite cause and effec chains in the would – the most importing thing is that you are clear on where you put it. The chain has a time dimention - of cause comes befor the effect. If that is not so you have probale mixed up your target group or population that shall have the effect.

24.06.2018 What is an indicator? A variable to verify and/or measure change or processes towards predefined ends. Measurement: continuous vs. discrete measures (not present/present) Indication: direct vs indirect Verification: evidence-based vs. perception-based

What is a good indicator? 24.06.2018 What is a good indicator? To be able to measure or verify the achievement of results, indicators should; Reflect the entire results statement in an adequate manner Be relevant – and give the needed (strategic) information Be at the right (corresponding) level Be realistic (“cost-efficient” and feasible)

1.0 Outcome ( Strengthened Institutions) 24.06.2018 z Impact 1.0 Outcome ( Strengthened Institutions) 1.1 Outcome 1.2 Outcome 1.3 Outcome E 1.1.1. Output 1.1.2 Output 1.1.3 Output 1.2.1 Output 1.2.2 Output 1.3.1 Output

There is no standard terminology 24.06.2018 Types of indicators Standard Indicators (core indicators) Key Indicators Indicators to demonstrate progress (sequenced outputs) Composite indicators (cluster) There is no standard terminology

Baseline – the current situation 24.06.2018 Baseline – the current situation Baseline data is narrower then a Problem Analyses Baseline is established when the Outcomes has been agreed upon. Baseline does not address causes Baseline data should be relevant to the Results Statements at each level Baselines may help reconfirm or adjust indicators and targets

Baseline values need to be ascribed to each indicator 24.06.2018 Baseline values Baseline values need to be ascribed to each indicator If you can’t find your baseline value you probabely have the wrong indicator

Global monitoring vs reporting on results of interventions 24.06.2018 Targets Targets on Output level; Set relative to resource input Targets on Outcome level; When is it feasible? Targets as symbols of a political consensus or visions Illustration: Unicef Global monitoring vs reporting on results of interventions

24.06.2018 The most common errors Starting by selecting indicators is starting in the wrong end Results Statements define indicators and not the other way around. Indicators should not be phrased as objectives Indicators are measurements. Sometimes what they measure have to be supplemented with a narrative description. Indicators are seen as “neutral” measures Indicators should reflect strategic choices, may have management implications and aim at learning. They may be and often are political

Measuring capacity by its effects 24.06.2018 Measuring capacity by its effects In NORHED the effects of capacity is identified as, more and better research produced more and better qualified graduates “At the Programme level it is evident that there is a need for more comprehensive indicators for project outputs, achievements and outcomes beyond an annual summery of graduates produced and research papers published” Norad, Evaluation report 7/2009

Illustration: Indicators to measure quantity and quality of research 24.06.2018 Illustration: Indicators to measure quantity and quality of research ARWU – Academic Ranking of World Universities (Shanghai Jiao) HEEACT – Higher Education Evaluation and Accreditation Council of Taiwan Number of article in the past 11 years Number of articles last year Number of citations for past 11 years Number of highly cited papers Number of articles in highly cited journals Papers published in Nature and Science Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index Staff/alumni winning Nobel Prices and Field Medals Does it say something about the quality of the institution?

Illustration: Indicators to measure institutions 24.06.2018 Illustration: Indicators to measure institutions ARWU – Academic Ranking of World Universities (Shanghai Jioa) THE - Higher Times Education Research Outputs Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index Per capita academic performance Teaching and learning environment Results on reputational survey on teaching Staff to student ratio Ratio PhD to BA degrees awarded by institutions measured against number of academic staff Research volume, income and reputation. Results of reputational survey Research income scaled against staff Number of articles in academic journals (Thompson Reuter) per staff member

1.0 Outcome ( Strengthened Institutions) 24.06.2018 Expanded and better qualified, gender balanced workforce Increased, better and relevant knowledge base Increased evidence base for policies and public management NORHED IMPACT 1.1.1. Output 1.1.2 1.1 Outcome 1.2 Outcome 1.3 Outcome 1.0 Outcome ( Strengthened Institutions) 1.2.1 1.2.2 1.3.1 Output 1.1.3 E Here we have a results framework again. A results framwork is made up if many results chain. I fyo only hav one, you do not need at framwork . I could have taken away the two lowest levels. We have decided that the Outcome i Strengthened Insitutions – within certan prgramm area - that is outcome level. That is a kind of a institutional outcome. The insitution is strentened so it is possible to produce more students and

Contribution to impact Are we missing some indicators 24.06.2018 Contribution to impact Are we missing some indicators Expanded and better qualified, gender balanced workforce Increased, better and relevant knowledge base Increased evidence base for policies and public management Two remaining questions; How to measure contribution to better qualified work force? How do we demonstrate the relevance of knowledge? How do we verify that there is and increased evidence base for polices? Can we demonstrate the influence of our research findings?

How to measure high quality teaching? 24.06.2018 How to measure high quality teaching? Suggestions for composite indicators Curriculum includes/updated with knowledge that is relevant for the local setting Curriculum in adapted to the global knowledge base Share of lecturers with PhD Share of lecturers that with research experience that is relevant for the MA programme

24.06.2018 How to demonstrate the relevance knowledge from the six sub-programmes and how it influence society? Suggestion: Reporting in narrative Can we organize the reporting on the relevance of research findings according to thematic categories or research questions? How to report on influence? Should we put different requirements for reports on different types of research? Basic Applied Technology and Innovation R&D for management of public and natural resources Action research

Do we need a communication strategy? 24.06.2018 Do we need a communication strategy? Good stories - Anecdotal evidence of relevance and influence A more systematic approach?