Risk Management Indicators – Verifying Change 07.04.2019 Risk Management Indicators – Verifying Change NORHED seminar in Addis Ababa 13 – 14 March 2014 Lill-Ann Bjaarstad Medina Department for Quality Assurance/Norad

07.04.2019 Risk Management Risk management as a method to address the risks of not reaching planned or expected results. A project risk management plan should include management risks Does not include risks for own staff

What are risks? What is risks management? 07.04.2019 What are risks? What is risks management? Risk Risk Management “Factors or events that may occur and influence the results achievements in a negative way». Risks surge from uncertainties that have negative impacts A step-wise method to be able to handle risks Identification Assessment Mitigation Tolerance

Are there risks of not “getting to” the next level? 07.04.2019 Impact 1.0 Outcome ( Strengthened Institutions) 1.1 Outcome 1.2 Outcome Are there risks of not “getting to” the next level? E 1.1.1. Output 1.1.2 Output 1.1.3 Output 1.1.4 Output Activities Resources

07.04.2019 Types of risks «Internal” vs. “external” risk factors Interne risiko Eksterne risiko

Main Principles in Risks Management 07.04.2019 Main Principles in Risks Management Risks must include both aspects; uncertainties and negative impact on achievement of results Risk management divides into “phases” (identification, assessment, mitigation and tolerance) of which all are necessary steps to take The assessment of probability and consequences helps identify the most significant risks to be prioritized when planning mitigating measures and discussing much risk may be tolerated to proceeded with the intervention

07.04.2019 Identification The purpose of the identification phase is to set up an “unedited” list with negative factors that may influence our ability to achieve our results. This list is the starting point to undergo an assessment to be able to manage the most significant risks We should attempt to describe; what we see as the source of the risk (the risk factor) the nature of the consequence we believe this may have on “our” project

Assessment of Probability and Consequence 07.04.2019 Assessment of Probability and Consequence

Mitigation and Tolerance 07.04.2019 Mitigation and Tolerance Mitigation: Actions and preventive measures to reduce likelihood of the risk occurring (ref. probability) Preventive measures to reduce the degree of the negative impact (ref. consequence) Are there ways to do things differently? Common mitigation measures: Early interventions, redesign Contingencies plans Acquire more knowledge Mitigation measures may reduce risks but not remove it. Mitigation measures reduces, but not necessarily removes the risk. What is your tolerance to the risk?

Some common flaws in risk management 07.04.2019 Some common flaws in risk management Risk factors are identified, but the consequences for the intervention are not identified nor assessed External risks are mistaken as not being possible to mitigate “Jumping to conclusions”; Risk factors are identified, and mitigation measures are discussed, without a sound assessment of probability and consequence Risk assessments needs to be updated . Problems can be removed and should be addressed.

Today’s mission Where are we? 07.04.2019 Today’s mission Where are we? Wen have formulated some generic Outcomes of strengthening capacity We have identified some essential Outputs to reach these Outcomes We need indicators to monitor progress in delivering these Outputs and to be able to demonstrate whether our interventions have the intended effects.

We need indicators at each results level 07.04.2019 Indicators Impact Impact indicators We need indicators at each results level Outcomes Outcome indicators Outputs Output indicators

07.04.2019 What is an indicator? A variable to verify and/or measure change or processes towards predefined ends. Measurement: continuous vs. discrete measures (not present/present) Indication: direct vs. indirect Verification: evidence-based vs. perception-based

What is a good indicator? 07.04.2019 What is a good indicator? To be able to measure or verify the achievement of results, indicators should; Reflect the entire results statement in an adequate manner Be relevant – and give the needed (strategic) information Be at the right (corresponding) level Be realistic (“cost-efficient” and feasible)

There is no standard terminology 07.04.2019 Types of indicators Standard Indicators (core indicators) Key Indicators Indicators to demonstrate progress (sequenced outputs) Composite indicators (cluster) There is no standard terminology

Baseline – the current situation 07.04.2019 Baseline – the current situation Baseline data is narrower then a Problem Analyses Baseline is established when the Outcomes has been agreed upon. Baseline does not address causes Baseline data should be relevant to the Results Statements at each level Baselines may help reconfirm or adjust indicators and targets

Baseline values need to be ascribed to each indicator 07.04.2019 Baseline values Baseline values need to be ascribed to each indicator If you can’t find your baseline value you probabely have the wrong indicator

07.04.2019 Targets Targets at output level; Set targets relative to resources/inputs of the Project (measures efficiency and organizational effectiveness ) Targets at outcome level; When building institutional capacity it is feasible to set targets for the expected effects of the Project (ex. qualified graduates, research produced). Targets at impact level; May serve as a vision or symbols of political consensus (like the MDG indicators)

07.04.2019 The most common errors Starting by selecting indicators is starting in the wrong end Results Statements define indicators and not the other way around. Indicators should not be phrased as objectives Indicators are measurements. Sometimes what they measure have to be supplemented with a narrative description. Indicators are seen as “neutral” measures Indicators should reflect strategic choices, may have management implications and aim at learning. They may be and often are political

Measuring capacity by its effects 07.04.2019 Measuring capacity by its effects In NORHED the effects of capacity is identified as, more and better research produced more and better qualified graduates “At the Programme level it is evident that there is a need for more comprehensive indicators for project outputs, achievements and outcomes beyond an annual summery of graduates produced and research papers published” Norad, Evaluation report 7/2009

Example: Indicators that measure quantity or/and quality of research 07.04.2019 Example: Indicators that measure quantity or/and quality of research ARWU – Academic Ranking of World Universities (Shanghai Jiao) HEEACT – Higher Education Evaluation and Accreditation Council of Taiwan Number of article in the past 11 years Number of articles last year Number of citations for past 11 years Number of highly cited papers Number of articles in highly cited journals Papers published in Nature and Science Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index Staff/alumni winning Nobel Prices and Field Medals Does the HEEACT indicators say anything about the quality of the research production?

Example: Indicators that measures quality 07.04.2019 Example: Indicators that measures quality ARWU – Academic Ranking of World Universities (Shanghai Jioa) THE - Higher Times Education Research Outputs Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index Per capita academic performance Teaching and learning environment Results on reputational survey on teaching Staff to student ratio Ratio PhD to BA degrees awarded by institutions measured against number of academic staff Research volume, income and reputation. Results of reputational survey Research income scaled against staff Number of articles in academic journals (Thompson Reuter) per staff member Does this ARWU indicators say something about the quality of the institutions research production ?

1.0 Outcome ( Strengthened Institutions) 07.04.2019 Expanded and better qualified, gender balanced workforce Increased, better and relevant knowledge base Increased evidence base for policies and public management NORHED IMPACT 1.1.1. Output 1.1.2 1.1 Outcome 1.2 Outcome 1.3 Outcome 1.0 Outcome ( Strengthened Institutions) 1.2.1 1.2.2 1.3.1 Output 1.1.3 E

Contribution to impact Are we missing some indicators? 07.04.2019 Contribution to impact Are we missing some indicators? Expanded and better qualified, gender balanced workforce Increased, better and relevant knowledge base Increased evidence base for policies and public management Two remaining questions; How to measure contribution to better qualified work force? How do we demonstrate the relevance of knowledge? How do we verify that there is and increased evidence base for polices? Can we demonstrate the influence of our research findings?

How to measure the quality in graduate teaching? 07.04.2019 How to measure the quality in graduate teaching? Illustration: Make a composite indicator that include (elements may be weighted): Curriculum includes/updated with knowledge that is relevant for the local setting Curriculum is adapted to standard of the global knowledge Share of lecturers with PhD Share of lecturers with research experience that is relevant for the graduate programme

07.04.2019 How to demonstrate the relevance knowledge from the six sub-programmes and how it influence society? Suggestion: Reporting in narrative Can we organize the reporting on the relevance of research findings into thematic categories, specific research questions or other common denominators? How to report on influence? Should we put different requirements for reports on different types of research? Basic Applied Technology and Innovation R&D for management of public and natural resources Action research

Do we need a communication strategy? 07.04.2019 Do we need a communication strategy? Narratives add important anecdotal evidence to document the relevance and influence of the research and the research findings Can we find a more systematic approach to communicate research findings, and how these contribute to a better and more relevant knowledge base? Good stories are always relevant