1. Risk Management Indicators – Verifying Change
Risk Management Indicators – Verifying Change
NORHED seminar in Addis Ababa 13 – 14 March 2014
Lill-Ann Bjaarstad Medina
Senior Adviser
Department for Quality Assurance/Norad

2.
Risk Management
Risk management as a method to address the risks of not reaching planned or expected results.
A project risk management plan should include management risks
Does not include
risks for own staff

3. What are risks? What is risks management?
What are risks? What is risks management?
Risk
Risk Management
“Factors or events that may occur and influence the results achievements in a negative way». Risks surge from uncertainties that have negative impacts
A step-wise method to be able to handle risks
Identification
Assessment
Mitigation
Tolerance

4. Impact Risks Resources 1.0 Outcome ( Strengthened Institutions)
Impact
1.0 Outcome ( Strengthened Institutions)
1.1 Outcome
1.2 Outcome
Risks E
1.1.1.
Output
1.1.2
Output
1.1.3
Output
1.1.4
Output
Activities
Resources

5.
Types of risks «Internal” vs. “external” risk factors Interne risiko Eksterne risiko

6. Main Principles in Risks Management
Main Principles in Risks Management
Risks must include both aspects; uncertainties and negative impact on achievement of results
Risk management divides into “phases” (identification, assessment, mitigation and tolerance) of which all are necessary steps to take
The assessment of probability and consequences helps identify the most significant risks to be prioritized when planning mitigating measures and how much risk you may tolerate

7. Identification
The purpose of the identification phase is to set up an “unedited” list with negative factors that may influence our ability to achieve our results.
This list is the starting point to undergo an assessment to be able to manage the most significant risks
We should attempt describing;
what we see as the source of the risk (the risk factor)
the nature of the consequence we believe this may have on “our” project

8. Assessment Probability and Consequence
Assessment Probability and Consequence

9. Mitigation and Tolerance
Mitigation and Tolerance
Preventive measures to reduce likelihood of the risk occurring (ref. probability)
Preventive measures to reduce the degree of the negative impact (ref. consequence)
Early intervention
Contingencies plans
More knowledge
Mitigation measures may reduce risks but not remove it.
Mitigation measures reduces, but not necessarily removes the risk. What is your tolerance?

10. Some common flaws in risk management
Some common flaws in risk management
Risk factors are identified, but the consequences for the intervention are not identified nor assessed
External risks are mistaken as not being possible to mitigate
“Jumping to conclusions”; Risk factors are identified, and mitigation measures are discussed, without a sound assessment of probability and consequence
Risk assessments needs to be updated .
Problems can be removed and should be addressed.

11. Today’s mission Where are we?
Today’s mission
Where are we?
Wen have formulated some generic Outcomes of strengthening capacity
We have identified some essential Outputs to reach these Outcomes
We need indicators to monitor progress in delivering these essential Outputs and to whether they have the intended effect.

12. We need indicators at each results level
Indicators
Impact
We need indicators at each results level
Outcomes
Outputs

13.
What is an indicator?
A variable to verify and/or measure change or processes towards predefined ends.
Measurement: continuous vs. discrete measures (not present/present)
Indication: direct vs. indirect
Verification: evidence-based vs. perception-based

14. What is a good indicator?
What is a good indicator?
To be able to measure or verify the achievement of results, indicators should;
Reflect the entire results statement in an adequate manner
Be relevant – and give the needed (strategic) information
Be at the right (corresponding) level
Be realistic (“cost-efficient” and feasible)

15. There is no standard terminology
Types of indicators
Standard Indicators (core indicators)
Key Indicators
Indicators to demonstrate progress (sequenced outputs)
Composite indicators (cluster)
There is no standard terminology

16. Baseline – the current situation
Baseline – the current situation
Baseline data is narrower then a Problem Analyses
Baseline is established when the Outcomes has been agreed upon.
Baseline does not address causes
Baseline data should be relevant to the Results Statements at each level
Baselines may help reconfirm or adjust indicators and targets

17. Baseline values need to be ascribed to each indicator
Baseline values
Baseline values need to be ascribed to each indicator
If you can’t find your baseline value you probabely have the wrong indicator

18. Global monitoring vs reporting on results of interventions
Targets
Targets on Output level; Set relative to resource input
Targets on Outcome level; When is it feasible?
Targets as symbols of a political consensus or visions
Illustration: Unicef
Global monitoring vs reporting on results of interventions

19.
The most common errors
Starting by selecting indicators is starting in the wrong end
Results Statements define indicators and not the other way around.
Indicators should not be phrased as objectives
Indicators are measurements. Sometimes what they measure have to be supplemented with a narrative description.
Indicators are seen as “neutral” measures
Indicators should reflect strategic choices, may have management implications and aim at learning. They may be and often are political

20. Measuring capacity by its effects
Measuring capacity by its effects
In NORHED the effects of capacity is identified as,
more and better research produced
more and better qualified graduates
“At the Programme level it is evident that there is a need for more comprehensive indicators for project outputs, achievements and outcomes beyond an annual summery of graduates produced and research papers published”
Norad, Evaluation report 7/2009

21. Example: Indicators that measure quantity or/and quality of research
Example: Indicators that measure quantity or/and quality of research
ARWU – Academic Ranking of World Universities (Shanghai Jiao)
HEEACT – Higher Education Evaluation and Accreditation Council of Taiwan
Number of article in the past 11 years
Number of articles last year
Number of citations for past 11 years
Number of highly cited papers
Number of articles in highly cited journals
Papers published in Nature and Science
Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index
Staff/alumni winning Nobel Prices and Field Medals
Does the HEEACT indicators say anything about the quality of the research production?

22. Example: Indicators that measures quality
Example: Indicators that measures quality
ARWU – Academic Ranking of World Universities (Shanghai Jioa)
THE - Higher Times Education
Research Outputs
Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index
Per capita academic performance
Teaching and learning environment
Results on reputational survey on teaching
Staff to student ratio
Ratio PhD to BA degrees awarded by institutions measured against number of academic staff
Research volume, income and reputation.
Results of reputational survey
Research income scaled against staff
Number of articles in academic journals (Thompson Reuter) per staff member
Does this ARWU indicators say something about the quality of the institutions research production ?

23. 1.0 Outcome ( Strengthened Institutions)
Expanded and better qualified, gender balanced workforce
Increased, better and relevant knowledge base
Increased evidence base for policies and public management
NORHED IMPACT
1.1.1.
Output
1.1.2
1.1 Outcome
1.2 Outcome
1.3 Outcome
1.0 Outcome ( Strengthened Institutions)
1.2.1
1.2.2
1.3.1 Output
1.1.3 E

24. Contribution to impact Are we missing some indicators
Contribution to impact Are we missing some indicators
Expanded and better qualified, gender balanced workforce
Increased, better and relevant knowledge base
Increased evidence base for policies and public management
Two remaining questions;
How to measure contribution to better qualified work force?
How do we demonstrate the relevance of knowledge?
How do we verify that there is and increased evidence base for polices?
Can we demonstrate the influence of our research findings?

25. How to measure high quality teaching?
How to measure high quality teaching?
Suggestions for composite indicators
Curriculum includes/updated with knowledge that is relevant for the local setting
Curriculum in adapted to the global knowledge base
Share of lecturers with PhD
Share of lecturers that with research experience that is relevant for the MA Programme

26.
How to demonstrate the relevance knowledge from the six sub-programmes and how it influence society?
Suggestion: Reporting in narrative
Can we organize the reporting on the relevance of research findings according to thematic categories or research questions?
How to report on influence? Should we put different requirements for reports on different types of research?
Basic
Applied
Technology and Innovation
R&D for management of public and natural resources
Action research

27. Do we need a communication strategy?
Do we need a communication strategy?
Narratives may add important anecdotal evidence of relevance and influence for research and research findings
Can we find a more systematic approach to communicate research findings, and how they contribute to a better and more relevant knowledge base?
Good stories are always relevant