1. Risk Management Indicators – Verifying Change
Risk Management Indicators – Verifying Change
NORHED seminar in Addis Ababa 13 – 14 March 2014
Lill-Ann Bjaarstad Medina
Department for Quality Assurance/Norad

2.
Risk Management
Risk management as a method to address the risks of not reaching planned or expected results.
A project risk management plan should include management risks
Does not include
risks for own staff

3. What are risks? What is risks management?
What are risks? What is risks management?
Risk
Risk Management
“Factors or events that may occur and influence the results achievements in a negative way». Risks surge from uncertainties that have negative impacts
A step-wise method to be able to handle risks
Identification
Assessment
Mitigation
Tolerance

4. Are there risks of not “getting to” the next level?
Impact
1.0 Outcome ( Strengthened Institutions)
1.1 Outcome
1.2 Outcome
Are there risks of not “getting to” the next level? E
1.1.1.
Output
1.1.2
Output
1.1.3
Output
1.1.4
Output
Activities
Resources

5.
Types of risks «Internal” vs. “external” risk factors Interne risiko Eksterne risiko

6. Main Principles in Risks Management
Main Principles in Risks Management
Risks must include both aspects; uncertainties and negative impact on achievement of results
Risk management divides into “phases” (identification, assessment, mitigation and tolerance) of which all are necessary steps to take
The assessment of probability and consequences helps identify the most significant risks to be prioritized when planning mitigating measures and discussing much risk may be tolerated to proceeded with the intervention

7.
Identification
The purpose of the identification phase is to set up an “unedited” list with negative factors that may influence our ability to achieve our results.
This list is the starting point to undergo an assessment to be able to manage the most significant risks
We should attempt to describe;
what we see as the source of the risk (the risk factor)
the nature of the consequence we believe this may have on “our” project

8. Assessment of Probability and Consequence
Assessment of Probability and Consequence

9. Mitigation and Tolerance
Mitigation and Tolerance
Mitigation:
Actions and preventive measures to reduce likelihood of the risk occurring (ref. probability)
Preventive measures to reduce the degree of the negative impact (ref. consequence)
Are there ways to do things differently?
Common mitigation measures:
Early interventions, redesign
Contingencies plans
Acquire more knowledge
Mitigation measures may reduce risks but not remove it.
Mitigation measures reduces, but not necessarily removes the risk. What is your tolerance to the risk?

10. Some common flaws in risk management
Some common flaws in risk management
Risk factors are identified, but the consequences for the intervention are not identified nor assessed
External risks are mistaken as not being possible to mitigate
“Jumping to conclusions”; Risk factors are identified, and mitigation measures are discussed, without a sound assessment of probability and consequence
Risk assessments needs to be updated .
Problems can be removed and should be addressed.

11. Today’s mission Where are we?
Today’s mission
Where are we?
Wen have formulated some generic Outcomes of strengthening capacity
We have identified some essential Outputs to reach these Outcomes
We need indicators to monitor progress in delivering these Outputs and to be able to demonstrate whether our interventions have the intended effects.

12. We need indicators at each results level
Indicators
Impact
Impact indicators
We need indicators at each results level
Outcomes
Outcome indicators
Outputs
Output indicators

13.
What is an indicator?
A variable to verify and/or measure change or processes towards predefined ends.
Measurement: continuous vs. discrete measures (not present/present)
Indication: direct vs. indirect
Verification: evidence-based vs. perception-based

14. What is a good indicator?
What is a good indicator?
To be able to measure or verify the achievement of results, indicators should;
Reflect the entire results statement in an adequate manner
Be relevant – and give the needed (strategic) information
Be at the right (corresponding) level
Be realistic (“cost-efficient” and feasible)

15. There is no standard terminology
Types of indicators
Standard Indicators (core indicators)
Key Indicators
Indicators to demonstrate progress (sequenced outputs)
Composite indicators (cluster)
There is no standard terminology

16. Baseline – the current situation
Baseline – the current situation
Baseline data is narrower then a Problem Analyses
Baseline is established when the Outcomes has been agreed upon.
Baseline does not address causes
Baseline data should be relevant to the Results Statements at each level
Baselines may help reconfirm or adjust indicators and targets

17. Baseline values need to be ascribed to each indicator
Baseline values
Baseline values need to be ascribed to each indicator
If you can’t find your baseline value you probabely have the wrong indicator

18.
Targets
Targets at output level; Set targets relative to resources/inputs of the Project (measures efficiency and organizational effectiveness )
Targets at outcome level; When building institutional capacity it is feasible to set targets for the expected effects of the Project (ex. qualified graduates, research produced).
Targets at impact level; May serve as a vision or symbols of political consensus (like the MDG indicators)

19.
The most common errors
Starting by selecting indicators is starting in the wrong end
Results Statements define indicators and not the other way around.
Indicators should not be phrased as objectives
Indicators are measurements. Sometimes what they measure have to be supplemented with a narrative description.
Indicators are seen as “neutral” measures
Indicators should reflect strategic choices, may have management implications and aim at learning. They may be and often are political

20. Measuring capacity by its effects
Measuring capacity by its effects
In NORHED the effects of capacity is identified as,
more and better research produced
more and better qualified graduates
“At the Programme level it is evident that there is a need for more comprehensive indicators for project outputs, achievements and outcomes beyond an annual summery of graduates produced and research papers published”
Norad, Evaluation report 7/2009

21. Example: Indicators that measure quantity or/and quality of research
Example: Indicators that measure quantity or/and quality of research
ARWU – Academic Ranking of World Universities (Shanghai Jiao)
HEEACT – Higher Education Evaluation and Accreditation Council of Taiwan
Number of article in the past 11 years
Number of articles last year
Number of citations for past 11 years
Number of highly cited papers
Number of articles in highly cited journals
Papers published in Nature and Science
Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index
Staff/alumni winning Nobel Prices and Field Medals
Does the HEEACT indicators say anything about the quality of the research production?

22. Example: Indicators that measures quality
Example: Indicators that measures quality
ARWU – Academic Ranking of World Universities (Shanghai Jioa)
THE - Higher Times Education
Research Outputs
Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index
Per capita academic performance
Teaching and learning environment
Results on reputational survey on teaching
Staff to student ratio
Ratio PhD to BA degrees awarded by institutions measured against number of academic staff
Research volume, income and reputation.
Results of reputational survey
Research income scaled against staff
Number of articles in academic journals (Thompson Reuter) per staff member
Does this ARWU indicators say something about the quality of the institutions research production ?

23. 1.0 Outcome ( Strengthened Institutions)
Expanded and better qualified, gender balanced workforce
Increased, better and relevant knowledge base
Increased evidence base for policies and public management
NORHED IMPACT
1.1.1.
Output
1.1.2
1.1 Outcome
1.2 Outcome
1.3 Outcome
1.0 Outcome ( Strengthened Institutions)
1.2.1
1.2.2
1.3.1 Output
1.1.3 E

24. Contribution to impact Are we missing some indicators?
Contribution to impact Are we missing some indicators?
Expanded and better qualified, gender balanced workforce
Increased, better and relevant knowledge base
Increased evidence base for policies and public management
Two remaining questions;
How to measure contribution to better qualified work force?
How do we demonstrate the relevance of knowledge?
How do we verify that there is and increased evidence base for polices?
Can we demonstrate the influence of our research findings?

25. How to measure the quality in graduate teaching?
How to measure the quality in graduate teaching?
Illustration:
Make a composite indicator that include (elements may be weighted):
Curriculum includes/updated with knowledge that is relevant for the local setting
Curriculum is adapted to standard of the global knowledge
Share of lecturers with PhD
Share of lecturers with research experience that is relevant for the graduate programme

26.
How to demonstrate the relevance knowledge from the six sub-programmes and how it influence society?
Suggestion: Reporting in narrative
Can we organize the reporting on the relevance of research findings into thematic categories, specific research questions or other common denominators?
How to report on influence? Should we put different requirements for reports on different types of research?
Basic
Applied
Technology and Innovation
R&D for management of public and natural resources
Action research

27. Do we need a communication strategy?
Do we need a communication strategy?
Narratives add important anecdotal evidence to document the relevance and influence of the research and the research findings
Can we find a more systematic approach to communicate research findings, and how these contribute to a better and more relevant knowledge base?
Good stories are always relevant