1. Risk Management Indicators – Verifying Change
Risk Management Indicators – Verifying Change
NORHED seminar in Addis Ababa 13 – 14 March 2014
Lill-Ann Bjaarstad Medina

2. Today’s mission Where are we?
Today’s mission
Where are we?
Wen have formulated some generic Outcomes of strengthening capacity
We have identified some essential Outputs to reach these Outcomes
We need indicators to monitor progress in delivering these essential Outputs and to whether they have the intended effect.

3.
Risk Management
Risk management as a method to address the risks of not reaching planned or expected results.
A project risk management plan should include management risks
Does not include
risks for own staff

4. What are risks? What is risks management?
What are risks? What is risks management?
Risk
Risk Management
“Factors or events that may occur and influence the results achievements in a negative way». Risks surge from uncertainties that have negative impacts
A step-wise method to be able to handle risks
Identification
Analysis
Mitigation
Tolerance
It is about danger and the unsecurtey of the unknown

5. Impact Risks Resources 1.0 Outcome ( Strengthened Institutions)
Impact
1.0 Outcome ( Strengthened Institutions)
1.1 Outcome
1.2 Outcome
Risks E
1.1.1.
Output
1.1.2
Output
1.1.3
Output
1.1.4
Output
Activities
Resources

6.
Types of risks «Internal” vs. “external” risk factors Interne risiko Eksterne risiko

7. Main Principles in Risks Management
Main Principles in Risks Management
Risks must include both aspects; uncertainties and negative impact on achievement of results
Risk management divides into “phases” (identification, assessment, mitigation and tolerance) of which all are necessary steps to take
The assessment of probability and consequences helps identify the most significant risks to be prioritized when planning mitigating measures and how much risk you may tolerate

8. Risk Assessment Probability and Consequence
Risk Assessment Probability and Consequence

9. Mitigation and Tolerance
Mitigation and Tolerance
Implementation of preventive measures to reduce likelihood (ref. probability)
Preventive measures to reduce the degree of negative impact (ref. consequence)
Early intervention
Contingencies plans
Mitigation measures may reduce risks but not remove it.
Mitigation measures reduces, but not necessarily removes the risk

10. Some common flaws in risk management
Some common flaws in risk management
Risk factors are identified, but the consequences for the intervention are not identified nor assessed
External risks are mistaken as not being possible to mitigate
“Jumping to conclusions”; Risk factors are identified, and mitigation measures are discussed, without a sound assessment of probability and consequence
Risk assessments needs to be updated .
Problems can be removed and should be addressed.

11. We need indicators at each results level
Indicators
Impact
We need indicators at each results level
Outcomes
Outputs
Let os turn to the terminology and the actual approcah.
I am using OECD DAC teminology
The abstaction that is the basis for the entier method. Some of thsi may be basic for you – and you have heard it before.
It is a chain of cause ena effect. Where you distingush between what you do and what you acheiye or come out of what you do .
Input – resurces, money, human resources, technlogy that make you able to undertake som kind of activies
This is the intervetnion – the aid intervention that we want to transform into products ans services that have some effects – these are results.
What is that product and services. Products – sees the production line terminology. The quality of that product is just as important as the quantety to be able to get an effct on the next result level
We distingush between interventions and results – but with we have come to the results level – there are three, we disinguish between Outputs ans effect – that are defned at two differnt levels,
Outcomes and Impact. Here there are som divergencies in how you define the tow highest levels.
OECD/Dac says Oucomes are short term effect, Impact are long term.
We say that Outcomes are effect on your target group – while Impact is effect on Society. So again, it is how you defien your unit or your populations so to say. Very often Impacat has normative aspect – we don’t want any change – we want a developmet change. That is way Impact often is substituted with the term Development Goal. Society – it can be global, regional, national.
Outcome is alos about effects on defined populations – UN call it behavioral outcomes – your tareget gropu act diffetnly or get access to a service etc.
Usually we are happy with this – but here I want to caomplicate it, the reason for that is that Insitutions can also be Outcomes. UN refer insitutional outcome – I am saying this because capacity maight be a kind of insitutional Outcome. A law can be an outcome.
Guatemala – Hag convention of adoption - you need to see an effect
You have to get your results chain right. The leogic – what leads to what – needs to be clear I am no so conserned about what is what – becaus often you gat confused. It is a chain, ans since there are infenite cause and effec chains in the would – the most importing thing is that you are clear on where you put it.
The chain has a time dimention - of cause comes befor the effect. If that is not so you have probale mixed up your target group or population that shall have the effect.

12.
What is an indicator?
A variable to verify and/or measure change or processes towards predefined ends.
Measurement: continuous vs. discrete measures (not present/present)
Indication: direct vs indirect
Verification: evidence-based vs. perception-based

13. What is a good indicator?
What is a good indicator?
To be able to measure or verify the achievement of results, indicators should;
Reflect the entire results statement in an adequate manner
Be relevant – and give the needed (strategic) information
Be at the right (corresponding) level
Be realistic (“cost-efficient” and feasible)

14. 1.0 Outcome ( Strengthened Institutions) z
Impact
1.0 Outcome ( Strengthened Institutions)
1.1 Outcome
1.2 Outcome
1.3 Outcome E
1.1.1.
Output
1.1.2
Output
1.1.3
Output
1.2.1
Output
1.2.2
Output
1.3.1 Output

15. There is no standard terminology
Types of indicators
Standard Indicators (core indicators)
Key Indicators
Indicators to demonstrate progress (sequenced outputs)
Composite indicators (cluster)
There is no standard terminology

16. Baseline – the current situation
Baseline – the current situation
Baseline data is narrower then a Problem Analyses
Baseline is established when the Outcomes has been agreed upon.
Baseline does not address causes
Baseline data should be relevant to the Results Statements at each level
Baselines may help reconfirm or adjust indicators and targets

17. Baseline values need to be ascribed to each indicator
Baseline values
Baseline values need to be ascribed to each indicator
If you can’t find your baseline value you probabely have the wrong indicator

18. Global monitoring vs reporting on results of interventions
Targets
Targets on Output level; Set relative to resource input
Targets on Outcome level; When is it feasible?
Targets as symbols of a political consensus or visions
Illustration: Unicef
Global monitoring vs reporting on results of interventions

19.
The most common errors
Starting by selecting indicators is starting in the wrong end
Results Statements define indicators and not the other way around.
Indicators should not be phrased as objectives
Indicators are measurements. Sometimes what they measure have to be supplemented with a narrative description.
Indicators are seen as “neutral” measures
Indicators should reflect strategic choices, may have management implications and aim at learning. They may be and often are political

20. Measuring capacity by its effects
Measuring capacity by its effects
In NORHED the effects of capacity is identified as,
more and better research produced
more and better qualified graduates
“At the Programme level it is evident that there is a need for more comprehensive indicators for project outputs, achievements and outcomes beyond an annual summery of graduates produced and research papers published”
Norad, Evaluation report 7/2009

21. Illustration: Indicators to measure quantity and quality of research
Illustration: Indicators to measure quantity and quality of research
ARWU – Academic Ranking of World Universities (Shanghai Jiao)
HEEACT – Higher Education Evaluation and Accreditation Council of Taiwan
Number of article in the past 11 years
Number of articles last year
Number of citations for past 11 years
Number of highly cited papers
Number of articles in highly cited journals
Papers published in Nature and Science
Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index
Staff/alumni winning Nobel Prices and Field Medals
Does it say something about the quality of the institution?

22. Illustration: Indicators to measure institutions
Illustration: Indicators to measure institutions
ARWU – Academic Ranking of World Universities (Shanghai Jioa)
THE - Higher Times Education
Research Outputs
Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation Index
Per capita academic performance
Teaching and learning environment
Results on reputational survey on teaching
Staff to student ratio
Ratio PhD to BA degrees awarded by institutions measured against number of academic staff
Research volume, income and reputation.
Results of reputational survey
Research income scaled against staff
Number of articles in academic journals (Thompson Reuter) per staff member

23. 1.0 Outcome ( Strengthened Institutions)
Expanded and better qualified, gender balanced workforce
Increased, better and relevant knowledge base
Increased evidence base for policies and public management
NORHED IMPACT
1.1.1.
Output
1.1.2
1.1 Outcome
1.2 Outcome
1.3 Outcome
1.0 Outcome ( Strengthened Institutions)
1.2.1
1.2.2
1.3.1 Output
1.1.3 E
Here we have a results framework again. A results framwork is made up if many results chain. I fyo only hav one, you do not need at framwork .
I could have taken away the two lowest levels.
We have decided that the Outcome i Strengthened Insitutions – within certan prgramm area - that is outcome level.
That is a kind of a institutional outcome. The insitution is strentened so it is possible to produce more students and

24. Contribution to impact Are we missing some indicators
Contribution to impact Are we missing some indicators
Expanded and better qualified, gender balanced workforce
Increased, better and relevant knowledge base
Increased evidence base for policies and public management
Two remaining questions;
How to measure contribution to better qualified work force?
How do we demonstrate the relevance of knowledge?
How do we verify that there is and increased evidence base for polices?
Can we demonstrate the influence of our research findings?

25. How to measure high quality teaching?
How to measure high quality teaching?
Suggestions for composite indicators
Curriculum includes/updated with knowledge that is relevant for the local setting
Curriculum in adapted to the global knowledge base
Share of lecturers with PhD
Share of lecturers that with research experience that is relevant for the MA programme

26.
How to demonstrate the relevance knowledge from the six sub-programmes and how it influence society?
Suggestion: Reporting in narrative
Can we organize the reporting on the relevance of research findings according to thematic categories or research questions?
How to report on influence? Should we put different requirements for reports on different types of research?
Basic
Applied
Technology and Innovation
R&D for management of public and natural resources
Action research

27. Do we need a communication strategy?
Do we need a communication strategy?
Good stories - Anecdotal evidence of relevance and influence
A more systematic approach?