Introduction to assessment performance Mikko Pohjola, THL

Contents Setting & concepts Common perspectives (& examples) Quality assurance/quality control Uncertainty analysis Model performance Properties of good assessment Summary & discussion

Setting Decision making under uncertainty – Information inputs Assessment results News Hearsay, gossip – Decision making Background knowledge Values/emotions Interpretation,cognition, communication – Outputs Decision(s) -> action(s) -> outcome(s)

Setting Assessment performance is about evaluating – Information...in use making of... – In a situation of incomplete knowledge Are there actually any other situations? How good is it? – Why? – How to evaluate? – What for?

Concepts Some basic concepts: Performance = goodness! Assessment, Management Model Process (making/using), Product Output, Outcome Assessor, Decision/Policy maker, Stakeholder Participant, User

Rationale Why evaluation of assessment performance is important? Efficient use of resources? Value of work done? Importance/meaning of information? Implications of information? Actual impacts of information? … …because funder, customer, user, boss, peer, stakeholder etc. wants/needs to know!

Roles and interests ExpertsData quality, analysis procedure, coherence, comprehensiveness, … FundersRelevance, efficiency, timeliness, importance, … Users (DM)Understandability, reliability (of source), acceptance, practicality, … Interested (SH)(same as DM, but different perspective)

General RA/RM framework Process, product, use

Common perspectives & examples Quality assurance/quality control Focus on assessment process An “engineering” perspective Uncertainty Focus on assessment output (product) A scientists perspective??? Model performance Focus on modelling and model Combines QA/QC and uncertainty perspectives A modellers perspective

Quality assurance/quality control Principle: “Good process guarantees good outputs/outcomes!” Question: “How should an assessment be done?” Examples: Ten steps by Jakeman et al.(2006) IDEA framework (Briggs, 2008) (Over)appreciation of randomized controlled trials (RCT’s)

Ten iterative steps in development and evaluation of environmental models Jakeman et al.: Ten iterative steps in development and evaluation of environmental models. Environmental Modelling & Software Issue 5, May 2006, Pages

IDEA framework (INTARESE) Briggs: A framework for integrated environmental health impact assessment of systemic risks. Environmental Health 2008, 7:61.

RCT as the ultimate study type? RCT results often regarded as the best possible kind of information – Does RCT suit all situations? – DO RCT's provide answers to all kinds of questions? – Are RCT results always better than results from any other kind of study? After all, RCT is just a study procedure!

Uncertainty analysis Principles: “Performance is an intrinsic property of an information product!” “The more accurate, the better!” Question: How good are the assessment results?

Uncertainty analysis Examples: Statistical uncertainty Mean, variance, confidence limits, distributions, … Cf. D. Lindley: Philosophy of Statistics, 2000 Sources of uncertainty For example model, parameter & scenario uncertainty (as applied e.g. by the U.S.EPA) Extensive approaches E.g. inclusion of qualitative aspects, sources of uncertainty as in NUSAP (

NUSAP N: numeral U: unit S: spread A: assessment (qualitative judgment) P: pedigree (historical path leading to result)

NUSAP - pedigree Jeroen van der Sluijs: NUSAP- some examples. Presentation. Available:

Model performance Principle: The model is the essence of the assessment! Question: How good is the model? Examples: Verification, validation, (reliability, usability, …) Outcome-oriented approach by Matthews et al. 2011

Outcome-oriented modelling approach Matthews et al.: Raising the bar? – The challenges of evaluating the outcomes of environmental modelling and software. Environmental Modelling & Software, March 2011, Pages

Summary of common perspectives Assessment process and outputs are addressed in many ways Use of results mostly not considered The link between outputs and outcomes (cf. Matthews et al. 2011) Evaluation often a separate process Expert processes of making assessments and using their results Expert processes of evaluating performance Alternative perspectives?

Properties of good assessment

Ex post (after assessment) evaluation Ex ante (before/during assessment) evaluation Guidance of design and execution Links process and output with use Thereby also linking them to outcomes

Example: what makes a good hammer?

How is the hammer made? By whom? What properties does the hammer have? What do you want to do with the hammer? How does the hammer help you do it? What is it that really makes a difference?

Summary Consideration of (intended) use is essential Consideration of process and product in light of use In policy-support information is a tool (a means to an end) In policy support, information is a tool – Consider the instrumental value of information Cf. absolute value (a common science view) Cf. Ad hoc solutions (a common practice view) Contextuality, situatedness, practicality, … A model is a tool for producing information How does this relate to the previous lectures about DA and the DA study plan exercise?

Discussion example: swine flu vaccination Because of urgence, swine flu vaccination was bought in Finland without a thorough testing. When narcolepsy cases were identified, the decision made without testing was seen as a major mistake. Was it a mistake? – How should we evaluate the situation to find an answer? – How did the decision-maker assess the situation? – How should she have assessed the situation?

Swine flu example: issues in performance? What are the critical issues in the assessment performance? Possibilities include e.g. – The assessment truthfully estimates the total health impact of swine flu. – The assessment truthfully estimates the health impact of a vaccination campaign. – The only tested vaccines are assessed. – The assessment does not underestimate potential side effects of the vaccine, whether tested or not. – Something else, what?

Swine flu example: follow-up as a part of assessment performance? What are the methods to identify if something starts to go on after the decision? Should these be assessed already in the assessment before the decision? How can this be done? Does this improve the assessment performance?

Setting