1. National Public Health Institute, Finland www.ktl.fi Open risk assessment Lecture 7: Evaluating assessment performance Mikko Pohjola KTL, Finland

2. National Public Health Institute, Finland www.ktl.fi Lecture contents Properties of good assessments Relations between properties and parts of assessments Assessment performance Evaluation of assessment performance Group work: evaluation of performance

3. National Public Health Institute, Finland www.ktl.fi Properties of good assessments

4. National Public Health Institute, Finland www.ktl.fi Properties of good assessments Quality of content –How good a description of the given part of reality this is? Applicability –How well does the information within the description transfer to its use process (decision making)? Efficiency –f(quality of content, applicability | purpose) / effort expenditure –How good output, given the effort, is/was produced?

5. National Public Health Institute, Finland www.ktl.fi Quality of content Informativeness –How exact is the description? Quantitatively: how tight is the spread, how narrow is the distribution? Calibration –How close to reality is the description? Quantitatively: distance of central estimate to ”true” value? Relevance –How well does the description answer to the specific question?

6. National Public Health Institute, Finland www.ktl.fi Applicability Usability –How well is the information understood by its intended users? Language, terms, representation and illustration, clarity, … Availability –How well do the intended users get access to the information according to their needs? Who, when, where, how receives the information? –In relation to identified information need

7. National Public Health Institute, Finland www.ktl.fi Applicability Acceptability –How well is the information, and conclusions based on it accepted (taken into use, internalized) By intended users and by stakeholders –Acceptability of premises: value judgments and assumptions behind the assessment E.g. severity weight or cost of specific disease, use of DALYs and/or money as aggregate measures, … –Acceptability of process: methods, means and materials used in making the assessment E.g. data and data sources, openness (transparency), scientific reliability of methods (and assessors), …

8. National Public Health Institute, Finland www.ktl.fi ”Effectiveness” Quality of content and applicability are together called effectiveness How much effect does (or can) the assessment have regarding its intended use? E.g. How great impact it has regarding the decision making where it is used? As the use process (decision making) is considered as being outside the sphere of assessment, effectiveness is thought of as ”potential for effect”

9. National Public Health Institute, Finland www.ktl.fi Efficiency Intra-assessment efficiency –How much effectiveness (quality of content and applicability) per spent effort is/was achieved in a particular assessment? Inter-assessment efficiency –How much effectiveness (quality of content and applicability) per spent effort is/was achieved in a series of assessments?

10. National Public Health Institute, Finland www.ktl.fi Relations Quality of content -> product Applicability –> product, process and use Efficiency – assessment process

11. National Public Health Institute, Finland www.ktl.fi Relations Quality of content –Informativeness, calibration -> assessment/variable result –Relevance -> assessment/variable scope Applicability –Usability -> presentation of content –Availability -> assessment process – use process interaction –Acceptability of premises -> scientific context, policy context –Acceptability of process -> assessment process, assessment/variable definition Effectiveness –Assessment process

12. National Public Health Institute, Finland www.ktl.fi Assessment performance Performance = f(quality of content, applicability, efficiency | purpose) Purpose must always be defined and explicated! –General purpose = to describe reality –Instrumental purpose = to answer to a specific need No deliberate distortion of results Specific assessment question must be defined

13. National Public Health Institute, Finland www.ktl.fi Evaluation of assessment performance Evaluation of product and process in terms of the properties against the purpose Quantitative evaluation where possible Informativeness, calibration, efficiency (effort expenditure) Qualitative or semi-quantitative evaluation for the rest No good semi-quantitative evaluation methods yet known to be available

14. National Public Health Institute, Finland www.ktl.fi Evaluation of assessment performance A priori: applying the performance principles in making an assessment –Continuous evaluation of assessment performance Definition of purpose and scope Identification and explication of assumptions Information processing means and methods Process management, e.g. openness, communication, tools Formulation and presentation of outputs …

15. National Public Health Institute, Finland www.ktl.fi Evaluation of assessment performance A posteriori: evaluating performance of an existing already made assessment –Same principles as a priori, but an afterward check of performance Evaluating only a posteriori is always too late –Evaluation part of the group work case is an a posteriori evaluation exercise

16. National Public Health Institute, Finland www.ktl.fi Group work: evaluation of performance Evaluate identification and explication of assessment purpose –Intended use process –Intended users –Important stakeholders –Policy context If identification is incomplete, consider how the purpose should/could have been identified and explicated

17. National Public Health Institute, Finland www.ktl.fi Group work: evaluation of performance Evaluate quality of content –Both on assessment and variable level: Scope Definition Result Evaluation can be done against both: –The purpose identified and explicated in the assessment –The purpose suggested by the evaluator

18. National Public Health Institute, Finland www.ktl.fi Group work: evaluation of performance Evaluate applicability against purpose –”Packaging” of the information –Assessment – use interaction During and after assessment –Identification and explication of assumptions –Explication and choice of used methods, means and materials Again evaluation against both ”real” and suggested purpose as seen necessary Also compare original format and new format

19. National Public Health Institute, Finland www.ktl.fi Group work: evaluation of performance Evaluate efficiency –In practice: effort expenditure Person time Money Consider both efforts in original assessment and conversion work

20. National Public Health Institute, Finland www.ktl.fi Group work: evaluation of performance Compile an overall evaluation of performance –Performance = f(quality of content, applicability, efficiency | purpose) Efficiency = f(quality of content, applicability | purpose) / effort expenditure –Performance = f(quality of content, applicability | purpose) / effort expenditure Consider possible improvements Does ORA appear useful in improving assessment performance?