National Public Health Institute, Finland Open risk assessment Lecture 7: Evaluating assessment performance Mikko Pohjola KTL, Finland

National Public Health Institute, Finland Lecture contents Properties of good assessments Relations between properties and parts of assessments Assessment performance Evaluation of assessment performance Group work: evaluation of performance

National Public Health Institute, Finland Properties of good assessments

National Public Health Institute, Finland Properties of good assessments Quality of content –How good a description of the given part of reality this is? Applicability –How well does the information within the description transfer to its use process (decision making)? Efficiency –f(quality of content, applicability | purpose) / effort expenditure –How good output, given the effort, is/was produced?

National Public Health Institute, Finland Quality of content Informativeness –How exact is the description? Quantitatively: how tight is the spread, how narrow is the distribution? Calibration –How close to reality is the description? Quantitatively: distance of central estimate to ”true” value? Relevance –How well does the description answer to the specific question?

National Public Health Institute, Finland Applicability Usability –How well is the information understood by its intended users? Language, terms, representation and illustration, clarity, … Availability –How well do the intended users get access to the information according to their needs? Who, when, where, how receives the information? –In relation to identified information need

National Public Health Institute, Finland Applicability Acceptability –How well is the information, and conclusions based on it accepted (taken into use, internalized) By intended users and by stakeholders –Acceptability of premises: value judgments and assumptions behind the assessment E.g. severity weight or cost of specific disease, use of DALYs and/or money as aggregate measures, … –Acceptability of process: methods, means and materials used in making the assessment E.g. data and data sources, openness (transparency), scientific reliability of methods (and assessors), …

National Public Health Institute, Finland ”Effectiveness” Quality of content and applicability are together called effectiveness How much effect does (or can) the assessment have regarding its intended use? E.g. How great impact it has regarding the decision making where it is used? As the use process (decision making) is considered as being outside the sphere of assessment, effectiveness is thought of as ”potential for effect”

National Public Health Institute, Finland Efficiency Intra-assessment efficiency –How much effectiveness (quality of content and applicability) per spent effort is/was achieved in a particular assessment? Inter-assessment efficiency –How much effectiveness (quality of content and applicability) per spent effort is/was achieved in a series of assessments?

National Public Health Institute, Finland Relations Quality of content -> product Applicability –> product, process and use Efficiency – assessment process

National Public Health Institute, Finland Relations Quality of content –Informativeness, calibration -> assessment/variable result –Relevance -> assessment/variable scope Applicability –Usability -> presentation of content –Availability -> assessment process – use process interaction –Acceptability of premises -> scientific context, policy context –Acceptability of process -> assessment process, assessment/variable definition Effectiveness –Assessment process

National Public Health Institute, Finland Assessment performance Performance = f(quality of content, applicability, efficiency | purpose) Purpose must always be defined and explicated! –General purpose = to describe reality –Instrumental purpose = to answer to a specific need No deliberate distortion of results Specific assessment question must be defined

National Public Health Institute, Finland Evaluation of assessment performance Evaluation of product and process in terms of the properties against the purpose Quantitative evaluation where possible Informativeness, calibration, efficiency (effort expenditure) Qualitative or semi-quantitative evaluation for the rest No good semi-quantitative evaluation methods yet known to be available

National Public Health Institute, Finland Evaluation of assessment performance A priori: applying the performance principles in making an assessment –Continuous evaluation of assessment performance Definition of purpose and scope Identification and explication of assumptions Information processing means and methods Process management, e.g. openness, communication, tools Formulation and presentation of outputs …

National Public Health Institute, Finland Evaluation of assessment performance A posteriori: evaluating performance of an existing already made assessment –Same principles as a priori, but an afterward check of performance Evaluating only a posteriori is always too late –Evaluation part of the group work case is an a posteriori evaluation exercise

National Public Health Institute, Finland Group work: evaluation of performance Evaluate identification and explication of assessment purpose –Intended use process –Intended users –Important stakeholders –Policy context If identification is incomplete, consider how the purpose should/could have been identified and explicated

National Public Health Institute, Finland Group work: evaluation of performance Evaluate quality of content –Both on assessment and variable level: Scope Definition Result Evaluation can be done against both: –The purpose identified and explicated in the assessment –The purpose suggested by the evaluator

National Public Health Institute, Finland Group work: evaluation of performance Evaluate applicability against purpose –”Packaging” of the information –Assessment – use interaction During and after assessment –Identification and explication of assumptions –Explication and choice of used methods, means and materials Again evaluation against both ”real” and suggested purpose as seen necessary Also compare original format and new format

National Public Health Institute, Finland Group work: evaluation of performance Evaluate efficiency –In practice: effort expenditure Person time Money Consider both efforts in original assessment and conversion work

National Public Health Institute, Finland Group work: evaluation of performance Compile an overall evaluation of performance –Performance = f(quality of content, applicability, efficiency | purpose) Efficiency = f(quality of content, applicability | purpose) / effort expenditure –Performance = f(quality of content, applicability | purpose) / effort expenditure Consider possible improvements Does ORA appear useful in improving assessment performance?