Management of assessments and decision making: execution, facilitation, evaluation Mikko V. Pohjola, Nordem Oy (THL)

Basic problem Lankinen, et al.: State research institutes and research funding: proposal on a comprehensive reform: – “Preparation, decision-making and implementation of policy should be based on researched information and evidence.…the use of research- based informati on in social policy decision-making, must be strengthened” If we want policy to be ”knowledge-based” and ”effective”… …what constitutes effective knowledge-based policy? and effective knowledge support to policy making?

Common perspectives to effectiveness Assessment (and modeling) Quality assurance / control Uncertainty analysis Technical assessment of models ”Effectiveness” Other Policy Outcomes (vs. objectives) Behavioral responses + CBA Monitoring Ex ante – ex post evaluation Factors limiting implementation ProcessUse Output Constraints Outcomes Context Modelling/assessment Enablers

Concepts Assessment – knowledge support for practical needs Policy – decisions and actions with societal relevance Outcomes – changes in the world resulting from decision and actions Practices – (common) ways of dealing with things Effectiveness – capability (or likelihood) of achieving intended outcomes Evaluation – analysis of the capability (or likelihood) of achieving intended outcomes

Framework for knowledge-based policy Policy making Assessment Outcomes Implementation (Knowledge) practices Evaluation & management Design Execution Follow-up Process Product Use Interaction Question Answer Interpretation

Framework for knowledge-based policy Focus on knowledge support to decision making – Decision makers' needs? – Societal needs? – Other needs? – Expert knowledge – Local knowledge – Opinions, views, perspectives, values – → collaborative search for questions and answers – → good decisions, desired changes in the world Evaluation and management of effectiveness -> proactive guidance of decisions and actions both now and in the future

Management of assessments and decision making ExecutionEvaluation AssessmentDecisionOutcomesAssessmentDecisionOutcomes Question Answer Interpretation Open assessment Openness Discussion Collaboration Shared understanding Information structure … Properties of good assessment Dimensions of openness Assessment-policy interaction Explicit purposes, aims … Monitoring Policy analysis … Explicit purposes, aims Common practices … Facilitation

Evaluation of decision support Five ”knowledge-based” approaches to effectiveness (cf. the Puijo-case): Dimensions of openness (DO) (Pohjola and Tuomisto 2011) Knowledge-policy interaction (KPI) (Pohjola manuscript) Properties of good assessment (PoGA) (Sandström et al. manuscript, Assessment effectiveness (Kauppinen et al. 2006) Relational evaluation approach (Koivisto and Pohjola 2011) Evaluation guides (the management of) execution! Not only afterwards, but also during and even before the making of an assessment

Evaluation Evaluation is possible only in relation to purpose! Taking account of (intended) use is essential Instrumental value of assessment Cf. absolute value (a common science view) Cf. Ad hoc solutions (a common practice view) => Assessment is a tool for policy-support (a means to an end) => A model is a tool for making assessments

Evaluation Why evaluation of decisions / decision support? Efficient use of resources? Value of work done? Importance/meaning of information? Implications of information? Actual impacts of information? … …because funder, customer, user, boss, peer, stakeholder etc. wants/needs to know!

Evaluation Assessment is business of creating understanding about reality – Making right questions – Providing good answers – Delivering the answers (and questions) to use Dual purpose of assessments: – Meeting the needs of use (decision making, policy) – Striving for truth (science)

KPI: how well does assessment address decision needs? Impacts Causes Problem owner Target Interaction

DO: how well can a decision process (or assessment) accept contributions? Scope of participation: who are allowed to participate in the process? Access to information: what information regarding the issue at hand is made available to participants Timing of openness: when are participants invited or allowed to participate? Scope of contribution: which aspects of the issue at hand participants are invited or allowed to contribute to? Impact of contribution: to what extent are participant contributions allowed to have influence on the outcomes, i.e. how much weight is given to participant contributions?

PoGA: how well can an assessment influence decisions and actions? Quality of content – Informativeness – Calibration – Coherence Applicability – Relevance – Availability – Usability – Acceptability Efficiency – Intra-assessment efficiency – Inter-assessment efficiencý

Properties of good assessment

Example: what makes a good hammer?

How is the hammer made? By who? What properties does the hammer have? What do you want to do with the hammer? How does the hammer help you do it?

Summary: evaluating decision support Purpose: Identify and explicate decision maker aims & decision criteria Identify and explicate other relevant needs Decision process: DO: Making use of assessments Decision support (assessment): PoGA & KPI: Serving decision needs DO: Making use of available knowledge

Homework 9 Look into two (other than your own) homework 3 draft assessments Characterize the assessment drafts according to Knowledge-policy interaction & Dimensions of openness Based on the characterization, evaluate the assessment drafts according to Properties of good assessment (modified) Write your development suggestions to the corresponding user pages Homework 3 deadline is Monday 4.2.! -> do your homework 9 after that!