The World Bank Monitoring and evaluation of science, technology & innovation An International Perspective

The World Bank M&E a core element in strengthening national S&T policies Accountability: Demand from elected officials and the public for accountability for the use of resources Accountability: Demand from elected officials and the public for accountability for the use of resources Policy-Making: Guidance on policy choices, reforms and innovations, and knowledge of bottlenecks in national innovation system Policy-Making: Guidance on policy choices, reforms and innovations, and knowledge of bottlenecks in national innovation system Program Improvement: Information to track implementation of S&T policies and to decide on improvements Program Improvement: Information to track implementation of S&T policies and to decide on improvements

The World Bank Key challenges to developing M&E systems in Latin America Lack of transparency in results achieved with public resources invested in R&D Lack of transparency in results achieved with public resources invested in R&D Fragmentation in responsibilities for collecting and reporting data Fragmentation in responsibilities for collecting and reporting data Unreliable statistics, notably in regard to private sector R&D Unreliable statistics, notably in regard to private sector R&D Low capacity to process and analyze data Low capacity to process and analyze data

The World Bank Monitoring and evaluation Monitoring: continuous process of collecting and analyzing information to compare how well a project, program or policy is being implemented against expected plans or results Monitoring: continuous process of collecting and analyzing information to compare how well a project, program or policy is being implemented against expected plans or results Evaluation: systematic assessment of planned, on-going or completed interventions to determine their relevance, appropriateness, effectiveness, efficiency, impact or sustainability Evaluation: systematic assessment of planned, on-going or completed interventions to determine their relevance, appropriateness, effectiveness, efficiency, impact or sustainability

The World Bank Responding to the demand for information Developing theory of change and sector strategy Developing theory of change and sector strategy Determining what and how to measure Determining what and how to measure Organizational arrangements for collecting information Organizational arrangements for collecting information Capacity to analyze and use collected information Capacity to analyze and use collected information

The World Bank Developing theory of change and sector strategy – determining what to monitor Strategy implies the movement of a sector from its present stage to a desirable but uncertain future stage. Because the country/sector has never been to this future position, its intended pathway involves a series of linked hypotheses Strategy implies the movement of a sector from its present stage to a desirable but uncertain future stage. Because the country/sector has never been to this future position, its intended pathway involves a series of linked hypotheses A M&E module is a method to document and test the assumptions inherent in the strategy A M&E module is a method to document and test the assumptions inherent in the strategy

The World Bank Measuring the middle ground Generally weak articulation of the mechanisms through which inputs and activities are expected to result in the desired impact Generally weak articulation of the mechanisms through which inputs and activities are expected to result in the desired impact Milestones and inter- mediate measures are important in S&T where final outcomes may take years to emerge Milestones and inter- mediate measures are important in S&T where final outcomes may take years to emerge Inputs Intervention Outcomes Outputs Goal (Impacts) Results Activities ? ?

The World Bank Transition to a knowledge- based economy Improved human capital Grants for international cooperation Cooperative research consortia Insertion of researchers into industry Sustained growth State of the art scientific community Improved private sector research capacity Enhanced public- private linkages and innovation Coherent strategy for innovation and human capital Conducive innova- tive policies and instruments Continual monitoring and evaluation Strengthened Chile’s national innovation system Acquisition of S&T equipment Advanced Human Capital Excellence in science Causality tree Represents the best guess as to an appropriate course of action, given the best available knowledge Represents the best guess as to an appropriate course of action, given the best available knowledge Make relationships explicit so they can be monitored, validated and managed Make relationships explicit so they can be monitored, validated and managed

The World Bank Indicators and incentives ‘Indicators’ rather than ‘measures’ ‘Indicators’ rather than ‘measures’ Risk: “What you measure is what you get” Risk: “What you measure is what you get” Academic research: risk of cementing an ‘ivory tower’ mentality by assigning too much importance to bibliometrics Academic research: risk of cementing an ‘ivory tower’ mentality by assigning too much importance to bibliometrics Develop indicators for desirable behavior, e.g. collaboration, entrepreneurship, mobility Develop indicators for desirable behavior, e.g. collaboration, entrepreneurship, mobility

The World Bank Apply international standards and data manuals (Oslo and Frascati) to facilitate international benchmarking Determining what and how to measure Types of indicators Direct measures, e.g. annual PhD graduation and bibliometrics Direct measures, e.g. annual PhD graduation and bibliometrics Proxies, e.g. average country citation impact factor Proxies, e.g. average country citation impact factor Perception variables, e.g. transparency of funding allocation mechanisms Perception variables, e.g. transparency of funding allocation mechanisms

The World Bank Challenge of measuring private sector innovation Understanding determinants for turning science into business essential to ensuring that investments in S&T translated into increased competitiveness and growth Understanding determinants for turning science into business essential to ensuring that investments in S&T translated into increased competitiveness and growth Company innovation surveys, e.g. Chile ( ; ; ) Company innovation surveys, e.g. Chile ( ; ; ) Challenge: To be useful, content needs to be agreed between several sectors and agencies, e.g. Ministry of Education, Ministry of Economy and National Statistical Agency Challenge: To be useful, content needs to be agreed between several sectors and agencies, e.g. Ministry of Education, Ministry of Economy and National Statistical Agency

The World Bank Marginal effect of company characteristics on patents, product innovation and process innovation Interpretation: Collaboration with universities increases the probability 35 percent of being involved in patenting, keeping all other factors constant

The World Bank Organizational arrangements for collecting information Assigning responsibilities: Clear agreement on who does what, when and how Assigning responsibilities: Clear agreement on who does what, when and how Consider the cost of the M&E system, e.g. build on existing reporting structures Consider the cost of the M&E system, e.g. build on existing reporting structures Providing an incentive to collect and report valid data by ensuring that requested information is useful for decision-making at both central and decentralized level Providing an incentive to collect and report valid data by ensuring that requested information is useful for decision-making at both central and decentralized level Avoiding heavy bureaucratic burdens on researchers taking time away from core activities Avoiding heavy bureaucratic burdens on researchers taking time away from core activities

The World Bank Systems of performance reporting Use standardized reporting to increase transparency, ensure comparability and reduce administrative load Use standardized reporting to increase transparency, ensure comparability and reduce administrative load Essential to ensure collection of longitudinal data rather than a number of cross-sections Essential to ensure collection of longitudinal data rather than a number of cross-sections

The World Bank National capacity for analyzing data Unit with skills and capacity to process and analyze data, undertaking studies, and suggesting policy implications Unit with skills and capacity to process and analyze data, undertaking studies, and suggesting policy implications Chile: National Observatory for S&T Chile: National Observatory for S&T Technical rigor in designing evaluations (process, impact): challenge of using quasi-experimental designs in small innovation systems using competitive resource allocation mechanisms Technical rigor in designing evaluations (process, impact): challenge of using quasi-experimental designs in small innovation systems using competitive resource allocation mechanisms

The World Bank Impact evaluation: Difference-in-difference design T1T1 T2T2 ΔYCΔYC ΔYTΔYT ΔYT- ΔYCΔYT- ΔYC Treatment group Competitively selected Control group Cost-effectiveness: Evaluate against best alternative investment rather than the no treatment scenario Cost-effectiveness: Evaluate against best alternative investment rather than the no treatment scenario

The World Bank Impact evaluation: Challenge of selection bias T1T1 T2T2 Treatment group Selection bias: Controlling for past trends Control group ΔYCΔYC ΔYTΔYT ΔYT- ΔYCΔYT- ΔYC Ex-ante ΔY T A baseline is a line, not a point A baseline is a line, not a point

The World Bank Summing up M&E essential for developing and maintaining effective ST&I polices M&E essential for developing and maintaining effective ST&I polices Need to ground M&E in national strategy for ST&I Need to ground M&E in national strategy for ST&I Indicators shape incentives Indicators shape incentives Base M&E systems on international data manuals Base M&E systems on international data manuals Place increased emphasis on understanding the determinants for turning science into business Place increased emphasis on understanding the determinants for turning science into business Develop capacity to use M&E for learning and improving ST&I policies Develop capacity to use M&E for learning and improving ST&I policies

The World Bank Thank you Kristian Thorn