Dr. Sanjaya Lall Professor, Developmental Economics Oxford University, UK

Former Senior Economist, World Bank Advisor, Technology Policy, Innovation, Competitiveness, Industrial Technology, Foreign Direct Investment, Trade, Monitoring & Evaluation Systems Former Senior Research Officer, Institute of Economics and Statistics, Oxford Dr. Sanjaya Lall

Monitoring and evaluation of innovation projects & programs Sanjaya Lall

Need for monitoring and evaluation (M&E) All investment proposals need to be evaluated if they compete for resources All projects and programs have to be monitored for effective implementation, meeting deadlines and ensuring discipline in spending, purchasing and so on All projects and programs must be evaluated during and after completion to ensure that direct and social objectives are achieved

M&E is exceptionally difficult for innovation projects & programs Highly speculative and uncertain nature of innovation results: problem exists even within enterprises in allocating R&D budgets But public projects face other problems: Public goods nature of many outputs Non-measurable nature of benefits of programs to Create inter-firm & inter-industry linkages Strengthen linkages between industry and research institutions Even more diffuse nature of stimulating technology culture, modernization etc.

Nature of M&E depends on nature of innovation project Upgrade S&T institutions Stimulate enterprise R&D Strengthen linkages of TIs with industry Improve IPR regime/ implementa tion Fund technology based firms (VCs, etc.)

Two major components Benefits to the economy Achievement of specific project objectives

Inherently qualitative and diffuse nature of innovation benefits gives rise to large number of possible M&E measures

Economic benefits: three levels of analysis Trends at the general economy-wide or industrial level Before and after project International benchmarks Analysis of enterprises and institutions covered by project Comparisons of enterprises within project with those outside it

Economic benefits: measures R&D spending Patents, publications Competitiveness (in technology products?) Labour productivity (or TFP) Spin-offs, technopreneurs Employment (total or skilled, mfg or R&D?) International R&D collaboration Technology balance of payments Inward and outward FDI (technology based)

Other possible measures, more industry specific… Quality standards and reject rates Order to delivery times Customer satisfaction ratings Inter-firm collaboration Cluster effects: local purchases, joint training and R&D, other links

Economic benefits: sources of data General industrial, trade and R&D data within country Benchmarks from other countries Industrial census (with additional data on technology) Questionnaires to firms & institutions within and outside project Interviews with firms and institutions in project

Economic benefits: methodology Benchmarking over time or across countries Econometric analysis of enterprises within & outside project Case studies

Project objectives: three components (for implementing agencies) Inputs Outputs Outcomes

Inputs… Extent to which implementation meets targets on schedule, e.g.: Disbursement of project funds Purchases of equipment Construction of facilities Other milestones

Outputs, depending on agency… Speed of processing patent applications Extent of metrology needs met, conformance to international standards % of budget earned from industry (contracts or research) Restructuring, international accreditation etc. of laboratories No. of R&D contracts, commercialization, no. of publications (impact), patents Number of technology loans, repayment rates Venture capital funding: extent, success

Outcomes (broad economic impact)… Part are already covered under analysis of economic benefits More specific measures of performance by institutions and firms directly involved in project

Specific performance measures Enterprises Growth and nature of R&D R&D management Access to technology financing Access to MSTQ services Efficiency of IPR regime Extent of interaction with technology institutions, universities, etc. Export and productivity performance

R&D: Internal institutional culture, responsiveness to and links with industry, spin-offs MSTQ: Increased efficiency of services, effects on productivity, competitiveness, FDI VC: Stimulation of hi-tech start-ups, IPOs General: Improvement of R&D performance and creation of technology culture, use of new technologies in production & exports, attraction of MNC R&D facilities and contracts, outward FDI, cluster effects. Specific performance measures: institutions

Who does M&E? Project agencies provide information on project inputs and outputs Data is collected by statistical & project agencies, questionnaires administered by independent researchers Analysis carried out by independent researchers

Dr. Sanjaya Lall Professor, Developmental Economics Oxford University, UK