What Might a Theory-Based Roadmap for Prospective Evaluation and Developing Innovation Policy Look Like? Presented at American Evaluation Association Conference November 2009 Gretchen Jordan, Sandia National Laboratories Portions of the work presented here were completed for the U.S. DOE Office of Science by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the author. SAND Number: C

G. Jordan AEA November Prospective evaluation in context National interest (SoSP, SciSIP) Models of what is known about innovation, what we need to know Theories and an example Conclusions Outline

G. Jordan AEA November Evaluation in the Policy Cycle Foresight Technology Roadmapping Technology Assessment Wolfgang Polt

G. Jordan AEA November National Interest: SoSP and SciSIP The science of science policy (SoSP) is an emerging field of interdisciplinary research, the goal of which is to provide a scientifically rigorous, quantitative basis from which policy makers and researchers can assess the impacts of the Nation’s scientific and engineering enterprise, improve their understanding of its dynamics, and assess the likely outcomes. A National Science and Technology Council (NSTC) Interagency Task Group (ITG) The Science of Science & Innovation Policy (SciSIP) program was established at NSF in John Marburger April 2005

G. Jordan AEA November SoSP Workshop in December 2008 Primary Conclusion of SoSP Roadmap: “Expert judgment” remains the best available decision support tool for science policy makers, but a nascent community of practice is emerging in the science policy arena that holds enormous potential to provide rigorous and quantitative decision support tools in the near future. ” The White House SoSP Interagency Task Group should take the lead to set the Federal agency research agenda.

G. Jordan AEA November White House S&T Priorities for the FY 2011 Budget Agencies should describe in their budget submission how they are prioritizing activities toward four challenges and strengthening four cross-cutting areas (which include productivity of research institutions) Expecting outcomes of research in above areas, providing quantitative metrics where possible Building capacity to rigorously evaluate programs, and how assessments have been used to eliminate or reduce programs Operating in the open innovation model and supporting long term high-risk, high payoff research Agencies will: Develop outcome oriented goals for S&T, target investment toward high performers, develop ‘science of science policy” tools that can improve management and assessment of impact -Peter Orszag, John Holdren, August 4, 2009

G. Jordan AEA November The SoSP Roadmap 10 Science Questions 1. What Are The Behavioral Foundations Of Innovation? 2. What Explains Technology Development, Adoption And Diffusion? 3. How And Why Do Communities Of Science And Innovation Form And Evolve? 4. What Is The Value Of The Nation’s Public Investment In Science? 5. Is It Possible To “Predict Discovery”? 6. Is It Possible To Describe The Impact Of Discovery On Innovation? 7. What Are The Determinants Of Investment Effectiveness? 8. What Impact Does Science Have On Innovation And Competitiveness? 9. How Competitive Is The U.S. Scientific Workforce? 10. What Is The Relative Importance Of Different Policy Instruments In Science Policy? Theme 1: Understanding Science and Innovation Theme 2: Investing in Science and Innovation Theme 3: Using the Science of Science Policy to Address National Priorities The National Imperative Science Questions Findings Recommendations Source: J. Lane, April 2009

G. Jordan AEA November Blind Men and the Elephant Parts are studied and understood better than the whole! Source: Bhavya Lal, STPI, at AEA 2006

G. Jordan AEA November A Science of Science and Innovation Policy must build a theory that connects levels Research Team Research Organization The Sector’s Idea Innovation Network The Sector’s National and Global Context micro meso macro

G. Jordan AEA November Anticipate effects of Scientific discovery Anticipate effects of science on R&D Understand technology development & diffusion Understand behavioral foundations Understand network behaviors Science Workforce competitiveness Impacts on competitiveness, etc. Assess real time value of new knowledge Relative importance of policy instruments Determinants of investment effectiveness SoSP Roadmap Questions Rearranged into a Three Level Logic Model Draft by G. Jordan 12/12/2008 Investment, incentives, Use People & organizational inputs & incentives The (non-linear) S&T, R&D process Understanding a multi-level eco-system Macro Micro Meso

G. Jordan AEA November The science must explain relationships among institutions Demand Consumers (final demand) Producers (intermediate demand) Industrial system Education and research system Political system Government Governance RTD Policies Professional education and training Higher education and research Public sector research Large companies Mature small/ medium enterprises (SMEs) New, technology- based firms Infrastructure Intermediaries Research institutes Brokers Banking, venture capital IPR and information Innovation and business support Standards and norms Framework conditions Financial environment; taxation and incentives; propensity to innovation and entrepreneurship; mobility A National Innovation System Model The potential reach of public policies… Demand Consumers (final demand) Producers (intermediate demand) Industrial system Education and research system Political system Government Governance RTD Policies Professional education and training Higher education and research Public sector research Large companies Mature small/ medium enterprises (SMEs) New, technology- based firms Infrastructure Intermediaries Research institutes Brokers Banking, venture capital IPR and information Innovation and business support Standards and norms Framework conditions Financial environment; taxation and incentives; propensity to innovation and entrepreneurship; mobility Source: Arnold and Kuhlman, 2001 A National Innovation System Model The potential reach of public policies…

G. Jordan AEA November Source: G. Jordan, Modified from R. Cooper/ Exxon’s Stage Gate, Hage & Hollingsworth’s Idea Innovation Network Marketing R&D, Quality R&D Diffusion and use Engineering & manufacturing R&D Connectivity and Throughput Production, Refinement Micro, meso, macro impacts 9 10 The science must explain connections among arenas of research and development

Confirmation Awareness Persuasion Decision Implementation Feedback Continued adoption Later adoption Discontinuance Continued rejection Adoption Rejection Product Characteristics Relative advantage Compatibility Complexity Trialability Observability Characteristics of the decision-making unit Adopter type Personality type Communication behavior Socio-economic status Socio-cultural/market environment Market structure Market segments Prior practice Culture and norms Innovativeness Communication field Broadcast Contagion Source: Everett Rogers 1994 as modified by Innovologie, LLC The science must understand Diffusion and relate it to R&D

G. Jordan AEA November All this information is useful to predict where and how policy makers can intervene to achieve desired goals Socio-cultural/market environment Market structure Market segments Prior practice Culture and norms Innovativeness Socio-cultural/market environment Market structure Market segments Prior practice Culture and norms Innovativeness Interventions at micro, meso, and/or macro levels?

G. Jordan AEA November Theories that could be integrated to understand how we can drive innovation Research Team –Management of innovation literature, learning theory Research Organization –Organizational innovation theories –Research Profiles theory Science/technological Sector –Idea Innovation Network on S&T/R&D process –Network theories –Diffusion theory –Sector economic models National and global context –Modes of coordination theories –Institutional and institutional change theory –Policy decision making –theories of

G. Jordan AEA November One possible decision tool to identify bottlenecks, policy objectives & effectiveness Socio economic outcomes Technical progress Network connectedness Organizational profiles – do attributes match the profile? RTD arenas – are there sufficient funds Portfolios - need more/ less radical, large scope? Modes of coordination – effective? Capabilities – Level, mix, availability High risk capital – available where Basic research Manufacturing research Applied research Development research Quality research Commercialization research Macro- Institutional Rules as they affect the sector Micro - funds allocation by arena and profile INNOVATION Meso - Performance by Tech sector and arena Policy Objectives -Structural -Technical Source: Jordan, Hage, and Mote, 2006, 2007, 2008

G. Jordan AEA November Conclusion Innovation occurs within a multi-level, complex, dynamic eco-system Prospective evaluation predicts Prediction requires understanding, characterization, theory There are theories that can be used now Synthesis of existing theories and building new theories are needed going forward.

G. Jordan AEA November Selected References Arnold, E. (2004). Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation, 13(1), 3-17 Hage, Jerry, G.B. Jordan and J. Mote (2007). A Theories-Based Innovation Systems Framework for Evaluating Diverse Portfolios of Research: Part Two - Macro Indicators and Policy Interventions. Science and Public Policy, 34(10): Jordan, G. B., Hage, J., & Mote, J A theories-based systemic framework for evaluating diverse portfolios of scientific work, part 1: Micro and meso indicators. In C.L.S. Coryn & Michael Scriven (Eds.), Reforming the evaluation of research. New Directions for Evaluation, 118, 7–24. Jordan, G.B Factors Influencing Advances in Basic and Applied Research: Variation Due to Diversity in Research Profiles. In Innovation, Science, and Institutional Change: A Handbook of Research, J. Hage and M. Meeus (eds). Oxford University Press: Oxford, Mote, J., Y. Whitestone, G. Jordan and J. Hage Innovation, Networks and the Research Environment: Examining the Linkages. International Journal of Foresight and Innovation Policy 4(3): Reed, John H, G. Jordan, Using Systems Theory and Logic Models to Define Integrated Outcomes and Performance Measures in Multi-program Settings, in Research Evaluation, Volume 16 Number 3 September 2007.