1. National Institute of Standards and Technology U.S. Department of Commerce Technology Program Evaluation: Methodologies from the Advanced Technology Program Richard N. Spivack Robert Sienkiewicz Impact Analysis Office Technology Innovation Program National Institute of Standards and Technology Collaborative Expedition Workshop National Science Foundation March 18, 2008

2. National Institute of Standards and Technology U.S. Department of Commerce ATP Mission To accelerate the development of innovative technologies for broad national benefit through partnerships with the private sector

3. National Institute of Standards and Technology U.S. Department of Commerce Economic Rationale for ATP Addresses market failure or market imperfections in early-stage, generic technologies –Due to high technical risk and uncertainty about outcome of R&D activity, long expected time to commercialization, and uncertain potential markets. –Also due to spillover effects and inability of innovator to fully appropriate returns on investment. –The above leads to disincentive for private sector to invest in technologies that could benefit society as a whole.

4. National Institute of Standards and Technology U.S. Department of Commerce Need to Measure Success “Scientists do, of course, make judgments all the time about promising lines of research … It makes sense for the world’s largest sponsor of research, the U.S. government, to want to make such choices as wisely as the most productive scientists do … But is it possible to decide rationally when to enhance or to terminate a project if we do not possess a way of measuring its success?” John Marburger, President’s Science Advisor (keynote speaker at the 2002 American Association for the Advancement of Science’s 27 th annual Colloquium on Science and Technology Policy)

5. National Institute of Standards and Technology U.S. Department of Commerce Evaluation is Important for Public Programs An agency should demonstrate that it has: a plan and organizational capacity to conduct assessment of its R&D activities; an explicit set of performance metrics; an organizational commitment to evaluate its performance; and credible results that it can produce. These are important elements in annual budget reviews by the executive and legislative branches in the United States as well as other countries where performance management is a priority.

6. National Institute of Standards and Technology U.S. Department of Commerce Evaluation Began Early ATP evaluation activities began early –pre-GPRA 1993, pre-President’s Management Agenda 2001 Economic Assessment Office charged with measuring the impact of the program. –Staff of 16 economists, statisticians, information specialists, plus contractors. ATP budget for evaluation had grown from $25K to $2-5M per year. EAO uses multiple methods to measure direct and indirect impacts of project funding.

7. National Institute of Standards and Technology U.S. Department of Commerce Multiple Approaches to Evaluation Statistical profiling of –applicants, projects, participants, technologies Real time monitoring of research and business activities –ATP project teams –technical and business progress reports Progress measures collected through –surveys –Business Reporting System –other databases Statistical and econometric analysis –research productivity –collaboration effects Developing and testing new tools and methods

8. National Institute of Standards and Technology U.S. Department of Commerce Multiple Approaches to Evaluation, con’t Microeconomic case studies of projects or clusters of projects –benefit and cost estimates –private, public, and social return on investment –follow on macroeconomic impact studies Policy/special issue studies –early-stage tech funding –regional effects Status reports on completed projects –mini case studies –star rating for portfolio analysis –cross-cutting analyses Expert reviews –ATP Advisory Committee –National Research Council Comparisons with foreign counterpart programs

9. National Institute of Standards and Technology U.S. Department of Commerce Evaluation Activities tied to Timing of Expected Results Ex ante peer review for project selection Survey tools to monitor project progress Performance measures Expert reviews Portfolio-wide analysis –status reports of completed projects that rate progress against ATP mission. Post-project surveys and data analyses In-depth and cluster case studies—return on investment –net present value –benefit-cost ratio –internal rate of return social, public, private Econometric analysis Macroeconomic analysis short and mid-term longer term

10. National Institute of Standards and Technology U.S. Department of Commerce Portfolio-wide Analysis Status reports of completed projects –descriptive mini-case studies. Provides a portfolio analysis of project performance. Each project receives a rating between 0 and 4 stars on how well it met mission objectives –overall project performance = knowledge creation and dissemination + commercialization progress and diffusion + future outlook. Aggregation of stars provides portfolio of program success. Patent trees show knowledge spillovers.

11. National Institute of Standards and Technology U.S. Department of Commerce Benefit-Cost Studies Microeconomic analytical framework using a multidisciplinary approach. –technology assessment; industry structure; competitive dynamics; microeconomic analysis; and corporate finance measures Uses widely accepted economic reasoning to develop estimates of impacts of program funding on project timing and success, based on: –spillovers; counterfactuals; sensitivity analysis; valuation metrics Calculates net present values, internal rates of return, and benefit-cost ratios. Aggregation of project impact studies point to an 8:1 return on program investments of $2.1B.

12. National Institute of Standards and Technology U.S. Department of Commerce Evaluation Results are Used for Program Management Fully integrated program evaluation –Design –Implementation –Assessment –Learning and Feedback Result: Continuous program improvement –Project selection –Project management –Outreach

13. National Institute of Standards and Technology U.S. Department of Commerce Additional Uses for Evaluation Results … Meet external stakeholder requests for program results and official requirements; Gain new insights into key relationships; Improve understanding of ATP’s contribution to the U.S. innovation system; Document performance; and Promote and maintain external support.

14. National Institute of Standards and Technology U.S. Department of Commerce Evaluation Best Practices Management and institutional commitment to performance evaluation. Dedicated and appropriate mix of expert staff. Committed and steady budget. Multi-faceted approach to evaluation. Commissioning external studies with experts.

15. National Institute of Standards and Technology U.S. Department of Commerce Evaluation Best Practices, con’t Matching assessment methods to questions posed. Systematic data collection and regular reporting systems. Pursuit of development and testing of innovative methods to evaluate impact. Examination of successful and unsuccessful projects. Strategic communication of results.

16. National Institute of Standards and Technology U.S. Department of Commerce Continuing Challenges for Program Evaluation Managing change. –Data continuity versus evolving information needs Driving survey redesign through key output measures. Balancing program needs with broader economic research objectives. Balancing stories and case studies with portfolio analysis. Effective internal feedback mechanisms for project management and program design. Benchmarking to other programs and industry. Knowledge management.

17. National Institute of Standards and Technology U.S. Department of Commerce Final Thoughts “The simple act of measuring and reporting on results will itself promote improvements.” “Too much monitoring can cripple efforts in the early stage; too little and the value will be lost.” Wholey, Hatry, Newcomer Handbook of Practical Evaluation, pp. 102, 110

18. National Institute of Standards and Technology U.S. Department of Commerce Contact Information Richard N. Spivack, PhD Technology Innovation Program National Institute of Standards and Technology U.S. Department of Commerce phone: (301) 975-5063 fax: (301) 975-4776 email: richard.spivack@nist.gov Robert Sienkiewicz, PhD Technology Innovation Program National Institute of Standards and Technology U.S. Department of Commerce phone: (301) 975-4969 fax: (301) 975-4776 email: robert.sienkiewicz@nist.gov