1. Evaluating Risk: The Experience of the Advanced Technology Program (ATP) Stephen Campbell Technology Innovation Program (TIP), NIST American Evaluation Association November 13, 2009

2. Outline Why so narrow a focus for this talk? – Session title – Small number of items can provide several paths of analysis Something concrete – Making risk operational in surveys – Findings from project selection – Findings from project outcomes Something abstract – Why should we care about risk? – Loose terminology

3. ATP Mission… To accelerate the development of innovative technologies for broad national benefit through partnerships with the private sector

4. Rationale for ATP Under-investment (from society’s perspective) in early-stage technologies – Inadequate information for making investment decisions (information asymmetries between entrepreneurs and investors) – Difficulty in appropriating benefits of early-stage, enabling technologies (externalities and spillovers)

5. Risk Questions (Two Dimensions) Competition Years 2000 and 2002 – “From 0 to 100 percent, what would you say is the approximate probability that your proposed ATP project could fully achieve its technical goals?” – “From 0 to 100 percent, what would you say is the approximate probability that a typical R&D project at your company could fully achieve its technical goals?” Competition Year 2004 – “For your ATP project, consider the technical risk in achieving your minimal technical goals. What is the probability, from 0% to 100%, that your company can achieve the minimal technical targets required for success on this project?” – “For your ATP project, consider the technical risk in achieving your maximal ‘stretch’ technical goals. What is the probability, from 0% to 100%, that your company can achieve the maximal ‘stretch’ technical goals for this project?” – “What would say is the probability, from 0 to 100 percent, that a typical R&D project at your company can achieve all technical targets required for project success?”

6. Project Selection – Meeting Mission Is there evidence that the program is selecting higher risk projects? Two dimensions – Absolute risk Awardees have 45% risk (55% probability of technical success) Non-awardees have 28% risk (72% probability of technical success) – Relative risk (differential between ATP project and typical project) Awardees have 14% ATP risk premium (45% versus 31%) Non-awardees have a 4% risk premium (28% versus 24%)

7. “Stretch” Goals Minimal, Maximal, and Typical – Awardees and non-awardees have similar relative profiles – Minimal is very close to typical (Minimal slightly higher than typical for awardees) Awardee versus Non-awardee – Awardees have higher minimal and maximal risk than non-awardees

8. Evaluation Panel “Value-Add” Leveraging data from NIST technical reviewers and evaluation panel recommendations Evidence of an additional “risk-premium” from evaluation panel above NIST technical reviews

9. Higher Risk and Project Outcomes Awarded projects have a higher risk profile than non-awarded projects Risk-Reward trade-off suggests that, among awarded projects, higher risk project should have lower rates of commercial success (with larger impacts is successful) Consistent findings that higher risk projects have higher rates of success – Commitment – Planning

10. Why are we worried about risk? Risk relevant through information asymmetries Risk versus uncertainty Future measure for new program – Probabilistic measures for consistency – Questions around timing and “nature” of risk