1. 1 Toward a standard benefit-cost methodology for publicly-funded S&T programs Crossing Borders, Crossing Boundaries 2005 Joint CES/AEA Conference Toronto, Canada October 26-29, 2005 Jeanne Powell Economic Assessment Office Advanced Technology Program National Institute of Standards and Technology

2. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 2 Outline Background AEA 2005: Analysis of Differences and Considering an Approach to Comparability Moving Forward

3. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 3 Background ATP Mission and Project Selection Criteria Evaluation Framework Role and Types of Benefit-Cost Analysis Performance Metrics Benefit-Cost Studies To Date Study Similarities

4. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 4 ATP Mission … ATP Mission … To accelerate the development of innovative technologies for broad national benefit through partnerships with the private sector. Chemistry Biotechnology Electronics Manufacturing AdvancedMaterials Photonics InformationTechnology

5. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 5 Scientific and Technological Merit (50%) – Technical innovation – High technical risk with evidence of feasibility – Detailed technical plan Potential for Broad-Based Economic Benefits (50%) – National economic benefits – Need for ATP funding – Pathway to economic benefits ATP Selection Criteria

6. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 6 What We Measure When ECONOMIC IMPACTS Short-TermMid-TermLonger-Term 10 or More Years -1 0 1 2 3 4 5 6 7 8 9 AnnounceCompe-titionAnnounceAward Complete Project Post-Project Period Total Economic Benefits Benefits to Awardees (Inputs/Outputs) Award/participant characteristics Project funding R&D partnering Acceleration of R&D Innovative technology development – Patents – Publications – Competitive advantage – Prototype products & processes (Outputs/Outcomes) Commercial activity – New products – New processes – Licensing Attraction of capital Strategic alliances Company growth (Impacts) Broad national economic benefits – Return on investment – Public – Private – Social – Inter-industry diffusion – Increased GDP/ tax base – Societal impacts

7. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 7 Types of B-C Case Studies in ATP Traditional cash-flow based, return on investment measures – Benefit-to-cost ratios – Internal rates of return – Net present value Hedonic price index models Macroeconomic models

8. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 8 Benefit-cost Analysis: Return on Investment Measures Private return – return to ATP project participants on their own investment Public return – return to nation on ATP investment Social return – return to ATP project participants and nation on total ATP and private investment

9. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 9 Performance Metrics Return on Investment Metrics Benefit-to-Cost ratio is computed by dividing the present value of benefits by the present value of investments Net present value (NPV) is the present value of benefits minus the present value of investments Internal rate of return is calculated by iterative solution for a rate at which the discounted value of investments equals the discounted value of benefits

10. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 10 ATP’s B-C Studies To Date 12 studies published, including quantitative case studies of 28 projects 2 studies underway, including quantitative case studies of 4 projects All studies performed by independent contractors

11. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 11 ATP’s B-C Studies To Date Short Title (# of Projects)NumberContractor Pub Date Study Cost-$ Photonics Cluster (2)GCR 05-879Delta Research2005 228,000 Composites Cluster (2) GCR 04-863Delta Research2004 200,000 2mm Project—Retrospective (1)GCR 03-856MIT2004 357,000 HDTV Joint Venture (1)GCR 03-859RTI International2004 75,000 A-Si Detectors for Digital Mammography (1) GCR 03-844Delta Research2003 136,000 Component-Based Software (8)GCR 02-834RTI International2002 154,000 Closed-Cycle Refrigeration (1)GCR 01-819Delta Research2001 77,000 Digital Data Storage (2)GCR 00-790RFF2000 138,000 Flow-Control Machining in Auto Industry (1) NISTIR 6373NIST-Building & Fire Research 1999 90,000 Tissue Engineering (7)GCR 97-737RTI International1998 122,000 2mm—Early Assessment (1)GCR 97-709CONSAD1997 25,000 Printed Wiring Board (1)GCR 97-722Albert N. Link1997 22,500

12. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 12 Study Similarities Address ATP Mission, emphasizing public benefits to industry customers and end users and need for ATP funding to achieve these benefits Cash-flow based approaches use: – Good practices consistent with public finance literature; NPV, B:C, IRR – OMB-mandated 7% real discount rate (all but 1 study) and constant $ estimates – Year of ATP project start for base year – Relatively short study periods Results are not presented as “representative”, but rather as portfolio minimums

13. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 13 AEA 2005 Analyzing Differences and Considering an Approach to Study Comparability

14. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 14 AEA 2005: Analyzing Differences and Considering an Approach to Study Comparability Analyzing four major differences : 1) Study timing relative to ATP funding and uncertainties about future outcomes 2) Identifying the counterfactual and attribution of benefits 3) Which metrics? Public versus social return on investment 4) Adjusting for timing differences across studies and projects

15. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 15 AEA 2005: Analyzing Differences and Considering an Approach to Study Comparability Considering a standard approach based on three criteria: 1) Meeting evaluation objective 2) Quality and accuracy 3) Comparability

16. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 16 1. Study Timing and Uncertainties About Future Outcomes Issues: Uncertainties concerning technology benefits are reduced over course of innovation process – At time of ATP funding, risks of technical failure (or of meeting only limited technical objectives) are very high. – Market/financial/business risks remain high during and after ATP funding phases of technology development and into commercialization phases – Innovation life-cycle/timing varies by technology area Earliest studies conducted early in innovation process – No projects had matured – Direct experience with range of possible outcomes (and empirical data about innovation process) was limited

17. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 17 1. Study Timing and Uncertainties About Future Outcomes: Contrasting Studies Study Timing Relative to ATP Funding Technology StatusTreatment of Uncertainty Tissue Engineering (7 cases) Early in ATP for 3 cases; soon after ATP for 4 cases Technologies not developed and/or not ready for commercialization (Biotech projects have very long timeline) Barriers to meeting technical and economic goals not assessed. Sensitivity analysis performed but showed results not very sensitive to input variables tested. Considered only one application per case in order to be conservative Component -Based Software (8 cases) Soon after ATPTechnologies complete; in commercialization (IT projects have short timeline) Selection emphasized projects with revenues to date and near- term prospects. Benefit analysis quantified only products actually on the market and assumed short product lives.

18. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 18 1. Study Timing and Uncertainties About Future Outcomes: Contrasting Studies Notes: Tissue Engineering: 3 technologies made considerable technical progress and commercialization continues to evolve but much more slowly than anticipated; 3 technologies failed to develop and companies dissolved; 1 is in transition to new company, future as yet is uncertain Component-Based Software: focus on existing products and short-term product life reduced the amount of research required per project and enabled study of more projects

19. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 19 1. Study Timing and Uncertainties About Future Outcomes: Conclusions Under conditions of considerable technical/business uncertainty about future outcomes: Rigorous, complex economic models may or may not generate useful performance metrics. Probability-based outcome measures that consider the broad technical, market, and business risks, as well as technology benefits, are likely needed Metrics are more reliable/capable of precision as projects overcome technical, market, and financial barriers

20. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 20 2. Identifying the Counterfactual and Attribution of Benefits Issue: Where does project being analyzed start and end? And what is the counterfactual, or baseline? – In theory, investment costs and benefits included in analysis need to measure increment over the counterfactual (what would have happened without the project being measured). – In practice, benefits (e.g., profit from sale of a product embodying new technology) derive from multiple public and private investments and from one or two of many product generations.

21. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 21 2. Identifying the Counterfactual and Attribution of Benefits—Contrasting Studies StudyApproach Tissue Engineering ATP project benefits are defined in terms of acceleration of benefits and increased probability of technical success relative to a hypothetical project without ATP. Public plus private benefits are compared to public plus private investments to compute social return on social investment. Component-Based Software Identified “project” that resulted in identifiable product sales and compared public/private benefits with all costs associated with that project to compute social returns A-Si Detectors for Digital Mammography and Composites Company interviews indicated “no project without ATP”; public benefits from product sales are compared to ATP investment to compute public return on ATP investment Photonics Technologies 50% attribution to ATP where two different federal programs providing similar levels of funding were deemed jointly responsible for realization of any benefits.

22. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 22 2. Identifying the Counterfactual and Attribution of Benefits—Conclusions Researchers use a variety of mechanisms to model benefits relative to a counterfactual (e.g., increased probability of achieving benefits, acceleration, allocation in accordance with amount or importance of funding sources) ATP-funded technologies are often embodied in multiple products, each of which resulted from a number of different public/private funding sources and “projects”. Researchers measure benefits of products with strongest ATP linkages and share attribution where needed Social rate of return has weaker assumption about “causation” than public return on ATP $ does.

23. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 23 3. Which metrics? Social return metrics – Economics literature compares social rate of return to private rate of return (to see economic spillover) and then further compares private rate of return to private hurdle rate to identify market failure that justifies public funding – Cost-shared projects imply comparison of benefits to both public and private recipients with investments from all sources AND/OR Return on ATP investment – Directly addresses: What is the ATP program impact? – Involves substantially less historical/company proprietary information than social return metrics

24. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 24 3. Which metrics? Contrasting Studies Issue: Some studies/contractors emphasize social return metrics; others emphasize public return

25. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 25 3. Which metrics? Contrasting Studies Social return metrics Public return metrics (NC = Not computed) NPV ($M) B:CIRR (%)NPV ($M)B:CIRR(%) Component- based Software 1 -1.2.37N/MNC 278939363NC 3-1.22.63N/MNC 429.69.6103NC 52.061.831NC 6217.651NC 75218136NC 81.2 13NC

26. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 26 3. Which metrics? Contrasting Studies Social return metrics (NC = Not Computed) Public return metrics (NC = Not Computed) NPV ($M)B:CIRR (%)NPV ($M)B:CIRR Digital Video (Midpoint) 165.94.2428.6NC A-Si Detector (base case) NC 21912569 Composites (base cases) 1NC 55222157 2NC 51018758 Photonics (base cases) 1NC 431847549 2NC 51201244350

27. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 27 3. Which metrics? Social versus Public Returns—Conclusions Both have merit for different purposes, and both have advantages and disadvantages – Social return metrics reflect cost-sharing aspect of ATP and other public-private partnership projects; however, they require company-proprietary data and historical data about multiple funding sources (generally only rough estimate is feasible). – Public return on ATP investment provides measure of direct program impact; however, it requires judgments about of “causation”

28. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 28 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Issue: Different case studies based cash flow estimates on constant dollars of different years Although nearly all studies used the same discount rate (7%-real), they discounted cash flow estimates to different points in time (start of each project). (This practice evolved in financial world’s prospective analysis of potential investments and is used in Excel.) Thus, reported NPV metrics are not comparable across projects.

29. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 29 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year E xample 1: Ranking does not change Component- Based Software 8 Cases Reported NPV ($M) Constant $ Year Base Year for Discounting Adj. NPV-- Constant 2005$ ($M) Adj. NPV-- 2005 as Base Year ($M) Aesthetic Solutions-1.2 (7)20001995-1.3-2.6 (7) Commerce One JV789 (1)200019978801,513 (1) Extempo-1.22 (8)200019951.36-2.68 (8) Intermetrics29.6 (3)2000199833.053.0 (3) Real-Time Innovations 2.06 (5)200019962.304.23 (5) SciComp21 (4)200019952346 (4) Tom Sawyer52 (2)2000199658107 (2) Xerox PARC1.2 (6)200019951.32.6 (6)

30. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 30 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example 2 : Ranking Changes Other Cash-Flow – Based Cases Reported NPV ($M) Constant $ Year Base Year for Discounting Adj. NPV-- Constant 2005$ ($M) Adj. NPV-- 2005 as Base Year ($M) Digital Video165.9 (6)20021995177.7349.5 (5) A-Si Detector219 (3)20021998236374 (4) Composites: Applied Sciences Lincoln Composites 552 (1) 510 (2) 2003 1997 1998 579 535 996 (1) 860 (2) Photonics: X-Ray Optics Ion Optics 184 (5) 201 (4) 2004 1994 2000 188 206 396 (3) 288 (6)

31. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 31 Adjusting to Constant 2005$ NPV in Constant 2005$ = Where: IPDGDP is Implicit Price Deflator for GDP and IPDGDPstudy is IPDGDP for Constant $ year used in study 1 + Q2 2005 IPDGDP – IPDGDPstudy IDPGDPstudy x Reported NPV

32. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 32 Adjusting Base Year to 2005 Adjusted NPV for Base Year 2005 = NPV in Constant 2005$ x 1.07 n Where: n = 2005 minus base year used in study

33. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 33 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example Note: Rankings “before and after” adjustment are for illustrative purposes only. – NPVs indicate net benefit magnitudes – NPVs do NOT indicate investment efficiency

34. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 34 4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Conclusions: Both computed value of NPV and rank order can change by adjustment to constant $ and common base year NPV adjustments to constant $ and common base year are easy to make—directly from reported NPV Comparability and aggregation of NPV results indicate need for such adjustments although practitioners may differ on best approach IRR and B:C do not require adjustment; however, they cannot be aggregated

35. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 35 Moving Forward Work with contractors/researchers toward greater understanding of need for comparability of B-C results while seeking improved methodologies Recognize and build on strengths of individual contractors/researchers ; help them overcome inconsistencies with comparability goals Improve understanding of B-C methodologies among S&T evaluation community Work with NIST experts in both B-C analysis and standards development

36. National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 36 Call Jeanne Powell: 301-975-4196 Send e-mail to: jeanne.powell@nist.gov Visit our website: www.atp.nist.gov View our publications: www.atp.nist.gov/eao/eao_pubs.htm For further information: