1. Benefit-Cost Analysis: ATP Experience Jeanne Powell Economic Consultant j.w.powell@verizon.net Technology Program Evaluation: Methodologies from the Advanced Technology Program March 18, 2008 National Science Foundation

2. Jeanne Powell, Economic Consultant2 Benefit-Cost Analysis: Outline Benefit-Cost Analysis of Federally-funded Technology: What is it? What is it good for? What metrics are computed? Whose return on investment? How does it work? ATP Studies Advantages Disadvantages Summary

3. Jeanne Powell, Economic Consultant3 Benefit-Cost Analysis What is it? Evaluation tool for: – Estimating net impacts of a federal technology development project or group of related projects according to program: Mission Logic Model Timeline – Comparing total estimated costs with total estimated benefits over a project’s useful life Can be Prospective (reflect projected future effects), or Retrospective (reflect past effects), or a mix of both

4. Jeanne Powell, Economic Consultant4 What is it good for? Quantitative case study Economic estimation Generally most suited to long-term evaluation – Risks and uncertainties of R&D are large in early phases; therefore, the probability distributions of the economic estimates are broad until technical and business uncertainties are reduced over the longer term Generally most suited to applied R&D, close to market

5. Jeanne Powell, Economic Consultant5 What does it measure? Return on Investment: – Net benefits – Benefit-to-cost ratio – Internal rate of return What investment? – Depends on whose return is being measured

6. Jeanne Powell, Economic Consultant6 Whose Return? Social Return Analysis Private return – return to project participants on their own investment/cost share Private profits to technology innovator Spillover return – return to others beyond the project participants, including broad societal benefits, on combined sources of investment Benefits to technology users (intermediate and final) Broader societal benefits (e.g., public goods) Social return – return to project participants and nation broadly on combined public program and private investments needed to generate return

7. Jeanne Powell, Economic Consultant7 Whose Return? Public Return Analysis Public return – Increased social return enabled by public investment relative to that public investment – Additionality effect – If multiple public sources contributed to the benefit, need to allocate benefit

8. Jeanne Powell, Economic Consultant8 How Does It Work? (1) Usually is based on monetary valuation of multiple types of benefits expressed as cash-flow analysis

9. Jeanne Powell, Economic Consultant9 Cash Flows for Project Time Investments Cash Flow Benefits Offsets

10. Jeanne Powell, Economic Consultant10 How Does It Work? (2) Cash flows are time adjusted – To constant dollars – For opportunity cost of money OMB Circular A-94 mandates use of a 7% real rate. As a result, ATP has used a 7% real rate in nearly all benefit-cost studies Discount rate consistency will be important for comparing results of different investments Cash flows are most commonly discounted back to the start of the project/investment being analyzed Cash flows are adjusted for risks and uncertainties using expected value analysis Metrics are computed—typically with Excel

11. ATP’s Studies To Date Short Title (# of Projects) Pub Number Contractor Pub Date Green Technologies Cluster (2)GCR 06-897Delta Research2007 DNA Diagnostics Cluster (2)GCR 06-898RTI International2007 Photonics Cluster (2)GCR 05-879Delta Research2005 Composites Cluster (2)GCR 04-863Delta Research2004 2mm Project—Retrospective (1)GCR 03-856MIT2004 HDTV Joint Venture (1)GCR 03-859RTI International2004 A-Si Detectors for Digital Mammography (1)GCR 03-844Delta Research2003 Component-Based Software (8)GCR 02-834RTI International2002 Closed-Cycle Refrigeration (1)GCR 01-819Delta Research2001 Digital Data Storage (2)GCR 00-790RFF2000 Flow-Control Machining in Auto Industry (1)NISTIR 6373NIST-BFRL1999 Tissue Engineering (7)GCR 97-737RTI International1998 2mm—Early Assessment (1)GCR 97-709CONSAD1997 Printed Wiring Board (1)GCR 97-722Albert N. Link1997

12. Jeanne Powell, Economic Consultant12 Advantages (Cash-flow analysis approach) Provides quantitative estimates of impact that complement qualitative analysis Based on standard financial analysis used by private businesses. (The public finance literature has extended them to the analysis of public and societal benefits.) Models are intuitive and relatively easy for policy makers and others to understand Can be applied systematically and relatively uniformly Can be extended to analyze multiple projects

13. Jeanne Powell, Economic Consultant13 Disadvantages (cash-flow analysis approach) Modeling of spillover benefits is challenging and expensive as with other tools Many important spillover benefits may be difficult to express in monetary terms Private benefits are often confidential Results are sensitive to uncertainties about technical success, market adoption, and inter- industry diffusion

14. Jeanne Powell, Economic Consultant14 Advantages and Disadvantages Summary Benefit-cost analysis is a practical choice for long- term evaluation of program impacts Utility increases as empirical basis increases: Technical accomplishment Product development Entrepreneurial activity (including financing) Market adoption Economic diffusion

15. Jeanne Powell, Economic Consultant15 Send e-mail to: j.w.powell@verizon.net Visit ATP/TIP website: www.atp.nist.gov View ATP/TIP publications: www.atp.nist.gov/eao/eao_pubs.htm For further information: