1 Toward a standard benefit-cost methodology for publicly-funded S&T programs Crossing Borders, Crossing Boundaries 2005 Joint CES/AEA Conference Toronto, Canada October 26-29, 2005 Jeanne Powell Economic Assessment Office Advanced Technology Program National Institute of Standards and Technology

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 2 Outline Background AEA 2005: Analysis of Differences and Considering an Approach to Comparability Moving Forward

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 3 Background ATP Mission and Project Selection Criteria Evaluation Framework Role and Types of Benefit-Cost Analysis Performance Metrics Benefit-Cost Studies To Date Study Similarities

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 4 ATP Mission … ATP Mission … To accelerate the development of innovative technologies for broad national benefit through partnerships with the private sector. Chemistry Biotechnology Electronics Manufacturing AdvancedMaterials Photonics InformationTechnology

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 5 Scientific and Technological Merit (50%) – Technical innovation – High technical risk with evidence of feasibility – Detailed technical plan Potential for Broad-Based Economic Benefits (50%) – National economic benefits – Need for ATP funding – Pathway to economic benefits ATP Selection Criteria

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 6 What We Measure When ECONOMIC IMPACTS Short-TermMid-TermLonger-Term 10 or More Years AnnounceCompe-titionAnnounceAward Complete Project Post-Project Period Total Economic Benefits Benefits to Awardees (Inputs/Outputs) Award/participant characteristics Project funding R&D partnering Acceleration of R&D Innovative technology development – Patents – Publications – Competitive advantage – Prototype products & processes (Outputs/Outcomes) Commercial activity – New products – New processes – Licensing Attraction of capital Strategic alliances Company growth (Impacts) Broad national economic benefits – Return on investment – Public – Private – Social – Inter-industry diffusion – Increased GDP/ tax base – Societal impacts

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 7 Types of B-C Case Studies in ATP Traditional cash-flow based, return on investment measures – Benefit-to-cost ratios – Internal rates of return – Net present value Hedonic price index models Macroeconomic models

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 8 Benefit-cost Analysis: Return on Investment Measures Private return – return to ATP project participants on their own investment Public return – return to nation on ATP investment Social return – return to ATP project participants and nation on total ATP and private investment

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 9 Performance Metrics Return on Investment Metrics Benefit-to-Cost ratio is computed by dividing the present value of benefits by the present value of investments Net present value (NPV) is the present value of benefits minus the present value of investments Internal rate of return is calculated by iterative solution for a rate at which the discounted value of investments equals the discounted value of benefits

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 10 ATP’s B-C Studies To Date 12 studies published, including quantitative case studies of 28 projects 2 studies underway, including quantitative case studies of 4 projects All studies performed by independent contractors

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 11 ATP’s B-C Studies To Date Short Title (# of Projects)NumberContractor Pub Date Study Cost-$ Photonics Cluster (2)GCR Delta Research ,000 Composites Cluster (2) GCR Delta Research ,000 2mm Project—Retrospective (1)GCR MIT ,000 HDTV Joint Venture (1)GCR RTI International ,000 A-Si Detectors for Digital Mammography (1) GCR Delta Research ,000 Component-Based Software (8)GCR RTI International ,000 Closed-Cycle Refrigeration (1)GCR Delta Research ,000 Digital Data Storage (2)GCR RFF ,000 Flow-Control Machining in Auto Industry (1) NISTIR 6373NIST-Building & Fire Research ,000 Tissue Engineering (7)GCR RTI International ,000 2mm—Early Assessment (1)GCR CONSAD ,000 Printed Wiring Board (1)GCR Albert N. Link ,500

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 12 Study Similarities Address ATP Mission, emphasizing public benefits to industry customers and end users and need for ATP funding to achieve these benefits Cash-flow based approaches use: – Good practices consistent with public finance literature; NPV, B:C, IRR – OMB-mandated 7% real discount rate (all but 1 study) and constant $ estimates – Year of ATP project start for base year – Relatively short study periods Results are not presented as “representative”, but rather as portfolio minimums

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 13 AEA 2005 Analyzing Differences and Considering an Approach to Study Comparability

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 14 AEA 2005: Analyzing Differences and Considering an Approach to Study Comparability Analyzing four major differences : 1) Study timing relative to ATP funding and uncertainties about future outcomes 2) Identifying the counterfactual and attribution of benefits 3) Which metrics? Public versus social return on investment 4) Adjusting for timing differences across studies and projects

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 15 AEA 2005: Analyzing Differences and Considering an Approach to Study Comparability Considering a standard approach based on three criteria: 1) Meeting evaluation objective 2) Quality and accuracy 3) Comparability

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Study Timing and Uncertainties About Future Outcomes Issues: Uncertainties concerning technology benefits are reduced over course of innovation process – At time of ATP funding, risks of technical failure (or of meeting only limited technical objectives) are very high. – Market/financial/business risks remain high during and after ATP funding phases of technology development and into commercialization phases – Innovation life-cycle/timing varies by technology area Earliest studies conducted early in innovation process – No projects had matured – Direct experience with range of possible outcomes (and empirical data about innovation process) was limited

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Study Timing and Uncertainties About Future Outcomes: Contrasting Studies Study Timing Relative to ATP Funding Technology StatusTreatment of Uncertainty Tissue Engineering (7 cases) Early in ATP for 3 cases; soon after ATP for 4 cases Technologies not developed and/or not ready for commercialization (Biotech projects have very long timeline) Barriers to meeting technical and economic goals not assessed. Sensitivity analysis performed but showed results not very sensitive to input variables tested. Considered only one application per case in order to be conservative Component -Based Software (8 cases) Soon after ATPTechnologies complete; in commercialization (IT projects have short timeline) Selection emphasized projects with revenues to date and near- term prospects. Benefit analysis quantified only products actually on the market and assumed short product lives.

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Study Timing and Uncertainties About Future Outcomes: Contrasting Studies Notes: Tissue Engineering: 3 technologies made considerable technical progress and commercialization continues to evolve but much more slowly than anticipated; 3 technologies failed to develop and companies dissolved; 1 is in transition to new company, future as yet is uncertain Component-Based Software: focus on existing products and short-term product life reduced the amount of research required per project and enabled study of more projects

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Study Timing and Uncertainties About Future Outcomes: Conclusions Under conditions of considerable technical/business uncertainty about future outcomes: Rigorous, complex economic models may or may not generate useful performance metrics. Probability-based outcome measures that consider the broad technical, market, and business risks, as well as technology benefits, are likely needed Metrics are more reliable/capable of precision as projects overcome technical, market, and financial barriers

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Identifying the Counterfactual and Attribution of Benefits Issue: Where does project being analyzed start and end? And what is the counterfactual, or baseline? – In theory, investment costs and benefits included in analysis need to measure increment over the counterfactual (what would have happened without the project being measured). – In practice, benefits (e.g., profit from sale of a product embodying new technology) derive from multiple public and private investments and from one or two of many product generations.

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Identifying the Counterfactual and Attribution of Benefits—Contrasting Studies StudyApproach Tissue Engineering ATP project benefits are defined in terms of acceleration of benefits and increased probability of technical success relative to a hypothetical project without ATP. Public plus private benefits are compared to public plus private investments to compute social return on social investment. Component-Based Software Identified “project” that resulted in identifiable product sales and compared public/private benefits with all costs associated with that project to compute social returns A-Si Detectors for Digital Mammography and Composites Company interviews indicated “no project without ATP”; public benefits from product sales are compared to ATP investment to compute public return on ATP investment Photonics Technologies 50% attribution to ATP where two different federal programs providing similar levels of funding were deemed jointly responsible for realization of any benefits.

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Identifying the Counterfactual and Attribution of Benefits—Conclusions Researchers use a variety of mechanisms to model benefits relative to a counterfactual (e.g., increased probability of achieving benefits, acceleration, allocation in accordance with amount or importance of funding sources) ATP-funded technologies are often embodied in multiple products, each of which resulted from a number of different public/private funding sources and “projects”. Researchers measure benefits of products with strongest ATP linkages and share attribution where needed Social rate of return has weaker assumption about “causation” than public return on ATP $ does.

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Which metrics? Social return metrics – Economics literature compares social rate of return to private rate of return (to see economic spillover) and then further compares private rate of return to private hurdle rate to identify market failure that justifies public funding – Cost-shared projects imply comparison of benefits to both public and private recipients with investments from all sources AND/OR Return on ATP investment – Directly addresses: What is the ATP program impact? – Involves substantially less historical/company proprietary information than social return metrics

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Which metrics? Contrasting Studies Issue: Some studies/contractors emphasize social return metrics; others emphasize public return

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Which metrics? Contrasting Studies Social return metrics Public return metrics (NC = Not computed) NPV ($M) B:CIRR (%)NPV ($M)B:CIRR(%) Component- based Software N/MNC NC N/MNC NC NC NC NC NC

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Which metrics? Contrasting Studies Social return metrics (NC = Not Computed) Public return metrics (NC = Not Computed) NPV ($M)B:CIRR (%)NPV ($M)B:CIRR Digital Video (Midpoint) NC A-Si Detector (base case) NC Composites (base cases) 1NC NC Photonics (base cases) 1NC NC

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Which metrics? Social versus Public Returns—Conclusions Both have merit for different purposes, and both have advantages and disadvantages – Social return metrics reflect cost-sharing aspect of ATP and other public-private partnership projects; however, they require company-proprietary data and historical data about multiple funding sources (generally only rough estimate is feasible). – Public return on ATP investment provides measure of direct program impact; however, it requires judgments about of “causation”

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Issue: Different case studies based cash flow estimates on constant dollars of different years Although nearly all studies used the same discount rate (7%-real), they discounted cash flow estimates to different points in time (start of each project). (This practice evolved in financial world’s prospective analysis of potential investments and is used in Excel.) Thus, reported NPV metrics are not comparable across projects.

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year E xample 1: Ranking does not change Component- Based Software 8 Cases Reported NPV ($M) Constant $ Year Base Year for Discounting Adj. NPV-- Constant 2005$ ($M) Adj. NPV as Base Year ($M) Aesthetic Solutions-1.2 (7) (7) Commerce One JV789 (1) ,513 (1) Extempo-1.22 (8) (8) Intermetrics29.6 (3) (3) Real-Time Innovations 2.06 (5) (5) SciComp21 (4) (4) Tom Sawyer52 (2) (2) Xerox PARC1.2 (6) (6)

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example 2 : Ranking Changes Other Cash-Flow – Based Cases Reported NPV ($M) Constant $ Year Base Year for Discounting Adj. NPV-- Constant 2005$ ($M) Adj. NPV as Base Year ($M) Digital Video165.9 (6) (5) A-Si Detector219 (3) (4) Composites: Applied Sciences Lincoln Composites 552 (1) 510 (2) (1) 860 (2) Photonics: X-Ray Optics Ion Optics 184 (5) 201 (4) (3) 288 (6)

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 31 Adjusting to Constant 2005$ NPV in Constant 2005$ = Where: IPDGDP is Implicit Price Deflator for GDP and IPDGDPstudy is IPDGDP for Constant $ year used in study 1 + Q IPDGDP – IPDGDPstudy IDPGDPstudy x Reported NPV

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 32 Adjusting Base Year to 2005 Adjusted NPV for Base Year 2005 = NPV in Constant 2005$ x 1.07 n Where: n = 2005 minus base year used in study

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example Note: Rankings “before and after” adjustment are for illustrative purposes only. – NPVs indicate net benefit magnitudes – NPVs do NOT indicate investment efficiency

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Conclusions: Both computed value of NPV and rank order can change by adjustment to constant $ and common base year NPV adjustments to constant $ and common base year are easy to make—directly from reported NPV Comparability and aggregation of NPV results indicate need for such adjustments although practitioners may differ on best approach IRR and B:C do not require adjustment; however, they cannot be aggregated

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 35 Moving Forward Work with contractors/researchers toward greater understanding of need for comparability of B-C results while seeking improved methodologies Recognize and build on strengths of individual contractors/researchers ; help them overcome inconsistencies with comparability goals Improve understanding of B-C methodologies among S&T evaluation community Work with NIST experts in both B-C analysis and standards development

National Institute of Standards and Technology Technology Administration U.S. Department of Commerce 36 Call Jeanne Powell: Send to: Visit our website: View our publications: For further information: