TIA Consulting, Inc. Portfolio Evaluation: Toward a Common Evaluation Methodology for Funding Decisions Across Multiple Interdependent R&D Portfolios of a U.S. Agency New Frontiers In Evaluation Plattform FTEval April 24-25, 2006 Vienna, Austria Rosalie T. Ruegg Managing Director TIA Consulting, Inc.

TIA Consulting, Inc. 2 Background: Retrospective Evaluation of DOE Applied Energy R&D Programs $Billions allocated to federal applied energy R&D programs. Evaluations used different models, data, and assumptions across programs. Congressional appropriators expressed frustration with evaluations offered in support of their budget decisions. 2000, Congress ordered comprehensive retrospective study of these programs (2001 NRC Retrospective Evaluation Report).

TIA Consulting, Inc. 3 Retrospective Evaluation Matrix Realized Benefits and Costs Options Benefits and Costs Knowledge Benefits and Costs Economic Benefits and Costs Environmental Benefits and Costs Security Benefits and Costs

TIA Consulting, Inc. 4 Accompanying Tool for “Loosely” Characterizing Uncertainty in Retrospective Study Technology Development Economic/ Policy Conditions Technology Developed Technology Development in Progress Technology Development Failed Will be favorable for commercial- ization Realized benefitsKnowledge benefits Might become Favorable for commercialization Options benefitsKnowledge benefits Will not become favorable for commercialization Knowledge benefits Source: NRC Report, 2001.

TIA Consulting, Inc. 5 More Recent Portfolio Assessment Congress continued to express interest in benefits assessment for these programs. 2003, Congress requested the NRC to - develop a comprehensive prospective evaluation methodology - apply it on a continuing basis to review the programs, - to inform Congressional decision making. 2005, NRC Prospective Evaluation Methodology Report Published. (Phase 1 of the study) 2006, NRC-implementation of new method to provide Congress a better understanding of portfolio benefits and costs is underway. (Phase 2 of the Study)

TIA Consulting, Inc. 6 NRC Committee & Staff on Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs Committee Members-Phase 1: Robert Fri, Chair, Resources for the Future Linda Cohen, University of California James Corman, Energy Alternatives Studies, Inc. Paul DeCotis, New York State Energy R&D Authority Wesley Harris, MIT George Norton, Virginia Tech Rosalie Ruegg, TIA Consulting, Inc. Maxine Savitz, Honeywell, Inc. (retired) Jack Siegel, Energy Resources International, Inc. James Smith, Duke University Terry Surles, Electricity Innovation Insitute James Sweeney, Stanford University John Wise, Mobil R&D Company (retired) NRC Staff: Martin Offutt, Study Director

TIA Consulting, Inc. 7 Priorities in Developing the Prospective Benefits Methodology Adoption to the extent feasible the work of the retrospective study and subsequent work done by DOE and OMB. Development of a methodology rigorous in its calculation of benefits and assessment of risks, but at the same time practical and consistent for applying across a variety of DOE programs. Development of a methodology that is transparent, easy to use, feasible to implement from the standpoint of resource requirements, and that produces results understandable by key stakeholders.

TIA Consulting, Inc. 8 Prospective Benefits Methodology: 6 Key Elements 1.Consistent definition of benefits 2.Common scenarios for analysis 3.Explicit accounting of risks & gov’t role 4.Consistent estimation of probabilities 5.Simplified models for calculating EV benefits 6.A results matrix for uniform summary

TIA Consulting, Inc. 9 Results Matrix for Summarizing Prospective Benefits and Costs PANEL NAME: Program Name: Program Goals: Year Goals Expected to be Achieved: Program Costs: Funding to Date:$ Current Funding:$ Proposed Year Funding: $ Expected Cost to Completion: $ Industry and Foreign Government Funding: $ Key Complementary/Interdependent DOE Programs: (First Part of Matrix)

TIA Consulting, Inc. 10 Results Matrix for Summarizing Prospective Benefits and Costs. Source: NRC Report, Reference Case High Oil & Gas Prices Carbon Constrained Program Risks Technical (cross ref to decision tree) Market (cross ref to decision tree) Expected Program Benefits Economic Environmental Security Global Scenarios (Second Part of Matrix) Note: All benefits are cumulative through 2050 and are reported in 20XX year dollars.

TIA Consulting, Inc. 11 Characterization of the Approach Harnesses the speed of expert panels Guides the panels to perform comprehensive decision tree analysis Specifies consistent scenarios and calculations Provides clear, consistent procedures for each step Yields cross-program comparable results

TIA Consulting, Inc. 12 Selected for Application of Methodology From DOE’s Office of Energy Efficiency & Renewable Energy: distributed energy program, vehicle technologies program, industrial technologies program; From DOE’s Office of Fossil Energy: integrated gasification combined cycle, sequestration, natural gas technologies (exploration and production).

TIA Consulting, Inc. 13 Implications Overview The developments described have implications for: State-of-the-art R&D evaluation Those who wish to organize and conduct portfolio-scale evaluation studies in support of real-time policy and budgetary decisions User communities who make policy and budgetary decisions based on the results of evaluation studies Suppliers of traditional federal agency R&D evaluation

TIA Consulting, Inc. 14 For further information… Information on the project is posted at the National Academies Website. Go to National Research Council and search “current projects” for “Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs—Phase 2” [