Download presentation
Presentation is loading. Please wait.
Published byGabriel Lloyd Modified over 9 years ago
1
Evaluating Research Portfolios I: An Analytical Perspective Impact Evaluation of Energy R&D Portfolios* Rosalie Ruegg, TIA Consulting, Inc. ruegg@ec.rr.com www.tiaconsultinginc.com American Evaluation Association Conference Washington, DC October 18, 2013 * Based on work funded by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy
2
Impact Evaluation of Energy R&D Portfolios Approach Background Why perform R&D evaluation at a portfolio level? Methodological extensions required for DOE/EERE applications Current status: (1) DOE/EERE Retrospective Portfolio Impact Studies (2) Revised B-C Guide (3) Aggregation Tool Development Ruegg 2
3
R&D Evaluation at a Portfolio Level: Approach Ruegg 3 Note: “Portfolio” is used here to mean a grouping of investments but not to imply risk analysis or diversification.
4
Background ATP, more than a decade ago, began performing traditional project-level impact studies. OMB response was that project-level studies were not meaningful because they represented “cherry picking.” ATP moved to evaluating portfolios of R&D projects to provide conservative estimates of return on larger pieces of its investment. DOE/EERE extended the portfolio approach by: a) using a more systematic approach according to a Guide, b) broadening coverage of benefits. Ruegg4
5
Advantages of DOE/EERE’s Portfolio Approach Rigorous evidence of return on EERE’s R&D investment Systematic treatment allows aggregation across portfolios Efficient use of the evaluation budget Ruegg5
6
Areas of Emphasis in DOE/EERE’s Evaluation Characterizing technology content by type (e.g., infratechnologies, platform or generic technologies, product technologies) Identifying next-best technology alternative Detailed treatment of additionality Data collection Estimation of: - Resource impacts - Environmental impacts - Energy security impacts - Life-cycle effects - Knowledge effects - Combined impacts Multiple economic performance metrics Lessons learned Ruegg6
7
Methodological Extensions to Broaden Coverage: Impacts Measured in Dollars Ruegg7
8
Methodological Extensions to Broaden Coverage: Impacts Measured in Physical Units Ruegg8
9
Methodological Extensions to Broaden Coverage: Impacts Measured Qualitatively Ruegg9
10
Methodological Extensions: Overview Ruegg10
11
Economic Performance Metrics Ruegg11
12
Evaluative Questions Addressed What has been the return on DOE/EERE R&D investment? To what extent has EERE's investment produced energy, environmental, energy security, knowledge, and economic benefits? Would today’s commercialized technologies likely have happened at the same time, and with the same scope and scale, without EERE’s efforts? Was the public investment worth it? What factors led to return on public investment? What lessons learned can be applied to future R&D investments? Ruegg12
13
Current Status Retrospective Impact Guide: 2008, 2011(R), 2013(R) 2008 & 2011 editions prepared by Ruegg & Jordan 2013 edition (in review) prepared by Ruegg, O’Connor, & Loomis Portfolio Studies Completed: 5 Data Aggregation Tool under Development Ruegg13
14
Portfolio Aggregation Tool Select from the following: SelectModuleImpact Studies of DOE/EERE R&D Portfolios Level of Analysis Detailed Summary 1Geothermal Program 2Solar Photovoltaic Subprogram 3Wind Energy Program 4Advanced Batteries for HEV & EV 5Advanced Combustion 6Aggregation Across Portfolio StudiesSummary
15
Case example Ruegg15
16
Example from DOE/EERE: Advanced Combustion Engine R&D Portfolio (Link, 2010) CRF & laser and optical diagnostics tools = Infratechnology “technologies that influence the R&D, production, or marketing of other technologies but are usually not physically embodied in resulting products” Next-Best Alternative Identified: Ruegg16 Technology Type Identified:
17
Comparing Actual and Counterfactual BTE (Link, 2010) Ruegg17
18
Analysis Linked MPG to Change in BTE and Translated Results to Fuel Savings and Reduced Air Emissions Experts estimated 4.5% improvement in Brake Thermal Efficiency (BTE) from DOE-supported R&D MPG was linked to BTE with an econometric model Fuel savings were calculated from MPG fuel economy improvements and miles driven. Resulting energy savings: 17.6 billion gallons of diesel fuel from 1995 to 2007. (Other effects were also calculated). Ruegg18
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.