March 5, 2002 Lessons Learned from GAO’s Evaluation of the Outcomes of R&D Programs Presentation to ORNL’s Conference on Estimating the Benefits of Government-Sponsored Energy R&D

Metrics for R&D Evaluation are Limited Spending on research was the primary indicator of the investment in research -- an input measure –Available in both the public and private sectors. –However, not a good indicator of the level of results achieved No single output measure captures the results of research well either. Some examples include: –Quantitative analyses of return on investment; patents, copyrights & technology licensing; technology partnerships; and bibliometrics –Qualitative assessments based on peer review.

Outcome Orientation Drives R&D Evaluation Outcomes is the focus of GPRA Outcomes have also been focus of GAO’s work at DOE GAO reports on the Department of Energy’s R&D efforts include –Energy Efficiency –Fossil Energy –Renewable Energy

Four Broad Themes from GAO Reports, 1991— Would private sector do the research? 2. Will consumers buy the product? 3. Do benefits exceed costs? 4.Are efforts consistent with agency’s mission and strategic plans? (1998)

1. Would the private sector do the research? Sample Findings: DOE has funded research that private industry may have performed. –Clean Coal Technology Program Other Programs, such as the Advanced Technology Program (1991) (1996)

2. Will consumers buy the product? Sample Findings: DOE has conducted research on technologies that –have already been developed and marketed elsewhere, Wind turbines, Clean Coal Technologies –and may not be marketable. PNGV, Advanced Batteries (1999) (1990) (2000) (1995)

3. Do benefits exceed costs? Sample Findings: In evaluating the results of its R&D, DOE –has not clearly stated the costs of its research –has overstated economic benefits DOE’s Success Stories Report (1996)

Sample Findings: DOE’s Performance Plans and Reports (energy efficiency, fossil energy, and science) do not convey a clear picture of what the department is trying to do or how it plans to get there. Problems inherent to science 4. Are efforts consistent with agency’s mission and strategic plans? ? Strategic Goal Performance Goal Annual Activity Progress Towards Strategic Goal 100% 75% 50% 25% (2000)