Building America: Retrospective Evaluation of a Unique DOE R&D and Market Diffusion Program As Jack and Daniel noted, the Guide for Evaluating Realized Impacts of DOE’s EERE R&D Program provides a standard method for conducting evaluations of R&D investments which it followed. However, this study is the first in a series of 7 EERE retrospective impact evaluations for which market-oriented components—rather than R&D--comprise the majority of the investment. Thus, while the evaluation team followed the Guide where applicable, it was required to decide if additional approaches were needed to address the market-oriented investment components. This required identifying where the differences lay and the nature of the differences. I will provide background to Daniel’s presentation by focusing on how evaluation of R&D and systems investments may differ from evaluation of market-oriented investment. Evaluating r&D & Market Diffusion benefits: Methodology Issues Rosalie Ruegg TIA Consulting *Based on work funded by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy.

Evaluating Impacts of Different Types of Investment—e. g Evaluating Impacts of Different Types of Investment—e.g., R&D, Systems Integration, & Market Stimulation: Methodology Issues How do different types of investments generate impacts? What difference does it make to evaluation? TIA Consulting, Inc.

DOE/EERE Investments Generally Share Long-term Goals More efficient use of energy and other resources. Reduction of air emissions from fossil fuel use. Energy security. Improved health, environment, and economic wellbeing of citizens. Generation of overall positive net benefits for the nation. Thus, when the time horizon is long, EERE investments of different types can often be evaluated using a single set of metrics. TIA Consulting, Inc.

Most DOE/EERE Investments Generate Costs in Similar Ways Budget items Measured in dollars (current converted to constant dollars) Weighed against benefits to assess if the investment has been worthwhile. So far, so good, but , now let’s consider differences that give rise to methodology issues. TIA Consulting, Inc.

Market-Oriented Investments mainly affect demand for product/process. Many Similarities, BUT Different Types of Investments May Differ in How They Generate Impacts R&D Investments (and certain types of Systems Investments) mainly affect supply of a resulting product/process - Improved performance - Reduced costs Market-Oriented Investments mainly affect demand for product/process. - Changed behaviors of builders, financiers, & homeowners by r raising their awareness/knowledge/attitudes TIA Consulting, Inc.

R&D & Systems Integration Investments Produce Changes in Supply Market-oriented Investments Produce Changes in Demand Demand/Supply/Price Supply 1 Supply 2 P3 P1 P2 Demand 2 Demand 1 Q1 Q2 Q3 TIA Consulting, Inc.

These Investments differ in timing Add Market Investments Add DOE R&D & Systems Investments Without DOE Investments   $ Net Cash Flow Time during which R&D & Market investments and impacts--& evaluation occur Market-oriented investments tend to lag R&D investments—and they don’t have the same period over which their benefits are realized. This difference in timing may affect the metrics and methods used to evaluated each type of investment –next pp. R&D & Systems Investments Market-Oriented Investments *NOTE: Framework Guide cover retrospective impact evaluation of R&D TIA Consulting, Inc.

How long are evaluation time horizons? Can Evaluation be Accomplished with a Single Set of Impact Metrics or Will Multiples be Needed? How long are evaluation time horizons? How far along are R&D and systems integration investments? How far along are market-oriented investments? How fragmented are the activities and targeted constituencies of the market-oriented investments? (e.g., some actions may affect financiers, others may affect some part of the building community, while others may affect some but not all homeowners.) TIA Consulting, Inc.

Impact Metrics are Sensitive to Type of Investment and Stage of Evaluation Evaluation Metrics for R&D & Systems Investments Affecting Mainly Supply Evaluation Metrics for Market-Oriented Investments Affecting Mainly Demand Early Stage Supply Chain metrics Net Chain metrics Social Network metrics Knowledge base metrics Behavioral Change metrics by group Replication after demonstrations A, B, C Take-up of recommended practices X, Y, Z Trend in sales before & after intervention Growth in different market sectors Later Stage Energy and other resource effects Environmental effects Energy security effects Knowledge diffusion effects Economic impacts Economic Impacts TIA Consulting, Inc.

When Faced with Multiple Investment Types… Develop a logic model of the program/portfolio, activities, outcomes Assess the timing of different types of investment and the timing of impacts For retrospective R&D investment, follow EERE guide for assessing impacts in terms of -Energy and other Resource Impacts -Environmental Healthcare Impacts -Environmental Climate Impacts -Energy Security Impacts -Knowledge Impacts If non-R&D investment, analyze the type, timing and appropriate treatment Systems & process investments can usually be treated same as R&D For market-oriented investments, develop a strategy that best fits the characteristics & timing of each component Report on investment impacts, combining impacts where feasible TIA Consulting, Inc.

Rosalie Ruegg, Managing Director TIA Consulting, Inc. ruegg@ec.rr.com Questions & Comments Rosalie Ruegg, Managing Director TIA Consulting, Inc. ruegg@ec.rr.com