Select Issues in Attribution and Net-to-Gross – Practical Examples for Discussion Presented at: CALMAC Meetings July 18, 2007 By: Daniel M. Violette, Ph.D. Summit Blue Consulting Boulder, Colorado

Summit Blue Consulting 2 Attribution and Net Savings through Self-reporting “Net Program Impacts” account for free ridership and spillover Self-reporting is often the least expensive option for estimating net program impacts, BUT… –Self-reporting may be biased (recall, self-selection) –How do we incorporate and quantify qualitative responses? Financial incentives for the Program Administrator –At NYSERDA, financial incentives for the Program Administrator are not tied to net savings –In California, if incentives are at stake, should net savings estimation follow a specified approach that limits investigator bias?

Summit Blue Consulting 3 Attribution and NTG Issues Two types of issues have been arising in our research: 1.Attribution of impacts to programs when more than one entity in the region is implementing a similar program 2.Within a specific program, how to address free ridership and spillover (or market effects) due to the program. As more entities are becoming involved in DSM implementation the first type of attribution is becoming more important in assessing organization-specific cost-effectiveness. Also, costs for kW and kWh savings may need to aggregate the costs from more than one program effort. Entities implementing programs include utilities, federal agencies (EPA), state governments, and regional energy agencies such as the NW Energy Efficiency Alliance (NW Alliance)

Summit Blue Consulting 4 Example from NW Alliance Periodic reviews of the Alliance are called for by the NW Alliance Board according to the Alliance charter to determine whether the Alliance benefits outweigh its costs. Approach: 1.Review of the Market Progress Evaluation Reports (MPERS) 2.Review of the most recent Alliance Cost Effectiveness (ACE) model 3.Determine threshold analysis – what is breakeven for that program 4.Determination of pivot assumptions 5.Dimension uncertainty around these assumptions 6.Simulate results under different values for the pivot assumptions. A concern was expressed by some Board members that the NW Alliance was claiming savings that were actually due to other effects or to other organizations.

Summit Blue Consulting 5 Examples of Alternative Hypotheses Energy Star Residential Lighting (most controversial program) 1. Energy crisis of 2001 drove sales (energy costs and media awareness = indicators) 2. BPA and local utility coupon program spillover 3. Field performance is not as anticipated (installation, removal rates, retention, …) 4. CFL stocking practices were driven by other market factors (availability, infrastructure) 5. Relationship to EPA/Energy Star programs drove sales Energy Star Residential Windows 1. Baseline assumptions were different (nationally and regionally) 2. Builders changed installation preferences for other reasons than Alliance activities 3. Manufacturers changed processes for other reasons than Alliance activities 4. Distribution of electrically heated home is different than assumptions

Summit Blue Consulting 6 Approach to Scenario Analysis A six-step approach characterized each program scenario analysis: Step 1: Begin with the cost-effectiveness analyses for the four programs identified for detailed analysis. Step 2: Select pivot assumptions that influence cost-effectiveness. Step 3: Trace assumptions to MPERs or other reference documents. Step 4: Conduct interviews with other organizations to bracket impacts, key assumptions and develop scenarios. Step 5: Seek ranges for key values. Step 6: Delineate breakeven scenarios and distributions of economic outcomes

Summit Blue Consulting 7 CFL Scenarios Estimated Value: Alliance receives credit for all CFL sales that were not utility coupon or giveaway sales minus assumed baseline of 100,000 CFL sales (this baseline comes from the ACE model for the CFLs project) Low Scenario: The low scenario might assume that the many utilities in the region that developed their own CFL programs actually were the more important driver. 30% of the CFLs that the Alliance is taking credit for in the “Estimated Value” case are actually spillover from the utility coupon and giveaway programs to other sales of CFLS. The awareness was created by the utility programs and that CFLs would have been available in adequate supply such that the utility programs were a more significant driver of total CFL sales than is assumed in the estimated value base case. High Scenario: The high scenario assumes that spillover goes in the other direction and that due to Alliance efforts. Utilities are able to sell 30% more CFLs than would otherwise have been the case since the Alliance helped set up coupon programs, the redemption center and encouraged retailers to stock CFLs. The end result is that without the Alliance efforts the utility achieved sales would have been 30% less

Summit Blue Consulting 8 Scenario Implications (1) LOW attribution scenario = (Total) - (Utility) - (30% spillover i.e., impact on non-utility sales cause by utility efforts) - (baseline) = 2.8 million (2) MEDIUM -- Estimated value = (Total) - (Utility) - (baseline) = 4.0 million. (3) HIGH attribution scenario = (Total) - (Utility) + (30% spillover, i.e., alliance efforts make utility sales 30% higher than would otherwise have been the case) – (baseline) = 5,170 thousand or roughly 5.2 million

Summit Blue Consulting 9 What would one want to know? How likely is each of these scenarios to occur? Are scenarios other than these three as likely or more likely to occur? What is meant by low, medium and high? –Is the low scenario the lowest conceivable value? –Is the high the highest conceivable value? Just knowing these three values may not tell us very much and might not capture the expert judgment and ancillary information available very well As a result, a distribution approach was used. This process was used for a small set of programs that accounted for the vast majority of the savings claimed by the NW Alliance.

Summit Blue Consulting 10 Distribution approach Interviews were conducted with regional experts familiar with the regional programs – no purely quantitative approach seemed adequate. Opinions were obtained about the likelihood of the three scenarios. Example of a distribution-base analysis:

Summit Blue Consulting 11 Completing the Analysis Two other distributions were selected for pivot factors for the CFL analysis: 1.Number of lamps sold due to Alliance activities. 2.Savings in Watts for each lamp sold (takes into account installation, retention, wattage, and other factors). 1.The final distribution was based on 5,000 random draws of values from each distribution 2.For each draw (or value) from that distribution, the final attribution value is calculated for that set of values. 3.This provides 5,000 values which are graphed to give us the final distribution of impacts attributable based on the literature review and expert judgment.

Summit Blue Consulting 12 Results – Most controversial Program Since the evaluation team could not directly measure who was responsible for each CFL sale, it relied upon responses from interviews with retailers, utility program managers, and other knowledgeable experts. Taking into account of all factors, the cumulative savings due the from 70.4 aMW to about 26 aMW. Due both to sales of CFL and lower savings per CFL. Other programs were much closer to initial Alliance estimates. Conclusions: –The program was cost-effective at these levels using mean values. –Levelized cost was still below the cost of power in the region. –Also, calculated program risk, i.e., the probability that the program was not cost-effective (a small but positive value).

Summit Blue Consulting 13 A NYSERDA Example Net-to-Gross Ratios w/o billing data Based on work on-going at NYSERDA No DSM incentives are dependent upon these results Goal is to make the best decisions regarding program implementation and attribution of impacts. Looked at both free-ridership and three different types of spillover: –Internal project spillover by participants –Participant spillover at other projects –Non-participant spillover CONCEPT – Prove existence first, then try to dimension effect – and incorporate uncertainty in estimates.

Summit Blue Consulting 14 Estimating a Net-to-Gross (NTG) Ratio Net Savings = Gross Savings (e.g., from program data base and infield validation) x NTG ratio Summit Blue’s Approach NTG Ratio = Net Factor x Market Factor Net Factor = [ 1 minus Free Ridership ] Market Factor = [ 1 + Spillover(1) + Spillover(2) + Spillover(3) ] where: –Spillover(1) is “Inside Spillover” within Program projects –Spillover(2) is “Outside Spillover” from Program participants –Spillover(3) is “Non-participant Spillover” (aka Market Effects)

Summit Blue Consulting 15 Estimating Free Ridership

Summit Blue Consulting 16 Estimating Spillover Inside and Outside Spillover for Participants 1.Is there evidence of spillover (Yes/No)? 2.What share of the market does it apply to? Number of projects/facilities Relative size of projects (physical and savings) 3.Attribution of savings to the program influence Non-Participant Spillover 1.Non-participant survey 2.Participating ESCOs, vendors, etc. 3.Baselines and market activity NOTES: 1. Constructed distributions around all estimates based on the dispersion of responses. 2. Distributions based on the assumption that if the population of participants responded to the same set of questions they would show a variance similar to that found in the sample. 3.Use of estimated based on conservative assumptions, e.g., the 33 rd fractal. 4.Now using integrated data collection (IDC) to get better estimates.

Summit Blue Consulting 17 Other Examples: Ontario –Shared savings incentives based on NTG calculations. –Some of the programs designed by the natural gas utilities were picked up and operated by Natural Resources Canada (NR Canada) –Attribution to the utilities made in a regulatory proceeding based on filed arguments. Other Northeast utilities also have shared savings and need attribution to their programs in a multi-entity delivery setting. Within NYSERDA, program-level attribution is difficult due to overlapping program marketing and measures – e.g., equipment rebates and new construction programs.

Summit Blue Consulting 18 Contact Information: Daniel Violette, Ph.D. Summit Blue Consulting th Street #230 Boulder, Colorado, Phone: