Non-Residential Program Impact Evaluation Methods Gary Cullen, Summit Blue Consulting (360)

2 Agenda Much of what has been provided by Mary and Katherine is also applicable to the non-residential sector I will try and avoid re-hashing the same issues I will focus on families of measures and the approaches to be taken

3 Current Non-Residential Offerings Reviewed the Energy Efficiency in Californias Public Power Sector 2006 Status Report The measure families most often identified include: 1. Indoor lighting as a retrofit, early retirement, and new construction application 2. HVAC equipment including air conditioners, heat pumps, and chillers 3. Energy efficient motors and pumps 4. A/C tune-ups and duct sealing 5. Shade trees

4 Indoor Lighting A number of different technologies could be included: > CFLs > T8/T5 fixtures w/electronic ballasts > High intensity discharge fixtures > LED exit signs > Lighting controls > Fluorescent fixture reflectors with de-lamping Key considerations are that they are non-weather sensitive and for the most part non-vintage sensitive. Are sensitive to building type

5 HVAC Technologies could include: > High efficiency air conditioners > High efficiency heat pumps > High efficiency chillers > Programmable thermostats > Building shell components such as windows and insulation > AC and Chiller tune-ups > Duct sealing > Cool roofs > Cooling circulation pumps Key considerations for these measures are that they are weather zone, vintage, and building type sensitive.

6 Efficient Motors Generally premium efficiency motors and variable speed drives Key considerations for these measures are that they are non- weather sensitive, non-vintage sensitive, but building type and application sensitive.

7 Shade Trees This is generally a residential and a commercial measure The means of saving energy from this program are unique and often other benefits support its implementation Energy impacts are generally HVAC related and therefore weather, building type, and building vintage dependent

8 Evaluation Decisions to be Made: First Assumption is it will be an Impact Evaluation Decisions to be made: 1. What Evaluation approach to take? May vary by type of program. Billing analysis? Metering analysis? Modeling analysis? Deemed savings analysis? 2. Do you want to estimate Net-to-Gross assumptions or utilize CPUC approved values?

9 Evaluation Approaches – Billing Analysis Advantages > Examine the billed energy use of the participants (pre and post participation) > Uses existing metered consumption data > Possible to have very large sample sizes Challenges > Data acquisition -Multiple fuel types provided by multiple sources > Data management > De-aggregating billing data (example: cooling) > Behavior – how do participant intentions and behavior affect energy savings? E.g., snap-back, energy efficiency attitudes, etc. > Insuring changes in building operation and assumptions are identified and included in the analysis (change in occupancy schedule, number of people, building size, etc.) > Weather differences need to be accounted > This method is more appropriate for the residential sector with relatively homogeneous participants and measures

10 Evaluation Approaches – Metering Analysis Advantages > Directly meter energy usage on specific end-uses of interest (lighting, space heating, cooling, and water heating) > Focuses on the affected measures and end-uses to reduce variation and bias from other factors > Examines the installed measures in a sample of the participants and uses ex-anti impact estimates and site-specific data to estimate overall program impacts. Challenges > Expensive > Data acquisition takes time > Sometimes difficult to obtain access to buildings > Variance of usage still very large > Sample size(s) – how many participants for meaningful results? > Sample bias – are the samples representative? > What are the key factors affecting use or savings? The technology improvement? Behavior such as snap-back, energy efficiency attitudes, etc. ?

11 Evaluation Approaches – Modeling Analysis Advantages > Use energy modeling to simulate usage and savings (either spreadsheet or building simulation modeling) > Can be performed on entire population of participants using average characteristics or can be applied to important population subsets with results weighted up to the population > Can take advantage of modeling performed during the program participation process Challenges > Data acquisition --Requires obtaining participant building characteristic data > Buildings often not built exactly as modeled > Buildings often not operated exactly as modeled

12 Evaluation Approaches – Deemed Savings Analysis Advantages > Low cost and can be quickly performed > Savings are based on stipulated values. Generally non-controversial to use. > Deemed savings available for a large number of weather sensitive and non-weather sensitive measures Challenges > Deemed savings often represent average building and operating characteristics. > The buildings or operating characteristics used to estimated the deemed savings may differ from your service area. > Still need to perform a verification study for actual installation and use.

13 Issue of Net Savings Net savings are the adjusted gross savings that take into account savings that would have occurred in the absence of a program Primary factors that account for the difference between net and gross savings: > Free ridership – free riders are program participants that would have implemented the measure even in the absence of the program. This effect reduces gross savings. > Spillover effects – Additional savings accrued beyond the effects of the measure implemented. Could be from increased awareness that causes reduced energy use or additional conservation measure purchases. This effect increases gross savings. > Rebound effects – A change in energy using behavior that increases the level of service and results from an energy efficiency action. The most common form is take back, where a consumers behavior results in the higher utilization of the efficient technology, such as using an air conditioner more because it is more efficient. This effect reduces gross savings.

14 Approaches for Estimating Net Savings Self Reporting Surveys – information is reported by participants and non- participants without independent verification. Enhanced Self Reporting Surveys – the self reporting survey results are enhanced by combining them with results from interviews. Econometric Methods – the previously mentioned billing analysis is enhance to include non-participants. Statistical models are then used to compare participant and non-participant energy use. Stipulated Net-to-Gross Ratios – ratios that are multiplied by the gross savings to obtain an estimate of net savings. In California, the CPUC has a set of stipulated values based on historical studies.

15 Estimating Your Own Net Savings Advantages > the California stipulated values are generically aligned to general program categories. Specific measures, especially measures new to the market place or relatively unknown, may have a higher net- to-gross value that the stipulated value. > Historically, the studies used to estimate the California stipulated values were weak in their estimation of spillover effects. Disadvantages > Fine tuning net-to-gross values may not be important to the utility. > There is generally the need for a non-participant survey and associated analysis. This adds costs to the evaluation effort.

16 CPUC Approved Net-To-Gross Values Existing programs not listed below should use a default value of 0.8 Residential > Appliance early retirement and replacement 0.80 > Residential Audits 0.72 Refrigerator Recycling/Freezer Recycling 0.35/0.54 > Residential Contractor Program 0.89 > Emerging Technologies 0.83 Nonresidential > Agricultural and Dairy Incentives 0.75 > Commercial and agricultural information, tools, or design assistance services 0.83 > Express Efficiency (rebates) 0.96 > Energy Management Services, including audits (for small and medium customers) 0.83 > Industrial Information and Services 0.74 > Large Standard Performance Contract 0.70 New Construction > Industrial and Agricultural Process 0.94 > Industrial new construction incentives 0.62 > Savings by Design 0.82

17 Recommended Evaluation Approaches Recommendations based on the following caveats: > Evaluation budgets are limited. > The recommendations are generic in scope. A specific program or group of measures may be better served by an alternative approach. Lighting: > Utilize deemed energy savings > Utilize stipulated net-to-gross values unless your program is promoting newer technologies HVAC: > Utilize deemed energy savings unless it is a complex system being installed. If complex, then metering should be followed. > Utilize stipulated net-to-gross values unless your program is promoting newer technologies Efficient Motors: > This depends on the importance of the program. > If a small contributor to your portfolio, utilize deemed energy savings that are adjusted by the verification survey that includes collection information on hours of operation. > If a large contributor, consider short term metering analysis or at a minimum modeling analysis that is independently reviewed for robustness > Utilize stipulated net-to-gross values unless your program is promoting newer technologies Shade Trees: > There are no deemed saving or appropriate stipulated net-to-gross values > I only know of the SMUD shade tree program evaluations. These should be reviewed and a similar methodology followed.

Non-Residential Program Impact Evaluation Methods Gary Cullen, Summit Blue Consulting (360)