EvergreenEcon.com ESA 2011 Impact Evaluation Draft Report Public Workshop #2 August 7, 2013 Presented By: Steve Grover, President

2 Agenda for Today Introductions (10 min) Presentation of Draft Evaluation Report (30-45 min) Q&A / Discussion (~2 hours) Wrap Up / Next Steps (5 min)

Evaluation Team The ESA Impact Evaluation team consists of the following: Evergreen Economics (Prime Contractor) Michaels Energy (Engineering Support) CIC Research (Phone Surveys) Betsy Wilkins (Technical Editor) John Stevenson (Survey Design) 3

ESA Impact Evaluation Tasks Task 1: Project Management Task 2: Kickoff Meeting Task 3: Research Plan Task 4: Public Workshop #1 Task 5: Data Collection and Analysis Task 6: Draft Report Task 7: Public Workshop #2 Task 8: Final Report Task 9: Data Documentation 4

Presentation Topics The Draft Report Public Workshop presentation will focus on: 1. Impact estimates – methods and results 2. Phone survey findings 3. Evaluation conclusions and recommendations Draft Report is available on the CPUC Energy Division website Public comment period ends: August 12 Final Report due: August 31 5

Impact Methods: Billing Regression Model Primary analysis method is a fixed effects billing regression model Fixed effects model advantages: Fixed effects specification contains the desirable statistical properties for producing unbiased estimates Allows for controlling some of the non-program influences that affect savings through the use of dummy variables Consistent with prior ESA Program impact evaluations 6

Fixed Effects Model (kWh) 7

Fixed Effects Model (therms) 8

Model Variations Three models estimated and included in the draft report: 1. Basic Model – includes all measures, uses entire sample 2. Measure Model – same model run separately for individual measures, uses sub-sample that installed each individual measure 3. Whole House Model – estimates total savings at the household level (separate models by house type, no breakdown by measure) 9

Savings Assignments Impacts assigned at the measure level based on the following algorithm: 1.If the 95 percent confidence interval of the impact estimate from the Basic Model includes the ex ante savings value, then the estimate from the Basic Model is used. 2.If the confidence interval for Basic Model estimate did not include the ex ante value, then evaluator judgment was used to assign an impact value from among the Basic Model, Measure Model, or ex ante values. 3.In a couple of instances, an engineering estimate was assigned when the ex ante values were unusually high and neither of the regression models could provide a more reasonable result. 10

Data Screening for Regression Models Screens used were very minimal, with observations deleted based on the following criteria: Master metered accounts Monthly observations > 10,000 kWh Monthly observations < 100 kWh Monthly observations > 5,000 therms Screens resulted in very few observations or households being excluded from the analysis dataset (1-3%). 11

Data Screening ( 10,000 kWh) 12

Data Screening (> 5,000 therms) 13

Weather variable calculation Typical weather variable calculations for cooling- degree days (CDD) and heating-degree days (HDD) use daily averages for temperatures (measuring deviations from 65 degrees F). Traditional method will miss some heating/cooling activity – there may be some brief periods of heating or cooling, even if daily average does not meet 65 degree threshold. The current evaluation calculates weather variables based on hourly temperature deviations from 65 degrees, then averaged for entire day. 14

Weather variable comparison (CDD) 15

Weather variable comparison (HDD) 16

Impact Estimation Results 17

Household Savings Estimates 18

Impact Estimates Over Time (All housing types) 19

Whole House Model (Annual kWh Impacts) 20

Whole House Model (Annual Therm Impacts) 21

Post-Participation Energy Use / Pre-Participation Use 22

Factors Influencing Savings Savings will vary across utilities and evaluation years for a variety of reasons: Weather Mix of measures installed Household energy usage Household demographics Home condition Regression model specification 23

Phone Survey Results 24

Phone Survey Overview Phone Survey sample = 150 participants per utility (600 total) Sample chosen from customers with a high increase (top 33% of increasers) in weather-normalized energy use before and after participating in the program. Questions focused on possible reasons for explaining increase in energy use Surveys fielded in April 2013 by CIC Research. 25

Phone Surveys Results: Cooling 26

Phone Survey Results: Heating 27

Phone Survey Results: Change in Occupancy 28

Conclusions Current impact estimates generally consistent with ex ante values Current estimates do deviate from prior impacts and DEER values, possibly due to multiple factors Whole House impact estimates consistently lower than those produced by assignments from Basic/Measure model. Consumers may be unaware that they are using more energy. 29

Recommendations Use hourly (rather than daily) method to calculate weather variables. If billing regressions used in future evaluations, having multiple models provides more flexibility on assigning final savings estimates. If wide variations in impacts is unacceptable, consider using deemed savings from DEER to provide stability. Allocate more time for evaluation, which will allow for more modeling options to be explored (including a multi-year analysis). 30

Remaining Project Timeline Key Dates: Comments on Draft Report due: August 12, 2013 Final Report: August 31, 2013 Database and documentation delivery: September