1. Residential Behavior Programs Ryan Firestone October 20, 2015

2. Presentation Outline RTF history 7 th Power Plan Where do savings come from? Modeling principals for the RTF Where does this measure fit within the Guidelines? Measure cost Next steps for staff and subcommittee(s) 2

3. Overview Two fundamental issues: Established impact evaluation methods do not get us the level of detail necessary to count savings appropriately – We’re unlikely to prescriptively measure our way out of this issue – RTF judgement here may be of great value for regional standardizing We expect behavior programs (and their implementation) to vary significantly enough that a prescriptive Standard Protocol is not practical – We’d like to provide guidance for impact evaluations – Guidelines identify Program Impact Evaluation as a savings estimation methodology, but don’t describe measure-specific RTF guidance products 3

4. RTF History Standard Protocol approved in March, 2010 Standard Protocol – Experimental design (treatment and control/comparison group) – Billing analysis – Normalization for weather and other factors, difference-in-difference measure of savings – More impact evaluation guidance than prescriptive methodology – “The RTF didn’t take up the issue of whether behavior-based programs meet the definition of conservation.” meeting minutes from that meetingmeeting minutes Status set to “Under review for compliance w/ RTF Guidelines” after the Guidelines were developed. Staff/CAT proposal: Deactivate this Standard Protocol 4

5. Power Plan Residential behavior is being modeled as a conservation resource for the first time in the 7 th Plan (draft). – Focus on long term savings Savings that aren’t expected to persist aren’t counted – Curtailment ≠ Conservation Savings from reductions in utility (e.g., thermostat adjustment) aren’t considered conservation under the Power Act. – Not double counting Potential from Plan-modeled equipment/appliances not re-counted in behavior analysis – Council staff estimated actions that would satisfy Plan requirements, and saving potential Reduce water heater setpoint – (BULK OF SAVINGS) Reduce lighting HOU when not in room Reduce HVAC usage when not at home – Up to 70% of homes (need control group) – 172 kWh/home – 48 aMW potential Analysis files (see “Residential Behavior” section) Analysis files 7 th Plan Conservation Resource Advisory Committee Presentations – December 17, 2014 December 17, 2014 – January 16, 2015 January 16, 2015 5

6. RTF Residential Behavior Subcommittee – Sept. 17 Meeting Meeting to prepare for October RTF discussion RTF history 7th Power Plan Where do savings come from? Modeling principals for the RTF Measure categorization (Standard vs. Custom Protocol) Next steps for staff and subcommittee (Meeting presentation and notes) [discussion from meeting covered later in presentation] 6 Attendees Jennifer Anziano (RTF Manager) Andie Baker (Abacus Research) Rebecca Blanton (PSE) David Bopp (Flathead Electric) Leona Doege (Avista) Ryan Firestone (RTF Contract Analyst) Lauren Gage (Bonneville) Mark Jerome (CLEAResult) Don Jones Jr. (PacifiCorp) Jim Perich-Anderson (PSE) Josh Rushton (RTF Contract Analyst) Mohit Singh-Chhabra (RTF Contract Analyst) Bonnie Watson (Bonneville) Gary Wood (Seattle)

7. Uniform Methods Project (UMP) DOE protocols for determining savings from energy efficiency measures and programs Chapter 17: Residential Behavior Programs Applicable to residential behavior programs with large (1,000’s-10,000’s) number of participants – Each with individual billing data (e.g., by house) Experimental Design: – Randomized Control Trial – subjects randomly assigned to group that gets or does not get messaging – Randomized Encouragement Design – all subjects can opt in, subjects randomly assigned to group that gets or does not get encouragement to participate. Analysis: – Difference (kWh control – kWh treatment ) – Difference-in-Difference ( (kWh pre - kWh post ) treatment – (kWh pre -kWh post ) control – Simple average, panel regression w/ or w/out fixed-effects – Avoid double counting of trackable program savings – analyze participation data – Avoid double counting of untrackable (upstream) program savings (e.g., lighting) – use surveys Similar methods in State and Local Energy Efficiency Action (SEE Action) “Issues and Recommendations” report, evaluations, etc. 7

8. Where Do Savings Come From? 8 Purchase BehaviorUsage Behavior Traditional difference-in-difference billing analyses measure total savings relative to control group Savings can be from changes in equipment and/or changes in the use of existing equipment Even more categorization is necessary to determine “claimable” savings and align them with other claimed savings Avoid double counting with non- behavior program claimed savings ConservationCurtailment Upstream program participation Non- program purchase Non- program removal Trackable program participation Avoid double counting with non- behavior program claimed savings Avoid double counting with Momentum savings Need to consider persistence Not considered conservation under standing interpretation of the Power Act. Not to scale

9. Modeling Principles for an RTF Behavior Measure 1)Don’t count savings from curtailment (e.g., thermostat adjustment) 2)Avoid double counting – program-incented equipment-based savings – momentum savings 3)Don’t apply risk mitigation and capacity credits to things we don’t expect to persist Subcommittee feedback: – Current evaluation methods are a good starting point – Disaggregating savings to address double counting issues and to exclude curtailment seems appropriate, but is it feasible? 9

10. Modeling Principles for an RTF Behavior Measure 1)Don’t count savings from curtailment (e.g., thermostat adjustment) 2)Avoid double counting – program-incented equipment-based savings – momentum savings 3)Don’t apply risk mitigation and capacity credits to things we don’t expect to persist Staff/CAT recommendation: Use persistent savings as a proxy for conservation (i.e., not curtailment) – Savings that do not persist are likely to reflect a compromise of utility – Addresses issues 1) and 3) above Double Counting issue would still need to be addressed 10

11. Persistence as a Proxy for Conservation 11 Avoid double counting with non- behavior program claimed savings ConservationCurtailment Upstream program participation Non- program purchase Non- program removal Trackable program participation Avoid double counting with non- behavior program claimed savings Avoid double counting with Momentum savings Need to consider persistence Not considered conservation under standing interpretation of the Power Act. Persistent savings

12. What Kind of Measure is This? Standard Protocol? – Requires prescriptive best practice method for estimating savings No best practice yet for disaggregating savings: Evaluations to date have not had much success – Method may be dependent on: Targeted behaviors Control vs. comparison group Granularity of billing data Normalization requirements Program size Subcommittee agreed that evaluation guidance was more appropriate than a prescriptive protocol 12

13. What Kind of Measure is This? RTF Staff/CAT recommend developing this as measure-specific Program Impact Evaluation guidance – Weigh-in on areas we have something to say, but leave details to programs/evaluators Experimental design: sample size, control and treatment group selection Normalization for weather, rate changes, etc. What actions to count/not count Estimating total savings Estimating disaggregated savings How to handle persistence – Would this be worthwhile to the region? – The Guidelines may need to be modified to accommodate this Program Impact Evaluation is identified as a savings estimation method, but Guidelines do not describe measure-specific guidance Custom Protocol is identified as a less-prescriptive protocol for site- specific savings estimation methods 13

14. Measure Cost For Standard Protocol and Custom Guidance, RTF does not estimate costs or cost effectiveness However, Guidelines say, “costs and benefits should be estimated and documented as described in these Guidelines, as appropriate.” Caution: Regional costs include not only program costs, but also customer costs to acquire new equipment 14

15. Next Steps Today – Deactivate the current Standard Protocol It does not meet the criteria of a Standard Protocol in our Guidelines It’s probably not as thorough as the guidance we would develop – it’s a good starting point, though – RTF gives Staff/CAT guidance on key issues: Should this be developed as Impact Evaluation Guidance (it’s not currently spelled out in the Guidelines)? – Would this be worthwhile to the region? How much disaggregation of savings is necessary? Is persistent savings a reasonable proxy for conservation (i.e., not curtailment)? Other recommendations? Staff/CAT to review recent evaluations and other material – Summarize methods, outcomes, and challenges Work with Residential Behavior Subcommittee to begin developing this measure 15

16. Motion “Deactivate the Standard Protocol measure: ‘Residential: Methods for Evaluating Behavior-Based Energy Conservation Programs in the PNW’.” and “Instruct Staff to develop a Residential-Behavior- Program Impact Evaluation Guidance document.” 16

17. Additional Material 17 Slides from January 16, 2015 CRAC Presentation

18. 18

19. 19

20. 20

21. 21

22. 22

23. Additional Material 23 Slide on Curtailment from March 2, 2010 RTF Meeting

24. Slide 24 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N The Power Act and RBBP  Historically, the Council has excluded measures and practices that reduced the level of service or utility (in economic terms) provided to consumers by the current measure or practice. – 839a(3). "Conservation" means any reduction in electric power consumption as a result of increases in the efficiency of energy use, production, or distribution. [Northwest Power Act, §3(3), 94 Stat. 2698.] – Savings from lowering thermostats for space heating have not been considered conservation under the Act – Savings from lowering the thermostat on water heaters from 140 F or 130 F to 120 F have been.  Rationale for behavioral change programs as “utility neutral”, or non-sacrificial? – Persistent savings are unlikely to be sacrifices – Programs are asking customers to reduce behavior when it does not change utility (e.g., turning off lights or thermostat down when not in the room) – Recent interest in sustainability has created utility for reducing energy consumption, particularly enabled through technology or information