1. Residential Behavior-based Programs Measure Development Update Ryan Firestone Regional Technical Forum March 15, 2016

2. Presentation Overview Overview of measure RTF history Next steps Additional details 2

3. Measure Overview Measure description: Provision of information and/or messaging to residential customers that leads to reduced energy consumption Typical program periodically provides participants with comparisons of their energy use to their neighbors’ and provides recommendations on how to save energy: – Random assignment of treatment and control group within population allows for savings estimation by comparison of billed energy 3

4. RTF History March 2010 - Standard Protocol approved (pre-Guidelines)Standard Protocol – Experimental design (treatment and control/comparison group) – Billing analysis – Normalization for weather and other factors, difference-in- difference measure of savings – More impact evaluation guidance than prescriptive methodology – “The RTF didn’t take up the issue of whether behavior-based programs meet the definition of conservation.” meeting minutes from that meetingmeeting minutes Sometime after Guidelines approval - Status of Standard Protocol set to “Under review for compliance w/ RTF Guidelines” after the Guidelines were developed 4

5. RTF History September 15, 2015 – RTF Residential Behavior Subcommittee meeting – Existing Standard Protocol does not meet Guidelines requirements (it’s not prescriptive) – Neither UES nor Standard Protocol are appropriate classifications for this measure – Impact evaluation guidance would be a useful product This is not described in the Guidelines. It would be new for the RTF. Existing Standard Protocol, current evaluation methods, and the Uniform Methods Project (UMP) are a good starting points – Discussed where savings come from … Conservation behavior Curtailment behavior Program participation Other equipment purchase – … and the priorities and challenges in estimating disaggregated savings 5

6. RTF History October 20, 2015 – RTF meeting – RTF deactivated the existing Standard Protocol The Standard Protocol does not provide prescriptive best practice method to estimate savings Developing such a Standard Protocol would be unlikely, given the variety of products and implementations covered by a the Protocol Without a Simplified Method for estimating savings, the usefulness of a Standard Protocol would be limited – Impact evaluation guidance would be a useful product It would help standardize how savings are estimated across the region, and what those savings mean RTF has struggled to develop Guidelines-compliant Standard Protocols; this may be a more realistic path for some measures – Discussed potential for measure savings to include curtailment, which may not count as conservation Limited curtailment may be OK to count – we think this happens for other measures – it’s not a large compromise, – it’s not possible to disaggregate this from other whole-home savings Persistent savings may be a reasonable proxy for conservation – “persistent savings” would need to be defined 6

7. RTF History February 2, 2016 – RTF Residential Behavior Subcommittee meeting – Review of UMP Protocol – Discussed what a path forward for an Impact Evaluation Guidance Document UMP Protocol as starting point Possible modifications: – More detail in some areas – NW specifics (curtailment, momentum) – New evaluation methods with proven track record – Discussion of what to do when applicability conditions cannot be met – Limits on applicability of results » To same cohort in other years » To other cohorts 7

8. Next Steps CAT reviews evaluations and methods provided by Subcommittee members, others (now) – If you’d like us to consider anything, contact Ryan and Josh CAT develops a straw-man Impact Evaluation Guidance Document (~April) Subcommittee reviews straw-man Impact Evaluation Guidance document and work towards a draft document to propose to the RTF (~April – June) – Please sign up for subcommittee if you’re interested 8

9. Additional Details 9

10. This Measure is Atypical Our current RTF measures have clear physical mechanisms for savings energy However, for behavior-based programs, we don’t know what mix of participant actions result in energy savings. For example, – turn off lights, turn down HVAC when not in a space – lower water heater setpoint – use less heat, light, or other services this could conflict with our notion of Conservation – buy more efficient products and services costs of products and services are unknown This mix can vary by program design, implementation, location, and time 10

11. What Kind of Measure is This? UES? – Evaluated savings (normalized to % of house load) vary significantly (~1% - 3%) – Programs are evolving – likely faster than RTF processes could keep up with the measure – Wouldn’t provide programs flexibility to target specific populations unless we had data on those populations Standard Protocol? – No simplified method to estimating savings has been proposed – Developing a prescriptive best practice method for estimating savings is not realistic. Method may be dependent on: Targeted behaviors Control vs. comparison group Granularity of billing data Normalization requirements Program size Evaluation Guidance? [Subcommittee/CAT recommendation & RTF decision] – No way around evaluating savings – RTF could help the region standardize how evaluations are done – This is something new for the RTF; it’s not in the Guidelines 11

12. Uniform Methods Project (UMP) DOE protocols for determining savings from energy efficiency measures and programs Chapter 17: Residential Behavior Programs Applicable to residential behavior programs with large (1,000’s-10,000’s) number of participants – Each with individual billing data (e.g., by house) Experimental Design: – Randomized Control Trial – subjects randomly assigned to group that gets or does not get messaging – Randomized Encouragement Design – all subjects can opt in, subjects randomly assigned to group that gets or does not get encouragement to participate. Analysis: – Difference (kWh control – kWh treatment ) – Difference-in-Difference ( (kWh pre - kWh post ) treatment – (kWh pre -kWh post ) control – Simple average, panel regression w/ or w/out fixed-effects – Avoid double counting of trackable program savings – analyze participation data – Avoid double counting of untrackable (upstream) program savings (e.g., lighting) – use surveys Similar methods in State and Local Energy Efficiency Action (SEE Action) “Issues and Recommendations” report, evaluations, etc. 12

13. Evaluated Savings Opower Home Energy Report, typical NW evaluated savings – 1% - 3% of whole-house consumption – persist/increase while messaging continues (findings limited by new-ness of programs) – decay after messaging stops – vary by demographics: e.g., larger energy consumers save proportionally more 13

14. Documentation Links Webpage for (Deactivated) Standard Protocol Subcommittee meeting documents RTF presentation, 10/20/2015 UMP Protocol 14