B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT: Methods for Evaluating Residential Behavior-based Programs RTF Presentation February 2,

Slides:



Advertisements
Similar presentations
NAESB Measurement and Verification Model Business Practice Retail Electric Demand Response 5/29/09 update.
Advertisements

SSC-NM0053 Determination of Greenhouse Gas Emissions Reductions Based on Whole- Building Simulation of Building Mitigation Efforts Using eQUEST/DOE-2.2.
Introduction Build and impact metric data provided by the SGIG recipients convey the type and extent of technology deployment, as well as its effect on.
1 Residential Weatherization Calculator Deriving Realistic Savings Estimates for the “Uninsulated” House.
Residential Refrigerators and Freezers UES Measure Update Regional Technical Forum October 14, 2014.
October 8, 2013 Eric Fox and Mike Russo. AGENDA »Recent Sales and Customer Trends »Preliminary State Sales and Demand Forecast »Building a No DSM Forecast.
Jim Pauley, P.E. Vice President, Industry and Government Relations – Schneider Electric Chair, NEMA High Performance Building Council Building Star An.
Designs to Estimate Impacts of MSP Projects with Confidence. Ellen Bobronnikov March 29, 2010.
Are Building Codes Effective at Saving Energy? Evidence from Residential Billing Data in Florida Grant D. Jacobsen UC Santa Barbara Matthew J. Kotchen.
Manufactured Homes Calibration: Existing and New Homes Mohit Singh-Chhabra & Josh Rushton RTF Update March 17, 2015.
Portfolio Manager—ICBA Members ENERGY STAR  Tools For Benchmarking and Tracking Energy Use.
Manufactured Homes Calibration: Existing and New Homes Mohit Singh-Chhabra & Josh Rushton RTF Update May 12, 2015.
Research Strategy Review: Heat Pump Water Heaters Thermostatic Restriction Valves Jennifer Anziano RTF R&E Subcommittee July 8, 2015.
Assessing Energy Savings Potential of Products and Technologies 2004 APPA Customer Connections Conference Rob Penney WSU Energy Program.
Farmers Rural Electric Cooperative Corporation 2006 Load Forecast Prepared by: East Kentucky Power Cooperative, Inc. Forecasting and Market Analysis Department.
1Managed by UT-Battelle for the Department of Energy Michael Blasnik M Blasnik & Associates Greg Dalhoff Dalhoff Associates, LLC David Carroll APPRISE.
Multifamily Air Sealing Program Bobbi Wilhelm/John Forde Implementation & Evaluation protocols March 20, 2013.
Guidelines for the Development and Maintenance of RTF- Approved Measure Savings Estimates December 7, 2010 Regional Technical Forum Presented by: Michael.
Overview of the Regional Technical Forum Guidelines January 22, 2013.
Common Carbon Metric for Measuring Energy Use & Reporting Greenhouse Gas Emissions from Building Operations A tool developed by GHG Protocol and UNEP-SBCI.
Non-Residential Network Computer Power Management UES Measure Update Regional Technical Forum July 16, 2013.
Non-Res Lighting Subcommittee Retrofit Lighting Standard Protocol Review 04/03/14.
DHP for Houses with Electric FAF Research Plan: Revisions Adam Hadley, Ben Hannas, Bob Davis, My Ton R&E Subcommittee February 25, 2015.
Residential Behavior Programs RTF Subcommittee Ryan Firestone September 17, 2015.
Agricultural Irrigation Pump Variable Frequency Drive Provisional Standard Protocol Proposal Regional Technical Forum April 16, 2013.
1 NORTHWEST ENERGY EFFICIENCY ALLIANCE Northwest Ductless Heat Pump Pilot Project Impact & Process Evaluation: Billing Analysis Ecotope, Inc. February.
M&V Part 2: Risk Assessment & Responsibility Allocation.
Northwest Power and Conservation Council Slide 1 Direct Use of Natural Gas Economic Fuel Choices from the Regional Power System and Consumer’s Perspective.
NEET Workgroup #3 - Residential Subgroup Snohomish County PUD November 2008.
EvergreenEcon.com ESA 2011 Impact Evaluation Research Plan Public Workshop #1 February 20, 2013 Presented By: Steve Grover, President.
Residential Single Family and Manufactured Home Heat Pump Water Heaters Christian Douglass Regional Technical Forum 4/14/2015.
Refrigerator Decommissioning: Measure Status Update Regional Technical Forum October 16, 2013.
1 Northwest Energy Efficiency Taskforce Workgroup # 1 Measuring What Matters Looking ahead, what data must we have to succeed?
EvergreenEcon.com ESA 2011 Impact Evaluation Draft Report Public Workshop #2 August 7, 2013 Presented By: Steve Grover, President.
Update: Grocery Refrigeration Provisional Standard Protocol for Site Specific Savings RTF Meeting June 28,
Demand Response and the California Information Display Pilot 2005 AEIC Load Research Conference Myrtle Beach, South Carolina July 11, 2005 Mark S. Martinez,
BPA M&V Protocols Overview of BPA M&V Protocols and Relationship to RTF Guidelines for Savings and Standard Savings Estimation Protocols.
Path for Multi-Family (MF) Weatherization and NC Measures Christian Douglass 8/18/2015.
SEEM Calibration: Phase II Single Family Heating Energy Regional Technical Forum September 17, 2013 Presented By: Josh Rushton and Adam Hadley Subcommittee.
SEEM Calibration: Phase II Single Family Heating Energy Regional Technical Forum August 20, 2013 Presented By: Josh Rushton and Adam Hadley Subcommittee.
Schools Lighting Hours of Use Data Regional Technical Forum May 13, 2014.
Experience you can trust. Phase 1: Cataloguing Available End-Use and Efficiency Load Data September 15, 2009 End-Use Load Data Update Project.
DRAFT Preliminary: BPA Summary of 6 th Plan Supply Curves May 15, 2009 Lauren Gage
Review of the New England “Mini-Pilot” DHP Evaluation Why we ignore this study.
2009 Impact Evaluation Concerns ESAP Workshop #1 October 17, 2011.
Northwest Power and Conservation Council 1 Distribution System Efficiency Potential & Conservation Voltage Reduction Power Committee April 2009.
Guidelines Revisions Defining What RTF Means by “Savings” December 17,
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT: Methods for Evaluating Residential Behavior-based Programs RTF Presentation January 5,
T12 to HPT8 Change-outs Small/Rural request for development May 15, 2012.
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Page 1 Pacific Northwest Smart Grid Demonstration Project  Largest Smart Grid Demonstration.
1 Case study: Sofia District Heating Dr. Venkata R. Putti Team Leader, Carbon Finance Assist The World Bank Minsk / Belarus, March 2007.
Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov February 16, 2011.
Utilities’ Update on Energy Savings Assistance Program Studies Ordered in D LIOB Meeting August 21, 2013 Sacramento, California.
Residential Behavior Programs Ryan Firestone October 20, 2015.
ENERGY STAR and Eco-Rated Homes: Planning Estimates and Research Strategy Regional Technical Forum December 8 th, 2015 Josh Rushton & Mohit Singh-Chhabra.
Multifamily Homes: Calibration Update Regional Technical Forum December 8 th, 2015.
Process for Review of 6 th Plan Supply Curves June 19, 2009 Lauren Gage
Scientific Irrigation Scheduling Subcommittee Thursday, August pm – 3pm, PDT.
Proposed 2015 Work Plan and 3 Year Look Back September 18, 2014.
Draft Seventh Power Plan Meets RTF. Key Finding: Least Cost Resource Strategies Rely on Conservation and Demand Response to Meet Nearly All Forecast Growth.
1 1 DISTRIBUTION EFFICIENCY INITIATIVE (DEI) Benefits on Both Sides of the Meter RTF MEETING February 5, 2008.
RTF Management Updates Jennifer Light Regional Technical Forum February 17, 2016.
You’re Getting Warmer... Smart Thermostat Pilot Update Presenters: Dan Rubado-ETO, Evaluation Project Manager Dennis Rominger-PSE, Market Manager Customer.
Residential Behavior-based Programs Measure Development Update Ryan Firestone Regional Technical Forum March 15, 2016.
Workshop on the Criteria to establish projections scenarios Sectoral projection guidance: Residential and services Mario Contaldi, TASK-GHG Emanuele Peschi,
Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov March 23, 2011.
Web-Enabled Thermostats
Best Practices in Residential Energy Efficiency
Industrial Assessment Center Database
Behavior Modification Report with Peak Reduction Component
Presentation transcript:

B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT: Methods for Evaluating Residential Behavior-based Programs RTF Presentation February 2, 2010 Lauren Gage Bobbi Wilhelm

Slide 2 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Goals for RTF Meeting  Introduce methods developed  Gain feedback from RTF members  Bring final methods to RTF in March for action/ approval

Slide 3 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Background Residential behavioral-based programs are gaining popularity and promise significant savings potential Multiple programmatic approaches are being tested throughout the country In-home feedback devices Energy benchmarking information Others (e.g., community-based programs). Current research on quantity and persistence of savings is sparse

Slide 4 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Process  Group of NW utilities and energy organizations held meetings to discuss regional evaluation strategies for behavior-based energy programs. – Puget Sound Energy, BPA, Seattle City Light, Snohomish County PUD, Energy Trust of Oregon, and Eugene Water & Electric Board.  The group agreed on the need for standard methods for evaluation of behavior-based conservation programs within the Pacific Northwest.  Group drafted previous version, received input from multiple national experts and revised to current version

Slide 5 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Outcomes and Caveats  These are recommended guidelines and/or methods for evaluating behavioral programs for Pacific Northwest utilities.  They consider the research completed to date and are designed to evolve as more studies are completed.  They should not be viewed as the only acceptable approaches to estimating savings.  Once reviewed and approved by the RTF, they would be an agreed-upon approach that provides guidance to future evaluations

Slide 6 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Residential Behavior-Based Programs (RBBP) Definitions  Programs designed to save energy through: – Changes in behavior - e.g., turning off lights, setting thermostats – Increasing investments in energy-efficiency measures  Program Examples: – Energy benchmarking compares participant consumption to historical consumption or to peers. – Feedback devices use monitoring or metering devices to provide information on instantaneous demand or consumption over time at the whole-house or end-use level – Others include: Information and training programs, Schools-based programs and marketing/community-based programs

Slide 7 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N The Power Act and RBBP  Historically, the Council has excluded measures and practices that reduced the level of service or utility (in economic terms) provided to consumers by the current measure or practice. – 839a(3). "Conservation" means any reduction in electric power consumption as a result of increases in the efficiency of energy use, production, or distribution. [Northwest Power Act, §3(3), 94 Stat ] – Savings from lowering thermostats for space heating have not been considered conservation under the Act – Savings from lowering the thermostat on water heaters from 140 F or 130 F to 120 F have been.  Rationale for behavioral change programs as “utility neutral”, or non-sacrificial? – Persistent savings are unlikely to be sacrifices – Programs are asking customers to reduce behavior when it does not change utility (e.g., turning off lights or thermostat down when not in the room) – Recent interest in sustainability has created utility for reducing energy consumption, particularly enabled through technology or information

Slide 8 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Contents of the Methods  Development of an evaluation plan  Participant and comparison group selection  Collect sufficient and relevant data  Conduct data cleaning consistent with best practices  Estimate program savings  Estimate savings from other non-RBBP programs  Estimate the persistence of savings

Slide 9 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Development of Evaluation Plan  Each program may call for different evaluation approaches. Therefore, first step is to develop an evaluation plan.  Evaluation plan should include: – Program design and logic model, if applicable – General research questions to be addressed in the evaluation – Program participants and comparison selection – Methods for data collection, cleaning – Analytical approaches – Multi-year approach DRAFT for RTF DISCUSSION

Slide 10 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Participant and Comparison Selection: Utility-selected Participants  If utility selects the program participants – Ideally, use experimental research design – If not possible, then should select comparison group at the time of program deployment.  In selecting participants, the evaluation plan should document – (1) sample selection criteria, – (2) estimated sample size and – (3) the methodology used to determine the appropriate sample size, including the assumed precision and confidence level.

Slide 11 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Participant and Comparison Selection: Comparison group (non-experimental design)  If participants self select or a comparison group is not chosen through experimental research design, then the comparison group should be chosen to be similar to the participant group and at least as large in size.  Minimally, this criteria would include home type, location, and total baseline consumption. – Additional characteristics may be necessary or may increase the relevance of the comparison group - home characteristics, heating fuels, household size, community characteristics, occupant demographics or motivations.  In selecting a comparison group, the evaluation plan should document (1) the timing of comparison group selection, (2) selection criteria, (3) estimated sample size and (4) the methodology used to determine the appropriate sample size, including the assumed precision and confidence level. DRAFT for RTF DISCUSSION

Slide 12 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Collect Sufficient and Relevant Data  Billing data comparison is primary method discussed in methods.  Evaluation plan should outline process to collect sufficient and relevant data for the study.  Minimum data requirements are: – Billing data: Daily (or monthly or bi-monthly) kWh and/or therm data for comparison and participant groups - at least one year prior to the study period, during and after participation – Weather data: HDD and CDD for all months/days used in the billing analysis – Participation group information: Participant or comparison group assignments, frequency of participation, participation start date – Energy-efficiency measure installation: Utility-sponsored installations, Optional survey data on other measure installation for a sub-sample  Other data may improve the analysis or be necessary depending on program design and evaluation, including household and building characteristics data and attitudinal or behavioral information.

Slide 13 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Conduct Data Cleaning Consistent with Best Practices  Data should be managed, cleaned and reviewed using industry- best practices.  All steps taken in data cleaning should be documented, attrition rates should be tracked and removed data should be available for review.  Some data cleaning methods to be documented in the evaluation report include households with zero energy reads, households with estimated reads, households lacking sufficient pre- or post- participation data, households with change in occupancy, and households with exceptionally high or low consumption. DRAFT for RTF DISCUSSION

Slide 14 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Estimating Savings  In general, these methods recommend a regression- based model that considers other factors that affect consumption. Program savings should, at a minimum, be measured annually.  At a minimum, the model(s) should estimate the difference in the change in consumption between participant and comparison groups, accounting for weather and other key factors.

Slide 15 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Estimating Savings: Models  PRISM-type approach - estimates Normalized Annual Consumption for pre- and post- periods for participant and comparison groups; calculate program savings as the difference- in-differences  Monthly billing data model: describes monthly energy usage as a function of weather, a participation indicator, a period indicator and relevant interaction variables.  Either model option could be expanded to include additional variables to improve the model’s explanatory power. DRAFT for RTF DISCUSSION

Slide 16 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Estimate Savings Achieved Through other Non-RBBP Programs  Estimating observed savings from EE measures installed with a utility incentive - avoids double counting of savings and facilitates understanding of investment behavior in participants.  Possible methods: – Calculate savings based on non-RBBP estimates. Match participant and comparison HHs with utility program participation information, using existing estimates – Utilize regression-based models. Variables for non-RBBP program participation can be included or could exclude any participants in non- RBBPs. DRAFT for RTF DISCUSSION

Slide 17 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N Persistence  At this point it is unclear how long expected behavior modifications, and the savings associated with those behavior modifications will last.  Evaluation plan, program design must account for persistence measurement.  Generally, methods recommend: estimate the same regression model for each evaluation year. Differences in the year-to-year savings of the participant group will indicate how the savings change over time.  For additional understanding of persistence and optimal program design, the evaluation could adjust the sample over time and/or add survey data. DRAFT for RTF DISCUSSION

Slide 18 B O N N E V I L L E P O W E R A D M I N I S T R A T I O N DRAFT for RTF DISCUSSION Q&A