1. Performance Assessment and Monitoring of Energy Efficiency Programs Mallika Nanduri Natural Resources Canada May 12 th, 2005

2. Presentation Overview  Background & context  Overview of OEE’s performance assessment requirements  Current OEE work on performance assessment  General  Determining program impacts  Work to date

3. Background & Context  In 1997 Canada’s Auditor General recommended that NRCan’s Office of Energy Efficiency improve the performance reporting information for its programs by:  Estimating the energy and greenhouse gas (GHG) emissions impacts of its programs; linking the impacts to Canada’s Kyoto committments  From 1998 on  Focus on measurement of program outcomes - accountability for outcomes has only increased (as CC$ have increased)  Development of sound, informative metrics  Subsequent increase in measurement issues to resolve

4.  The OEE needs to:  Understand the extent to which it is meeting its mandate to improve energy efficiency  Incremental EE improvement  Incremental energy savings  Incremental GHG emissions reductions  Understand how OEE can better meet its mandate (program planning, redesign)  Meet its external monitoring and reporting obligations OEE Performance Assessment Requirements

5. Improved Program Effectiveness Increased Market Effects of Programs Measurement Performance Assessment Program Changes OEE Performance Assessment Cycle Business Plan Report to ParliamentReport to Parliament CC Reporting Energy Efficiency Trends in CanadaEnergy Efficiency Trends in Canada Performance Indicators Prospective Impact Analyses Retrospective Impact Analyses Metrics Qualitative Quantitative Metrics External Requirements Internal Requirements

6. Key Measurement Issues  Auditor General (AG) guidelines recognize that  a program’s ability to influence global (aggregate) objectives and outcomes is limited  isolating that program influence from other factors is difficult  OEE efforts include:  Measuring program outcomes and performance  Addressing attribution through various means

7. Current Approaches  Performance indicators/metrics  Qualitative and quantitative indicators of annual program progress  Tracks key program activities, outputs, intermediate and final outcomes  E.g., # of workshops held, # of rebates issued  Used mainly for planning; internal and external reporting  Report to Parliament, State of Energy Efficiency Report, OEE Business Plans

8. Current Approaches  Prospective analyses  RIAS – for forecasting net impacts of EE regulations  Cost-benefit analysis framework  Energy models – for establishing longer-term outlooks on energy efficiency at the economy, sector, sub-sector level  Internal energy models  External work with energy modellers and analysts across Canada

9. Current Approaches  Retrospective analyses  Factorization methods for analyzing net trends in energy efficiency over time at the economy/sector level  Isolate energy efficiency improvements from other changes in the economy – changing activity levels, structural changes, weather  Collection and analysis of actual market data for regulated products  Track changing product efficiencies, sales over time  E.g., appliances  Estimate associated energy savings

10. Current Approaches  Impact and attribution studies  For determining net impacts of programs  Estimating free-ridership, spillover effects  Discrete choice-based surveys and analysis  Technologically and behaviourally realistic models (CIMS at Simon Fraser University)  Focus on market transformation-type effects of OEE programs

11. Estimating Program Impacts Types of changes in EE/energy use Gross Change (Observed) Gross Program Impact Net Program Impact (incremental) Total observed change in EE/energy use between two specific time periods. Includes changes resulting from the program, but also “non-program” changes that would have occurred even without the program. E.g., weather, energy prices, etc. The gross change in EE/energy use, net of non- program factors like weather and economic changes. While a good measure of overall program success, it does not account for free- rider effects. Change in EE/energy use solely attributable to the program, I.e., that would not have occurred without the program. Net of free-rider effects, natural EE improvements, etc.

12. Estimating Program Impacts  Identification and quantification of free-rider effects critical for OEE, as programs focus on changing consumer behaviour  OEE has had some progress with discrete choice methods  Can look at behavioural change more closely  What might have occurred in the absence of the program?  More insight into effects/impacts of the program on public Net Program Impacts = Gross Program Impacts – (free-rider effects) + (spillover/indirect impacts) GJ or Kwh saved

13. Studies to Date & Ongoing Work  5 program specific studies to date – all based on DCM  Equipment, industry, housing, transport  More studies to come  One-Tonne Challenge  Energy Star  Industry  Upcoming federal processes involve ongoing and new work  Program review  UNFCCC reporting

14. End