Measuring the Effectiveness of Your Energy Efficiency Efforts Carol Mulholland American Public Power Association National Conference June 16, 2009

Contents  Why Implement Energy Efficiency Programs?  How Do You Know They Work?  Evaluation Overview  Process Evaluation  Impact Evaluation

Why Implement Energy Efficiency Programs?  Rising fuel costs  Rising demand for electricity  More concern over emissions leading to difficulty building new power plants  More global competition for energy  More profitable from utility business perspective  Environmental (global warming) concerns

Policy Changes We Are Seeing Today  Cost Recovery  Program spending on the rise (elec/gas)  Looking at DSM as alternatives or deferral of T&D upgrades  RPS and EPS on the rise  First forward capacity market  Local politicians in the GHG arena  Potential for carbon tax

Cost of EE vs. Price of Electricity: July 2005 Source: State of Delaware, Sustainable Energy Utility Task Force Briefing Book,

Energy Efficiency Uncertainties  Motivation -Regulatory, capacity, T&D or energy affordability?  Focus-Internal (system needs) external (customer needs)?  Policy Context-State Energy Efficiency Portfolio Standards, Federal Carbon tax?  Market -Size and make-up of the market for EE by sector?  Financials -Cost and benefits, avoided costs, compared to what?

Contents  Why Implement Energy Efficiency Programs?  How Do You Know They Work?  Evaluation Overview  Process Evaluation  Impact Evaluation

How Many of You... ...Are counting on energy efficiency/DSM programs to offset increased demand? ...Are counting on energy efficiency to help offset clip peaks? ... Are implementing energy efficiency/DSM programs because someone has told you to? ... Have detailed program operations manuals that explain how your utility's energy efficiency programs work? ...Maintain a database of all program expenditures and estimated savings?

 If you answered “YES” to the questions in BLUE, you need to evaluate your programs to find out whether you’re getting the results you expect.  If you answered “YES” to the questions in GREEN, you’re already part way there.

Contents  Why Implement Energy Efficiency Programs?  How Do You Know They Work?  Evaluation Overview  Process Evaluation  Impact Evaluation

The Importance of Program Evaluation Definition: Evaluation is a systematic process for an organization to obtain information on its activities, its impacts, and the effectiveness of its work, so that it can improve its activities and describe its accomplishments.* Benefits: An effective evaluation will help you learn about your successes [and shortcomings], share information with key audiences, and improve your services.* Other reasons:  To assign a value to energy efficiency as a potential asset  To verify impacts for capacity and carbon trading  To compare EE investments against alternative investments for meeting demand (integrated resources planning)

Why Evaluate? Evaluation:  Measures program impacts  Provides valuable information about program operations  Offers a positive customer contact opportunity  Shows what you’ve accomplished to all interested parties, including City Councils and Utility Boards  Meets regulatory requirements, for those who have them

What are you measuring?  Energy savings  Demand Savings  Customer satisfaction  Technology penetration

The Way it Should be  However, the role of evaluation can go well beyond simply documenting savings to actually improving programs and providing a basis for future savings estimates  If applied concurrently with program implementation, evaluations can provide information in real time to allow for course correction if needed  In summary, evaluation fosters more effective programs and justifies increased levels of energy-efficiency investment. Perhaps this was best put by Economist John Kenneth Galbraith, “Things that are measured tend to improve.”

Interaction Process Evaluation Market Assessment Impact Evaluation

When To Evaluate? ProgramImplementationProgramImplementation Program Design ProgramEvaluationProgramEvaluation is what makes evaluation useful The feedback loop…

Evaluate Early and Often  Evaluation should be an ongoing process  Feedback loop – measure change over time  Use the information – What you discover may cause you to alter your strategy as you go. Make mid-course corrections as necessary.  Interpretation is the most important step – Be careful how the data you have collected is interpreted (and by whom). You should carefully control the outcomes of evaluation such that raw data does not get misinterpreted by non-professionals.

Contents  Why Implement Energy Efficiency Programs?  How Do You Know They Work?  Evaluation Overview  Process Evaluation  Impact Evaluation

Process Evaluation  Assesses the process a program undergoes during implementation  Documents program goals and objectives from a variety of perspectives  Describes program strengths and weaknesses so that success is highlighted and improvements can be made

Designing a process evaluation Identify target audiences Determine what type of research to undertake with each (e.g. Interviews? Surveys? Focus groups?) Lay out the evaluation timeline, taking into consideration both internal and external deadlines Important to seek out independent 3 rd party for an effective evaluation – even more than impact work  To encourage candor on the part of delivery agents and customers alike  To provide credibility with outside parties that an unbiased view will be offered

Market Assessment and Market Transformation Market Assessment looks at the broader market for EE products and services within which a program operates  It is typically done BEFORE or AS PART OF program design  It can be done as an EVALUATION ACTIVITY, when looking at the broader market effects of an EE program  (e.g, sometimes rebate programs may increase product availability and drive product prices down, resulting in…) Market Transformation  Market transformation is a goal for EE programs that seek to overcome significant barriers to adoption in the marketplace

Market Assessment Goals  To determine the size of the program market  To assess the market’s characteristics  To find out what impact, if any, the energy efficiency programs had

Barriers to Market Transformation  High prices  Low availability  Low awareness  Lack of education as to use of product or its merits  Poor infrastructure for proper installation and support of product

Content  Why Implement Energy Efficiency Programs?  How Do You Know They Work?  Evaluation Overview  Process Evaluation  Impact Evaluation

What do we mean by impacts?  Demand and energy savings, market effects, and the environmental and economic costs and benefits that result from an energy services program.  Savings are typically reported on a gross and net basis. “Net energy and demand savings” adjusts the gross savings for a variety of factors.

Methods of Estimation Program Impacts Energy –Engineering Simple Engineering Simulation Models –Statistical Simple Pre/Post (difference of means) Regression (Acct for weather) Comparison Group (Quasi Experimental Design) Detailed Regression Demand –Existing Load Factors –Secondary Load Shapes –End-Use Metering –Simulations Data Collection –Surveys –Billing Data –Metering –Site Visits

What Do You Measure in Impact Evaluation?  Tracking system gross energy and demand savings  Adjusted gross energy and demand savings  Realization rates  Program verified gross energy and demand savings 4. Program Verified 3. Realized 2. Adjusted Gross 1. Tracking System Gross Energy and Demand Savings

What Do You Measure in Impact Evaluation? Net-to-Gross or Attribution Factors are typically measured through customer surveys. These factors are then applied to program verified gross savings to estimate the net energy and demand savings attributed to the Program’s activities. Net-to-gross factors include:  Free ridership  Spillover BECAUSE these can cancel each other out, some studies do not bother to measure either one!

What Do You Measure in Impact Evaluation? Economic costs and benefits  net energy and demand savings, program costs, participant costs, societal costs and benefits – all are estimated to calculate benefit-cost ratios for various cost-effectiveness tests Environmental benefits  Sum up electricity savings (and/or on-site fuel savings, e.g., natural gas)  multiply those savings by emission factors to produce pounds of emissions saved or avoided  emission factors are expressed as pounds of pollutant per MWh of electricity or per therm of natural gas

Approach to Impact Evaluation A typical impact evaluation requires the following activities or elements:  Gather/review program background and savings information  Develop sampling/analysis approach  Conduct surveys and interviews  Conduct project site data collection  Analyze data to develop measure and project level gross savings  Develop gross and net energy and demand savings by program

Final Comments  Set up the evaluation when you set up the program  Get your vendors to do your data tracking, provide monthly status reports (the program will evaluate itself!)  Make sure your processes are seamless and transparent, so someone else can easily get what they need  Remember this is art, not science!

Carol Mulholland