Electric / Gas / Water MAESTRO Evaluation Showcase July 26-27, 2006 Project Manager: Pierre Landry, SCE Lead Consultants: Mike Rufo, Itron; Keith Rothenburg, Southern Exposure Engineering; KEMA Inc. What a Realization: An Ex Post Impact Evaluation of a Performance-Based Program (2002/2003 SPC Evaluation)

Knowledge to Shape Your Future 2 Overview of Presentation Evaluation Context and Background Approach Results and Key Findings Recommendations

Knowledge to Shape Your Future 3 Context: Custom, Bidding, and SPC Type Programs History and types –custom rebates, bidding, SPC Goals –resource acquisition and market transformation Common features –focus on custom efficiency measures in large C&I –encouragement of comprehensive projects –inclusion of technical engineering review –requirements for proof of project installation

Knowledge to Shape Your Future 4 Common Issues for Large C&I Programs Uncertainty in savings estimates Risk of gaming and fraud Costs of measurement and verification Keeping application requirements manageable yet effective Distributing funds equitably Minimizing free riders/maximizing net impacts Supporting the efficiency services market

Knowledge to Shape Your Future 5 Background on CA SPC 3 IOUs (PG&E, SCE, SDG&E) Targeted at Large C&I Incentives by end use –Lighting ($0.05), HVACR ($0.14), Process/Other ($0.08) Calculated or Measured Savings Path –Determined by utility –90+% calculated All projects require Site Installation Report

Knowledge to Shape Your Future 6 Why An Ex Post Evaluation of “Performance Contract” Program? PY2002 & PY2003 first years SPC had independent ex post impact evaluation Originally (’98/’99), all SPC projects required in-program M&V –Initially, 2 years of M&V, then 1 year MT policy environment in CA ( ) –Virtually no attention to ex post impact evaluation M&V dropped as requirement in 2000 –Utilities retained right to require –Most projects done under calculated savings path Shift back to resource acquisition focus with energy crisis and program admin decision

Knowledge to Shape Your Future 7 Objectives –Re-establish process for site-specific ex post impact evaluation –Implement process for representative sample –Estimate program realization rate –Estimate net-to-gross –Recommendations for improvement Constraints –Total budget vs. program size, project complexity, # of utilities –Per site budgets Limited sub-metering –Took 2 years for most of sample to install projects –Limited pre-metering available –Self report NTGR method PY2002 SPC Evaluation Objectives and Constraints

Knowledge to Shape Your Future 8 Approach Develop and implement sample design (ratio estimation) Obtain files & savings documentation Review applications and prepare ex post analysis plans Conduct on-site data collection (limited metering) Develop ex post impact estimates Prepare detailed, site-specific documentation Carry out quality control review Extrapolate final ex post results to the population Estimate free ridership –self-report method CADMAC protocol-level battery

Knowledge to Shape Your Future 9 Population Combined SPC Population (apps): Combined SPC Population (kWh):

Knowledge to Shape Your Future 10 Sample Design PY2002-PY2003 SPC Sample (apps) PY2002-PY2003 SPC Sample (kWh)

Knowledge to Shape Your Future 11 Gross Realization Rate Results

Knowledge to Shape Your Future 12 Gross Realization Rate Results

Knowledge to Shape Your Future 13 Freeridership and Net-to-Gross DEER SPC NTGR = 0.7

Knowledge to Shape Your Future 14 Key Findings PY SPC more like a custom rebate than performance contract program Overall realization rate reasonably high but below one Wide range in individual realization rates Importance of influence of largest projects Gross savings may not be as conservative as SPC program managers intended NTGR slightly lower than period

Knowledge to Shape Your Future 15 Other Savings-Related Key Findings Wide range in the quality of applications and supporting documentation Need for increased verification and documentation of assumptions in project files Experience and expertise levels of the reviewers vary widely Difficulties in assessing complex industrial process projects Limited estimation of kW peak demand savings

Knowledge to Shape Your Future 16 Impact-Related Recommendations Consider targeted increases in the level of technical documentation Consider a stronger application affidavit statement regarding savings assumptions Further standardize the review approach and documentation requirements for recurring complex projects Consider providing or requiring more technical support for applicants for complex projects

Knowledge to Shape Your Future 17 Improve reviewer documentation Consider increasing: –conservatism for calculated path savings estimates –measurement for large complex projects –incentive premium for measured projects Increase pre-installation measurement for very large projects with highly uncertain baseline conditions Consider independent review of the SPC calculator Savings-Related Recommendations (Cont.)

Knowledge to Shape Your Future 18 Increase efforts to reduce free ridership Consider: –Higher incentive levels for higher payback measures or emerging technologies –Incorporating a payback floor –Bonus payments for first-time participants –Custom baselines for process improvements –Excluding projects that are obvious free riders Free Ridership-Related Considerations

Knowledge to Shape Your Future 19 Consider shifting to ex post impact evaluations from program-year to paid-year basis –or a combination Expand scope of ex post measurement –more measurement per site –more sites in samples –incorporate uncertainty in ex post estimates into program realization rate confidence interval Integrate evaluator early into program process to enable pre-measurement where necessary Evaluation-Related Recommendations