TOPICS 2013 Custom Impact Overview 2013 Custom Impact Elements Evaluation Results Gross Impact Findings Net Impact Findings Project Practices Assessment (PPA) Findings Selected IALC 2013 Findings and Recommendations Q&A/Discussion Emphasis on new sampling and evaluation approaches, tailored scope allowing for measure- and program-level results, and new recommendations INDUSTRIAL, AGRICULTURAL AND LARGE COMMERCIAL (IALC) ROADMAP DRAFT REPORT WEBINAR PY2014 CUSTOM IMPACT EVALUATION April 1, 2016

IALC CUSTOM IMPACT EVALUATION AGENDA »IALC Evaluation Overview and Objectives »Data Collection and Approach »Evaluation Sample by PA »Gross Impact Results Discrepancy Factor Impacts Discrepancy Factor Assessment »Net Impact Results »Project Practices Assessment (PPA) Results »IALC 2014 Recommendations 2

2014 IALC CUSTOM IMPACT OVERVIEW 3 »Evaluation of custom (non-deemed measures) across multiple programs that were run by four California program administrators (PAs) Core calculated programs Third party programs Local government partnerships and universities / colleges Over 100 programs within the IALC 2014 scope »Custom measures account for 24% of electric [477 GWh] and 61% of statewide natural gas savings claims [33 Million therms] »History of low gross and net impact realization rates for savings claims

IALC EVALUATION OVERVIEW »IALC Roadmap is composed of nonresidential, non-deemed claims: Non-deemed lighting measures fall under the Commercial RM »PY2014 IALC savings claims: 4

CUSTOM EVALUATION OBJECTIVES »Custom priorities and researchable issues Estimate achieved gross impacts  identify factors affecting gross realization rates (GRRs)  identify opportunities for improvement Estimate free ridership  identify factors affecting free ridership  identify opportunities for improvement Evaluate PA conformance with CPUC policies, decisions, and best practices Collect data and information to assist with other research or study areas 5

DATA COLLECTION AND APPROACH »The gross impact evaluation developed gross realization rates (GRRs) for statistically representative samples of custom projects The gross impact evaluation utilized project-specific measurement and verification (M&V) to estimate GRRs »Similar sampling approach used for net impact evaluation The net impact evaluation estimated free ridership using telephone survey self-reports, reported as net-to-gross (NTG) ratios »Gross and net impact sampling by PA Electric and gas savings were collapsed Sampling and analysis used source energy equivalents* 6 * Conversion rates obtained from “2001 Energy Efficiency Standards for Residential and Non-residential Buildings, California Energy Commission,” June 2001: 1 kWh = 10,239 Btu source energy; 1 Therm = 100,000 Btu source energy. 1 MMBtu =1,000,000 Btu

DATA COLLECTION AND APPROACH 7

»150 PY2014 gross M&V points were targeted and achieved »200 PY2014 NTG points were targeted and 196 were achieved 8

GROSS IMPACT RESULTS Lifecycle Gross Realization Rates – All IOUs 9

GROSS IMPACT RESULTS Lifecycle Gross Realization Rates – All PAs < 1,000,000 MMBtu detail 10

GROSS IMPACT RESULTS 11

GROSS IMPACT RESULTS Mean Lifecycle Gross Realization Rates by PA and Energy Metric (MMBtu and kW) »Mean lifecycle GRR by PA and energy metric all less than 0.70 (except GRR for PG&E kW = 0.74) »All weighted lifecycle GRRs lower than first year GRRs except PG&E »Primarily due to adjustments in EUL (overstated for 70 of 150 sampled projects, understated for 20 of 150 projects) »Across all PAs and sampled measures, EULs overstated by a total of 421 years and understated by 101 years »All PAs had projects with negative or zero GRRs (22 of 150 – 15%) 12

GROSS IMPACT RESULTS Discrepancy Factors 13 n = 75 n = 55 n = 31 n = 20 n = 16 n = 27 n = 10

NET IMPACT RESULTS 14

NET IMPACT RESULTS »As in 2013, existence of a corporate policy is a key source of decision influence for more than half of the projects. »Factors contributing to low program influence Measures added to control or work directly with existing equipment when the addition of the measure is necessary for desired operations Corporate standard practice (i.e., the measure is installed elsewhere at other locations by the customer, including those without rebates) Measures installed to replace old or failing equipment without the explicit demonstration of program influence on selection »Factors contributing to higher program influence Timing of decision follows discussions with program staff Energy efficiency is a key motivation Program rebate is a high percentage of the project cost (>25 percent) Most Influential Factors across PAs 15

PPA Overview PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS »PPA analysis first introduced in IALC 2013 »Systematically examines PA ex-ante savings development practices »Designed to compare and contrast ex-post and ex-ante project-level conclusions »Analysis segregated for pre-2013 and project agreement date »Capture effects of EAR policy and guidance in post-2012 projects »Clear improvement in project practices between periods not observed for any PA except SCG, but this did not translate into a relatively good GRR 16

PPA Overview PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS » Includes examination of: »Project eligibility »Project type »Baseline selection »Project RUL, EUL and cost assessment »Calculation methods, inputs and assumptions » Rating-based examination of: »Quality and comprehensiveness of documentation »Accuracy and appropriateness of ex-ante inputs, assumptions, results, and conclusions » General rating scale: »1 – Does not meet basic expectations »2 – Minimal effort to meet expectations »3 – Meets expectations »4 – Exceeds expectations »5 – Consistently exceeds expectations 17

Project Eligibility PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS » SCG had few eligibility issues overall » SDG&E eligibility treatment improved in relative to pre-2013 » PG&E and SCE had significant eligibility issues that often led to zero or negative savings » Primary reason for eligibility failure is the requirement that measures exceed code / ISP baseline » Other reasons include: CPUC guidance and decisions, previous EAR guidance, non-PA fuels and ancillary impacts, program rules, and EE policy manual stipulations 18

Project Types – All PAs PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS 19

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS » Generally good agreement  Add-on, ER, and ROB most frequently overturned project types 20

Project Baselines – All PAs PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS 21

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS » Generally good agreement  Existing equipment most commonly overturned project baseline (to ISP, most frequently) 22

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS » EUL is not well documented in the project application files » 48% of measures had different PA tracking and evaluation-sourced EULs (on average, evaluation EUL was 4.2 years less than tracking EUL) 23

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS 24 » Only SCG regularly documented incremental costs where it was appropriate to do so

OVERARCHING FINDINGS AND RECOMMENDATIONS » Lifecycle MMBtu GRRs remain low (0.49 to 0.63) » Comparison of , 2013, and 2014 LC MMBtu GRRs: 25

OVERARCHING FINDINGS AND RECOMMENDATIONS » First year MMBtu GRRs range from 0.58 to 0.73 » Comparison of , 2013, and 2014 FY MMBtu GRRs: 26

OVERARCHING FINDINGS AND RECOMMENDATIONS » NTG ratios range from 0.46 to 0.62 » Comparison to and 2013: 27

OVERARCHING FINDINGS AND RECOMMENDATIONS » To more accurately estimate ex-ante savings, the PAs should: Improve documentation and reporting of project EUL, including a review of evaluation EUL conclusions/rationale Improve quality control of: determining project operating conditions, ex- ante baseline determinations, savings calculations, and eligibility rules Ensure adjustments to project savings based on post-installation inspections and M&V. »To improve quality control, PAs should increase due diligence on accuracy, comprehensiveness and documentation in project application files. »To reduce continued moderate free ridership, PAs should test changes to program features designed to increase program-induced savings. 28

KEY RECOMMENDATIONS BY TOPIC AREA »All PAs had projects with negative and/or zero GRRs 22 of 150 M&V points – 15% of the sample »To screen out ineligible projects PAs should improve assessment, monitoring and communication surrounding: program eligibility requirements, manuals, training, and quality control procedures »PAs should regularly review the measures/technologies that are eligible for incentives and eliminate eligibility for those that become standard practice »PAs should review the 22 zero/negative savers and use lessons learned to improve related project practices »The PAs internal communication and coordination efforts for disseminating, implementing, and overseeing implementation of CPUC guidance should be increased »PAs should carefully review applications to ensure incented measures exceed the ISP/code baseline Underperforming Projects 29

KEY RECOMMENDATIONS BY TOPIC AREA » Increase focus on: Accuracy of operating conditions, Use of pre- and post-installation data and information, and Keeping project documentation and tracking claims up to date with field information »The evaluation team recommends that the PAs ensure that savings calculations are based on actual equipment-use schedules and reflect any changes to the post-installation operating parameters »PAs should use longer-term pre- and post-installation M&V activities and wait for measure operation to stabilize and become typical before using post installation data to true-up ex-ante models Operating Conditions 30

KEY RECOMMENDATIONS BY TOPIC AREA »Increase efforts to ensure conformance with CPUC baseline policies and make a greater effort to examine existing equipment RUL »Clearly identify project events in terms of natural replacement, replace on burnout, early replacement, new construction, and add- on equipment, and set the appropriate baseline accordingly »Disseminate information on baseline selection to ensure best practices across program staff, implementers and customers »Ensure that baseline equipment specifications are capable of meeting post-installation operating requirements, that the baseline selected is consistent with the project type, and that regressive baseline considerations are examined »Where applicable, the PAs need to carefully investigate and document the age, condition and functionality of existing equipment and operations, and use these to establish proper baselines. Baseline Conditions 31

KEY RECOMMENDATIONS BY TOPIC AREA »Continue to review and improve impact methods and models through review of evaluation results, industry best practices, and the CPUC’s ex-ante review process »Ex-ante savings estimates and calculation methods should be more thoroughly reviewed and approved by PA technical staff prior to finalization of incentives and savings claims »Conduct periodic due diligence to ensure programs adhere to PA and CPUC impact estimation policies, guidelines, and best practices »Continue to work closely and collaboratively with the CPUC’s ex- ante review process Calculation Methods 32

KEY RECOMMENDATIONS BY TOPIC AREA »It is recommended that a statewide document, similar to the PPA form, be developed for use by all PAs for custom claims; this form has been provided to the PAs for their use »Better ex-ante documentation is needed supporting key project parameters identified in the PPA analysis including cost- effectiveness and lifecycle savings estimation Cross-Cutting and Other Gross Impact-Related 33

KEY RECOMMENDATIONS BY TOPIC AREA »Adopt procedures to screen for and increase efficiency levels for high likelihood free ridership projects. Advise customers to move to a higher level of efficiency undertake a bundled retrofit to ensure deeper savings move to another EE project under consideration »Carefully review the list of qualifying measures; eliminate eligibility for standard practice measures »Make changes to the incentive design to maximize net (not gross) program impacts. Tier incentives by technology class. Consider adopting a payback floor. »Use a comprehensive mix of program features and leverage an array of outreach channels to engage individual customers and encourage a long-term energy efficiency-based focus. Program Influence/NTG 34

THANK YOU CONSULTING & ANALYSIS GROUP 1111 Broadway Oakland, CA