1. 1 Quality Control Review of E3 Calculator Inputs Comparison to DEER Database Brian Horii Energy and Environmental Economics, Inc. November 16, 2006

2. 2 Overview Purpose was to review how well data entered into the E3 Calculator matches the DEER database. A review of all IOU submissions reveals that very few measures actually state the DEER Run ID or Measure ID The review also shows many measures that were entered, but had no planned installations. –9,236 total measures for all IOUs 6,136 with some installations –2,424 RunID matches 1,304 with installations –155 Measure ID matches 151 with installations The lifecycle and peak impacts of these measures are shown in the next slides

3. 3 Lifecycle Gross kWh Small share of measures have DEER identifiers in E3 Calculator. Values not adjusted for NTG ratio. Values use entered EUL

4. 4 Gross Peak Reduction

5. 5 Lifecycle Gross Therms

6. 6 Run ID Matching For the subset that had Run ID inputs, we compared how the entries match the DEER database. Several criteria for “matches” were used. All matches were to only 2 or 3 significant digits to allow for rounding 1.Matches for either DEER common or code impacts AND either Incremental or Full costs. 2.Matches that ignore a commodity (G or E) if no savings claimed 3.Matches on ratios of Impacts to Costs. Either Common or Code can match. Choice of Incremental or Full costs for denominator based on DEER Application and CostBasis. –Full costs used with Common impacts 86% of the time. –Incr costs used with Code impacts 69% of the time. 4.Matches if entered ratios are lower than DEER ratios. –Note that these methods will NOT match cases where direct install costs are excluded from IMC and put into Admin.

7. 7 RunID Details 994 measures match DEER 1,070 match if entry ignored when no savings claimed 1,077 measures match for Impact/Cost ratios 1,893 match if all impact/cost ratios LOWER than DEER are deemed OK 244 measures do not match under any test. –41 measures have negative peak impacts that were not entered (3.4 MW) –244 measures have cost and impact units that do not match. Of those, 149 passed one of the matching tests, we did not perform any unit reconciliation tests. Impact of changing inputs for the 244 non-matches is shown on the next table. (i.e.: how different are those inputs?)

8. 8 Run ID Subset Impacts for the measures with a DEER Run ID that “match” the DEER database, versus the non-matches are shown in the top half of the table. The bottom half of the table shows the change in impacts if we modified the inputs for the non-match cases to match the DEER database. There is little effect on the kWh forecast, but significant impact on kW, and less so on Therms.

9. 9 kWh Detail for Measures w/ RunIDs Match includes measures where entered ratios are less than DEER ratios. Bottom table shows the change in kWh needed to match the DEER ratios. Negative value indicates that entered values have larger impacts than DEER

10. 10 kW detail SCE overestimates are largely (if not entirely) due to “pasting error” mentioned at the prior QA/QC workshop.

11. 11 Lifecycle Therm Detail Note that negative correction values for Sempra “matches” are due to cases where the measure had a negative therm reduction, but no therm impacts were claimed in the input section.

12. 12 Other Run ID Findings kW (Impact of matching to DEER shown in parentheses) –104 cases where entered kW significantly above DEER. (228 MW) –2 cases where DEER Watts entered as kW, plus 4 other conversion errors. (0.4MW) –6 additional cases where negative DEER impact is entered as positive (0.7 MW) –41 cases where negative kW impacts are not entered. Of those, 30 have no installations. (3.2 MW) Therms –5 cases where DEER values are Therms, not kBTU (24.4 MTh) –4 cases where impact and cost units differ despite DEER indicating “same” (12.1 MTh). Other findings –33 case where measure is RETROFIT, but base for impacts is CODE, not COMMON. This is conservative, and probably not an error, but highlights that users could use the DEER database to arrive at very different results by mixing the two sets of inputs and costs.

13. 13 Review of Measures with No Run ID

14. 14 Process Develop avg & max ratios of impacts per $GIMC –Utility submissions that match DEER RunID, by measure end use categories (based on end use shapes in the E3 calculator) –DEER measures by 60 subcategories. Map Measures –Match measure end use shapes. –Manually map 1739 cases. Compare entered ratios to maximums from step 1

15. 15 Ratio test overview 6916 measures did not have RunID matches in the input data 4681 have some installations 317 have no GIMC Initial filtering using DEER ratios –579 measures with kWh ratios in excess of the DEER sample’s maximum But if average ratios are used, the utility submissions are very conservative in aggregate for kWh and kW. –419 measures with Therm ratios in excess of the DEER sample’s maximum The therm “overestimate” would be slightly higher in aggregate if the average ratios were used for all measures. Problems with filtering analysis –Impact and cost unit mismatches make comparisons difficult –53,014 DEER runs have different units (out of about 120K) –Results shown exclude all DEER runs where units are not the same –Mapping of utility measures to categories is imprecise –Aggregation into categories is imprecise

16. 16 Impact Ratios by End Use Average and Maximum kWh ratios

17. 17 Watts Max & Average kBTU Max & Average

18. 18 Results based on DEER Extract Based on reductions relative to CODE. Similar results if larger of CODE or COMMON is used

19. 19 kWh ratios for DEER sub categories

20. 20 Watt ratios by DEER subcategories

21. 21 Using DEER groups yields comparable results Based on ratios using 60 DEER subcategories, results are similar to end-use matching Matches for gas measures remains poor

22. 22 Unmatched Measures We used the maximum impact per GIMC as a very generous criteria Yet, even with that criteria, 562 cases where kWh ratio is exceeded, 406 cases for kW ratio, and 490 cases for Therm ratio. (1,045 unique cases). –Cases by utility PG&E: 393 SCE:121 SDG&E:167 SoCal Gas:364 Note: may indicate an input problem, or a problem with the assumed mapping or a problem introduced by the large number of DEER runs excluded due to unit mismatches.

23. 23 With better data, the ratio test could be a useful screen Distribution of kWh ratios from the utility measures Largest ratios are for measures such as –Pre-rinse spray valve – electric water heating –Strip curtains –Lighting

24. 24 Identification of measures with largest ratio mismatches Based on sorting lifecycle amounts The measures will differ from column to column # of measures kWhkWTherms 10088%95%93% 5079%88%81%

25. 25 Some DEER Measure Gaps Non-Res freezers and refrigeration Non-Res ovens Non-Res pool heaters Computers Non-Res gas measures

26. 26 Summary of Findings 76% of cases with some installations have no easy links to RunIDs. –Note that for PG&E, 27% of their 1668 cases w/o DEER RunIDs are for calculated measures The Max Ratio test is a blunt instrument, but more precision would be very time consuming given the data in DEER and in the E3 calculators. With that caveat… –KWh estimates appear reasonable. (3% above max) –kW appears high, compared to DEER data (11% above max) –Therms are hard to judge via DEER (54% above max) –The max ratios “pass” 80% of the measures (4,274 out of 5,319) Recommendations –Future tool should require users to explicitly indicate if savings are relative to common or code, and if costs are installation or incremental. –Need a way to link to DEER sub categories, at a minimum, to allow for automated checks –Need a process for creating new approved measures, and for updating DEER. –EULs should be reconsidered for retrofits (remaining life gets common benefit, and EUL- remaining life gets code benefit) –Secondary impact on other fuels should be considered. –Direct install costs should be input on a measure basis (not moved into lump sum admin costs) to allow for QA/QC review.