1. SEEM 94 Calibration to RBSA Data Progress Report on Phase 2 (Digging a Little Deeper) Subcommittee Regional Technical Forum March 20, 2013

2. Background Ran SEEM on 289 RBSA houses. Full RBSA sample (n = 1404) whittled down due to – Presence of non-gas / non-electric heat, – Poor billing fits, – More than one foundation type, – Other… T-Stat “Calibration” by heat source only  – Aligns SEEM with PRISM kWh averages for each heating-system group. RTF requested further research: – Assess calibration needs related to other variables (climate, measure parameters, etc) in addition to heating system. See January 23, 2013 presentation and minutes for more detailspresentationminutes 2

3. Research Objectives Identify and quantify any systematic patterns (trends) in the differences, ∆ kWh = SEEM kWh ‒ PRISM kWh. – “Systematic” means “explained by known variables.” (Example: SEEM kWh tends to exceed PRISM kWh in cooler climates.) – Problem is multivariate. A single underlying trend (for example, ∆ increasing with heating costs) may be apparent in multiple guises (∆ increases with HDD, or with U-value, or fuel type.) Develop calibration procedure to remove systematic differences. – Express calibration in terms of T-stat adjustments. – Understand calibration impact on measure-related parameters. 3

4. General Approach Regression analysis – Compare PRISM output with SEEM output when SEEM is run at with a constant T-stat setting. All 289 SEEM values generated with T-stat = 68°F. – Y-value is the percent difference between SEEM kWh and PRISM kWh. – X-values are parameters known through RBSA. (Discovering explanatory variables is part of the goal.) Assumptions: – Billing (PRISM) results are generally unbiased (excepting obvious errors) 4

5. Data Issues Heteroskedasticity. – The SEEM/PRISM differences generally increase in magnitude in proportion to SEEM kWh (or PRISM kWh) Measurement error (random noise). – As estimates of heating kWh, SEEM and PRISM both have substantial standard errors. 5

6. 6 Heteroskedasticity…

7. Regression Step 0: The y-variable 7

8. Regression Step 1: The x-variables Goal: Identify variables having systematic influence on y-values. – Substantial random noise in both y-values and x-values, so “influence” is only seen in rough trends (think: correlation). Prominent x-variable candidates: – Measure-related parameters (U-values, duct tightness, infiltration, …) – Heating system type – Climate parameters (esp. HDDs) – Others? Model development is iterative. – A variable may be weakly correlated with raw y-values but strongly correlated with y’s that have been adjusted to account for some other variable’s influence. Need a single regression model that includes all important x- variables at once (different models for separate calibrations okay). 8

9. Limitations to keep in mind… Multicollinearity. When a potential x-variable closely tracks some combination of variables that are already included. – This redundancy leads to unstable model fits. – Threshold for “tracks too closely” gets low when the usual suspects are around: High noise / faint signal / small sample. Parsimony. General principle: Don’t over-fit the data (by including too many explanatory variables). Some variables aren’t known for many houses. 9

10. 10

11. Current Status There are substantial systematic patterns (in the y-values) related to U-value and heating system type. Final model must account for these. – Each variable’s influence persists after controlling for the other variable. – Wall-, ceiling-, and floor- U-values are too strongly correlated to separately include in one model. (Must combine into one or two coarser variables). Several other variables are influential too, but we can’t include them all. Still developing variables and comparing models. We present a particular linear model with variables for: Insulation, Heating System Type, and Heating Degree Days Purpose is to convey trade-offs and solicit input. 11

12. Insulation Variable(s) Model uses an indicator variable to capture “Poor Insulation”. – Definition: Insulation is “poor” if: (Wall u-value > 0.25) OR (Ceiling u-value > 0.25) OR both. – Judgment call in choosing this indicator variable versus the (continuous) weighted-average u-value, U 0. – Still checking options for floor- and window u-values. Floor-u variable is tricky because of different foundation types. 12

13. 13

14. Heating System Variable Four distinct heating systems in the sample: Electric zonal Electric FAF Gas FAF Heat pump After controlling for insulation, heating system effect mostly captured with just two groups: “Electric Resistance” = Electric zonal / Electric FAF “Gas/HP” = Gas FAF / Heat Pump Parsimony: two is better than four! 14

15. 15

16. 16

17. 17 Regression fit for this model

18. 18

19. 19 Interpreting the fitted model

20. Interim Calibration Results (for this example) 20

21. And Finally, Translate percent kWh adjustments into adjustments in daytime t-stat setting (from 68 °F). Do we interpret t-stat calibrations literally (as in, homes like this tend to maintain lower winter temps)? (Refer to morning session!) No data limitations here: we can directly observe SEEM’s sensitivity to t-stat settings. – Just run SEEM multiple times, with a different setting on each. 21

22. 22

23. 23 Calibration Results (for this example) There were no cases of poor insulation in heating zone 3 within the filtered sample (289).

24. Discussion Is this the correct approach? Keep in mind we’re still working on incorporating other variables into the model Duct tightness Infiltration Floor u-value Window u-value 24