1. SEEM 94 Calibration to Single Family RBSA Data Analysis and proposed actions Regional Technical Forum May 21, 2013

2. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal Overview - 2

3. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal Purpose - 3

4. Purpose: Align SEEM with Measured Energy Use The SEEM model is used to estimate energy savings for most space-heating-affected residential UES measures using the “calibrated engineering” estimation procedure (see section 2.3.3 of guidelines)guidelines – Heat Pumps and Central AC (ASHP, GSHP, DHP) – Weatherization – New Homes – Duct Sealing – Space Conditioning Interaction Factor Goal: Ensure SEEM94’s results are grounded in measured space heating energy use of single family homes. Use RBSA as source of measured data. Purpose - 4

5. RTF Savings Guidelines Purpose - 5

6. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal History - 6

7. RTF Decision History History - 7 Date RTF Decision Summary Housing TypeT-stat ResultsData Sources Used in Calibration Nov-2009 SEEM 92 model is calibrated. Single Family HP & Gas FAF 70°F Day ; 64°F Night Electric FAF and Zonal 66°F Day & Night 1. Res New Const. Billing Analysis (RLW 2007) 2. SGC Metered Data 3. NEEA Heat Pump Study (2005) Note: Very limited representation of Zones 2 & 3 Apr-2011 SEEM 93 model is calibrated. (implicit decision) Single Family with GSHP 70°F Day ; 64°F Night 1. Missoula GSHP Study (1996) Dec-2011 Use updated SEEM94 model Single Family, Manufactured Home n/a Ecotope updated SEEM code to model the physics of the house infiltration, rather than rely on a constant stipulated infiltration rate input in previous versions of SEEM. Dec-2011 SEEM 94 model is calibrated Manufactured Home 69.4°F Day 61.6°F Night 1. NEEM 2006 2. NEEA Heat Pump Study (2005) 3. MAP 1995 4. RCDP (manufactured homes) Sep-2012 SEEM 94 model is calibrated Multifamily Walk-up and Corridor 68°F Day& Night Townhouses 66°F Day & Night 1. Multifamily MCS (SBW 1994) 2. MF Wx Impact Evaluation for PSE (SBW 2011) 3. New Multifamly Building Analysis (Ecotope 2009) 4. ARRA Verification for King County (Ecotope 2010)

8. RTF Decision History (Continued) For “model is calibrated” decisions… Calibration Methodology: 1.Use available house and operation characteristics data from billing/metering studies to develop inputs to SEEM runs; 2.Adjust SEEM thermostat setting input to achieve a good match (on average) between SEEM output (annual heating energy use) and billing/metering study results. Note: The data sources used were free of (or mostly free of) supplemental fuel usage (wood, propane, oil, etc.) Collection of reliable electric and gas usage data for space heat consumption is relatively easy compared to other fuels. History - 8

9. DateForumTopicOutcomeLinks 1/23/13Full RTF Proposal to adopt calibration: Send staff back to assess calibration needs related to climate and measure parameters; and engage subcommittee. Presentation Minutes 3/20/13 Sub- committee Status update and check in. Presentation Minutes 5/7/13 Sub- committee Review staff’s proposal in detail. Decide whether to recommend RTF adoption. See next slide Presentation Minutes SF Calibration to RBSA - Recent History History - 9

10. May 7, 2013 Subcommittee Attendees Adam Hadley Josh Rushton Bob Tingleff Jim Maunder Rick Knori Bill Koran Mike Lubliner David Bopp Jeff Maguire Nick O’Neil Christian Douglass Dave Roberts Scott Horowitz Ben Larson David Baylon Mark Jerome Mohit Singh- Chaabra Debra Bristow Mark Johnson Peter Miller Cory Read Paulo Tabares Tom Eckman History - 10 3.5 hour meeting Summary: – The group reviewed an earlier version of this presentation, along with the details of the regression development. – The group gave recommendations and discussed next steps. Subcommittee Recommendations (staff completed these) Describe the regression development in a separate report/memo Correct the uninsulated wall u-value Re-calculate regression using a binned HDD variable, rather than continuous Major Issue: Many subcommittee members were very uncomfortable with some of the very low thermostat setting results. – Problem: No alternative calibration method identified. Conclusion: Move forward with presentation to the RTF.

11. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal Methodology - 11

12. Methodology Overview – Data (Step 1) Create two data sources to compare estimates of space heating for homes in RBSA dataset: – RBSA Billing Analysis: Estimates of annual “space heating use” for each house determined by using VBDD VBDD is a “change-point” regression model which uses billing histories to estimate temperature sensitive use VBDD analysis is based on monthly billing data (at least 2 years) – SEEM Simulation Analysis: Estimated annual space heating energy use for each house based on SEEM engineering model RBSA individual home characteristics (e.g., thermal envelope, heating system type, duct tightness) used as model inputs; Initial model runs use thermostat set to 68°F day & night – SEEM is a one-zone model, so t-stat setting input represents the average for the entire house – Actual t-stat settings are not well documented (occupant reported settings are unreliable, especially for zonal systems) – Thermostat setting will be used (step 3) as the “calibration knob”. Methodology - 12

13. Methodology Overview – Regression (Step 2) Use regression techniques to identify building characteristics that drive systematic differences between SEEM(68°F) and Billing Analysis space heating energy use estimates. Methodology - 13

14. Methodology Overview – “Calibrate” (Step 3) Use regression results to determine thermostat set- point that will align (i.e., “calibrate”) SEEM with Billing Analysis annual space heating use. – Calibration based on comparing average of all SEEM annual estimates to average of all Billing Analysis annual estimates. – Calibration is based on building characteristics identified in regression. – SEEM run for each house at varying “day-time” thermostat settings, with “night-time” thermostat settings based on occupant-reported setbacks. Methodology - 14

15. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal Methodology - Data - 15

16. Data Sources Data Source used in this calibration: – Underlying database* for the Single Family Residential Building Stock Assessment (2012) RBSA study’s database offers recent billing analyses results and detailed house characteristics on 1404 houses in the Region. This allows well-defined SEEM runs for each individual house. Methodology - Data - 16 * Using a pre-release version of the database for this analysis.

17. Key Model Input Parameters RBSA Data Availability UAAvailable for each house. WeatherZip code (available for each house) linked to nearest TMY3 weather station. Gas Heating EfficiencyAvailable for some houses; used average for remaining houses. HP Operation & EfficiencyNot readily available. Used ARI control & 7.9 HSPF. Duct System Leakage and Surface Area Available for some houses; used average for remaining houses with ducts. Duct System Insulation and Location Available for each house. InfiltrationAvailable for some houses; used a floor area-scaled average (by foundation type) for remaining houses Mechanical VentilationNot available. Assumed 2 hours /day at 50 cfm. Non-Lighting Internal GainsNot available. See next slide for details. Lighting Internal GainsLPD available for each house; assumed 1.5 hours/day. T-stat SettingAvailable based on interviews, but used this as the “calibration knob”. Methodology - Data - 17

18. Detail: Non-Lighting Internal Gains Methodology - Data - 18 *Hendron, Robert. "Building America Research Benchmark Definition, Updated December 20, 2007." NREL/USDOE EERE. January 2008. NREL/TP-550-42662

19. Realistic SEEM Simulations Not Feasible/Possible for All Homes in RBSA; Some Homes were Filtered Out Filter# of Sites More than one foundation type331 25% > Ceiling Area to Floor Area > 200%, or Missing Ceiling U-value36 Footprint Area to Floor Area < 20%36 30% > Wall Area to Floor Area > 200%, or Missing Wall U-value24 Missing Floor U-value for Crawlspace Foundation5 Window Area = 03 Window u-value = 03 Resulting House Count: 1011 – These issues overlap on some houses, so the sum of the counts cannot be subtracted from 1404 to get 1011. Methodology - Data - 19

20. Data Filters Excluded some RBSA Homes Note: Gas Billing converted to kWh/year using reported AFUE Methodology - Data - 20

21. Methodology - Data - 21 Final Data Set n = 429

22. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal Methodology – Regression - 22

23. Regression Overview Analysis Identify and quantify any systematic patterns (trends) in the differences between SEEM(68°F) and Billing Analysis heating use estimates (∆ kWh = SEEM kWh ‒ Billing Analysis kWh) Systematic means “explained by known variables.” (Example: SEEM(68°F) kWh tends to exceed Billing Analysis kWh in cooler climates.) Tacit assumption: Billing Analysis estimates roughly unbiased. Definitions “Billing Analysis kWh” = Heating energy use estimated using the variable-base degree day method; given in RBSA SF dataset. “SEEM(68°F) kWh” = Heating energy use via SEEM runs using house-specific characteristics data from the RBSA SF dataset with thermostat set to 68°F Methodology – Regression - 23

24. Regression Overview Problem is multivariate… – A single underlying trend (example: ∆ increasing with heating costs) may appear in multiple guises (∆ increasing with HDD, or with U-value, or with building heat loss) Approach is multiple regression… – Compare Billing Analysis kWh with SEEM kWh when SEEM is run with a constant T-stat setting (68°F day, 68°F night.) – Y-variable is the percent difference between SEEM(68°F) kWh and Billing Analysis kWh. – X-variables are physical characteristics known through RBSA. (Specifying the x-variables is a large part of the work of setting up the regression.) Methodology – Regression - 24

25. Setting up the Regression 1.The regression is not a physical model – it is intended to capture unknown effects. 2.The y-variable must capture the differences between SEEM kWh and Billing Analysis kWh. 3.Need to deal with Heteroskedasticity. 4.Need to acknowledge substantial measurement error (random noise in both SEEM and Billing Analysis. 5.Identify x-variables that “lead to” systematic differences between SEEM(68°F) kWh and Billing Analysis kWh. A.Process is iterative: A variable may be weakly correlated with raw y-values but strongly correlated with y’s that have been adjusted to account for some other variable’s influence. B.Colinearity is to be avoided. Example: Including both heat loss rate and vintage. C.Pursuing Parsimony: don’t include too many variables. D.Some variables (duct tightness, infiltration) aren’t known for all houses. E.Prominent candidates would have characteristics that likely influence differences between SEEM(68°F) and Billing Analysis estimates (i.e.: Thermal efficiency drivers (U-values, duct tightness, infiltration), Heating system type, Climate (HDDs) Methodology – Regression - 25

26. Steps to Generating the Regression 1.Define y-variable 2.Identify candidate x-variables – Consideration of physical “common sense” important – Tools: Correlation between y-variable and candidate y-variable plotted vs. candidate 3.Run regression; Check results – Rule of thumb: x-variable “checks out” ok if p-value < 0.05 and no systematic pattern is evident in a plot of the residuals against the x- variable. If it does not check out, the variable should be dropped or reformulated to reflect the pattern in the residuals. 4.Look for other x-variable candidates – Use same tools, but apply to regression-adjusted values 5.Iterate Methodology – Regression - 26

27. The y-variable The chosen x-variables (all indicator variables) – Electric Resistance Heating System Type Value of 1, if Heating System type = Electric Zonal or Electric FAF; Otherwise, value of 0 (if Heating System type = Gas FAF or Heat Pump). – Poor Wall/Ceiling Insulation Value of 1, if Wall u-value > 0.20, or Ceiling u-value > 0.20; Otherwise, value of 0. – Poor Floor Insulation Value of 1, if Floor u-value > 0.25, and Foundation type = vented crawlspace; Otherwise, value of 0. – Climate Zone (2 variables) Heating Zone 2 = 1, if 6000 < HDD 65 < 7500; Otherwise value of 0. Heating Zone 3 = 1, if HDD 65 > 7500; Otherwise, value of 0. Final Regression Definitions Methodology – Regression - 27

28. Variable Estimated Coefficient Standard Error p-value Intercept-0.050.030.05 Electric Resistance0.280.040.00 Poor Ceiling or Wall Insulation0.290.060.00 Poor Floor Insulation0.160.050.00 Heating Zone 20.070.060.24 Heating Zone 30.180.080.02 Regression Results Methodology – Regression - 28 Adjusted R-square = 0.18

29. Other prominent x-variables considered (but not included) Insulation interaction term – Relationship too poor to include (low p-value) Duct Leakage – Too little data to support inclusion Infiltration – Didn’t show a trend Billing Analysis’ variable-base heating degree days – Its inclusion would result in circular logic Methodology – Regression - 29

30. Translating the Coefficients Methodology – Regression - 30

31. Specific Example (House ID: 21233) Methodology – Regression - 31

32. Regression Results Adjustment factors for all possible cases Methodology – Regression - 32 0.84 Example Case

33. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal Methodology – Calibrate - 33

34. T-Stat Calibration We then need to translate the adjustment factors into “calibrated” SEEM thermostat settings. Method: 1.Run SEEM for each house at multiple temperature settings in 2 degree increments – Daytime Settings: … 58, 60, 62, … – Nighttime Setting = Daytime setting – Average Setback(heating system) » Average Setback: Use average difference between reported daytime and nighttime t- stat settings in RBSA dataset; by heating system type: 2.Determine relationship of calibration adjustment factors to temperature settings for each of the 24 scenarios. 3.Interpolate to determine “calibrated” t-stat settings. 4.Note: 5 of the 24 possible scenarios have n=0 houses. In those cases, the average ratio of daytime temperature between the next zone was used to determine the temperature setting for that scenario. Methodology – Calibrate - 34

35. Step 1: Run each house in SEEM at multiple t-stat’s w/setback Methodology – Calibrate - 35 - Each line represents the SEEM runs with setback for one of the 12 individual houses within this case. - Each triangle represents the SEEM(68°F) run for that house.

36. Step 1a: Take the Average Methodology – Calibrate - 36 - Each line represents the SEEM runs with setback for one of the 12 individual houses within this case. - Each triangle represents the SEEM(68°F) run for that house.

37. Methodology – Calibrate - 37

38. Methodology – Calibrate - 38

39. Target Adjustment Factor (from regression): 0.84 Adjustment factor = SEEM(t-stat with setback)/SEEM(68) Methodology – Calibrate - 39 66.8 Step 3: Determine case’s calibrated t-stat setting

40. Proposed “Calibrated” Thermostat Settings Note: Categories with transparent bars had zero houses. Methodology – Calibrate - 40 66.8 Example Case

41. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal Discussion - 41

42. Next Steps If the RTF agrees it’s calibrated, the RTF will be able to use SEEM94 to help estimate energy savings for residential single family – Heat Pump Conversions Upgrades Commissioning, Controls, and Sizing – Weatherization Insulation Windows Infiltration reduction – Duct Sealing – New Home Efficiency Upgrades “Help” is used here because we will still need to deal with “non-electric benefits” for these measures. – This topic is out of scope for today’s discussion. The goal today is simply to determine whether SEEM has been calibrated to provide reliable results. Discussion - 42

43. Discussion Discussion - 43

44. Overview Purpose History Methodology – Data – Regression – “Calibrate” Discussion Proposal Discussion - 44

45. Proposed Motion “I _______ move that the RTF consider SEEM94 calibrated for single family houses.” Proposal - 45