1. Empirical Data on Settlement of Weather Sensitive Loads Josh Bode, M.P.P. ERCOT Demand Side Working Group Austin, TX September 20, 2012

2. Presentation Overview  Why is settlement of weather sensitive loads an issue?  Testing accuracy of settlement methods  Empirical results  Using smart meter data and control groups for evaluation Page 1

3. Baselines are a tool to estimate demand reductions  Measuring demand reductions is an entirely different task than measuring power production  Power production is metered and thus is measured directly.  Demand reductions cannot be metered. They must be estimated by indirect approaches.  In principle, the reduction is simply the difference between electricity use with and without the load curtailment  However, it is not possible to directly observe or meter what electricity use would have been in the absence of the curtailment – the counterfactual.  Instead, the counterfactual must be estimated. Page 2

4. The accuracy of baseline estimates for large C&I customers has been studied multiple times Page 3 KEMA Baseline Analysis for CEC WG2 Baseline accuracy analysis (Quantum) WG2 Report Baseline Accuracy Analysis (Quantum) LBNL Study (proxy events) Ontario Power Authority Study (FSC) California Aggregator and DBP Evaluations (CAEC) Highly volatile load customer study (CAEC) ISO-NE Baseline Study (KEMA) PJM Baseline Study (KEMA) 2003200420052006200720082009201020112012 California Aggregator Programs (FSC)

5. Settlement of reductions from weather sensitive loads like AC has been studied less  Weather sensitive loads have demonstrated the ability to support multiple grid functions  4.8 million residential AC units and more than 500,000 water heaters in the U.S. have load control devices  Recent technological innovations enable aggregation and real time visibility of small scale loads  Many load control devices now include over and under frequency relays, providing an automated fail-safe mechanism that is synchronized with the grid Page 4

6. Visibility of loads has been tested  Because of the sheer number of AC units, it is not practical to monitor all data points  Real time monitoring of AC units is expensive, even for a sample Page 5 Feeder Load AC end use sample Estimated loads for population

7. Load control programs have shown the ability to provide contingency reserves Page 6 Fast start up time Fast ramp up to full resource capability  Highly granular dispatch is possible – it is possible to dispatch all or some of the resources in a specific area  No one has tested the built-in under frequency relays which provide a failsafe mechanism, do not rely on central dispatch and should respond even faster  Customers that were curtailed 68-71 over the summer report same comfort and frequency of events as customers that were curtailed once

8. Water heaters have demonstrated the ability to follow regulation signals Page 7  Graph is from PJM pilot (Joe Callis)  The initial test was for a unit and has since expanded to wider scale testing

9. However, these loads are highly variable Average Hourly Residential AC Loads by Temperature Page 8 The fact that some loads are very weather sensitive does not mean they are unpredictable

10. Testing the accuracy of settlement methods Page 9

11. We tested 11 different settlement alternatives for baselines for short curtailments Type of Estimator MethodNo.Calculation Data Source Individual AC Aggregate AC Feeder House Data Within- subject estimators Day-matching baseline 110-in-10 with a 20% in-day adjustment capXXXX 210-in-10 without an in-day adjustment capXXXX 3Top 3 in 10 without an in-day adjustment capXXXX Weather- matching baseline 4 Profile selected based on daily maximum temperature without an in-day adjustment cap XXXX Regression 5Treatment variables and no day or hourly lags or leadsXXXX 6Treatment variables with a day lagXXXX 7Treatment variables with hourly lags and leadsXXXX 8No treatment variables but use of hourly lags and leadsXXXX Between- subject estimators Random assignment of load control operations 9Comparison of MeansXX 10Difference-in-differencesXX Pre-calculated load reduction estimate tables 11 Multiply the number of AC units in each geographic location by the corresponding estimate of demand reductions per AC unit for the corresponding area, hour of day, and temperature bin. XX Page 10

12. To test accuracy, one needs to know the correct values Page 11 Because the demand reductions values are artificially introduced and known, we can determine the accuracy of each baseline alternative

13. For the tables and approaches that relied on control groups, we used a split sample approach 1.Randomly split data into two groups 2.Simulate the reduction in one group 3.Use the second group to produce the counterfactual or baseline 4.Calculate impacts and store 5.Repeat 100 times Page 12

14. There are two key issues in assessing accuracy – bias and goodness of fit Type of MetricMetricDescription Bias Mean Percentage Error (MPE) The mean percentage error (MPE) indicates the percentage by which the measurement, on average, tends to over or underestimate the true demand reduction. Goodness-of-Fit Mean Absolute Percentage Error (MAPE) The mean absolute percentage error (MAPE) is a measure of the relative magnitude of errors across event days, regardless of positive or negative direction. It is normalized allowing comparison of results across different data sources. CV(RMSE) This metric is similar to MAPE except that it penalizes large errors more than small ones Page 13

15. Limitations of analysis  The analysis of settlement approaches focuses on ancillary services  Short term reductions to stabilize the grid  Usually triggered by generation or transmission outages and sometimes unexpected changes in wind or loads  Not always in the hottest hours  The errors in the demand reduction estimates depend on the magnitude of the reduction signal  The estimates were based on 50% standard cycling  Air conditioner use is lower in California than in Texas  The direction of the findings likely hold up but the magnitude of the errors will be different Page 14

16. How does the magnitude of the demand reduction signal affects accuracy? Example MetricCustomer ACustomer B Baseline Estimate294 kW True Reference load300 kW Load with DR270 kW225 kW Demand reduction estimate24 kW (294-270) 69 kW (294-225) True demand reduction30 kW (10%)75 kW (25%) % Error-20%-8% Page 15 The customers are nearly identical, but the estimation error differ because of they reduce ad different amount of demand

17. Empirical Results Page 16

18. What is the value of more complex approaches?  In each case, we compare results to the most simple approach – the pre-calculated load reduction tables  We present the results for within-subject and control group approaches separately  We present the bias and goodness of fit metrics separately  All graphs used the same scale Page 17

19. Matching and regression approaches with individual AC data Page 18

20. Matching and regression approaches with aggregated AC data Page 19

21. Matching and regression approaches with whole house data Page 20

22. Matching and regression approaches with feeder data Page 21

23. why are feeder results so inaccurate? Example  Feeder characteristics  2,672 accounts on feeder  266 AC load control accounts (10% of feeder)  292 AC controllable AC units  Likely includes commercial  Penetration higher than 90% of feeders  Event day characteristics  August 24, 2010, max temp 103 F  Simulated event period 12:00-14:00 AC load per unit 0.63 kW Load Impact 35% Controllable AC load 183.3 kW (0.63 kW per unit x 292 AC units) Feeder Impact 64 kW Actual load without DR 7772 kW Simulated load with DR 7708 kW  Percent impact on feeder 0.83%!!! Page 22

24. Control group methods with AC end use data (500 control group, 500 treatment) Page 23

25. Control group methods with whole house data (2,000 control group, 2,000 treatment) Page 24

26. Implications of study  Don’t rely on feeder data for settlement  Day matching baselines are the least accurate approach with weather sensitive loads  Day-matching baselines are not well suited for measuring demand reductions from highly weather sensitive loads  More granular meters do not necessarily increase the accuracy of demand reduction measurement because measuring demand reduction is fundamentally different  Complex methods provide limited improvement  Pre-calculated load reduction tables can produce results that on average are correct, but may err for individual days, especially if they are cooler  Methods with control groups and large sample sizes perform best Page 25

27. Using Smart Meter Data and Control Groups for Evaluation Page 26

28. Impact estimate tables are not developed in a vacuum  They should be based on a history of results  Ideally this includes systematic testing of load control devices under different condition and with different operation and control strategies  The estimates from the operations underlying the tables need to be unbiased and, ideally, precise  The more data points, the better the results  One may need to account for changes in the customer mix, if relevant Page 27

29. With large samples and random assignment, estimation error is virtually eliminated  Actual example for 2011 PG&E SmartAC evaluation  Wide availability of smart meter data and individually addressable devices are a pre-requisite Page 28  Randomly assign population into 10 groups  For each test event, one group was activated and the other 9 were held as a control groups  For a few events, we tested different operation strategies side-by- side

30. It enables side by side testing of different operation strategies Page 29

31. It also enables side by side testing of different control strategies Page 30

32. By using control groups and short events, once can get a substantial history of results  In the PG&E study, each customer only experienced one event, but we obtained results from 7 days, including 3 with side by side testing  It is reasonable to call up to 10 events per customers, especially if the curtailments are short (e.g. 1-2 hrs)  This can yield results under 100 different curtailments to inform the impact estimate tables Page 31

33. Page 32 For any questions, feel free to contact Josh Bode, M.P.P. Freeman, Sullivan & Co. 101 Montgomery Street 15 th Floor, San Francisco, CA 94104 joshbode@fscgroup.com 415.777.0707