Introduction Second report for TEGoVA ‘Assessing the Accuracy of Individual Property Values Estimated by Automated Valuation Models’ Objective to identify criteria which will provide an appraiser with information on the attendant uncertainty associated with individual property AVM generated valuations We need statistics to help interpret the results (the valuation) of an AVM Ten items of information requested for inclusion in AVM reports Refer to the Appendix for statistical definitions

Recommendation 1 Confidence Intervals for the AVM valuation at: i) the 50% level and ii) the 95% level

A little statistics: the normal distribution http://www.muelaner.com/wp-content/uploads/2013/07/Standard_deviation_diagram.png

The empirical rule for a normal distribution The empirical rule approximates the variation of data in a bell-shaped distribution Approximately 50% of the data in a bell shaped distribution is within a 0.67 standard deviation of the mean, or µ ± 0.67σ Approximately 68% of the data in a bell shaped distribution is within 1 standard deviation of the mean, or µ ± 1σ Approximately 95% of the data in a bell-shaped distribution lies within 2 standard deviations of the mean, or µ ± 2σ Approximately 99.7% of the data in a bell-shaped distribution lies within 3 standard deviations of the mean, or µ ± 3σ Note, although the normal distribution is convenient and simple to use, other distributions/measures reflecting the ‘uncertainty’ surrounding valuations can be employed

16.17 10.61 14.03 8.26 11.08 8.4 11.91 7.44 14.34 12.44 12.13 10.16 10.2 9.28 10.44 8.91

Recommendation 2 A clear explanation, accompanied by a ‘legend’, of the ‘confidence score’ or ‘confidence level’ ‘Confidence score’ and ‘confidence level’ represent the AVM vendor’s assessment of how accurate, in their view, the AVM valuation is. It provides useful information on how much confidence the AVM vendor has in the resulting valuation figure Each vendor will have their own definition, for example, it could be on a scale of 1-9 or (a,b,c,d,e etc) or some other scale. The basis of the calculations will differ for different AVM vendors Some vendors base their definition on Forecast Standard Deviation (FSD) Recommendation 2 asks for a clear explanation together with a legend showing the range of values

Forecast Standard Deviation (FSD) FSD is a statistical measure which provides the probability that the sale price falls within a range of the estimated AVM value The lower the FSD the smaller the range in within which sale price may lie Example: If the FSD is 10%, there is a 68% (2/3rds) probability that the sale price will fall within +/-10% of the AVM estimate If the AVM returns an estimate of €100,000, there is a 68% chance that the sale price will lie in the range €90,000 and €110,000 The FSDs for the four models are: Target analogy for FSD Scale The FSD tells you, with 68% statistical certainty, into which ring the sale price is likely to fall. Model FSD (%) Model 1 9.5 Model 2 10.9 Model 3 8.8 Model 4 6.5

Evaluating AVM valuations The statistical measures are based on AVM ‘error’ This measures the difference between the AVM Value and the Sale Price error = AVM Value – Sale Price Various accuracy measures using the ‘error’ can be calculated In Europe, appraiser valuations are typically used in place of Sale Price Refer to Table 2 in the Appendix for the different accuracy measures

TABLE 1: Summary of required information 1: The i) 50% and ii) 95% confidence intervals of the AVM valuation  2: A clear explanation, accompanied by a ‘legend’, of the ‘confidence score’ or ‘confidence level’  3: Confirmation that comparable have been used in the AVM valuation. If not, what method was used in the AVM valuation? 4: The standard deviation and the skewness of the comparable sales prices, or appraised values, used in the AVM valuation 5: The AVM model’s overall accuracy, based on the comparable sales sample using: i) Mean Absolute Error ii) Median Absolute Error iii) ‘Error Buckets’ for the percentage of valuations lying within +/- 5%, +/- 10% and +/- 20% of the Sales Price 6: The number and the overall geographic distribution of the comparable used in the AVM valuation 7: The range of comparable sales prices used in the AVM valuation 8: Confirmation of the earliest and most recent sales dates of the comparable used in the AVM valuation 9. If ‘adjusted’ comparable sales prices have been used, explanation of how they were adjusted 10: Confirmation of the Benchmark used in arriving at the figures in 4: and 5 above, sales prices or valuations, in arriving at the overall accuracy figures

Conclusions The recommendations are not about judging the accuracy of AVM models per se They are directed towards making an assessment of the accuracy reported for a specific situation, i.e. the property in question For different properties,different conclusions about accuracy, based on the same model, could result

Appendix Statistical measures

Table 2: Definitions and measures of AVM accuracy What is being measured Absolute Error The difference between the sales price and the original valuation, ignoring whether it’s a positive or negative amount. The value of |AVM Value - Sale Price|. Mean Absolute Error The average of the Absolute Errors. Absolute Error/N, where N = the number of valuations. Absolute Percentage Error Absolute Error as a % Sale Price. 100*Absolute Error/Sale Price. Median Absolute Error The value which splits the errors, such that 50% are less than the middle value and 50% are greater than the middle value. Order the Absolute Errors from the lowest value to highest value, then select the middle value, which is known as the median. (If there are an even number of values take the average the two middle values.) The Absolute Errors are thus split into two groups containing an equal number of errors. Percentage Error 100*(AVM Value – Sale Price)/Sale Price. Mean Percentage Error The average of the Percentage Errors. Percentage Error/N, where N = the number of AVM valuations. Standard Deviation of the Percentage Errors The spread of the Percentage Errors around the Mean Percentage Error. Variance = 𝟏 𝐍 (𝐏𝐞𝐫𝐜𝐞𝐧𝐭𝐚𝐠𝐞 𝐄𝐫𝐫𝐨𝐫−𝐌𝐞𝐚𝐧 𝐏𝐞𝐫𝐜𝐞𝐧𝐭𝐚𝐠𝐞 𝐄𝐫𝐫𝐨𝐫) 𝐍 𝟐 Standard Deviation = 𝐕𝐚𝐫𝐢𝐚𝐧𝐜𝐞 Forecast Standard Deviation (FSD) A measure of the spread surrounding the AVM valuation figure. This is the ‘uncertainty value’ which surrounds the AVM valuation, based on the model’s standard deviation error. Confidence Interval The range within which the sales price is likely to be. For a 95% Confidence Interval, the range: (AVM Value-1.96*SD to AVM Value+1.96*SD), where SD= standard deviation error. Vendor’s bespoke ‘Confidence Score’ or ‘Confidence Level’ The vendor’s assessment of the how accurate the AVM valuation actually is. Vendors may/will have their own scale which needs to be clearly explained in the report. ‘Error Buckets’: % of AVM valuations that fall within +/-5%, +/-10% and +/- 20% of the sales price The percentage of comparable valuations falling within the Error Buckets sales price ranges.

Mean (Arithmetic Mean) Mean (arithmetic mean) of data values Sample mean Population mean Sample Size Population Size

Mean (Arithmetic Mean) The most common measure of central tendency Affected by extreme values (outliers) 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 12 13 14 Mean = 5 Mean = 7

Advantages of using the mean A mean can be calculated for every set of data Every value of each data point has an equal weight in determining the value of the arithmetic mean

Problems with the mean The mean is based on all the observations in the set of data but can be greatly affected by any extreme value or values For example, in the sample 1, 2, 3, 4, 5 and 99 the mean is 19, although 5 of the 6 observations are 5 or below The mean cannot be determined for open-ended frequency distributions without additional information

The Median The median is the middle value in an ordered sequence of data We say that half the observations will be above the median and half below The median is thus unaffected by extreme observations in the data set In cases where there are outliers the median is often the best measure

The Median To compute the median we have to order the data We then define the positioning point formula - (n + 1)/2 This finds the median value in an ordered array The computation depends on whether the sample is an odd number of an even number

The Median Rule for odd samples: Rule for even samples: the median is represented by the numerical value corresponding to the positioning point - the (n + 1)/2 ordered observation Rule for even samples: the positioning point will lie between the two middle observations in the ordered array. The median is then the average of the numerical values corresponding to these middle observations

Example - Odd sample Take an odd sized sample like 24 78 92 115 290 24 78 92 115 290 The positioning point formula gives: (5 + 1)/2 = 3 The median is the third observation or 92

Example - even sample Take an even sized sample such as this: 5.5 9.6 8.4 3.5 7.5 6.5 We order the array as: 5.5 3.5 6.5 7.5 8.4 9.6 The positioning point is (6 + 1)/2 = 3.5 The median thus lies between the third and the fourth observation. We average the 3rd and 4th observation to get (6.5+ 7.5)/2 = 7.0 = median value

The Median Robust measure of central tendency Not affected by extreme values In an ordered array, the median is the “middle” number If n or N is odd, the median is the middle number If n or N is even, the median is the average of the two middle numbers 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 12 14 Median = 5 Median = 5

Comparing Standard Deviations Data A Mean = 15.5 s = 3.338 11 12 13 14 15 16 17 18 19 20 21 Data B Mean = 15.5 s = .9258 11 12 13 14 15 16 17 18 19 20 21 Data C Mean = 15.5 s = 4.57 11 12 13 14 15 16 17 18 19 20 21

Range Measure of variation Difference between the largest and the smallest observations: Ignores the way in which data are distributed Range = 12 - 7 = 5 Range = 12 - 7 = 5 7 8 9 10 11 12 7 8 9 10 11 12

Variance An important measure of variation Shows variation about the mean Sample variance: Population variance:

Standard Deviation Simply, the square root of the Variance Shows variation about the mean Has the same units as the original data Sample standard deviation: Population standard deviation: Measured in the same units as the mean and is useful for making probability statements when using distributions

The shape of a distribution: skewness Describes the amount of asymmetry in the distribution symmetric or skewed? Skewness Statistic < 0 0 >0