3. Introduction Second report for TEGoVA
‘Assessing the Accuracy of Individual Property Values Estimated by Automated Valuation Models’
Objective to identify criteria which will provide an appraiser with information on the attendant uncertainty associated with individual property AVM generated valuations
We need statistics to help interpret the results (the valuation) of an AVM
Ten items of information requested for inclusion in AVM reports
Refer to the Appendix for statistical definitions

4. Recommendation 1 Confidence Intervals for the AVM valuation at:
i) the 50% level and
ii) the 95% level

5. A little statistics: the normal distribution

6. The empirical rule for a normal distribution
The empirical rule approximates the variation of data in a bell-shaped distribution
Approximately 50% of the data in a bell shaped distribution is within a 0.67 standard deviation of the mean, or µ ± 0.67σ
Approximately 68% of the data in a bell shaped distribution is within 1 standard deviation of the mean, or µ ± 1σ
Approximately 95% of the data in a bell-shaped distribution lies within 2 standard deviations of the mean, or µ ± 2σ
Approximately 99.7% of the data in a bell-shaped distribution lies within 3 standard deviations of the mean, or µ ± 3σ
Note, although the normal distribution is convenient and simple to use, other distributions/measures reflecting the ‘uncertainty’ surrounding valuations can be employed

7. 16.17
10.61
14.03
8.26
11.08
8.4
11.91
7.44
14.34
12.44
12.13
10.16
10.2
9.28
10.44
8.91

8. Recommendation 2
A clear explanation, accompanied by a ‘legend’, of the ‘confidence score’ or ‘confidence level’
‘Confidence score’ and ‘confidence level’ represent the AVM vendor’s assessment of how accurate, in their view, the AVM valuation is. It provides useful information on how much confidence the AVM vendor has in the resulting valuation figure
Each vendor will have their own definition, for example, it could be on a scale of 1-9 or (a,b,c,d,e etc) or some other scale. The basis of the calculations will differ for different AVM vendors
Some vendors base their definition on Forecast Standard Deviation (FSD)
Recommendation 2 asks for a clear explanation together with a legend showing the range of values

9. Forecast Standard Deviation (FSD)
FSD is a statistical measure which provides the probability that the sale price falls within a range of the estimated AVM value
The lower the FSD the smaller the range in within which sale price may lie
Example: If the FSD is 10%, there is a 68% (2/3rds) probability that the sale price will fall within +/-10% of the AVM estimate
If the AVM returns an estimate of €100,000, there is a 68% chance that the sale price will lie in the range €90,000 and €110,000
The FSDs for the four models are:
Target analogy for FSD Scale
The FSD tells you, with 68% statistical certainty, into which ring the sale price is likely to fall.
Model
FSD (%)
Model 1
9.5
Model 2
10.9
Model 3
8.8
Model 4
6.5

10. Evaluating AVM valuations
The statistical measures are based on AVM ‘error’
This measures the difference between the AVM Value and the Sale Price
error = AVM Value – Sale Price
Various accuracy measures using the ‘error’ can be calculated
In Europe, appraiser valuations are typically used in place of Sale Price
Refer to Table 2 in the Appendix for the different accuracy measures

11. TABLE 1: Summary of required information
1: The i) 50% and ii) 95% confidence intervals of the AVM valuation 
2: A clear explanation, accompanied by a ‘legend’, of the ‘confidence score’ or ‘confidence level’ 
3: Confirmation that comparable have been used in the AVM valuation. If not, what method was used in the AVM valuation?
4: The standard deviation and the skewness of the comparable sales prices, or appraised values, used in the AVM valuation
5: The AVM model’s overall accuracy, based on the comparable sales sample using:
i) Mean Absolute Error
ii) Median Absolute Error
iii) ‘Error Buckets’ for the percentage of valuations lying within +/- 5%, +/- 10% and +/- 20% of the Sales Price
6: The number and the overall geographic distribution of the comparable used in the AVM valuation
7: The range of comparable sales prices used in the AVM valuation
8: Confirmation of the earliest and most recent sales dates of the comparable used in the AVM valuation
9. If ‘adjusted’ comparable sales prices have been used, explanation of how they were adjusted
10: Confirmation of the Benchmark used in arriving at the figures in 4: and 5 above, sales prices or valuations, in arriving at the overall accuracy figures

12. Conclusions
The recommendations are not about judging the accuracy of AVM models per se
They are directed towards making an assessment of the accuracy reported for a specific situation, i.e. the property in question
For different properties,different conclusions about accuracy, based on the same model, could result

13. Appendix Statistical measures

14. Table 2: Definitions and measures of AVM accuracy
What is being measured
Absolute Error
The difference between the sales price and the original valuation, ignoring whether it’s a positive or negative amount.
The value of |AVM Value - Sale Price|.
Mean Absolute Error
The average of the Absolute Errors.
Absolute Error/N, where N = the number of valuations.
Absolute Percentage Error
Absolute Error as a % Sale Price.
100*Absolute Error/Sale Price.
Median Absolute Error
The value which splits the errors, such that 50% are less than the middle value and 50% are greater than the middle value.
Order the Absolute Errors from the lowest value to highest value, then select the middle value, which is known as the median. (If there are an even number of values take the average the two middle values.) The Absolute Errors are thus split into two groups containing an equal number of errors.
Percentage Error
100*(AVM Value – Sale Price)/Sale Price.
Mean Percentage Error
The average of the Percentage Errors.
Percentage Error/N, where N = the number of AVM valuations.
Standard Deviation of the Percentage Errors
The spread of the Percentage Errors around the Mean Percentage Error.
Variance = 𝟏 𝐍 (𝐏𝐞𝐫𝐜𝐞𝐧𝐭𝐚𝐠𝐞 𝐄𝐫𝐫𝐨𝐫−𝐌𝐞𝐚𝐧 𝐏𝐞𝐫𝐜𝐞𝐧𝐭𝐚𝐠𝐞 𝐄𝐫𝐫𝐨𝐫) 𝐍 𝟐
Standard Deviation = 𝐕𝐚𝐫𝐢𝐚𝐧𝐜𝐞
Forecast Standard Deviation (FSD)
A measure of the spread surrounding the AVM valuation figure.
This is the ‘uncertainty value’ which surrounds the AVM valuation, based on the model’s standard deviation error.
Confidence Interval
The range within which the sales price is likely to be.
For a 95% Confidence Interval, the range:
(AVM Value-1.96*SD to AVM Value+1.96*SD), where SD= standard deviation error.
Vendor’s bespoke ‘Confidence Score’ or ‘Confidence Level’
The vendor’s assessment of the how accurate the AVM valuation actually is.
Vendors may/will have their own scale which needs to be clearly explained in the report.
‘Error Buckets’: % of AVM valuations that fall within +/-5%, +/-10% and +/- 20% of the sales price
The percentage of comparable valuations falling within the Error Buckets sales price ranges.

15. Mean (Arithmetic Mean)
Mean (arithmetic mean) of data values
Sample mean
Population mean
Sample Size
Population Size

16. Mean (Arithmetic Mean)
The most common measure of central tendency
Affected by extreme values (outliers)
Mean = 5
Mean = 7

17. Advantages of using the mean
A mean can be calculated for every set of data
Every value of each data point has an equal weight in determining the value of the arithmetic mean

18. Problems with the mean
The mean is based on all the observations in the set of data but can be greatly affected by any extreme value or values
For example, in the sample 1, 2, 3, 4, 5 and 99 the mean is 19, although 5 of the 6 observations are 5 or below
The mean cannot be determined for open-ended frequency distributions without additional information

19. The Median
The median is the middle value in an ordered sequence of data
We say that half the observations will be above the median and half below
The median is thus unaffected by extreme observations in the data set
In cases where there are outliers the median is often the best measure

20. The Median To compute the median we have to order the data
We then define the positioning point formula - (n + 1)/2
This finds the median value in an ordered array
The computation depends on whether the sample is an odd number of an even number

21. The Median Rule for odd samples: Rule for even samples:
the median is represented by the numerical value corresponding to the positioning point - the (n + 1)/2 ordered observation
Rule for even samples:
the positioning point will lie between the two middle observations in the ordered array. The median is then the average of the numerical values corresponding to these middle observations

22. Example - Odd sample Take an odd sized sample like 24 78 92 115 290
The positioning point formula gives:
(5 + 1)/2 = 3
The median is the third observation or 92

23. Example - even sample Take an even sized sample such as this:
We order the array as:
The positioning point is (6 + 1)/2 = 3.5
The median thus lies between the third and the fourth observation.
We average the 3rd and 4th observation to get ( )/2 = 7.0 = median value

24. The Median Robust measure of central tendency
Not affected by extreme values
In an ordered array, the median is the “middle” number
If n or N is odd, the median is the middle number
If n or N is even, the median is the average of the two middle numbers
Median = 5
Median = 5

25. Comparing Standard Deviations
Data A
Mean = 15.5
s = 3.338
Data B
Mean = 15.5
s = .9258
Data C
Mean = 15.5
s = 4.57

26. Range Measure of variation
Difference between the largest and the smallest observations:
Ignores the way in which data are distributed
Range = = 5
Range = = 5

27. Variance An important measure of variation
Shows variation about the mean
Sample variance:
Population variance:

28. Standard Deviation Simply, the square root of the Variance
Shows variation about the mean
Has the same units as the original data
Sample standard deviation:
Population standard deviation:
Measured in the same units as the mean and is useful for making probability statements when using distributions

29. The shape of a distribution: skewness
Describes the amount of asymmetry in the distribution
symmetric or skewed?
Skewness
Statistic
< >0