1. ESS Quality and Performance Indicators
ESTP Training Course
“Quality Management and survey Quality Measurement”
Rome, 24 – 27 September 2013
Giovanna Brancato,
Senior Researcher
Head of unit “Auditing, Quality and Harmonisation“
Istat
Giorgia Simeoni,
Researcher
Unit “Auditing, Quality and Harmonisation“
Istat

2. ESS Quality and Performance Indicators
Set of standard quality indicators developed by an expert group
Indicators for each ESS quality dimension, if possible “product” oriented
To be included in ESS standard quality reporting

3. Measuring product quality
Not all quality dimensions have quantitative measurements associated (e.g.: accessibility)
Some components are difficult to measure in an objective way (e.g.: relevance)
Direct measurement of accuracy is costly, indirect measurement is proposed

4. Accuracy and reliability Timeliness and punctuality
ESS Quality and Performance Indicators
Relevance
Data completeness - rate
Accuracy and reliability
Sampling error - indicators
Over-coverage - rate
Unit non-response - rate
Item non-response - rate
Imputation - rate
Common units - proportion
Data revision - average size
Timeliness and punctuality
Time lag - first results
Time lag - final results
Punctuality - delivery and publication
Accessibility
Data tables - consultations
Metadata - consultations
Metadata completeness - rate
Comparability and coherence
Asymmetry for mirror flows statistics - coefficient
Length of comparable time series

5. A similar indicator can be defined at national level.
Relevance
Measures for the relevance can be obtained by means of the customer satisfaction survey, as degree of satisfaction with regard to given statistics.
Indirect Relevance measures are obtained looking at the consultation of the data from the users.
Within the relevance dimension, Eurostat defines a Data Completeness rate as:
The ratio of the number of data cells provided to the number of data cells required (by regulation) or considered relevant by Eurostat.
Eurostat indicator can assume value 1, indicating that all the required or relevant data cells are available.
A similar indicator can be defined at national level. 5

6. collected data are referred at time TX
Timeliness
collected data are referred at time TX T0 TX T1 T2 TE TP TF
planning
data collection
data processing
results dissemination
to Eurostat
provisio-nal data
final data 6

7. Punctuality TE TP TF TE1 TP1 TF1 scheduled results dissemination
actual
results dissemination
to Eurostat
provisio-nal data
final data
to Eurostat
provisio-nal data
final data 7

8. Timeliness
In general, the timeliness of statistical outputs is the length of time between the end of the event or phenomenon they describe and their availability
Eurostat defines Timeliness of first results and Timeliness of final results
It is computed as the number of days (or weeks or months) from the last day of the reference period to the day of publication of first/final results
The indicator has to be interpreted considering the periodicity of the estimates, i.e. the interval of time the estimates refer to.
Example: The monthly estimates for the Italian Survey on Dispatch/arrival of goods with EU Countries (Intrastat System) has a timeliness of final results around 65 to 70 days. 8

9. The target value for this indicator is 0.
Punctuality
Punctuality is the time lag between the delivery/release date of data and the target date for delivery/release as agreed for delivery or announced in an official release calendar, laid down by Regulations or previously agreed among partners.
It is computed as the difference (days) from actual date of the effective provision of the statistics and scheduled date
The target value for this indicator is 0.
Eurostat indicator is focused on the proportion of punctual deliveries. i.e. on the Rate of punctuality of data publication. 9

10. Accessibility and clarity
Data table consultation: Number of consultations of data tables within a statistical domain for a given time period.
It is computed as number of data tables views for on-line databases.
Metadata consultation: Number of metadata (ESMS)* consultations within a statistical domain for a given time period.
It is computed as number of times a metadata file is viewed
No target value of reference.
The indicators contribute to the assessment of users' demand of data and metadata (level of interest), for the assessment of the relevance of subject-matter domains.
* Euro SDMX Metadata Standard: Structure proposed by Eurostat for documenting data files, dealt with in «Quality Assessment» (day 4 of the course) 10

11. Accessibility and clarity
Rate of Metadata Completeness: The ratio of the number of metadata elements provided to the total number of metadata elements applicable.
Computed separately for three subgroups of concepts: metadata on statistical outputs, statistical processes and quality.
The reference is ESMS structure. The target value is 1 meaning that all the Euro-SDMX Metadata concepts are provided.
It reflects the extent of the availability of the metadata, not the quality. 11

12. Geographical comparability
Coefficient of asymmetry for mirror flows statistics: discrepancies between data related to flows, e.g. for pairs of countries.
Is computed as the difference or the absolute difference of inbound and outbound flows between a pair of countries divided by the average of these two values. It refers to any kind of flow, e.g. amounts of products traded, number of people visiting a country for tourism purposes, etc.
The target value should be as close to zero as possible, since – at least in theory – the value of inbound and outbound flows between pairs of countries should match.
These indicators can help checking the consistency of data reporting, of the reporting process and the definitions used. They can also help to estimate missing data. 12

13. Comparability over time
Length of comparable times series: number of reference periods in time series from last break.
Breaks in statistical time series may occur when there is a change in the definition of the parameter to be estimated (e.g. variable or population) or the methodology used for the estimation. Sometimes a break can be prevented, e.g. by linking.
A long time series may seem desirable, but breaks may be motivated by the introduction of new concepts due to reality change or by the need to achieve coherence with other statistics.
If there has not been any break, the indicator is equal to the number of the time points in the time series. 13

14. Reliability: Data revision indicators
Revisions analysis shows the degree of closeness of initial estimates to subsequent or final estimates. Since all estimates are affected by error, this type of analysis cannot definitively demonstrate the accuracy of initial estimates. However, the amount of revision is still an indicator of accuracy, since it is reasonable to assume that estimates are converging towards the true value as estimates are based on more and more reliable sources.
Mean Revision:
The mean absolute revision:
where:
XLt - “later” estimate, Lth release of the item at time reference t
XPt - “earlier” estimate, Pth release of the item at time reference t
n = number of estimates (reference periods) in the time series taken into account 14

15. Reliability: Data revision indicators
Another indicator [RMAR: Relative Mean Absolute Revision] considers the relative differences between the estimates.
Both MAR and RMAR indicators provide information on the stability of the estimates. They do not provide information on the direction of revisions, since the absolute values of revisions are considered.
Information on the direction of the revision is instead provided by MR:
positive sign means upwards revision (underestimation)
negative sign indicates overestimation in the first case
MR sometimes is referred to as ‘average bias’, but a nonzero MR is not sufficient to establish whether the size of revisions is systematically biased in a given direction. To ascertain the presence of bias it has to be assessed whether MR is statistically different from zero.
The OECD has a considerable experience on Revision Indicators (website methodology section available and revision analysis database: ex. of Industrial production index) 15

16. Accuracy: Sampling error indicators
Sampling errors are commonly reported in relative terms,
Coefficient of variation:
Or in terms of confidence intervals, i.e. an interval that includes with a given level of confidence the true value of a parameter

17. Accuracy: Non sampling error indicators
Direct measurement
Set of methodologies to estimate components of the MSE
Measurements of the real impact of the error on the estimates produced from the survey.
Indirect measurement
Quality indicators obtained as a byproduct of the production process
Give an indication of the “amount” of error occurred and not of its impact on the estimates

18. Direct and indirect quality measurement
Direct measurement
 direct measure of bias and variance components
 for each error typology or error source
 expensive
 no overall measure of impact of errors on survey estimates
Quality indicators
 indirect measure of impact of errors on survey estimates
 cheap
 way for monitoring the process
 alarm bells for further studies

19. Standard outcome classification
Total Units
Resolved Units
Unresolved Units
In-Scope Units
Out-of-Scope Units
Respondents
Nonrespondents
Adapted from Hidiroglou et al. (1993).

20. Standard outcome definitions
Total units: total number of units belonging to the survey of interest. For sample surveys, it corresponds to the number of sampling units
Resolved Units: it has been possible to ascertain their eligibility status (vs unresolved)
In-Scope Units: units belonging to the population of interest for the survey
Out-of-Scope Units: unit not belonging to the population of interest for the survey, although included in the frame
Nonrespondents: units for which it has not been possible to obtain information
Respondents: units for which it has been possible to obtain information

21. Coverage indicators Over-coverage rate
Definition: The rate of over-coverage is the proportion of units accessible via the frame that do not belong to the target population (are out-of-scope).
ESS standard over-coverage rate*:
O = set of out-of-scope units
E = set of in-scope units
Q = set of units of unknown eligibility (unresolved units)
α = the estimated proportion of cases of unknown eligibility that are actually eligible. It should be set equal 1 unless there is strong evidence for assuming otherwise
wj= weight of unit j
*Eurostat, 2010

22. Coverage indicators ESS standard over-coverage rate*:
Three main cases:
wj= 1: the unweighted over-coverage rate gives the number of units that have been found not belonging to the target in proportion to the total number of observed units. The number refers to the sample, the census or the register population studied.
wj= dj ,dj= 1/πj: the design-weighted overcoverage rate is an estimate for the frame population in comparison with the target population, based on the information at hand, usually a sample.
wj= dj xj, xj is the value of an auxiliary variable X: the size-weighted overcoverage rate expresses the rate in terms of a chosen size variable, e.g. turnover in business statistics. (This case is less interesting for overcoverage than for non-response.)
*Eurostat, 2010

23. Unit nonresponse indicators
Unit nonresponse rate
Definition: The ratio of the number of units with no information or not usable information to the total number of in-scope units.
ESS standard unit nonresponse rate*:
R = set of respondent units
NR = set of non respondent units
Q = set of units of unknown eligibility (unresolved units)
α = the estimated proportion of cases of unknown eligibility that are actually eligible. It should be set equal 1 unless there is strong evidence for assuming otherwise
wj= weight of unit j
*Eurostat, 2010

24. Unit nonresponse indicators
ESS standard unit nonresponse rate:
Three main cases:
wj= 1: the un-weighted unit nonresponse rate shows the result of the data collection in the sample, rather than an indirect measure of the potential bias associated with non-response. If α=1, it assumes that all the units with unknown eligibility are eligible, so it provides a conservative estimate with regard to other choices of α
wj= dj ,dj= 1/πj: the design-weighted unit nonresponse rate shows how well the data collection worked considering the population of interest
wj= dj xj, xj is the value of an auxiliary variable X: the size-weighted unit nonresponse rate would represent an indirect indicator of potential bias caused by nonresponse prior to any adjustments
*Eurostat, 2010

25. Item nonresponse rate*
Definition: The item nonresponse rate for a given variable is defined as the ratio between in-scope units that have not responded and in-scope units that are required to respond to the particular item.
ESS standard item nonresponse rate*:
RY = the set of eligible units responding to item Y (as required)
NRY = the set of eligible units not responding to item Y although this item is required. – The denominator corresponds to the set of units for which item Y is required. (Other units do not get this item because their answers to earlier items gave them a skip past this item; they were “filtered away”.)
wj = weight of unit j
*Eurostat, 2010

26. Item non response rate Weights
wj= 1: the un-weighted item nonresponse rate
wj= dj ,dj can be = 1/πj or the final-weight after adjustments for unit nonresponse and coverage errors: the design-weighted item nonresponse rate
wj= dj xj, xj is the value of an auxiliary variable X: the size-weighted item nonresponse rate
Interpretation: A high item non-response rate indicates difficulties in providing information, e.g. a sensitive question or unclear wording for social statistics or information not available in the accounting system for business statistics. The target value for this indicator is as close to 0 as possible.

27. Imputation rate* Definition :
Imputation is the process used to assign replacement values for missing, invalid or inconsistent data that have failed edits. This excludes follow-up with respondents and manual review and correction (if applicable). Thus, imputation as defined above occurs after data collection, no matter from which source or mix of sources the data have been obtained, including administrative data.
After imputation, the data file should normally only contain plausible and internally consistent data records.
This indicator is influenced both by the item non-response and the editing process. It measures both the relative amount of imputed values and the relative influence on the final estimates from the imputation procedures.
*Eurostat, 2010

28. Imputation rate ESS standard imputation rate*:
IY = the set of units for which variable Y is imputed
KY = the set of units for which the value of variable Y is kept
wj =weight of unit j
wj= 1: the un-weighted imputation rate
wj= dj ,dj can be = 1/πj or final-weight after adjustments for unit non response and coverage errors: the design-weighted imputation rate
wj= dj xj, xj is the value of an auxiliary variable X: the size-weighted imputation rate
The un-weighted imputation rate for a variable is the ratio of the number of imputed values to the total number of values requested for the variable.
The weighted rate shows the relative contribution to a statistic from imputed values; typically a total for a quantitative variable. For a qualitative variable, the relative contribution is based on the number of units with an imputed value for the qualitative item.
*Eurostat, 2010

29. Common units – proportion
Definition:
This indicator is applicable to mixed statistical processes where some variables come from survey data and others from administrative source(s).
The indicator is defined as the proportion of units covered by both the survey and the administrative sources in relation to the total number of units in the survey.
ESS standard Common units – proportion*:
Interpretation:
The indicator is used when administrative data is combined with survey data in such a way that data on unit level are obtained from both the survey and one or more administrative sources (some variables come from the survey and other variables from the administrative data). The indicator provides an idea of completeness/coverage of the sources – to what extent units exist in both administrative data and survey data.
*Eurostat, 2010

30. References
Eurostat (2010) ESS Guidelines for the implementation of the ESS quality and performance indicators
FCSM (2001) “Measuring and Reporting Sources of Error in Surveys”. Federal Committee on Statistical Methodology, Statistical Policy Working Paper 31.
Hidiroglou MA, Drew DJ, Gray GB (1993). “A Framework for Measuring and Reducing Nonresponse in Surveys”. Survey Methodology, 19, 1, pp
OECD Eurostat. OECD / Eurostat Guidelines on Revisions Policy and Analysis