Q2010, Helsinki Development and implementation of quality and performance indicators for frame creation and imputation Kornélia Mag László Kajdi Q2010, Helsinki

Introduction, background  Eurostat financed grant (planned finish: August 2010.)  The final report of the project will contain the following results regarding to frame creation and imputation:  Identification of the inputs, the result and the impacts of process steps and the flow charts.  Identification of the dimensions of process quality  Detailed description of the specified standard variables like required metadata, calculation and the possible variations of the variables, as well as recommendations for their application. The possible implementation of the indicators and the assessment of the measurement capability will be presented in the case of a concrete survey.  The development is based on the methods discussed in „Handbook on improving quality by analysis of process variables” (Eurostat [2003])”

Q2010, Helsinki International experiences Most often used and recommended indicators:  For frame selection: coverage errors (over- and undercoverage, misclassification and duplication)  For imputation: imputation rate and ratio  International experiences  Eurostat Standard Quality Indicators  ONS indicators  Hungarian experiences Process Quality Strategy Project:  Process variables for the production process steps

Q2010, Helsinki Frame creation The fact that we can not observe the whole target population causes different errors on survey estimations. Quality components  Relevance: any differences in the definition of the target population and the frame population or in the auxiliary information which are used in the survey can generate deficiencies on the relevance of the estimation.  Accuracy: any coverage problem that arises from not being able to sample from the whole of the target population can lead to the bias of the final estimation.  Timelines of the frames: the time lag between the reference date of the frame and the survey can cause coverage problems and consequently, it can have an effect on the final accuracy.

Q2010, Helsinki Frame creation – process flow Survey information:  Target population definition  Auxiliary information  Reference period Frame description:  Unit definition  Auxiliary information definition  Reference periods of the units/items Asessment procedures Differences of the unit definitions Differences of the auxiliary information definitions Time lag between the reference periods Deviation in the data Errors Missing units Non population elements Multiplicity elements Incorrect or missing auxiliary information

Q2010, Helsinki Process indicators  Overcoverage I.: Number and percentage of exisiting units in the frame that are not elements of the target population  Overcoverage II.: Number and percentage of non-existing units in the frame  Undercoverage.: Number and percentage of units which are not in the frame, but are part of the second frame  Misclassification: Number and percentage of units that are misclassified  Number of multiplied units  Number and percentage of the units excluded because of the time lag between reference dates of the survey and the frame Calculation: Number of units in the frame constructed at a later stage, but excluded from the original frame / (Number of units in the frame used for the survey) * 100 Metadata: Elements of the frame used for the survey; elements of the frame constructed at a later stage Comment: This can be calculated if we use the quality check approach, which means portions of the frame are independently constructed at a time close to the survey.  Number and percentage of units with missing auxiliary information (by type)

Q2010, Helsinki Imputation Imputation is the method which serves the handling of problems concerning the missing, false or inconsistent responses identified during data-processing, editing. Imputation takes place after the data collecting, data entry, editing, consistency checking, and outlier filtering phases. The quality of all these phases can affect all the forthcoming processing phases, including imputation. Quality components affected by imputation  Accuracy: at origin imputation is used to improve the accuracy and completness of the data, but obviously, as a part of the processing it can cause errors. For example: Imputed data usually exhibit less variability than true data.  Timeliness: If we introduce an imputation method then the data capturing phase needs to be shorter. This ha an impact on the quality of the data capture phase. For example, if we plan to use the imputation we do not necessarily need to re- contact data suppliers for the missing or wrong data restoration.

Q2010, Helsinki Imputation indicators  Imputation rate, item level  Imputation rate, unit level  Rate of correction with previously imputed data Calculation: number of cells corrected with previously imputed data / number of imputed cells. Metadata: Data used for imputation with flags, imputed dataset with flags, description of the imputation method. Comment: In case of using several data for the correction of a cell and one of them is corrected, than to imputation with previously imputed data must be considered.  Imputation ratio, item level  Change in standard error due to imputation in case of main indicators

Q2010, Helsinki The next steps of the project  Analysis and assessment of the measuring capability of the above identified indicators  Testing in the case of specific surveys  EU-SILC  Integrated business statistic survey

Q2010, Helsinki Thank you for your attention!