© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt DESAP - A New Self Assessment Checklist for Survey Managers Manfred Ehling Thomas Körner Anja Nimmergut

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt DESAP is the first generic checklist for a simple self assessment programme for survey managers in the European Statistical System (ESS ) goes back to recommendation no. 15 of the Leadership Group (LEG) on Quality: “A generic checklist should be developed for a simple self assessment programme for survey managers in the ESS” applies to individual statistics collecting micro data is fully compliant with the ESS quality criteria and comprises the main aspects relevant to the quality of statistical data

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Main components of quality („quality vector”) in the ESS: Relevance Accuracy Timeliness and Punctuality Accessibility and Clarity Comparability Coherence

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Simple quality checklists: focus on the processes and products; working with these increases the awareness of quality issues and reveals areas in need of improvement (specific) Self assessments (DESAP) amongst other assessment tools: EFQM model: tool of self- assessment for the whole organisation (with help from experienced quality award examiners) Internal or external audits: assessment from outside Documentation/ Quality reports: are more detailed then simple checklist and hence it is more effort to produce them Complement any measure or first step before

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Objectives of DESAP: Raising awareness for the quality components and survey quality concepts to provide a tool for a systematic, even though subjective, assessment of statistical products and processes to provide helpful guidance in the consideration of improvement measures Additional potential applications: assistance for a basic appraisal of the risk of potential quality problems to provide a means for simple comparisons of the level of quality over time and across domains to provide support for resource allocation within statistical offices or for the training of new staff

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt (4) Data Capture and Data Processing (7) Improving statistics (5) Data Analysis and Output quality (6) Documenta -tion and Disse- mination Structure of DESAP: process-orientation (3) Data collection (2) Survey Design (1) Decision to undertake a survey

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt (III) Data collection (IV) Data Capture and Data Processing (V) Data Analysis and Output quality (VI) Documenta -tion and Disse- mination (II) Survey Design (I) Decision to undertake a survey (VII) Improving statistics 24 Process elements of the checklist (1) Analysis of user needs Survey objectives (2) Survey concepts Sample design Development and testing of the measurement instruments (3) Data sources Interviewer deployment and training Reducing non-response Fieldwork (4) Data capture Editing procedures Imputation procedures (5) Relevance Accuracy Timeliness and punctuality Comparability Coherence Data analysis Disclosure control (6) Metadata documentation Dissemination strategies Data management (7) Adaptability / flexibility Expertise in relevant areas Quality management

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Typology of DESAP questions In order to fulfil the various function of the checklist different types of questions have been developed and integrated in the checklist: questions with numerous response categories (“improvement questions”) assessment questions open questions

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Questions with numerous response categories sum up to an inventory for (data) quality assurance measures in NSIs make the process of completing the checklist less time consuming because the survey managers just have to tick the answer(s) that apply to their survey regarding the different alternatives might stimulate some ideas of possible improvements for the survey production process separate page where the ideas for possible improvements can be noted down

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Example question with numerous response categories :

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Assessment questions: questions are worded in a harmonised way, using the same terms for the same quality levels questions are highlighted using a special colour and design (deep blue background and dark blue bar at the side) in order to facilitate the evaluation five point scale of the answers score of the respective answers can be drawn in a cobweb diagram which finally gives a graphical feed- back of the quality profile for the respective survey

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Example for an assessment question:

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt The DESAP assessment diagram: Section II/6 overcoverage Section II/7 undercoverage Section II/8 misclassification Section IV/4 necessity of editing Section V/6 CVs for key variables Section V/15 unit non-response Section V/21 time lag reference period/ publication Section V/25 comparability over time Section V/18 item non- response Section V/27 coherence for results of different frequencies Section V/24 comparability across non-geographical domains Section V/28 Coherence within the same socio- economic area Section V/2 user satisfaction Section V/3 overall relevance Section V/22 Publication dates kept? Accuracy Relevance Comparabilit y Timeliness & Punctuality Coherence Accuracy The larger this area - the better the quality profile

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Open questions: sometimes not possible to give response categories aim to confront the survey manager with pertaining questions to develop improvement ideas

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Example for an open question:

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Items which could be solved in co-operation: Experiences faced during the implementation of self assessment programmes (joint workshop?) Methods of evaluation esp. of the improvement questions should be further developed (OECD or IMF experiences?) Are there similar problems encountered in the framework of the OECD self assessments? How could a comprehensive assessment (judgement/ grading) be achieved that gives a better feed-back to the survey managers? Could this problem be solved by refining/ tailoring checklists for statistic areas following recommendations for best practice?

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Recommendations of the DESAP group for further steps concerning the checklist: recommendation of the implementation of the checklist among the members of the Statistical Programme Committee (SPC) checklist in all languages of the EU broad publication of the project results IT implementation of the checklist (electronic version could facilitate the checklists use) workshop concerning improvements of the checklist after one or two years of experiences with its use

© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Thanks for your attention! Further questions?