Institut de la statistique du Québec (ISQ) surveys: compendium of good practices and quality self-assessment checklist European Conference on Quality in Official Statistics Rome, July 8-11, 2008 Louise Bourque France Lapointe Institut de la statistique du Québec

2 Presentation outline Context Compendium of good survey practices –Development –Content Survey quality self-assessment checklist –Objectives –Development –A few observations by users (pre-test) Challenges in applying the tools Conclusion

3 Context The Institut de la statistique du Québec (ISQ) –Its enabling legislation  “... to provide reliable and objective statistical information”  “... in such a manner that comparisons in and outside Québec may be made” –Organization of work  A number of units work together in conducting a survey

4 Context (cont.) Specific work dealing with quality incorporated into everyday activities –Variation according to the project and unit (neither systematic nor harmonized) The ISQ has adopted an Integrated Quality Management Framework –General Quality Management Policy (November 2005)

5 Context (cont.) General definition of quality Quality is composed of all the characteristics that affect the capacity to meet the needs of users Six dimensions of quality –Relevance –Reliability and objectivity –Comparability –Timeliness –Intelligibility –Accessibility

6 Context (cont.) The policy refers to many documents (policies, procedures, programs, reports, strategic declarations, etc.) More specifically, survey quality assurance tools –Compendium of good practices –Self-assessment checklist

7 Compendium of good survey practices Objective –Promote the use of good practices in conducting surveys, at all stages, in order to ensure the quality of the results Development –Multidisciplinary team ▪Bringing together all the entities involved in the carrying out of surveys ▪Mandated by top manager ▪Work directed and monitored by a senior management committee

8 Compendium of good survey practices (cont.) –Inventory of reference practices  Statistics Canada, Statistics Finland...  Division of the survey process into 16 stages –Observation of ISQ practices  10 or so surveys selected as representative of diversity  Application of a common observation checklist oAdaptation of the European DESAP checklist –Comparison  ISQ practices vs. reference practices  Choice of selected practices

9 Compendium of good survey practices (cont.) Content of a section related to a stage –Introduction  Overview of the work to be done and links with quality dimensions –Definitions (non-specialized reader) –Principles (reader knows how a survey is conducted)  General quality assurance policy directions –Good practices (specialized reader) with ratings (1) Essential practices (2) Highly recommended practices (3) Recommended practices

10 Self-assessment checklist Objectives –Survey self-assessment and quality measurement according to the six quality dimensions selected –Checklist at the time of design, and throughout the conducting of a survey –Verification of the suitable application of the compendium’s good practices –Training for new employees –Etc.

11 Self-assessment checklist (cont.) Development –Multidisciplinary team –In three stages  Adapt the DESAP checklist for observation of practices at the ISQ (done)  Review adaptation following use for observation (pre-test) with the comments received (in progress)  Ensure consistency with the compendium's good practices (in future)

12 Self-assessment checklist (cont.) A few observations further to the pre-test –Logical structure of the questionnaire –Recognized usefulness of the checklist –High response burden  Long questionnaire  Complex questions, difficult to interpret –Questionnaire format to be revised

13 Tool application challenges The tools must... –Be in keeping with the ISQ’s reality –Gain the approval of the various entities The tools must be developed... –At the least cost –While ensuring that everyday activities continue

14 Tool application challenges (cont.) Tool introduction must... –Ensure the tools’ appropriation and use by employees Tool use must not... –Be too constraining –Require too many resources

15 Tool application challenges (cont.) General approach selected –Not to reinvent the wheel –Actively involve the units concerned –Be guided and supported by senior management –Inform and train the staff –Incorporate means of monitoring application

16 Conclusion Current and future work... –Compendium of good practices  Information and training activities for employees (in progress) –Finalize the self-assessment checklist (in progress) –Develop and adopt a survey quality policy (in future)  Monitor the use of the compendium and checklist

17 Thank you! Louise Bourque France Lapointe