1. Implementation of quality indicators in the Finnish statistics production process Kari Djerf Statistics Finland Q2008, Rome Italy

2. 2 Contents 1. Current situation 2. Challenges 3. Steps to proceed and time schedule

3. 3 1. Current situation Data exist - but: lots of data are collected for various reporting purposes which does not necessarily serve this purpose, and (too) much of data stored in inconsistent form 3

4. 4 Current situation – Strategic and/or performance indicators Collected centrally for all follow-up operations (now 2/4 times a year) Partly at the agency level, partly at department level Many of these indicators can be retrieved from a general planning and performance database

5. 5 Current situation – Strategic and/or performance indicators - 2 Examples of indicators: Public confidence (every 2 nd year) Delays in publications and releases Nonresponse rates of some key sample surveys Share of electronic data collection of all data collection Response burden →Not very suitable for continuous follow-up of quality of individual statistics!

6. 6 Quality indicators to be collected and compiled Product quality indicators Process indicators Very often these two are inseparable – a matter of opinion to which category an indicator actually belongs Goal: accumulate information, do not add unnecessary burden on subject-matter departments!

7. 7 Quality indicators to be collected and compiled - 2 Structure to follow the ESS quality dimensions: Relevance Accuracy Timeliness and punctuality Accessibility and clarity Comparability Coherence

8. 8 Quality indicators to be collected and compiled - 3 Obviously focus on accuracy and timeliness/punctuality but probably all dimensions will be covered Most ESS standard quality indicators are suitable as such, some may not be directly applicable

9. 9 Traditional indicators from sample surveys Unit response/nonresponse rates household surveys: long time-series business surveys: incomplete data Unit nonresponse rates divided into some key domains or classifications: Reason for nonresponse Demographics (gender, age, region, education, industry, size…)

10. 10 Traditional indicators from sample surveys - 2 Item response/nonresponse rates not sufficiently calculated in household and business surveys Evaluation of both types of nonresponse effects on survey results of key parameters: some results exist both on household and busness surveys

11. 11 Traditional indicators from sample surveys - 3 Reliability estimates (standard errors, CV’s or CI’s) have been reported from most household surveys lesser extent in business surveys (cut-off samples problematic) Survey specific indicators most probably to be included Editing and imputation indicators currently under development: indicators to be retrieved from the validation and editing process e.g. edit failure rates, imputation rates and their effect (esp. important in business statistics)

12. 12 Traditional indicators from sample surveys - 4 Response burden of household surveys measured since 1970s as interview time + occasional evaluation of self- completeted questionnaires or diaries Measurement of response burden of business and institutional surveys is currently under development Cost model to be developed

13. 13 Traditional indicators from censuses and administrative data Coverage rates – to be evaluated with respect to critical contents! Measurement errors, esp. correspondence between administrative and statistical concepts – important but normally they stay stable unless changes occur Editing (and imputation) rates 13

14. 14 2. Technical challenges Periodicity Requirements by various stakeholders Metadata standard(s) Various data sources 14

15. 15 Technical challenges - periodicity Example: Labour Force Survey In Finland a monthly survey since 1959: many indicators in comparable since 1984 Current EU-LFS regulations: quarterly with annual combination of data →Indicators must be calculated monthly, quarterly and annually – some can be aggregated, mostly not 15

16. 16 Technical challenges - stakeholders EU regulations, IMF, OECD etc. different in definitions and requirements EU regulations differ VERY much from each other on the extent of quality reporting and derivation of the indicators (EU-SILC, LFS, PEEIs etc.) → New Statistical law may improve the situation in general, but some very subject-dependent indicators might be left aside 16

17. 17 Technical challenges – different types of statistics Sample surveys Censuses and other total enumeration Administrative sources and registers Indices National accounts → Technical solutions must be flexible to allow different types of indicators 17

18. 18 Technical challenges - metadata SDMX standard to take over New ESMS is to include some indicators which may or may not be similar between different statistical domains →Technical allowance to retrieve directly as many of the required indicators as possible 18

19. 19 Technical challenges – existing data sources Obviously the biggest challenge! Subject-matter statistics do not compile and store data in a similar manner: many data warehouse systems were developed for one purpose only. New harmonised statistics production model will improve it gradually. Next proper database tools must be found to store data and facilitate easy reporting 19

20. 20 3. Steps to proceed - Cross-sectional data collection A self-assessment of all statistics in next autumn: Quality reports Available indicators Available metadata Obviously it will resemble the DESAP approach in contents Analysis of indicators to include important ones and exclude redundancies 20

21. 21 Cross-sectional data collection - 2 Find a ”good cocktail” of indicators and start retrieving them Database construction 2008/2009 Programs for reporting … and system to work in 2-3 years!

22. 22 THANK YOU VERY MUCH FOR YOUR ATTENTION !