1. Statistics Estonia on its way to improving efficiency UN ECE Seminar on New Frontiers for Statistical Data Collection Geneva, 31.10 ‒ 02.11.2012 Tuulikki Sillajõe

2. Outline Organization of the data collection Production system as a whole Conclusions and future plans 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

3. Data Entry Service Central Data Collection Department since 2004 Seminar on New Frontiers for Statistical Data Collection 02.11.2012 Data Collection Development Manager Data Collection Service Fieldwork Organisation Service Data Collection Department Manager

4. Collection of administrative data (I) Data Collection Department is not responsible for collection of administrative data Data Processing Systems Department is responsible for a single entry point for administrative data runs pre-agreed data processing makes the data available for statistical domains 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

5. Collection of administrative data (II) Methodology Department consolidates the needs of statistical domains conducts negotiations with the holders of administrative registers organises the conclusion of agreements with these holders is in charge of the description of data in a central metadata system Statistics Estonia used about 100 different administrative registers (2012) 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

6. Functionalities of ADAM (data collecting system for administrative data) Automatic extraction of detailed personalized data from administrative sources using X-road (data exchange layer) ftp, etc. Storing data in raw data databases Data processing coding duplicate removal Making data available for in-house applications 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

7. Milestones of Data Collection in Statistics Estonia, 2000 ‒ 2012 2000200120022003200420052006200720082009201020112012 HOUSEHOLDSHOUSEHOLDS ENTERPRISESENTERPRISES ABOLISHM ENT OF REGIONAL BUREAUS CENTRAL DEPARTMENT CAPI WEB-BASED COLLECTION (eSTAT) CALL CENTER FOR ENTERPRISES FAILURE WITH 1st CAWI 1st MIXED MODE CAPI+ CATI TERMINATI ON OF SENDING Q.-S PILOT PHC 2011 CAWI+CAPI AGRI- CENSUS CAWI+CAPI PRE-FILLING WITH ADMIN- DATA PHC 2011 PAPER- LESS OTHER CAWI PILOTS 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

8. Functionality of eSTAT for external users (I) to view the list of statistical questionnaires, which a particular economic entity has to present to Statistics Estonia during the current year to view deadlines for presenting these statistical questionnaires to order reminders, which notify by e-mail about upcoming deadlines to compile statistical questionnaires, i.e. to fulfil cells on the web with data or download and upload CSV-files to run controls, i.e. check whether they have compiled the statistical questionnaires required 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

9. Functionality of eSTAT for external users (II) to correct statistical questionnaires immediately upon compilation thereof to submit statistical questionnaires to look at all earlier statistical questionnaires submitted to Statistics Estonia via eSTAT by a respondent concerned to print out a paper copy of a compiled statistical questionnaire to administer users, i.e. to create, change and cancel rights and access to accept or correct one’s contact information 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

10. Functionality of eSTAT for internal users (I) to define statistical questionnaires in the system, i.e. to describe and add them to define controls for statistical questionnaires described in the system to follow the inflow of statistical questionnaires and send reminders to register contacts with respondents to see the time and the content of contacts with respondents to see the same information as external user and help them online 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

11. Functionality of eSTAT for internal users (II) to see and correct contact information of economic entity who can deliver questionnaires via the system to compile statistical questionnaires (e.g. when receiving them by phone) to view and correct statistical questionnaires compiled by respondents to administer external main users and internal users to create empty statistical questionnaires in pdf- format for printing out or saving them as a file for different administrative purposes 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

12. Questionnaires received from economic entities by channel, 2008–2012 Seminar on New Frontiers for Statistical Data Collection 02.11.2012

13. Pre-filling of questionnaires (I) Annual statistical questionnaires for the year 2012, i.e. for the reference year 2011 data collection, are prefilled using administrative data  structural business statistics (EKOMAR)  agriculture, forestry and fishing  financial intermediation and activities auxiliary to financial services and insurance activities  non-profit institutions Data providers have to fill in only the gaps, i.e. information not available from annual bookkeeping report 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

14. Pre-filling of questionnaires (II) The information from annual reports is preloaded to eSTAT every hour First results 52% of the questionnaires for structural business statistics had been pre-filled from annual reports 80% of the fields were pre-filled, and respondents had to fill in the remaining 20% (only fields filled in with a number other than zero are taken into account) Average compiling time of a statistical questionnaire has been reduced twice (from 3 hours to 1,5 hour) compared to previous year 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

15. Mixed mode was used for PHC 2011 Modes of questioning e-Census on the web (CAWI) → 66% Interview (CAPI) Institutions filled in a special questionnaire Different data sources Population and Housing Census 2000 Administrative registers Data collection from persons New software was developed next to eSTAT 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

16. New software for data collection (VVIS) Supports data collection process for various surveys and census, dividing it into three sub-processes Preparation work for data collection Data collection on the web (CAWI) and fieldwork (CAPI) Support and management of the whole process 02.11.2012

17. Seminar on New Frontiers for Statistical Data Collection Applications of VVIS Questionnaire definition application Interviewers’ (enumerators’) application Public web application Management application Interfaces with external systems 02.11.2012

18. Seminar on New Frontiers for Statistical Data Collection Questionnaire definition application A tool for preparing questionnaires Built on top of Eclipse Functionality Every survey can use a different, custom data model Several questionnaires can be built on top of data model (e.g. different questionnaires for CAWI and CAPI) Advanced navigation and validation rules within questionnaires Use of common classifications in different surveys (e.g. occupations: ISCO, education: ISCED, etc.) Data model and questionnaires saved in open XML format Multi-language support for questionnaires 02.11.2012

19. Seminar on New Frontiers for Statistical Data Collection Interviewers’ application Stand-alone desktop application can be used in laptop or desktop computer (Delphi) Synchronises all data for offline work over encrypted channel (HTTPS) Functionality Advanced validation and navigation rules within questionnaire Navigation between different questionnaires Questionnaires in multiple languages Planning / scheduling of work Reminders Communication with direct supervisor Map info for location of subjects, GPS positioning, work planning Overview of general fieldwork progress for interviewers Help information for interviewers Automatic software updates (distributed from central server) 02.11.2012

20. Seminar on New Frontiers for Statistical Data Collection Public web application Used by survey subjects independently Public system, accessible over the web (Java, Weblogic) Functionality: Authentication with ID card or through bank link Choice between questionnaires assigned to the subject Support for all rules used in questionnaires, the same questionnaires can be used that are defined for interviewers’ application Background information and help texts about the surveys and questionnaires 02.11.2012

21. Seminar on New Frontiers for Statistical Data Collection Management application Used by fieldwork management, statisticians, help desk Internal web application (Java, Weblogic), accessible from within internal network Functionality: Authentication using LDAP or any other authentication method Creation of survey object, configuration of methodology, fieldwork hierarchy and other characteristics of survey Role management Possibility to work simultaneously with several surveys Overview of fieldwork progress, task management Definition of milestones (deadlines) for fieldwork organisation Help desk functionality Data processing tools (e.g. for classification of data) Communication (messages from one user to another) 02.11.2012

22. Seminar on New Frontiers for Statistical Data Collection Interfaces with external systems Information database Meta-info about surveys Background survey information (displayed on the web) Classifications used in questionnaires Authentication system (e.g. Active Directory) Statistical register Import of sample Pre-filling of questionnaires Export of sample (changes in subjects’ information) GIS database Maps Location info of buildings Hierarchy of district division 02.11.2012

23. Generic Statistical Business Process Model Seminar on New Frontiers for Statistical Data Collection 02.11.2012

24. Architecture of the information system Seminar on New Frontiers for Statistical Data Collection 02.11.2012 Dissemination Statistical registers Metadata system Data collection Processing Statistical analysis Persons Administrative registers Users Economic entities Data Warehouse iMETA VVIS ADAM eGeostat SRS VAIS eSTAT PX-Web Census-HUB KUNDE Analyse

25. Seminar on New Frontiers for Statistical Data Collection 2002 Statistical Farm Register 1993 1994 Statistical Business Register 1994 economic entities 2004 persons 2004 2004 economic entities 2006 economic entities 2001 metadata management 2011 iMeta project started 2011 system for statistical registers project started 2011 project started 2011 persons planned Generic Statistical Business Process Model project started 2012 02.11.2012

26. Sources for efficiency gains of Statistics Estonia Central data collection department, i.e. standardisation of processes Generic office-wide software Administrative data instead of survey data if applicable Pre-filling of statistical questionnaires with administrative data 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

27. Small developments, big efforts Reminders sent before the deadline instead of after the deadline Informing economic entities simultaneously about all the questionnaires they have to fill in next year Centralisation of the preparation of questionnaires from statistical departments to IT Department and within a few years to Methodology Department Standardisation and simplification of instructions about questionnaires for economic entities Creation of a list of input variables Seminar on New Frontiers for Statistical Data Collection 02.11.2012

28. Next practical steps Introduction of CAWI as the main data collection method for surveys on individuals (2013) Introduction of CATI for data collection from both types of respondents: economic entities and individuals (step by step, starting from 2012) Implementation of generic office-wide software for other functions than data collection Reuse of data within the statistical office 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

29. Strategic directions Training of data suppliers (economic entities, individuals, registers, etc.), incl. about their personal and public gains Closer cooperation between the data collection function and dissemination function within the organisation, for better communication with data suppliers (based on the experience of PHC 2011) Simplification of statistical reports (harmonisation of concepts, deadlines, practices, etc.) Development of infrastructure for selling data collection services Seminar on New Frontiers for Statistical Data Collection 02.11.2012

30. Further challenges Wider use of administrative and commercial data Do we need two data collection tools? Centralization of data processing function? 02.11.2012 Seminar on New Frontiers for Statistical Data Collection

31. Thank you for your attention! tuulikki.sillajoe@stat.ee 02.11.2012 Seminar on New Frontiers for Statistical Data Collection