1. Using survey data collection as a tool for improving the survey process Silvia Biffignandi, Antonio Laureti Giulio Perani University of Bergamo Istat Istat EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

2. Outline  Introduction  Survey  Questionnaire  Preliminary results  Research in progress

3. Introduction: Paradata and web surveys Web surveys allow for the collection of paradata during web questionnaire completion; paradata are data generated during the fieldwork of the survey. Two types of paradata:  server-side paradata are collected by software tools running at the server. They relate mainly to the questionnaire compilation process. This data are contained in the so called logfiles.  client-side paradata describe how the respondents are answering the questions (order, questions skipped, keys that have been pressed and so on). (Biffignandi and Bethlehem, 2012; Biffignandi, 2010) Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

4. Analysis of paradata for:  questionnaire improvement (identification of the most difficult questions in the survey form, of some missing (or too restrictive) checks )  quantification of the burden on respondents (from the degree of ready availability of R&D data in enterprises, to the time needed to fill in the questionnaire )  timeliness? costs? identification of the characteristics of late respondents. Using survey data collection as a tool for improving the survey process Introduction: Tasks of paradata analysis EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

5. The survey  business R&D activities  yearly, census  target population: enterprises known, or assumed, to be R&D performers  many administrative sources to identify the target population (20000 “potential R&D performers” ) (previous Istat R&D surveys, other Istat business surveys with R&D-related questions, ASIA business register, fiscal data, Italian Register of R&D performing institutions,data on national and EU funding to research projects, patent databases, private business reports)  no a cut-off point in term of size, i.e. considered all enterprises (including micro- enterprises, provided that they are employing at least one researcher)  reference year 2008; overall response rate 55% Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

6. Future R&D performers (who are just asked to report about their future investment plans) Potential R&D performers Questionnaire Actual R&D performers (those who have to provide extensive information on the R&D activities undertaken in the reference year) Actual R&D performers (those who have to provide extensive information on the R&D activities undertaken in the reference year) Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss Non-R&D performers (no data requested)

7. R&D personnel Actual R&D performers Actual R&D performers Questionnaire R&D expenditures R&D expenditures Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss Qualitative questions on the R&D projects

8. Web survey administration Advanced web questionnaire design  questionnaire mainly based on previous paper questionnaire  innovative software architecture : two physically distinct components electronic form: delivered via Web on the remote PC of the respondent checking tool; resident in a Web-server in the Istat’s premises. This is a tool for the identification of the respondent and of the correct sequencing of the questionnaire’s provision. Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

9. Available paradata Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss Timing of use:  Entry and exit date.  Days in which the tool has been accessed.  Hours of use during the day. Intensity of use:  Number and typology of actions (events).  Processing time associated to each single action.  Number of actions per unit of time (e.g. per day or fraction of day).  Outcome of the compilation process. Compilation routes:  Sequencing in accessing the questions.  Changes to previously compiled questions. Generation of errors:  Number and typology of errors.  Questions (or groups of questions) mostly affected by errors.

10. Definitions “ events “ events”: each time a respondents is interacting with the electronic form (basically, by typing a figure or a word, or even, by scrolling down the form itself). We can identify each single “event” by its nature and by the time when it took place, as well as by its duration in time (usually, from a fraction to a few seconds). Events could be either “correct” (according to the rationale behind the structure of the questionnaire) or leading to the generation of “errors”. The questionnaire is “completed” only when all “errors” are properly fixed, the generation of “errors” will have as a result the need for new events aimed at correcting them. In the process of numbering the errors produced by each respondent, only the first appearance of an error type was taken into consideration, even in the event that the same error type would have been repeated in more than one session. Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

11. Definitions Each event – or a group of events – is associated to an “access”. “access”: itself to the questionnaire cannot be identified in a straightforward way as “log-ins” and “log-outs” are not recorded in the system. We know that a respondents is (was) connected to the server because of the activity carried out on the questionnaire. An access without any activity on the questionnaire will not be taken into consideration. An access has to be qualified in terms of time. Conventionally, all events taking place within one hour (or three, or six hours) could form an “access”. “daily accesses” Only “daily accesses” are taken into consideration in this study, i.e. an “access” will be equivalent to the set of all events having taken place in a day. Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

12. Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss Results. Results. Business R&D survey 2008 data collection. Paradata indicators: Number of respondents*, daily accesses and errors. Paradata indicators Actual R&D performers Future R&D performers Non R&D performers Respondents who have accessed the questionnaire without completing it Total N. of respondents* 2.601 268 1.086 324 4.279 Number of events 614.273 10.942 30.577 5.274 661.066 Average n. events per respondent 236,17 40,83 28,16 16,28 154,49 Number of daily accesses 4.471 371 1.359 391 6.592 Average n. accesses per respondent 1,72 1,38 1,25 1,21 1,54 Average number of events per access 137,39 29,49 22,50 13,49 100,28 Number of errors** 48.291 860 1.852 468 51.471 Average number of errors per respondent 18,57 3,21 1,71 1,44 12,03

13. Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss Results

14. Type of errors. Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

15. Daily accesses EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

16. Final delivery of questionnaire EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

17. Actual R&D performers: events accesses and errors Using survey data collection as a tool for improving the survey process EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

18. Research in progress EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss  deeper analyses on delivering time and on sectoral break-down  more detailed cross checking of the errors and mapping  data processing of the recent R&D survey edition  conclusions on questionnaire improvement  conclusions/suggestions for more efficient follow up strategies

19. EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss Suggestions? ….. Thank you for your attention! Thank you for your attention!