1. www.uis.unesco.org Innovation Surveys: Advice from the Oslo Manual National training workshop Amman, Jordan 18-20 October 2010

2. www.uis.unesco.org Ch 8 OM - Innovation Survey Procedures Guidelines on central elements for the collection and analysis of innovation data; Following these guidelines will generally lead to comparable results over time and across countries; Particular circumstances may require other methodology comparability should be in mind.

3. www.uis.unesco.org Populations The target population: The target population: Innovation surveys should refer to innovation activities in the business enterprise sector (goods-producing and services industries); It should include, at a minimum, all statistical units with at least ten employees.

4. www.uis.unesco.org Populations The frame population: The frame population: The units from which a survey sample or census is drawn form the frame population; It is based on the last year of the observation period for surveys; Ideal frame = up-to-date official business register established for statistical purposes NSOs; If the register forms the basis for several surveys (innovation, the R&D, general business), the information collected in the innovation survey can be restricted to issues specific to innovation.

5. www.uis.unesco.org Survey methods Mandatory Mandatory surveys increase response rates; Census or sample survey? Census or sample survey? Sample surveys should be representative of the basic characteristics of the target population (industry, size, region) a stratified sample is necessary; Census, though costly, might be unavoidable in some cases (legal requirement, small frame population, inclusion of all units in the frame with a certain number of employees).

6. www.uis.unesco.org Survey methods Domains Domains (sub-populations) are subsets of the sampling strata; Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active; Guidelines: » Statistical units and classifications should be the same; » Methods used to calculate results for subsets should be consistent with those used for results from the main sample. » Deviations in data treatment or differences in the quality of the results from the domains should be documented.

7. www.uis.unesco.org Survey methods Sampling techniques: Sampling techniques: Stratified sample surveys (reliable results): based on the size and principal activity of the units; Sampling fractions should not be the same for all strata: the sampling fraction of a stratum should be higher for more heterogeneous strata and for smaller strata. Cross-sections: Cross-sections: standard approach - a new random sample is drawn from a given population for each innovation survey; panel data Alternative/supplementary approach: panel data.

8. www.uis.unesco.org Survey methods Suitable respondents: Suitable respondents: A variety of methods can be used to conduct innovation surveys, including postal surveys and personal interviews; Choosing the units most suitable respondent is particularly important in innovation surveys (questions are very specialised and can be answered by only a few people in the unit); It is highly recommended to make a special effort to identify respondents by name before data collection starts. It is highly recommended to make a special effort to identify respondents by name before data collection starts.

9. www.uis.unesco.org Survey methods The questionnaire: The questionnaire: Pre-test each questionnaire before fieldwork; Keep it as simple and short as possible; Pay attention to the order of the questions; Questions on a number of qualitative indicators can use either a binary or an ordinal scale; In the case of international innovation surveys, special attention should be given to the translation and design of the questionnaire; Short-form questionnaires for units with little/no innovation activity previously reported.

10. www.uis.unesco.org Survey methods Combination of Innovation and R&D surveys: Combination of Innovation and R&D surveys: Reduction in the overall response burden of the reporting units; Scope for analysing the relations between R&D and innovation activities at the unit level; Efficient method of increasing the frequency of innovation surveys; Country experiences indicate that it is possible to obtain reliable results for R&D expenditures; Longer questionnaire; Units not familiar with the concepts of R&D and innovation may confuse them; The frames for the two surveys are generally different.

11. www.uis.unesco.org Survey methods Guidelines for conducting combined surveys: Guidelines for conducting combined surveys: The questionnaire should have two distinct sections; Individual sections for R&D and innovation should be smaller than in separate surveys; Comparisons of results from combined surveys with those from separate innovation surveys should be done with care, and surveying methods should be reported; Samples to carry out such surveys should be extracted from a common business register.

12. www.uis.unesco.org Estimation of results Weighting methods: Weighting methods: The simplest one is weighting by the inverse of the sampling fractions of the sampling units, corrected by the unit non-response; If a stratified sampling technique with different sampling fractions is used, weights should be calculated individually for each; Commonly based on the number of enterprises in a stratum. In international and other comparisons, be sure that the same weighting method is used.

13. www.uis.unesco.org Estimation of results Non-response: Non-response: Unit non-response: a reporting unit does not reply at all; Item non-response: refers to the response rate to a specific question and is equal to the percentage of blank or missing answers among the reporting units; biased results » Disregarding missing values and applying simple weighting procedures based only on the responses received implicitly assumes that non-respondents are distributed in the same way as respondents biased results; imputation methods » Possibility: imputation methods to estimate missing values on the basis of additional information.

14. www.uis.unesco.org Presentation of results Descriptive analysis: Descriptive analysis: Description of the statistical units in terms of their innovative or non- innovative activities without drawing any conclusions about the underlying survey or target population; No generalisation of the results; Unit non-response rate is of minor importance. Inferential analysis: Inferential analysis: Drawing of conclusions about the target population; The results should give a representative estimation of the situation; Weighted results; Unit non-response rate is very important.

15. www.uis.unesco.org Presentation of results Errors: Errors: Random errors due to the random process used to select the units; Systematic errors containing all non-random errors (bias); » To get at least an idea of the variance for the results, it is recommended to calculate both (average) values for innovation indicators and also their coefficients of variation and/or confidence intervals; » Results presentation should contain: metadata (including information on data collection procedure), sampling methods, procedures for dealing with non-response and quality indicators.

16. www.uis.unesco.org Frequency of data collection Innovation should be conducted every two years; Where this is not economically feasible, a frequency of three or four years may be chosen; Surveys must specify an observation period for questions on innovation; The length of the observation period for innovation surveys should not exceed three years nor be less than one year.

17. www.uis.unesco.org Annex A - 5. Methodological issues for developing country contexts Information system specificities: Relative weakness of statistical systems Absence of linkages between surveys and data sets; lack of official business registers information from other surveys cannot be used; Involvement of NSOs; When lacking, basic variables about firms performance can be included in the innovation survey - to enable further analysis.

18. www.uis.unesco.org Annex A - 5. Methodological issues for developing country contexts General methodological considerations: Questionnaire design: » Sections can be separated to allow different persons in the firm to reply them; » Guidance / definitions; » Language and the translation of technical terms; Survey application: » In-person; » Trained personnel.

19. www.uis.unesco.org Annex A - 5. Methodological issues for developing country contexts General methodological considerations: Frequency: CIS CIS » Every three to four years (e.g., timed to CIS rounds);CIS » Try to update a minimum set of variables every year; The purpose of surveys needs to be clearly stated and the questions clearly formulated; An adequate legislative base for the collection of innovation statistics can help ensure the success of such an exercise; The results should be published and distributed widely. The results should be published and distributed widely.

20. www.uis.unesco.org Thank you! http://www.uis.unesco.org m.schaaper@uis.unesco.org CIS: http://www.oecd.org/dataoecd/37/39/37489901.pdf