1. www.uis.unesco.org Innovation Surveys: Advice from the Oslo Manual South Asian Regional Workshop on Science, Technology and Innovation Statistics Kathmandu, Nepal 6-9 December 2010

2. www.uis.unesco.org Ch 8 OM - Innovation Survey Procedures Guidelines - collection and analysis of innovation data; Comparable results; Particular circumstances may require other methodology comparability!

3. www.uis.unesco.org Populations The target population: The target population: Business enterprise sector (goods-producing and services industries); At a minimum, all statistical units with at least ten employees.

4. www.uis.unesco.org Populations The frame population: The frame population: Units from which a survey sample or census is drawn = frame population; Basis: last year of the observation period; Ideal frame = up-to-date official business register established for statistical purposes NSOs; If the register forms the basis for several surveys (innovation, R&D, business), the information collected can be restricted to issues specific to innovation.

5. www.uis.unesco.org Survey methods Mandatory Mandatory surveys increase response rates; Census or sample survey? Census or sample survey? Sample surveys should be representative of the basic characteristics of the target population (industry, size, region) a stratified sample is necessary; Census, though costly, might be unavoidable in some cases (legal requirement, small frame population, inclusion of all units in the frame with a certain number of employees).

6. www.uis.unesco.org Survey methods Domains Domains (sub-populations) are subsets of the sampling strata; Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active; Guidelines: » Same statistical units and classifications; » Consistent methods; » Documentation of deviations in data treatment or differences in the quality of the results (from the domains).

7. www.uis.unesco.org Survey methods Sampling techniques: Sampling techniques: Stratified sample surveys (reliable results): based on the size and principal activity of the units; Sampling fractions should not be the same for all strata: the sampling fraction of a stratum should be higher for more heterogeneous strata and for smaller strata. Cross-sections: Cross-sections: standard approach - new random sample drawn from a given population; panel data Alternative/supplementary approach: panel data.

8. www.uis.unesco.org Survey methods Suitable respondents: Suitable respondents: Various methods: postal surveys, electronic surveys, personal interviews; Questions are very specialised and can be answered by only a few people in the unit; It is highly recommended to make a special effort to identify respondents by name before data collection starts. It is highly recommended to make a special effort to identify respondents by name before data collection starts.

9. www.uis.unesco.org Survey methods The questionnaire: The questionnaire: Pre-test before fieldwork; Simple and short; Order of the questions; Questions on qualitative indicators - binary or ordinal scale; International innovation surveys: translation and design of the questionnaire; Short-form questionnaires for units with little/no innovation activity previously reported.

10. www.uis.unesco.org Survey methods Combination of Innovation and R&D surveys: Combination of Innovation and R&D surveys: Reduction in the overall response burden; Scope for analysing the relations between R&D and innovation activities at the unit level; Efficient method of increasing the frequency of innovation surveys; It is possible to obtain reliable results for R&D expenditures; Longer questionnaire; Units not familiar with the concepts of R&D and innovation may confuse them; The frames for the two surveys are generally different.

11. www.uis.unesco.org Survey methods Guidelines for conducting combined surveys: Guidelines for conducting combined surveys: The questionnaire should have two distinct sections; Individual sections for R&D and innovation should be smaller than in separate surveys; Comparisons of results from combined surveys with those from separate innovation surveys should be done with care; Surveying methods should be reported; Samples to carry out such surveys should be extracted from a common business register.

12. www.uis.unesco.org Estimation of results Weighting methods: Weighting methods: The simplest one is weighting by the inverse of the sampling fractions of the sampling units, corrected by the unit non-response; Stratified sampling technique with different sampling fractions weights should be calculated individually; Commonly based on the number of enterprises in a stratum; In international and other comparisons, be sure that the same weighting method is used. In international and other comparisons, be sure that the same weighting method is used.

13. www.uis.unesco.org Estimation of results Non-response: Non-response: Unit non-response: a reporting unit does not reply at all; Item non-response: response rate to a specific question / % of blank or missing answers among the reporting units; biased results » Disregarding missing values and applying simple weighting procedures based only on the responses received implicitly assumes that non-respondents are distributed in the same way as respondents biased results; imputation methods » Possibility: imputation methods to estimate missing values on the basis of additional information.

14. www.uis.unesco.org Presentation of results Descriptive analysis: Descriptive analysis: Description of the statistical units in terms of their innovative or non- innovative activities without drawing any conclusions about the underlying survey or target population; No generalisation of the results; Unit non-response rate is of minor importance. Inferential analysis: Inferential analysis: Drawing of conclusions about the target population; The results should give a representative estimation of the situation; Weighted results; Unit non-response rate is very important.

15. www.uis.unesco.org Presentation of results Errors: Errors: Random errors due to the random process used to select the units; Systematic errors containing all non-random errors (bias); » Results variance: it is recommended to calculate both (average) values for innovation indicators and also their coefficients of variation and/or confidence intervals; » Results presentation: metadata (including information on data collection procedure), sampling methods, procedures for dealing with non-response and quality indicators.

16. www.uis.unesco.org Frequency of data collection Innovation surveys: every two years; When not economically feasible three or four years; Surveys must always specify an observation period for questions on innovation; The length of the observation period for innovation surveys should not exceed three years nor be less than one year.

17. www.uis.unesco.org Annex A - 5. Methodological issues for developing country contexts Information system specificities: Relative weakness of statistical systems Absence of linkages between surveys and data sets; lack of official business registers information from other surveys cannot be used; Involvement of NSOs; When lacking, basic variables about firms performance can be included in the innovation survey - to enable further analysis.

18. www.uis.unesco.org Annex A - 5. Methodological issues for developing country contexts General methodological considerations: Questionnaire design: » Separated sections - different respondents; » Guidance / definitions; » Language and the translation of technical terms; Survey application: » In-person; » Trained personnel.

19. www.uis.unesco.org Annex A - 5. Methodological issues for developing country contexts General methodological considerations: Frequency: CIS CIS » Every three to four years (e.g., timed to CIS rounds);CIS » Try to update a minimum set of variables every year; The purpose of surveys needs to be clearly stated and the questions clearly formulated; An adequate legislative base for the collection of innovation statistics can help ensure the success of such an exercise; The results should be published and distributed widely. The results should be published and distributed widely.

20. www.uis.unesco.org Thank you! http://www.uis.unesco.org l.marins@uis.unesco.org (CIS: http://www.oecd.org/dataoecd/37/39/37489901.pdf)