Measuring innovation CARIBBEAN REGIONAL WORKSHOP ON SCIENCE, TECHNOLOGY AND INNOVATION (STI) INDICATORS St Georges, Grenada 1-3 February 2011

Measuring Innovation Oslo Manual: Guidelines for collecting and interpreting innovation data UIS - Annex (OM, 2005): Innovation Surveys in Developing Countries

What is innovation? Innovation and economic development; Innovation is more than R&D; Innovation is the implementation of: (Technological innovation) New or significantly improved product (good/service) or process; (Non-tech. innovation) New marketing or organisational method. Types of innovation: Product; Process; Marketing; Organisational.

Why measure innovation? Innovation policy should be evidence-based; Innovation data... to better understand innovation and its relation to economic growth; to provide indicators for benchmarking national performance.

The innovation measurement framework

Diffusion and degree of novelty Diffusion… How innovations spread; Economic impact; New to the Firm / Market / World; Disruptive innovations; Significant impact on a market; Impact of innovations (as opposed to their novelty); May become apparent only long after introduction.

Innovation activities Innovation activities: All scientific, technological, organisational, financial and commercial steps which (intended to) lead to the implementation of innovations; Some innovation activities are themselves innovative, others are not novel activities but are necessary; R&D that is not directly related to the development of a specific innovation.

Innovation activities For product and process innovations: Intramural (in-house) R&D; Acquisition of R&D (extramural R&D); Acquisition of other external knowledge; Acquisition of machinery, equipment and other capital goods; Other preparations for product and process innovations; Market preparations for product innovations; Training. Preparations for marketing innovations: Activities related to the development and implementation of new marketing methods. Preparations for organisational innovations: Activities undertaken for the planning and implementation of new organisation methods. * Expenditures

Kinds of innovation activities Successful - in having resulted in the implementation of a new innovation (though not necessarily commercially successful); Ongoing - work in progress, which has not yet resulted in the implementation of an innovation; Abandoned - before the implementation of an innovation.

Classifying firms by degree of innovativeness Innovative firm: The innovations need not have been a commercial success; Innovation-active firm: Regardless of whether the activity resulted in the implementation of an innovation; Potentially innovative firm: Innovation efforts but no achieved results (period); Key element for innovation policy; (Annex).

Factors influencing innovation Objectives: Motives for innovating; Effects: Outcomes of innovations; Competition, demand and markets; Production and delivery; Workplace organisation; Other (Table 9) ; Hampering factors: Reasons for not starting innovation activities at all, or factors that slow innovation activity or have a negative effect on expected results; Cost, Knowledge, Market and Institutional factors + Reasons not to innovate (Table 10).

Linkages Linkages connections with other agents; Source, cost, level of interaction; Types of external linkages: Open information sources; Acquisition of knowledge and technology; Innovation co-operation.

Sources for transfers of knowledge and technology Open information sources Sources for purchases of knowledge & technology Co- operation partners Internal sources within the enterprise: R&D / Production / Marketing / Distribution Other enterprises within the enterprise group ****** External market and commercial sources: Competitors Other enterprises in the industry Clients or customers Consultants / consultancy firms Suppliers Commercial laboratories ********** ********** ************ Public sector sources: Universities and other higher education institutions Government / public research institutes Private non profit research institutes Specialised public innovation support services ******** ******** ******** General information sources: Patent disclosures / Professional conferences, meetings, literature and journals / Fairs and exhibitions / Professional associations, trade unions / Other local associations / Informal contacts or networks / Standards or standardisation agencies / Public regulations *

Data collection The subject approach Innovative behaviour and activities of the firm as a whole Should R&D and innovations surveys be combined?

Example - pdt innov/degree of novelty

Example - innovation activities and expenditures for pdt and pcs innov

Example - organisational innovation

Example - co-operation

Example - hampering factors

Developing countries Developing countries 3 rd OM standards, but adaptations; LA: the Bogotá Manual (RICYT, 2001); UIS: Annex to 3 rd OM… Innovation Surveys in Developing Countries.

Characteristics of innovation in developing countries Size and structure of markets and firms; Instability; Informality; Particular economic and innovation environments; Reduced innovation decision-making powers; Weak innovation systems; Elements of innovation.

Innovation measurement in developing countries Incorporation of the concept of potentially innovative firm; Measurement priorities - why / what / how: Innovation capabilities (HR, Linkages, Quality assurance systems, ICTs); Expenditure on innovation activities; Organisational innovation.

Adaptations ICTs in innovation surveys Strategic use of new technologies (Front office vs. Back office); Linkages Agents + Types + Location; Innovation Activities Hardware purchase and Software purchase (split); Industrial design and Engineering activities (split); Lease or rental of machinery, equipment and other capital goods; In-house software system development; Reverse engineering; Human resources and training Quality and environmental management

Methodological issues for developing country contexts Weakness of statistical systems; Questionnaire design; Survey application; Frequency; Publication; Difficulties… Lack of appreciation of the importance of innovation; Managers are secretive about finance; Lack of adequate legislative base. Will be discussed later

Innovation Surveys: Advice from the Oslo Manual

Ch 8 OM - Innovation Survey Procedures Guidelines on central elements for the collection and analysis of innovation data; Following these guidelines will generally lead to comparable results over time and across countries; Particular circumstances may require other methodology comparability should be in mind.

Populations The target population: The target population: Innovation surveys should refer to innovation activities in the business enterprise sector (goods-producing and services industries); It should include, at a minimum, all statistical units with at least ten employees.

Populations The frame population: The frame population: The units from which a survey sample or census is drawn form the frame population; It is based on the last year of the observation period for surveys; Ideal frame = up-to-date official business register established for statistical purposes NSOs; If the register forms the basis for several surveys (innovation, the R&D, general business), the information collected in the innovation survey can be restricted to issues specific to innovation.

Survey methods Mandatory Mandatory surveys increase response rates; Census or sample survey? Census or sample survey? Sample surveys should be representative of the basic characteristics of the target population (industry, size, region) a stratified sample is necessary; Census, though costly, might be unavoidable in some cases (legal requirement, small frame population, inclusion of all units in the frame with a certain number of employees).

Survey methods Domains Domains (sub-populations) are subsets of the sampling strata; Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active; Guidelines: » Statistical units and classifications should be the same; » Methods used to calculate results for subsets should be consistent with those used for results from the main sample. » Deviations in data treatment or differences in the quality of the results from the domains should be documented.

Survey methods Sampling techniques: Sampling techniques: Stratified sample surveys (reliable results): based on the size and principal activity of the units; Sampling fractions should not be the same for all strata: the sampling fraction of a stratum should be higher for more heterogeneous strata and for smaller strata. Cross-sections: Cross-sections: standard approach - a new random sample is drawn from a given population for each innovation survey; panel data Alternative/supplementary approach: panel data.

Survey methods Suitable respondents: Suitable respondents: A variety of methods can be used to conduct innovation surveys, including postal surveys and personal interviews; Choosing the units most suitable respondent is particularly important in innovation surveys (questions are very specialised and can be answered by only a few people in the unit); It is highly recommended to make a special effort to identify respondents by name before data collection starts. It is highly recommended to make a special effort to identify respondents by name before data collection starts.

Survey methods The questionnaire: The questionnaire: Pre-test each questionnaire before fieldwork; Keep it as simple and short as possible; Pay attention to the order of the questions; Questions on a number of qualitative indicators can use either a binary or an ordinal scale; In the case of international innovation surveys, special attention should be given to the translation and design of the questionnaire; Short-form questionnaires for units with little/no innovation activity previously reported.

Survey methods Combination of Innovation and R&D surveys: Combination of Innovation and R&D surveys: Reduction in the overall response burden of the reporting units; Scope for analysing the relations between R&D and innovation activities at the unit level; Efficient method of increasing the frequency of innovation surveys; Country experiences indicate that it is possible to obtain reliable results for R&D expenditures; Longer questionnaire; Units not familiar with the concepts of R&D and innovation may confuse them; The frames for the two surveys are generally different.

Survey methods Guidelines for conducting combined surveys: Guidelines for conducting combined surveys: The questionnaire should have two distinct sections; Individual sections for R&D and innovation should be smaller than in separate surveys; Comparisons of results from combined surveys with those from separate innovation surveys should be done with care, and surveying methods should be reported; Samples to carry out such surveys should be extracted from a common business register.

Estimation of results Weighting methods: Weighting methods: The simplest one is weighting by the inverse of the sampling fractions of the sampling units, corrected by the unit non-response; If a stratified sampling technique with different sampling fractions is used, weights should be calculated individually for each; Commonly based on the number of enterprises in a stratum. In international and other comparisons, be sure that the same weighting method is used.

Estimation of results Non-response: Non-response: Unit non-response: a reporting unit does not reply at all; Item non-response: refers to the response rate to a specific question and is equal to the percentage of blank or missing answers among the reporting units; biased results » Disregarding missing values and applying simple weighting procedures based only on the responses received implicitly assumes that non-respondents are distributed in the same way as respondents biased results; imputation methods » Possibility: imputation methods to estimate missing values on the basis of additional information.

Presentation of results To get at least an idea of the variance for the results, it is recommended to calculate both (average) values for innovation indicators and also their coefficients of variation and/or confidence intervals; Results presentation should contain: metadata (including information on data collection procedure), sampling methods, procedures for dealing with non-response and quality indicators.

Frequency of data collection Innovation should be conducted every two years; Where this is not economically feasible, a frequency of three or four years may be chosen; Surveys must specify an observation period for questions on innovation; The length of the observation period for innovation surveys should not exceed three years nor be less than one year.

Annex A - 5. Methodological issues for developing country contexts Information system specificities: Relative weakness of statistical systems Absence of linkages between surveys and data sets; lack of official business registers information from other surveys cannot be used; Involvement of NSOs; When lacking, basic variables about firms performance can be included in the innovation survey - to enable further analysis.

Annex A - 5. Methodological issues for developing country contexts General methodological considerations: Questionnaire design: » Sections can be separated to allow different persons in the firm to reply them; » Guidance / definitions; » Language and the translation of technical terms; Survey application: » In-person; » Trained personnel.

Annex A - 5. Methodological issues for developing country contexts General methodological considerations: Frequency: CIS CIS » Every three to four years (e.g., timed to CIS rounds);CIS » Try to update a minimum set of variables every year; The purpose of surveys needs to be clearly stated and the questions clearly formulated; An adequate legislative base for the collection of innovation statistics can help ensure the success of such an exercise; The results should be published and distributed widely. The results should be published and distributed widely.

Thank you! CIS: