1. Introduction Extensive Experience of ex-post evaluation of national support programmes for innovation; less experience at regional level; Paper aims to indentify lessons from evaluation at the national level for innovation support at regional level; Starts from discussion of some key features of innovation; Followed by a description of innovation policy instruments; Next there is a discussion of an integrated approach to ex-ante appraisal, monitoring and ex-post evaluation; Some suggestions are then offered about how evaluation should be undertaken. Some additional issues relevant to evaluation of innovation support are then discussed; The paper finishes with a short list of main conclusions. 30 November 20091 Lessons from Evaluation of National Innovation Support Programmes, John Barber

2. Key Features of Innovation The process by which firms master product designs & production processes that are new to them, if not to the world, nation or sector; Includes the introduction of novel products & processes but also their subsequent diffusion throughout the economy & society; Innovation involves a combination of one or more of new technology, new business practices, new markets, organisational change and upgrading of work-force knowledge and skills; Technology consists of knowledge, artefacts, software, skills, designs & prototypes & routines and ways of doing things; Firms obtain technology and knowledge from a wide range of sources; the mix of sources varies systematically across sectors; Innovation is non-linear interactive process involving the individual firm with a wide variety of outside organisations & influences; An innovation system is a network of tangible & intangible institutions & the interactions between them. 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 2

3. Main Elements of National Innovation Policy Support for collaborative research into advanced technologies; Support for mission oriented research, development & systems; Support for creation & development of new technology based small companies; Support for single company R&D &innovation usually via grants, loans or tax breaks; Support for the adaptation & transfer of technology, knowledge & business best practice; Creation of centres of excellence, networks & other sources of technological & business advice; Most can be used or delivered at the regional level 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 3

4. Evaluation Consists of ex-ante appraisal, in-flight monitoring & ex-post evaluation which should be undertaken as elements of a holistic process such the ROAMEF:  Rationale: why a programme is necessary & should offer value for money;  Objectives: operational, near term (results), impact (ultimate or final);  Appraisal: the process which determines what is to be funded;  Monitoring: collection of data on the achievements against objectives;  Ex-post evaluation: analysis of the outcomes of the programme;  & Feedback: Use of the results in future policy-making. Ex-post evaluation should serve a number of purposes:  Improving programme management;  Informing the design of new programmes & policies;  Demonstrating that the programme has yielded value for money;  Informing budget allocation;  Increasing knowledge about how the economy & society work. 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 4

5. How should we undertake Ex-post Evaluation 1? Both the design & ex-post evaluation of a programme (or policy) should be based on an analysis of the innovation processes being supported; A well-designed programme with clear objectives & a sound mode of operation will suggest what the parameters of an evaluation should be; Four main questions should be addressed in the ex-post evaluation of a programme:  Was the programme appropriate? Did it address a significant weakness in the innovation performance of the region or country concerned? Was it based on a well-founded analysis of the innovation behaviour it was aiming to influence?  Was the programme effective in achieving its objectives?  Did it yield value for money? Do the estimated additional benefits of the programme exceed the identifiable costs or are they expected to do so in the foreseeable future? Was the programme rationale sound?  Was the programme efficient? Did it achieve the estimated benefits at the lowest possible costs? 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 5

6. How should we undertake Ex-post Evaluation 2? Experience of evaluation of national innovation support programmes shows that their success depends on them being appropriate to the needs & characteristics of the firms or sectors they are intended to help; Programmes also need to address issues which are important to the innovation & competitive performance of the nation, region or sector concerned; If a programme meets these two conditions & is clear about what it is trying to achieve then it has a good chance of success. A key part of any evaluation is to collect evidence about whether its objectives have been achieved; Evidence on operational objectives and (near-term) results is usually obtained fairly quickly, evidence on the (long-term) impacts may only be available some years after the programme is finished; Many (long-term) impacts may be indirect; In some cases an early interim evaluation may need to be followed by a full evaluation some years later. 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 6

7. How should we undertake Ex-post Evaluation 3? Value for money depends on whether a programme has a valid rationale & yields additional benefits which would not occur in its absence & whether these additional benefits exceed any additional costs; Identification of additional benefits depends on whether the correct framework of analysis is applied; The more selective is a programme the higher will be the proportion of supported activity which is additional but the higher will be the costs of programme administration; Both additional benefits & additional costs depend on assumptions about what might have happened in the absence of the programme; specification of this ‘counter-factual’ is often very difficult in the case of innovation; One element in the indirect costs of innovation support is the displacement of similar activities by non-supported firms. This is also very difficult to identify and assess. Programme efficiency depends on minimising the sum of the cost of support provided under the programme plus the amount of the operational costs of the programme for a given level of benefits. 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 7

8. Technology Transfer In an evaluation of technology transfer (TT) the UK DTI devised a simple framework which set out what is involved in a successful transfer of technology; This has five phases in the TT process:  Awareness 1 – the firm becomes aware of a new technology;  Awareness 2 – the firm considers the potential benefits in detail;  Transfer – the technology is acquired from an identified source;  Absorption – the technology is incorporated in designs of products & processes;  Exploitation – the firm learns much more about the technology through use; Successful public support for one or more phases needs the other phases to be addressed other by other means. Ex-post evaluation needs to examine whether this has happened. 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 8

9. Delivery of Innovation Support The success of public support for innovation depends on the effectiveness of the means of delivery; ex-post evaluation should consider this. Programmes may be delivered by public departments, other public agencies, research institutes, universities, consultants etc but in each case the ability to manage & deliver programmes can only be built up over time; Evaluations should examine not only effectiveness of delivery but also the impact on the capabilities & knowledge of the delivery organisation including whether it has learnt from any mistakes & what suggestions it has for improved delivery in the future; Frequent changes in the mix of public support or in the means of delivery prevents this necessary expertise from being built up and may also result in a fragmented delivery infrastructure consisting of a large number of small R&D institutes/technology transfer organisations; These will seek funding from wherever they can & will lack the clear mission statements & objectives essential to effective ex-post evaluation. 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 9

10. Conclusions Innovation is a complex interactive process which differs considerable across firms & sectors; Evaluation consists of ex-ante appraisal, in-flight monitoring & ex-post evaluation. These should be undertaken as part of an integrated process running from the initial conception of the programme through to the final verdict on its performance and should inform future policy-making; Both the design & ex-post evaluation of a programme (or policy) should be based on a thorough analysis of the innovation processes it is trying to influence & of the relevant aspects of the innovation system; Ex-post evaluation should consider whether a programme was appropriate (addressed a significant feature of innovation performance & fitted the needs of those it was trying to help), whether it met its objectives, whether it offered value for money & whether it was operated efficiently; Ex-post evaluation should consider how a programme was delivered & the impact on the capabilities & knowledge of the delivery organisation(s). Evaluation of technology & knowledge transfer programmes should analyse their operation in the context of the overall transfer process. 30 November 2009 Lessons from Evaluation of National Innovation Support Programmes, John Barber 10