1. Technopolis Group 1 Advantages and limitations of trans-national benchmarking in policy evaluation Patries Boekholt Technopolis Group Amsterdam patries.boekholt@technopolis-group.com

2. Technopolis Group 2 This presentation Background on benchmarking Methodological issues Case: ‘centres of excellence’ Some considerations on policy benchmarking

3. Technopolis Group 3 Policy benchmarking en vogue in Europe Joint ambition: increase S&T expenditures to 3% of GDP by 2010 EU Member States: common understanding to benchmark policy performance Motivations for policy benchmarking: To understands where improvements have to be made Understand factors involving performance of policies Learning from ‘best practices’ or ‘good practices’ Setting standard and targets for performance Taking part in the process is already helping to learn: ‘naming and shaming’ Larger demand to integrate cross-border benchmarking in evaluations of RTD policy instruments

4. Technopolis Group 4 Methodological issues (1) RTD policy instruments operate in a specific national/ federal or regional institutional setting and governance structure The effectiveness of policies depends on their role in a national innovation system R&D instruments part of a ‘policy mix’: their effectiveness and relevance depend on other policy measures Transferability /Diffusion of policies limited

5. Technopolis Group 5 Methodological issues (2) Comparability of data is limited due to many differences Objectives of instruments differ even within ‘classes of instruments’ Time frames of instrument Financial investments Implementation models ‘Zero base’ levels are not similar Influence of socio-economic context (economic climate, ….) Important part of the effects are behavioural Networking effects Change in culture

6. Technopolis Group 6 Example: ‘Centre of Excellence’ programmes Netherlands: Evaluation of Leading Technological Institutes (LTIs) Typical example of ‘centres of excellence programme’ Inspiration came from Engineering Research Centres in US (1995) Benchmarking of LTIs programme and LTI centres as part of the evaluation Benchmark partly based on previous study for Norwegian Research Council

7. Technopolis Group 7 Competence Centre programmes are spreading

8. Technopolis Group 8 Problems encountered Despite comparable ‘centres of excellence programmes’ in various countries, large differences in the detail Policy objectives determine the positioning of Competence Centres on the ‘scale’ of excellent science and industry oriented research This effects the performance and evaluation criteria On the individual centre level: scientific/technological scope and focus do not match e.g ‘Telematics’ Institute Particularly configuration of academia - competence centre relation had many different models Different emphasis on changing the academic research culture versus providing useful results for industry

9. Technopolis Group 9 Competence centres on Stokes’ Quadrants Quest for fundamental understanding Yes Pure basic research (Bohr) Use inspired basic research (Pasteur) Pure applied research (Edison) Yes No Considerations of use Not a place you’d want to be … Competence centre focus Increasing market failure

10. Technopolis Group 10 Positioning of the Benchmarks Centres

11. Technopolis Group 11 Some lessons Cross-border benchmarking of RTD programmes is particularly beneficial in the early design phase of RTD policies: to learn from success factors and pitfalls in other countries and adapt to own situation Benchmarking as part of ex-post programme evaluations has many methodological pitfalls Context dependency hampers the transferability of ‘best practices’ Beware of simplistic use of results by politicians Nevertheless lessons can be learnt using benchmarking: Failures and pitfalls in the design of existing instruments Policy decisions on strategic positioning of RTD programmes