1. 1 new frontiers in evaluation Evaluating the RTD policy portfolio the Austrian experience Leonhard Jörg Andreas Schibany 24. April 2006

2. 2 Road map Why portfolio evaluation? Basic dimensions for evaluating RTD policy portfolios Some observations from Austria Limitations and practical problems

3. 3 Why should we look more systematically on RTD policy portfolios?... without a portfolio manager... as long budgets keep expanding End of catching up process is in sight Attention may shift again from “how much we spend” to “how we spend” There might be quite some room for increasing the effectiveness of the funding system

4. 4 Some remarks on the context Portfolios are not designed on the drawing table but the result of Changing perceptions of needs and problems Changing ways of how R&D is undertaken (mode 1  mode 2) Policy making in competitive environments There is no optimal portfolio Portfolios are usually messy with single instruments addressing multiple goals  We are looking after improvements rather than for THE optimal portfolio

5. 5 Road map

6. 6 Basic dimensions for evaluating RTD policy portfolios (i) Coverage: What policy goals are covered? Are there gaps? Proportions: Follow the money: How do the financial proportions fit to the policy agenda? Follow the debate: Does the amount of attention devoted to single instruments correspond with „importance“

7. 7 Basic dimensions for describing RTD policy portfolios (ii) Appearance/Visibility: Are differentiations between neighbouring instruments/brands clear to the clients? How many brands does the funding system communicate? Take the perspective of beneficiaries/clients: How many schemes/ programmes are available for specific RTD activities of specific groups: One? More than one? None? Patterns of usage: What instruments are used in parallel? Are there migration patterns between instruments?

8. 8 What indicates quality? Overall R&D-performance of the innovation system (hopefully) Responsiveness to changing environments and needs Interrelation between instruments (supporting complimentarity vs. interference and overlapping/competition) Interrelation between different levels of RTD-policy (regional, national, international) Entry rules and conditions for new instruments/programmes Exit strategies

9. 9 Road map

10. 10 Growing budgets

11. 11 Catching-up GERD/GDP

12. 12 Expanding policy portfolio 196519701975198019851990199520002005 Funding of institutions (universities, CRO’s) bottom-up project funding (ERP, FWF, FFF) first thematic programmes (energy) run by ministries Soft measures (coaching, information, IPR) more thematic programmes (transport, Flex-Cim,..) fiscal measures programmes … programmes Kplus, Kind/net, Fhplus, NW, NANO... Research infrastructure, investments education diffusion Industry structure high-tech sectors Critical masses excellence leverage effects science-industry linkages clusters Technology centres

13. 13 Committee for science, industry and economic affairs Government BMWABMVIT Firms ARC Polyt. ERP Fund National Research Fund Austrian Science Council Bottom-up project funding Universities Parliament AoS BMF LB-S BMBWK Start-up, IPR, PE/VC R&D-projects Structural Programmes Mobility/ Talent Thematic Programmes CD-L. Anniversary Fund Research projects KFI FFG Programme funding Institutional funding (colour of funding ministry) Catalytic financial measuresfiscal measures Policy Programmes / Agencies Performers

14. 14 Financial and Fiscal Measures: Objectives and Instruments Instruments/ primary goals RTD programmesBottom-up project funding Fiscal measures Institutional funding thematicfunctional Keeping the baseline       Increasing private R&D- investment           Broaden the innovation base    Enhancing entrepreneurship    Improving science industry linkages        Creation of excellence poles        Improving quality and relevance of scientific research          improving innovation support infrastructure     Exploiting specific new technology options     

15. 15 Financial Resources for main funding instruments

16. 16 Focus: direct funding

17. 17 Observations on the Austrian policy portfolio High level of diversification Strong in mobilising communities Significant improvements in management and evaluation standards Fragmentation – Tendency for establishing new programmes for ever smaller target groups Increasing competition between programmes – competing for beneficiaries Lack of portfolio management

18. 18 Road map

19. 19 Limitations and practical problems International benchmarking: New collections of “good practice” examples usually remain vague on the portfolio side “it’s the recipe not the ingredients”) Information base is dispersed and messy: Monitoring routines at programme level can rarely be combined/matched Evaluations on programme level usually address question of external coherence. However the big picture remains a patchwork Where is the customer?

20. 20 Thank you for your attention !