Eft/New frontiers in evaluation Conf. April fteval - New Frontiers in Evaluation Conference April 24th-25th 2006 Vienna - Austria Laurent Bach, Nicolas Carayol*, Patrick Llerena BETA L. Pasteur University of Strasbourg and CNRS (UMR 7522) *ADIS Université Paris Sud Current reforms in the French evaluation system : the growing role of indicators and measurement of PROs’ TT activities

Eft/New frontiers in evaluation Conf. April OUTLINES Reforms of the French public research system Law on Innovation 1999, various gvtal measures , Law on research March 2006 Focus on S-I relations / TT st implementation of New Law on Budgeting Procedures based on target / performance evaluation Indicators of TT activities of public research actors 2005/06 : 1st comprehensive study on TT activities of French universities (BETA-April 2006)

Eft/New frontiers in evaluation Conf. April OUTLINES the TT activities of French university : first results of the Beta study what can we learn from it about the performance indicators implemented by the New Law of Budgeting ? 1.The context : French public research system : main features, recent trends New law on budgeting : performance indicators 2.Study on the TT activities of French universities 3.An assessment of the indicators system of the New Law on Budgeting

Eft/New frontiers in evaluation Conf. April CONTEXT : French public research system Main features of the French public research system Extreme complexity "double" duality (still prevalent) : Research : PROs (± 25) vs universities (81) … BUT "mix labs" H.E. : universities vs Grandes Ecoles "mixed" roles : PROs = funding agencies, programmes agencies, research operators Largely dominating the overall research system Structure of fundings / of staff (civil servant status) Poor use of evaluation Lot of levels, lot of bodies Lack of coherence, of clarity, of impact on the system

Eft/New frontiers in evaluation Conf. April CONTEXT : Recent trends 80s - Law on Innovation governmental measures (innovation side) Law on research 2006 Development of competition-based programmes PROs, Ministerial funds…National Research Agency (ANR) Fostering of Science-Industry relations and TT 2006 law : approx. 2/3 of the "new" funds + almost all measures Concentration of resources on a local/regional basis "top-down" clusters (Pôles de Compétitivité) H.E/Research grouping under "new" administrative structures (PRES, Réseau Thématique de Recherche Avancée...) Slow development of New Public Management approaches Contractualisation, evaluation bodies…

Eft/New frontiers in evaluation Conf. April CONTEXT : The LOLF LOLF : Loi Organique sur la Loi de Finance Public Budgeting procedures / Parliementary level Adopted first implemented 2005 for 2006 budget Objectives : control of public money /actions multi-year strategic planning more flexibility of funds allocation Gvtal action : very broad missions->programmes->actions Performance targets and related evaluation system : objectives / performance indicators Budget year N+1 voted by Parliament depending on evaluation of success in year N (+ 5 years horizon)

Eft/New frontiers in evaluation Conf. April CONTEXT : The LOLF University research and the LOLF : mission "Research and Higher Education" (encompasses most of the public research activities, including the activities of PROs) 13 general objectives - 30 indicators (+ sub-indicators) research : 6 general objectives - 12 indicators programme "Higher education programmes and university research" various actions, including those related to research activities (broken down in 6 broad disciplinary fields + 1 interdisciplinary)

Eft/New frontiers in evaluation Conf. April CONTEXT : The LOLF Research-related objectives & corresponding indicators: 1

Eft/New frontiers in evaluation Conf. April CONTEXT : The LOLF Research-related objectives & corresponding indicators: 2

Eft/New frontiers in evaluation Conf. April CONTEXT : The LOLF Research-related objectives & corresponding indicators: 3 For PROs, most of these indicators are the same + some others are added depending on the field of research

Eft/New frontiers in evaluation Conf. April STUDY ON UNIVERSITY TT : Presentation performed : BETA Univ. L. Pasteur + French TTOs network CURIE for assembly of university presidents (CPU) and French Ministry of Education, Higher Education and Research (MENESR) survey by e-questionnaire filled out by universities' TTOs General organisation and functionning of TTOs (status, expend. and resources, policy, forms of contracts, personal, etc) Measurement of activities of TTOs situation as end time-based variables first of a regular evaluation in the future sent to all universities (88, incl. INP) + 11 members of CURIE; base : 2003 Ministry data The study on TT activities of universities

Eft/New frontiers in evaluation Conf. April STUDY ON UNIVERSITY TT : Main results Sample and representativity : 74 answers (66 univ. + 3 INP + 5 others) Rate of return : 74.7 % 78% of the universities + INP between 71% and 100% according to the type of universities between 8 and 10 out of the 10 Top universities (18 out of Top 20) according to Ministry data on research-oriented resources, number of teacher-researchers, industrial contracts, IPR revenues sample mean ≈ poulation mean for those criteria Not 100% answers on 100% questions => following results on sub- sample of respondants (not detailed here) First results of the study

Eft/New frontiers in evaluation Conf. April STUDY ON UNIVERSITY TT : Main results Creation of TT activities : more than 50% of univ. since 1999 (law on innovation), some very old ones Size of TTOs : 263 Full-Time Equivalent; 3.9 FTE per university; 1.2 TTOs staff for 100 teacher-researchers (exl. PROS staff of university labs) Total 2004 expenditures million €; 148 K€ per university; 2% of 2003 research-oriented resources Organisation : in 86% of the case TTOs = department or internal service from the university (1/5 : the 1999 born SAIC) 31 universities have at least two TTOs (2nd = very often external entity) 54% of the TTOs employees are civil servants Head : engineer (57% of univ), half have experience in private business.

Eft/New frontiers in evaluation Conf. April STUDY ON UNIVERSITY TT : Main results Research contracts and services : : M€ Average per university per year : 2.6 M€ (research contracts = 10 x service contracts) / 73 contracts Results on "outputs"

Eft/New frontiers in evaluation Conf. April STUDY ON UNIVERSITY TT : Main results Patent (applying – French priority/EPO) : : 1104 patents = 3.3 patent per university per year (≈ 20% of univ.: no patent) Patent (ownership – French + extension) End 2004 : 464 patents = 9.9 patents per university 58% fully owned are extensions (European patent or national ones)

Eft/New frontiers in evaluation Conf. April STUDY ON UNIVERSITY TT : Main results Licensing : 351 licenses awarded (1.1 license per year per university 50% on patent and related know-how 30% awarded to less than 3 years old university spin-off Revenues from license : 22.1 M€ ((90% on patent & related know-how) ) average : 0.2 M€ per university per year Other results on creation of firms, spin-off shareholding, "entrepreneurship" of civil servants, etc End 2004 : 243 "active" licenses (5.2 per university) 126 licenses are generating money (2.7 per univiversity)

Eft/New frontiers in evaluation Conf. April LOLF vs TT study : Some elements on the relevance of indicators per year on ("flow" data) : number of patent claims number of copyright on software number of licenses awarded per year income from licenses per year (€) amount of industrial contracts (€) 2004 ("stock data") number of patents owned number of current licenses number of licenses generating incomes TT performance indicators from the study :

Eft/New frontiers in evaluation Conf. April patent indicators : what about patent claim property regime ? - 1 LOLF vs TT study : Some elements on the relevance of indicators Performance as effectiveness : coverage of indicators : on 1091 patent claims, 79% (859 patents) full or co-ownership ownershipwaive

Eft/New frontiers in evaluation Conf. April patent indicators : what about patent claim property regime ? - 2 LOLF vs TT study : Some elements on the relevance of indicators Performance as effectiveness : coverage of indicators

Eft/New frontiers in evaluation Conf. April patent indicators : what about patent claim property regime ? - 3 LOLF vs TT study : Some elements on the relevance of indicators Performance as effectiveness : coverage of indicators

Eft/New frontiers in evaluation Conf. April patent indicators : what about patent claim property regime ? - 4 LOLF vs TT study : Some elements on the relevance of indicators Performance as effectiveness : coverage of indicators LOLF indicators : all ownership regime ? Patent invention vs ownership ? "share of patents" : net increase vs relative importance ? increasing proportion of co-ownership between university and PROs => may make the negociation of license with firms difficult

Eft/New frontiers in evaluation Conf. April Contracting activities indicators : what about distribution according to the different sources ? - 1 LOLF vs TT study : Some elements on the relevance of indicators Performance as effectiveness : coverage of indicators

Eft/New frontiers in evaluation Conf. April Contracting activities indicators : what about distribution according to the different sources ? - 2 LOLF vs TT study : Some elements on the relevance of indicators Performance as effectiveness : coverage of indicators LOLF indicators : industrial contracts compared to research-oriented resources : decreasing share ? distribution of different resources ? service contracts compared to the operating budget (with a certain % as objective), and not to any research-oriented resources.

Eft/New frontiers in evaluation Conf. April LOLF vs TT study : Some elements on the relevance of indicators performance as effectiveness : link between TT activities correlation bewteen TT activities ? positive correlation : no displacement effects ? high :industrial contracts - patent claim industrial contracts - licences awarded weak : industrial contracts - licences generating revenues industrial contracts - revenues form license => patent / licenses for money ? Signalling ? Secure collaboration ? weak for software with all => separate activity ?

Eft/New frontiers in evaluation Conf. April LOLF vs TT study : Some elements on the relevance of indicators Towards a first measurement of a "TT chain-link "

Eft/New frontiers in evaluation Conf. April LOLF vs TT study : Some elements on the relevance of indicators Towards a first measurement of a "TT chain-link " , one license => 119 K€ as revenue (137 K€). As for 2004 one patent owned => 0.34 (0.35) license 42% (38%) of the licenses are generating revenues, ie one patent => 0.14 (0.13) license generating revenues revenue : 78 K€ per license (81 K€) = 11 K€ (11 K€) per patent owned LOLF : no indicator on this

Eft/New frontiers in evaluation Conf. April LOLF vs TT study : Some elements on the relevance of indicators performance as efficiency Positive correlations between performance indicators and TT resources, but Inputs ? Outputs ? Low correlation between TT resources and IPR revenues LOLF : no efficiency what is called efficiency = IPR revenues / research-oriented resources

Eft/New frontiers in evaluation Conf. April LOLF vs TT study : Some elements on the relevance of indicators Universities differ : Size Scientific fields TT strategy, TT organisation, TT resources, etc => TT performance : indicator one by one + "performance pattern" aggregation issue : the heterogeneity of French universities LOLF : not taken into account risk over-using benchmark analysis risk to try to concentrate means on a few universities structural indicators ? reductions of gap …

Eft/New frontiers in evaluation Conf. April LOLF vs TT study : Some elements on the relevance of indicators Example of asymetric distribution of performance : patents owned

Eft/New frontiers in evaluation Conf. April LOLF vs TT study : Some elements on the relevance of indicators "average university" is far behind "frontier university" Top universities perform differently on different activities

Eft/New frontiers in evaluation Conf. April complementary remarks use of "ratio", "share of"… ≠evaluation in value ratio "against" other components of the French system (PROs, firms) publications : social sciences, mix labs, 2 years lag reactivity : priority thematics : too broad vs too narrow contribution to the construction of ERA : number of EC projects => scattering of resources, transactions costs organisational performance ? networking ? learning ?… LOLF vs TT study : Some elements on the relevance of indicators

Eft/New frontiers in evaluation Conf. April CONCLUDING REMARKS understanding TT activities from universities : 1st study => exploitation of data => future studies : more data to fill some gaps "optimal" organisation, given : various contexts, various activities (scope economies) "optimal" scale : critical mass ? return to scale ? cf same problem in production of knowledge LOLF : severe shortcomings key question : connection with other components of the evaluation system, notably at programme level (cf competition- based programmes) and at new structures level (cf new agencies)

Eft/New frontiers in evaluation Conf. April THANK YOU FOR YOUR ATTENTION