1. The evaluation of publicly funded research activities An overview Luis Sanz Menéndez OECD Symposium Madrid 3 July 2008

2. Issues for debate Implications of the use of evaluation as a management tool Changing evaluation "objects". –systems and policies, –programmes, –research 'collectives', –individuals and –institutions.

3. Evaluation as a management tool Characterizing evaluation activities - objects - aims - approaches - articulation with decision making processes Different objects ( Chabbal's terminology (1987)) –Actors: individuals & research collectives, –Operators: institutions, procedures, programmes and agencies/services, Policies

4. Evaluation objects: changing focus From the beginning: actors at the core of the process Researchers and research teams, research results and projects Organizing the Peer review process OECD and the evaluation of national systems Peer reviews and indicators The eighties and the focus on research operators Programmes at the core of new methodological developments The nineties (1) and the fashion of privatisation Focusing on the performance of research institutions The nineties (2) up to … : the fashion of 'excellence' and the growing between multiple missions and one sided criteria of performance

5. Evaluation as a management tool Different aims 1- Audits Objective: compliance to pre-established rules, often administrative. Main effect: sanctions. 2 - Assessment of Performance Problems: measures, differentiating outputs from outcomes. Approaches: "Summative"/"ex-post". Effects: hierarchy/positioning, gratifications/ rewards. 3- Relevance of action/activities Objective: the adequate "implementation structures". Approach: "pro-active" (focusing on the definition of future action). Effects: changes/adaptations in the course of action 4- Appropriateness (overall strategy) Objective: discussing the aims (against the changing environment) and/or their translation into goals and objectives. Effects: redefinition of the action and of its course.

6. Evaluation as a management tool Approaches to evaluation - evaluation process versus tools mobilized - "characterization" (of a situation) versus "judgment" - relations between approaches and couples "object/aim" --> typical articulations The articulation with decision making processes - 2 different approaches: support to the “boss” or as a means for collective learning. - 4 major constraints --> - Timing--->to feed back in the decision making process - Relevance--->to address issues at stake - Robustness--->to resist critics from evaluees - Credibility --->to be taken up by stakeholders & decision makers

7. Objects & approaches: Typical articulations Object/Aims/Approach/ TargetIssuesProcess ActorsPerformanceScientometrics Projects Quality Peer Reviews SystemPerformanceIndicators AppropriatenessAdvisory councils OperatorsPerformanceAudits RelevanceRenewed schemes Appropriateness& approaches Source: Callon, Laredo and Mustar (eds), 1997, the Strategic Management of Research and Technoloy, Paris: Economica International

8. Issues for debate related to 'objects' National policies (OECD role) The evaluation of operators Approaches adopted and problems raised by the evaluation of 'research collectives' (labs, groups, centers, institutes…). RAE in UK; INSERM in France, CSIC in Spain, etc. Problems raised around the evaluation of individuals: CNRS in France; CNAEI in Spain; SNI in México, etc The specific case of the strategic management of research institutions: how to go from the micro to the meso level?