Session 4. Evaluation of publicly funded research development and innovation: The Integral Monitoring and Evaluation System (SISE) Alfonso Beltrán García-Echániz Director General Adjunto Fundación Española para la Ciencia y la Tecnología

SISE was launched by the Central Government together with the National Plan for RD&I It is a tool for: Publicizing their activities and making them more transparent, thus providing the Spanish society with a better understanding of RD&I activities funded with public money Controlling the management of publicly funded RD&I programs Producing all data required for evaluating the RD&I public policy The rationale behind SISE is the need for decision makers to count in with a constant feedback about the implementation and the outcomes of their policies The Integral Monitoring and Evaluation System (SISE) WHAT IS SISE?

SISE works as the permanent observatory on the Spanish Science and Technology System (SECYT) SISE promotes analysis and studies that provides decision makers with a deeper and more accurate understanding of the Spanish RD&I System The Integral Monitoring and Evaluation System (SISE) WHAT IS SISE?

1.Identifying which are the main science and technology indicators 2.Evaluating the Spanish RD&I public policies and monitoring their impacts on the evolution of the science and technology system 3.Developing recommendations and proposals for improving the efficiency of these public policies 4.Disseminating all SISE outputs among the actors of the Spanish Science and Technology System 5.Contributing to a better data provision, including information related to regions’ performance, and women’s participation in the RD&I system The Integral Monitoring and Evaluation System (SISE) MAIN GOALS:

STRUCTURE: Program Monitoring Commissions 34 commissions, one per each National Program and Strategic Action included in the National Plan for RD&I Shaped by decision makers and by scientists Each commission produces a monitoring report on its Program, that includes information on funding schemes, on the amount of available resources, on the entities involved in its management, on the applications’ rate of success, etc. National Plan Monitoring Commission (COSEP) Shaped by scientists, by public policies assessors, and by decision makers from the Regional Governments The COSEP Report includes the information contained in each Program Report, together with further analysis on public funding and management of the National Plan for RD&I areas The Integral Monitoring and Evaluation System (SISE)

SISE in the National Plan for RD&I : CICYT (Interministerial Commission for Science and Technology) SISE Report Monitoring Reports (National Programmes) COSEP Report (Monitoring Plan Commission) National Plan for RD&I

SISE in the National Plan for RD&I : The Integral Monitoring and Evaluation System (SISE) SISE has been included in the National Plan for RD&I as the key mechanism for monitoring and evaluating research and innovation policies, in order to review current activities and identify the need for new initiatives Three evaluation levels operate: Ex-ante evaluations: Focused on evaluating proposals On-going evaluations: Through the RD&I National Plan’s Annual Report, and the Working Program’s Monitoring Report Ex-post evaluations: Focused on the outcomes and the outputs of publicly funded RD&I activities.  PROEVAR

SISE OUTPUTS: The Integral Monitoring and Evaluation System (SISE) Three SISE Reports have been produced so far: The SISE Report 2005 The SISE Report 2006 The SISE Report 2007 Each of these Reports contained specific recommendations for the Interministerial Commission for Science and Technology (CICYT). Most of the recommendations were converted into specific political measures.

Main Outputs and Outcomes - SISE Report 2005: The Integral Monitoring and Evaluation System (SISE) 1.To create a reflection forum for the promotion of a National Strategy on Science and Technology that encompass a new National Plan for Science, Technology and Innovation. (December 2006 ENCYT  July 2007 National Plan ) 2.To design an All-Round Popular Science Communication Plan that fostered scientific and technological culture. (  2007 Science Year) 3.To publish before January the annual Working Program of the National Plan, containing all the forthcoming RD&I calls and official announcements, as well the amount of money awarded to them, and what are the management units responsible for each of them. (  Working Program 2006 and Working Program 2007)

Main Outputs and Outcomes - SISE Report 2005 (II): The Integral Monitoring and Evaluation System (SISE) 4.To take into consideration a division between civil RD&I entries and military RD&I entries in the National Budget. (  National Budget 2007) 5.To try to increase the participation of public research centres and private enterprises in the European Framework Program (  EUROINGENIO 2007)

Main Outputs and Outcomes - SISE Report 2006: The Integral Monitoring and Evaluation System (SISE) 1.The National Plan for RD&I should include a set of indicators to measure its management and its outputs. 2.The National Plan for RD&I should be submitted to the Permanent Commission before the end of July 2007, in order for it to be passed in accordance with the Advisory Board and the Science and Technology Governing Body. (  Passed through the Permanent Commission, the Advisory Board, and the Standing Commission in June and July 2007; Passed through the Cabinet in September 2007)

Main Outputs and Outcomes - SISE Report 2006 (II): The Integral Monitoring and Evaluation System (SISE) 3.To develop an electronic platform for collecting data regarding all public calls for RD&I projects, both from the Central Administration and from the Regional Governments. (  SISE Platform 2007) 4.To include in the annual Working Program all Region’s RD&I public policies, in order to promote a fluent dialogue between the Autonomous Regions and the Central Government. (  Working Program 2008)

Main Outputs and Outcomes - SISE Report 2007: The Integral Monitoring and Evaluation System (SISE) 1.To carry out a pilot study on public sector technology purchase under the legal figure of public-private collaboration contract. (The Secretary of State for Telecommunications and the Information Society, together with the Public Administration Ministry). 2.To develop an All-Round Popular Science Communication Plan (PICODI) for disseminating scientific and technological culture. (The Spanish Foundation for Science and Technology - FECYT). 3.To develop an in-depth evaluation analysis on the National Plan. (FECYT).

Main Outputs and Outcomes - SISE Report 2007 (II): The Integral Monitoring and Evaluation System (SISE) 4.To analyse which are the most suitable mechanisms to integrate the Autonomous Regions within the National Plan’s actions towards human resources, infrastructures, and institutional strengthening. (Ministry for Science and Innovation). 5.To evaluate the use of the financial expenditures allocated to R&D through the National Budget (Ministry For Economy and Finances). All these recommendations have been included in the Working Program for 2008

The Working Program for 2008 The Working Program is the annual updating tool of the National Plan. It includes: A full timetable with all forthcoming public calls, including deadlines and management units. An expenditures’ distribution per knowledge areas and priority programs. A review on the strategic actions. The evolution of the carry out of the Working Program in followed through the Monitoring Report. It includes: A follow-up of the public calls published on the National or Regional Bulletins An analysis of each call content. Half-yearly reports on the Working Program development. The Integral Monitoring and Evaluation System (SISE)