Research assessment for at the Academy of Sciences of the Czech Republic Eva Zažímalová The Academy of Sciences of the Czech Republic

 Aims  Principles  Organizational structure, disciplines  Qualitative profile of teams and Institutes (Phase I)  Evaluation of teams and Institutes – other aspects (Phase II)  Time schedule The Academy of Sciences of the Czech Republic Research Evaluation Seminar 2014, Prague, April 17, 20142/26

 Aims  Principles  Organizational structure, disciplines  Qualitative profile of teams and Institutes (Phase I)  Evaluation of teams and Institutes – other aspects (Phase II)  Time schedule The Academy of Sciences of the Czech Republic 3/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Aims  To obtain qualitative and quantitative information about science in the Academy of Sciences of the Czech Republic, in national, European and worldwide context, including differentiation of teams and Institutes according to quality.  To obtain complex information for strategy management of the Academy as a whole, including financing of the Institutes.  To mediate independent and comparable evaluation and feed-back for management of individual Institutes and research teams.  To meet the demands of responsibility in using public resources, including publishing the evaluation results. The Academy of Sciences of the Czech Republic 4/26Research Evaluation Seminar 2014, Prague, April 17, 2014

 Aims  Principles  Organizational structure, disciplines  Qualitative profile of teams and Institutes (Phase I)  Evaluation of teams and Institutes – other aspects (Phase II)  Time schedule The Academy of Sciences of the Czech Republic 5/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Previous evalutation ( ), performed in th Evaluation exercise at the Academy Increasing quality of evaluation exercises, including impact(s) on the Institutes First evaluation on the level of research teams, but evaluation of Institutes as a whole was also included. Evaluation on-site, based on peer review, international Evaluation Committees organized according to Sections within the Research Areas of the Academy (9 sections – 9 Committees) 6/26Research Evaluation Seminar 2014, Prague, April 17, 2014 The Academy of Sciences of the Czech Republic

Previous evalutation ( ) - drawbacks and imperfections:  Definitions of research teams – too different in some cases  Steering Committee – unclear competences  Establishment of Evaluation Committees (conflict of interest, insufficient knowledge re bibliometry etc.)  Establishment of Evaluation Committees according to Sections (not according to disciplines)  Uneaqual rigidness between Evaluation Committees (comparison between Sections not possible)  Insufficient knowledge about interpretation of scientometric data in Institutes, teams, and Evaluation Committees (summ of IF per capita, IF as the main indicator of quality, underestimation of trends, etc.)  It was not obligatory for foreing evaluators to evaluate all teams within the Institute (free choice of only some teams)  Short evaluation on-site (one day)  Vague definition of rules for appeal/rebuttal 7/26Research Evaluation Seminar 2014, Prague, April 17, 2014 The Academy of Sciences of the Czech Republic

5 Principles: 1.Evaluation within research disciplines 2.Informed peer review 3.Two-phase evaluation 4.Evaluation of scientific quality separated from financing 5.Transparentness The Academy of Sciences of the Czech Republic 8/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Principle 1 - Evaluation within research disciplines  Quality of outputs and results of teams will be evaluated within disciplines (associated into groups of disciplines) by Panels (phase I) and Review Committees (phase II) consisting of foreign evaluators.  Evaluation within disciplines will respect specific features of particular disciplines and will enable comparison with worldwide and European level of the disciplines. The Academy of Sciences of the Czech Republic 9/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Principle 2 – Informed peer review  Based on evaluation of quality of particular outputs and results submitted by individual researchers and teams (publications, citations, bibliometric data), and evaluation on-site.  Bibliometric evaluation: standard for natural sciences, not for humanities and social sciences. The Academy of Sciences of the Czech Republic 10/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Principle 3 - Two-phase evaluation Phase I: Peer review of research outputs and results – foreign evaluators organized into Panels (remote, web-based). Criterion – quality in international context, regardless the Institutes. Phase II: Peer review of Institutes – international Review Committees, evaluation results from the Phase I will be summarized at Institutes and integrated into the context of evaluation of other aspects of Intitutes’ activities. Evaluation on-site. The Academy of Sciences of the Czech Republic 11/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Principle 4 – Evaluation of scientific quality separated from financing  Evaluation of scientific quality without direct relationship to financing.  Final report about the evaluation exercise will not contain any financial advice and recommendations.  However: Information about utilization of financial resources in relation to financial demands of particular discipline and to scientific outputs and results will be one of the outputs of evaluation exercise. The Academy of Sciences of the Czech Republic 12/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Principle 5 – Transparentness  Preparation and arrangement: Institutes will be informed regularly. Trial registration to disciplines.  Publishing: Results of the evaluation exercise will be published in the frame of the Academy, and presented on the web-site of the Academy. Co-operation with PR-department.  Enhancement of quality of evaluation and operational communication with Institutes (in Phase II). The Academy of Sciences of the Czech Republic 13/26Research Evaluation Seminar 2014, Prague, April 17, 2014

 Aims  Principles  Organizational structure, disciplines  Qualitative profile of teams and Institutes (Phase I)  Evaluation of teams and Institutes – other aspects (Phase II)  Time schedule The Academy of Sciences of the Czech Republic 14/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Organizational structure Within the Institutes: Assessment units (each usually composed of one research team) INSTITUTE Teams Team member – Researcher (categories 3 to 5 according to Career Structure of the Academy Team member The Academy of Sciences of the Czech Republic 15/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Disciplines  Based on the official international lists (ERC, OECD, REF/RAE) in context to JCR (WoS) categories.  OECD („Revised field of science and technology (FOS) classification in the Frascati manual”): 6 groups of disciplines / 42 disciplines  for the Academy 5 groups of disciplines / 27 disciplines (based on trial registration): 1. Natural sciences (including Agricultural sciences) 2. Engineering and technology 3. Medical and health sciences 4. Social sciences 5. Humanities The Academy of Sciences of the Czech Republic 16/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Disciplines – Natural sciences (example) Discipline Group of disciplinesPanel 1 Natural sciences 1.1Mathematics1 1.2Computer and information sciences2 1.3Physical sciences3 1.4Chemical sciences4 1.5Earth and related environmental sciences5 1.6B Biochemistry and molecular cell biology, biophysics, … 6 1.6O Biological sciences including biotechnology and agricultural sciences 7 The Academy of Sciences of the Czech Republic 17/26Research Evaluation Seminar 2014, Prague, April 17, 2014

OECD List of disciplines and JCR categories Example: Discipline 1.4 Chemical sciences – sub-disciplines OECD classification WoS (JCR categories) 1Organic chemistry CHEMISTRY, ORGANIC2929 2Inorganic and nuclear chemistry CHEMISTRY, INORGANIC & NUCL Physical chemistry CHEMISTRY, PHYSICAL30 4Polymer science POLYMER SCIENCE148 5 Electrochemistry (dry cells, batteries, fuel cells, corrosion metals, electrolysis) ELECTROCHEMISTRY4747 6Colloidochemistry 7Analytical chemistry CHEMISTRY, ANALYTICAL CHEMISTRY, APPLIED CHEMISTRY, MULTIDISCIPLINARY2828 The Academy of Sciences of the Czech Republic 18/26Research Evaluation Seminar 2014, Prague, April 17, 2014

The Academy: Disciplines and outputs Počty publikací WoS v oborech 1 – 27 ( ) Natural sciences Engineering & technology Medical and health sciences Social sciences Humanities Multidisciplinary sciences Number of papers (WoS) The Academy of Sciences of the Czech Republic 19/26Research Evaluation Seminar 2014, Prague, April 17, 2014

 Aims  Principles  Organizational structure, disciplines  Qualitative profile of teams and Institutes (Phase I)  Evaluation of teams and Institutes – other aspects (Phase II)  Time schedule The Academy of Sciences of the Czech Republic 20/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Phase I: Qualitative profile of teams and Institutes Each researcher: Maximum 4 scientific outputs Team: The best result(s) (consisting of outputs) for a team (in relation to number of researchers in a team). Remote evaluation Qualitative profile will be generated (outputs and results assorted into 5 classes) The Academy of Sciences of the Czech Republic 21/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Evaluation: Data and requirements for evaluators Bibliometry for all submitted outputs All outputs at the disposal (web links), for team results all relevant outputs Requirement: Evaluation of content in context with the discipline The Academy of Sciences of the Czech Republic 22/26Research Evaluation Seminar 2014, Prague, April 17, 2014

 Aims  Principles  Organizational structure, disciplines  Qualitative profile of teams and Institutes (Phase I)  Evaluation of teams and Institutes – other aspects (Phase II)  Time schedule The Academy of Sciences of the Czech Republic 23/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Phase II: Evaluation of teams and Institutes – other aspects o In Phase II, other aspects of teams and Institutes will be evaluated, such as: - grants, projects, collaborations - management (human resources, career policy) - education (teaching, supervision of students) - other scientific activities (scientific infrastructures, popularization etc.) o The Institute and teams will submit self-evaluation report o Evaluation on-site: International Review Committees The Academy of Sciences of the Czech Republic 24/26Research Evaluation Seminar 2014, Prague, April 17, 2014

 Aims  Principles  Organizational structure, disciplines  Qualitative profile of teams and Institutes (Phase I)  Evaluation of teams and Institutes – other aspects (Phase II)  Time schedule The Academy of Sciences of the Czech Republic 25/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Time schedule - tentative Key dates BeginningEnd Period for evaluation Documentation finished Functionality of information system Registration Impact on financing The Academy of Sciences of the Czech Republic 26/26Research Evaluation Seminar 2014, Prague, April 17, 2014

Acknowledgements: prof. Jana Musilová, prof. Jiří Chýla, prof. Petr Ráb Members of Working Group of the Council for Sciences Members of Working Group of the Academy Council The Academy of Sciences of the Czech Republic Research Evaluation Seminar 2014, Prague, April 17, 2014