1. Evaluation of SRNSF Proposals ISB meeting February 25, 2016 Manana Mikaberidze Deputy Director General Giorgi Khishtovani Head of Grants and Programs Administration, Evaluation and Analysis Department Nino Gachechiladze, Deputy Head of Grants and Programs Administration, Evaluation and Analysis Department

2. 1.Previous evaluation schemes (2010-2014) - major problems and challenges 2.2015 Reforms related to evaluation procedures 3.ISB recommendations 17/07/2015 4.New evaluation procedures 5.Outsourcing of big grants’ evaluation 2016 6.Small grants’ evaluation 2015-2016 7.Experts’ national system development & Experts’ selection criteria 8.Efficacy evaluation & experts’ involvement 9.Evaluation work plan 2016 Content of presentation

3. 1.Scale of intervals (1-10; 11 – 21, 31 – 41 …..) 2.Score Inflation (scores artificially increased … 98, 99, 100!) 3.Incapability of comparison between proposals in one field 4.Unsatisfactory and irrelevant comments Previous evaluation schemes(2010-2014): major problems and challenges

4. SRNSF structural changes and new units involved in evaluation process administration: Grants and Programs Administration, Evaluation and Analysis Department IR and Fundraising Unit Science Communication Unit IT unit Revision of experts’ database – Grants Management Unified System (GMUS) TOR development integrating new standards and requirements for Programs Department Diverse approach to ‘small’ grants, and ‘big grants’ evaluation New evaluation criteria 2015 Reforms related to evaluation procedures

5. ISB Recommendations (17/07/2015) More transparency during evaluation process (well developed and explained evaluation criteria, experts’ detailed comments), Disclosure SRNSF major experts’ names in the end of each year (without indicating the calls and programs) Outsourcing of evaluation of ‘big’ grants for 3 years until launching new national experts data base Rotation of panel members, involvement of different stakeholders and target groups Revised evaluation criteria – sharing EU funds experience Towards National Experts’ System ( in 2018) Grants’ efficacy evaluation using external experts’ services

6. 1.Revised evaluation procedures and criteria complementary to EU, US standards 2.New Schemes for transparency and communication with scientific community 3.New schemes of interaction with universities and research institutions/centers 4.Outsourcing of the big grants’ evaluation (2016-2018) 5.Experts’ Panels for evaluation of small grants (2015- 2018) 6.First steps towards National Experts’ Data base (2018) New Evaluation Criteria and Procedures (2015-2016)

7. Cooperating with external reviewers And sharing experience in STI management: ERC (European Research Council) ESF (European Science Foundation) – working on EU fundraising schemes for cooperation with ESF ETAG (Estonian Research Council) Using sources of international projects for attracting international experts ELSEVIER SciVal Reviewer finder IncoNet EaP project (FP7) Black Sea Horizon (H2020) IncoNet EaP+ (H2020 from 2016) Outsourcing of big grants (FR, AR, DI, YS) and PhDF

8. MoU signed (CRDF Global; ORAU) On-line registration and evaluation database/platform modified in accordance to terms of SRNSF calls will be provided 1.NSF - CRDF Global – Spin off organization of NSF, US Programs: FR, AR, YS One stage evaluation by field panels, + inviting additional experts 2. ORAU – Oak Rage Associated Universities Programs: DI One stage evaluation by 2 experts + inviting additional experts 3. Prospect partner: ESF - PhD Fellowships 2016-2018 Outsourcing of big grants (FR, AR, DI, YS)

9. Prospects of Using expertise of international partners (bi and multilateral agreements) University of Oxford, UK JUELICH CENTER Germany CNR Italy CNRS France STCU Ukraine CRDF-Global US TUBITAC, Turkey ISTC EU, US, Japan EU multilateral framework H2020 EU ERC NCP ELSEVIER CERN DUBNA

10. Since summer 2015, the Foundation has invited 58 experts in the panel and additionally 20 independent experts from the various specific subfields in the process of evaluation of the so- called small grants Most successful Georgian scientists were chosen As experts Panels have evaluated the following small grants (11) Small Grants Evaluation – Selection Panels 2015 (1) 1.Travel Grant 2.Young Scientists Grants 3.MA Students Research Grants 4.Georgian Cultural Heritage Research Grants 5.Seasonal Schools Grants 6.YS Internship Grants 7.Conference Grants 8.Oxford University GSP grants 9.Young Inventors Grants (Leonardo Da Vinci) 10.Joint Research Grants (CNR) 11.Joint Research Grants (CNRS PICS)

11. For experts selection previous database was used and new experts added In particular cases, additional independent experts were involved; Based on the experts’ evaluation, the panel takes final decision (at the same time, panel had an opportunity to modify experts’ marks) Information about 2015 panel members was disclosed on the website of the Foundation with their prior approval Small Grants Evaluation – Selection Panels 2015 (2) Only one member of the panel refused to put his name on the list of the Foundation’s panel members on SRNSF website in 2015

12. In every subfield of the newly introduced classifier a ranking of Georgian Scientists will be developed on the basis of the various measurable, internationally recognized indices such as h- and g-indexes To select experts for the rankings information from the previous experts database will be referred; Experts from 2015 panels will also be considered as candidates Panel members in 2016 will be selected from the developed rankings; Foundation will first contact experts with higher science productivity data (higher position in the rankings) Small Grants Evaluation - Selection Panels (3) - Plans for 2016

13. In case of almost every small grant, the selection criteria has been widely altered; for the newly introduced small grants, new selection criteria has been adopted Changes has been applied to the following small grants: Heritage Grants, Seasonal Schools, PhD Scholarships Internship Grants, Joint calls CNR and CNRS PICS New selection criteria have been introduced for the following small grants: New PhD Programmes, Master Students Research Grants, Young Scientists (PostDocs) Research Grants Small Grants Evaluation – Selection Criteria (1)

14. Changes in the selection criteria was based on the selection schemes of the various international organizations: Horizon 2020: impact, academic component, management Selection schemes of the European Cultural Heritage grant programs (in case of heritage grant program) ERC starting grants Small Grants Evaluation – Selection Criteria (2)

15. Revised Evaluation Criteria - applying H2020 standards 1.Scientific Excellence and Impact State of art and topic actuality/importance Innovation and originality of approach Relevance of proposed methodology with goals and objectives Importance of expected results (for AR importance for Georgia’s economic development, potential for commercialization) 2. Human Resource/academic component: Competence, Expertise and experience of participants Competence, capacity and science productivity of PI and involved researchers Competence, capacity and science productivity of international and local partners Involvement of young scientists Complementarity of the participants within the consortium (when relevant); Inter-disciplinarity (when appropriate) 3. Project management and feasibility: Quality and efficiency of the implementation Relevance of human and financial resources and institutional infrastructure Implementation plan and budget justification (timeframe, division of responsibilities, relevance of materials, equipment and other resources requested for project) International and local cooperation Dissemination of results

16. Science Productivity: 1.Publications in international peer-reviewed journals, patents 2.c, h, and g indexes by different data bases (ELSEVIER ScienceDirect, Scopus, Thomson and Reuther, Springer, Google Scholar Publish or Perish for Humanities and Soc. Sciences …. ) 3. Acknowledgment from international scientific community Participation in multilateral international projects Scientific Awards and prizes International patents 4. Experience of working with young scientists (PostDocs, PhD, MA) 5. Experience of working with interdisciplinary teams and projects Developing Georgian Experts National System - GENS 2018 Experts’ selection criteria

17. Preparatory work Introducing evidence based matrix / measurable criteria for GENS experts’ selection Revising existing database and identifying possible candidates as experts. (ELSEVIER SciVal Reviewer GS/PR) Identifying and using other data-bases (Springer, Thomson Reuters Web of Knowledge, ISI- International Scientific Indexing etc.) Developing Georgian Research Portal in 2016 Developing Georgian Experts National System - GENS 2018

18. Work in progress: Working on Grants efficacy evaluation Schemes Involving external experts for grants’ efficacy evaluation – site visits of local and international experts Analyzing data and revising needs assessments systematically Revising and modifying SRNSF strategy, programs and funding schemes based on evaluation results. Launching new programs and targeted calls when necessary Experts’ involvement in efficacy evaluation

19. SRNSF 2016 Calls Calendar Programmes /Calls Preliminary announcement Call announcement Deadline for electronic submission Deadline for Result announcement 1FR - Fundamental/Basic research (Institutional) grants JanuaryFebruaryApril August 2AR - Applied research (Institutional) grants JanuaryFebruaryApril August 3DI - Research with participation of compatriots residing abroad (Institutional) grants JanuaryFebruaryApril September 4DO - PhD students' research (individual) grants JanuaryFebruaryMay August 5YS - Young scientists' (Postdocs) research (individual) grants 30.12.201520.01.201621.03.201623.03.201623.05.2016 6MR - Master students' research (individual) grants 30.12.20159.02.20166.10.20167.10.201624.11.2016 7IG - Internship grants for young researchers (individual) 30.12.201520.01.201617.05.201619.05.201612.07.2016 8TG - Travel grants (I announcement 15.03-30.06.2016) 30.12.201518.01.201615.02.201617.02.20167.03.2016 9TG - Travel grants (II announcement 1.07-31.10.2016) 30.12.201506.04.20161.06.20163.06.201623.06.2016 10TG - Travel grants (III announcement 1.11.2016-15.03.2017) 30.12.20151.08.201612.09.201614.09.201613.10.2013 11UO GSP - SRNSF/Oxford University joint programme for Georgian Studies 5.01.21611.01.201630.05.21631.05.201617.09.2016 12GEO CONF - Georgian Studies target conference 29.01.201622.02.20168.04.201611.04.201630.05.2016 13CONF - International conference 30.12.201515.01.20162.03.20163.03.201624.03.2016 14SS - Seasonal school grants 30.12.201511.01.201615.03.201617.03.201627.04.2016 15LEO - 'Leonardo da Vinci' -young inventors programmes (for secondary school students) January4.02.2016 10.05 (1) 19.09 (2) 16.05.(1) 22.09(2) June. (1), November(2) 16SCR -Research with the participation of secondary school students 30.12.201529.01.201623.02.201625.02.20167.04.2016 17PhD programme calls 20.01.20164.03.201618.04.201620.04.201615.06.2016 18STCU/ SRNSF joint callJanuaryFebruary MarchApril July(1)- December(2) 19JUELICH PHD - Julich Center programme for PhD studentsAprilJuneJuly September 20JUELICH PHD - Julich Center programme for MA studentsAprilSeptemberOctober November 21CNR -CNR/SRNSF joint call FebruaryMaySeptember October 22CNRS PICS - CRNS PICS/SRNSF joint call AprilMayJune October 23DAAD - SRNSF/DAAD joint call for young researchers FebruaryMarchSeptember October 24VW - SRNSF/VW joint call for doctoral programmes FebruaryMarchSeptember November 25FR INST - SRNSF/French Institute joint call for young researchers FebruaryMarchSeptember October 26Target programme for science popularization March23.05.201622.06.201624.06.201622.07.2016 27Annual stipends/awards in science and technology March11.05.201616.06.201617.06.201617.06.2017

20. ‘Big’ grants – May, June, July, August ‘Small’ grants – February – November Evaluation of local resources for GENS 2018– January – December Efficacy evaluation – June -December 2016 calls Evaluation timeframe

21. 2016 Calls Evaluation Timeframe Programs/CallsFebrMarchAprilMayJuneJulyAugSepOctNov 1TG 1 2CG - 3SS - 4LEO 5TG 2 6GEO CONF 7STCU 8PhD Fellowships 9IG 10New PhD programs 11FR 12AR 13DI 14YS 15Targeted call for Sc. Popularization 16Annual prizes for Geo Scientists 17UO GSP 18JUELICH MA 19TG 3 20DAAD 21VW 22FR INST 23MR 24JUELICH PHD - 25CNR 26CNRS PICS 27SCR

22. Thank you for your attention mikaberidze@rustaveli.org.ge khishtovani@rustaveli.org.ge gachechiladze@rustaveli.org.ge gachechiladze@rustaveli.org.ge www.rustaveli.org.ge