1. Bureau du F.R.S. – FNRS 23 juin 2009 Possible role of a research funding agency The role of evaluation and ranking of Universities in the quality culture Prof. Véronique Halloin; General Secretary of F.R.S.-FNRS Dr. Pascal Perrin; head of research evaluation department

2. The quality of a University depends on: Academic/Faculty performance Research performance Quality of education Library Supports (administration, IT, infrastructure, …) Campus culture Quality of life … Ranking: need for a multidimensional framework (see Ph Vincke; see report of the EC Expert Group on Assessment of University-based Research) addressing users’ needs and interests (students, scientists, governments, universities, industries) 1. Research performance in ranking of universities

3. 3 The present contribution of research performances: Shangai : 80% research Nobel prizes: 20% (by staff) Highly cited staff in 21 disciplines: 20% Articles published: 20% (Nature, Science) + 20% (citation indexes) THES : <= 40% research Academic peer review : 40%/2 Citation per Faculty: 20% CHE : no weights; 4 fields: biology, chemistry, mathematics, & physics Size indicator: ouput volume in citation indexes Perception indicators: citations Number of often-cited staff & Nobel prize winners at the university Europe indicator: # projects in Marie Curie SHS poorly considered

4. 1. Research performance in ranking of universities 4 IndicatorIndicators used for ResearchRanking System (Country) Overall grants (money amount)Slovakia Grants per faculty (money amount)Austria, Germany, Italy Grants per faculty (absolute numbers)Italy Research projects funded by EUItaly Participation in int’l research programmesPoland No. of publicationsSweden Publications per researcherGermany, Slovakia, Switzerland Citations per facultyUK Citations per publicationGermany, Slovakia, Switzerland No. of int’l publicationsPoland % articles cited within 1 st two years after publicationSweden No. of publications with 5+ citationsSlovakia % articles belonging to top 5% most cited articles (HiCi)Sweden No. of patents (absolute number)Germany Patents per facultyGermany Ratio of pg research studentsUK Research qualityGermany, UK Reputation for researchAustria, Germany Ellen Hazelkorn; Dean of the Graduate Research School, Dublin Institute of Technology, Ireland; International Symposium on University Rankings; University of Leiden, 6-7 February 2009

5. 1.Research performance in ranking of universities 2.The possible role of a research funding agency 3.Challenges of bibliometrics 4.Research and science mapping 5.Ranking of research actors 6.Pending questions The role of evaluation and ranking of Universities in the quality culture

6. CAF - 26.06.09 - V3 6 2. The possible role of a research funding agency Funding evaluation is needed for allocation of money Research evaluation might be outsourced (if bibliometrics)but used to describe the global landscape of research to be funded Research policy is fed by funding evaluation and research evaluation A research funding agency is a university stakeholder: a llocation of money & research policy making UNIVERSITIES POLITICS, CITIZENS

7. 2. The possible role of a research funding agency INPUT OUTCOME OUTPUT Data bases, Researchers, Institutional repositories Indicator – metrics Graphs - maps FRS Methodology: bibliometrics Universities: monitoring Ranking makers: indicators; possible counterweight? Funding agency: research policy research evaluation indicators (economy & society) bibliometrics (research community) studies (retrospective & prospective) academic research landscape at international, national & community levels FNRS Aggegation to be avoided!! Multi-dimensional space crashed into linearity, information is definitely lost!!

8. 1.Research performance in ranking of universities 2.The possible role of a research funding agency 3.Challenges of bibliometrics 4.Research and science mapping 5.Ranking of research actors 6.Pending questions 8 The role of evaluation and ranking of Universities in the quality culture

9. 3. Challenges of bibliometrics DATA: many problems linked to the collection of data FIELDS – DISCIPLINES : various classifications, not satisfactory INDICATORS : which ones are valid/needed to quantify research performance? MAPPING : to assess potential rather than past performances (metrics) workshop

10. 3. Challenges of bibliometrics DATA Various databases : Web of Science (ISI); Scopus; Google scholar Overlap/coverage: strong dependency with the scientific field (Exemple: for management, only less than 20% of publications are published in the ISI listed journals ; Harzing & van der Wal, Ethics in science and environmental policy, 2008) Do not include productions such as art pieces or books, and limited open access publications (Scopus, Google) or conference proceedings institutional repositories to capture ouputs! Choice linked to what measurement is needed for (Google Scholar not suited to macro-level studies) Realibility of data is a precondition for all ranking exercices, the best methodology and soundest mathematical approach cannot correct what data collection might have distorsed…

11. 3. Challenges of bibliometrics DATA: problems of identification Spelling variances of ULB on WoS (1st encoder) 2001 – 2008: # publications? First step: 20.545 publications o ULB, univ, univ libre, free univ o IRIBHM, IBMM, ECARES, IIHE, erasme, Brugmann, St Pierre, Bordet Second step: - 3.850 “false” publications o Vrije or dutch or flemish or vlaams or UZ or AZ or ziekenhuis o St Luc, St Louis, catholique, ….. Re-coding errors (last encoder) o Univ. Zimbabwe, Dept Thorc.Surg., Leuven,Belgium ( Ziekenhuis ) o Free Univ. Brussels, Inst Math 1, D-14195 Berlin, Germany ( Freie Universität Berlin ) Belgium, Brussels Bruxelles Anderlecht, 1070… Except co- publications 16.695

12. 3. Challenges MAPPING: a field of bibliometrics in very active development (to detect clusters, new disciplines, emergence…) scientific fields mapped along time  evidence of change in networks each block in a column represents a field (ordered by size) the height of the block reflects citation flow through the field apparition of a new scientific field, neuroscience, from the convergence of neurology, psychology, and a part of molecular & cell biology

13. Map of science 2007 - 2008 Los Alamos on-line behaviour of scientists accessing different scientific journals (on-line access of a paper recorded by the servers of scholar web portals) objective : to visualize the links between disciplines circles = individual journals colours = disciplines lines = scientists navigation between publications unexpected relations revealed : connection between ecology and architecture prominent and central position for HSS (while maps based on citations favor natural sciences)

14. 1.Research performance in ranking of universities 2.The possible role of a research funding agency 3.Challenges of bibliometrics 4.Pending questions 14 The role of evaluation and ranking of Universities in the quality culture

15. 6. Pending questions Research evaluation is usefull But Needs to be carefully handled And Raises several questions Impact of evaluation on research activity, strategy (researchers, universities) Impact of evaluation on research funding (disciplines?) Risk to miss emergence of teams, themes (bibliometrics assess the past) Choice of methodology (peer review? bottom-up?) How to feed the research part of the (probably unavoidable) rankings with indicators developed as a support of research policy Possible role of networks (Eurohorcs, Unica, …)