1. UNIVERSITIES EVALUATIONS AND RANKINGS Philippe VINCKE Rector of the Université Libre de Bruxelles

2. Evolution of higher Education  New Actors of higher education and research  Increasing mobility of students and researchers  Accountability of the universities, transparency  Evaluations, comparisons, rankings

3. Criticisms of the existing rankings (1)  Competencies of the authors of the rankings  Impossibility, for the reader, to reconstruct and verify the results (rankings are not« scientific »)  No information about the goals, the intended uses, the aimed public  Precise definition of « university » : are they all comparable ?

4.  Choice of the criteria and of their relative importance  Research  Education  Costs  Services  Social aspects  National context, legislation  Financial ressources  Choice of the indicators  Data validation Criticisms of the existing rankings (2)

5. Criticisms of the existing rankings (3)  Bibliometry  Quality of the data  Discrimination among the scientific fields  Different traditions (journals, books, proceedings, number of authors, time span of valid research)  Supremacy of the publications in English  Which indicators ? (IF, citation index, h index, …)  Experts  Do they exist ?  How to choose them ?  Which questions ? How to treat the answers ?

6. Numerical « manipulations » (1)  How is it possible to imagine that complex objects such as universities can be characterized by one number ?  The weighted mean can exclude good candidates Example : A4197 B10038 C6868  Curious effects of normalization

7. (1)(2) A2000500 B1120175 C400370 D160045 E880240 F160435 G1360110 H640305

8. (1)(2) Global score A100 B563545,5 C207447 D80944,5 E444846 F88747,5 G682245 H326146,5 RANKING : A, F, C, H, E,B,G,D

9. (1)(2) A1600500 B1120175 C400370 D160045 E880240 F160435 G1360110 H640305

10. (1)(2) Global score A100 B703552,5 C257449,5 D100954,5 E554851,5 F108748,5 G852253,5 H406150,5 RANKING : A,D,G,B,E,H,C,F

11. Before : A, F, C, H, E, B, G, D One modification of the score of A on one criterion. No change in the scores of the other universities After : A, D, G, B, E, H, C, F Inverse ranking !!

12. Other comments  Rankings are contested but used  Rankings have an influence on reality  Excesses are possible (financial bonus, or incitements,…)  Standardization effect

13. Conclusions  The rankings relayed by the media are not scientifically valid at this stage  Evaluation of research and higher education is a necessity  But it must be realized by competent people in the context of a clear policy and with explicit goals  There does not exist a unique method applicable in all institutions

14. Main questions (1)  Wich « objects » ?  Universities (definition?)  Education programmes  Diploma’s  Research centers  Research programmes  …

15. Main questions (2)  What does one want to do ?  To compare  To select the « best(s) »  To rank  To define « homogene » categories  To detect strong and weak points  To assign ressources  …

16. Main questions (3)  For whom ?  External autorities, government, …  Potential partners (universities, research centers, companies,…)  External teachers or researchers  Potential students  Funding agencies  Sponsors  Public opinion, media  Alumni  Internal authorities  Internal teachers or researchers  Internal students  …

17. Main questions (4) For each « situation » (characterized by the answers to the 3 previous questions):  Which indicators ?  Quality of the data ?  Numerical treatment of the data !

18. Different approaches for different concrete questions  Choose the « best » education programme for this student ? (« best » for him)  Allocate financial resources to research centers  Select the universities which could be « good » partners for this company  Identify the strong points of these universities for students interested in studies in that field  Necessity of an interactive decision-aid toolbox for each possible user and question.