Mapping Diversity – The U-Multirank Approach to Rankings Gero Federkeil Workshop Universidade Nova de Lisboa, 29th June 2012

Presentation Universidade Nova de Lisboa, 2012/06/29 2 Portuguese Universities in Global Rankings A (Short) Critique of Existing Global Rankings An Alternative Approach: U-Multirank

Portuguese Universities in Global Rankings Universidade Nova de Lisboa, 2012/06/293 Shanghai RankingQS RankingTHE Ranking U Lisboa394U Coimbra U Aveiro U Porto U Nova de Lisboa U Porto U Porto U Catolica Portuguesa Portuguese universities are not well represent edin international rankings Hence those rankings are not a good instrument to look at the Portuguese HE system

Presentation Universidade Nova de Lisboa, 2012/06/294 Portuguese Universities in Global Rankings A (Short) Critique of Existing Global Rankings An Alternative Approach: U-Multirank

The coverage of Global Rankings Universidade Nova de Lisboa, 2012/06/295 Existing global rankings cover only a small minority of all universities EUA: Global University Rankings and their Impact. Bruxelles, 2011.

The Indicators Universidade Nova de Lisboa, 2012/06/296 Existing global rankings are rankings of internationally oriented research universities only

The methodology: Ranking orthodoxy Universidade Nova de Lisboa, 2012/06/297 Ranking of whole institutions Composite overall score League table approach  Most users are interested in information about “their” field”  Institutional rankings give misleading averages across fields/units  Most users are interested in information about “their” field”  Institutional rankings give misleading averages across fields/units Composite indicators blur profiles and strengths & weaknesses  There are neither theoretical nor empirical arguments for assigning specific pre-defined weights to single indicators Composite indicators blur profiles and strengths & weaknesses  There are neither theoretical nor empirical arguments for assigning specific pre-defined weights to single indicators Small differences in the scores of indicators lead to big differences in league table positions  Give false impression of exactness (“Number 123 is better than number 127”) Small differences in the scores of indicators lead to big differences in league table positions  Give false impression of exactness (“Number 123 is better than number 127”)

Conclusions Universidade Nova de Lisboa, 2012/06/298 1.Existing global rankings cover only a small minority of universities 2.Due to their indicators they only measure research and cover internationally oriented research universities only 3.They devaluate other profiles and missions (teaching, LLL, regional engagement) 4.They are a threat to diversity in higher education

Presentation Universidade Nova de Lisboa, 2012/06/299 Portuguese Universities in Global Rankings A Critique of Existing Global Rankings An Alternative Approach: U-Multirank

The Project Universidade Nova de Lisboa, 2012/06/2910 Commissioned by the European Commission 2-year feasibility project, 2009 – June 2011 Report now available: Ján Figel, the former European Commissioner for Education, Training, Culture and Youth: “- to allow stakeholders to make informed choices; - to help institutions to position themselves and improve their performance” Two phases: o Design of new instrument o Testing the feasibility of new instrument

Specification of the Project Universidade Nova de Lisboa, 2012/06/2911 Five dimensions: o Teaching & learning o Research o Knowledge transfer o International orientation o Regional engagement Development of a list of indicators to be tested in pilot project Development of data collection tools and processes (question- naires, definitions, FAQs, communication + feedback processes) Done by CHERPA consortium: CHE, CHEPS, CWTS, INCENTIM; OST

Specification of the project Universidade Nova de Lisboa, 2012/06/2912 Two levels: Institution (FIR) Fields (FBR) Global sample of higher education and research institutions: 159 (target: 150), 2/3 Europe, 109 completed institutional questionnaires Two pilot fields: Business studies Engineering (electrical and mechanical)

The Basic Methodology Universidade Nova de Lisboa, 2012/06/2913 Multi-dimensional ranking Multi-level ranking Multi-level ranking Grouping approach Different levels of analysis are relevant for different users  Field-based and institutional rankings Different levels of analysis are relevant for different users  Field-based and institutional rankings Rank groups instead of league tables provide more meaningful and valid information  There is no single objective ranking  Each ranking reflects the ideas and preferences of those doing them  The decision about the relevance of indicators should be left to the user  There is no single objective ranking  Each ranking reflects the ideas and preferences of those doing them  The decision about the relevance of indicators should be left to the user

Mapping Diversity Universidade Nova de Lisboa, 2012/06/2914 A basic aim of U-Multirank is to make visible the diversity of Higher Education Institutions and to show excellence beyond research excellence HOW ?

Universidade Nova de Lisboa, 2012/06/29 15 Starting point: Diversity of higher education institutions in Europe & the world Identifying comparable institutions that can be compared in one ranking Description of horizontal diversity  Types/profiles Description of horizontal diversity  Types/profiles Assessment of vertical diversity  Performance Assessment of vertical diversity  Performance Complementary instruments of transparency +

Mapping and Ranking Universidade Nova de Lisboa, 2012/06/ Step: Mapping: Selection of a comparable set of universities based on institutional profiles 1.Step: Mapping: Selection of a comparable set of universities based on institutional profiles Teaching and learning Research involvement Knowledge exchange Regional engagement International orientation Student profile Example: Comprehensive, teaching oriented institution Mainly undergraduate education Low research orientation Low international orientation Regionally embedded (e.g. recruiting) 2. Step: Ranking of subset of comparable institutions with similar profiles 2. Step: Ranking of subset of comparable institutions with similar profiles

Multi-Dimensional Ranking for Subset of Comparable Institutions Universidade Nova de Lisboa, 2012/06/2917 No composite indicator! No number 1 !

User-driven, Personalised Ranking Universidade Nova de Lisboa, 2012/06/2918

A Personalised Ranking … Universidade Nova de Lisboa, 2012/06/2919 … Helping to make an informed choice based on invidual preferences … Helping to make an informed choice based on invidual preferences

Results of the feasibility study Universidade Nova de Lisboa, 2012/06/2920 Generally the concept, indicators and instruments of U- Multirank are feasible both on the institutional and the field level There are some problems concerning indicators, mainly on Issues of employability Knowledge transfer, and, Regional engagment There has to be some refinement of concepts and indicators

Outlook : U-Multirank II Universidade Nova de Lisboa, 2012/06/2921 New call for tender launched by the European Commission CHE applied with partners (CHEPS, CWTS, …) Decision at the end of July (?) Start of the project on 1 October (?) 2 * 2 years Implementation of the concept: First publication of ranking at the end of 2013 Min. 500 institutions, institutional and 4 fields Annual extension of number of institutions and fields Development of a business model for a sustainable system

But after all, there still might be some limits to ranking… 22 „You‘re kidding! You count publications?“

Muito obrigado! For more information:

Mapping Diversity – The U-Multirank Approach to Rankings Gero Federkeil, CHE Centre for Higher Education Workshop Universidade Nova de Lisboa, 29th June 2012