On the feasibility of a new approach U-Multirank On the feasibility of a new approach Frans Kaiser, CHEPS Mini-seminar organised by the Transparency Tools Working Group for the BFUG Cracow 12 October 2011

Rankings in higher education instruments to compare higher education institutions and to judge their relative position, usually in league tables based on the actual performance of these institutions

The rise of global rankings Academic Ranking of World Class Universities (ARWU) Shanghai Jiaotong University, since 2003 Times Higher Education Supplement World Rankings (THE) Times Higher Education, since 2004 Higher Education Evaluation and Accreditation Council of Taiwan Ranking (HEEACT), since 2007 The Leiden Ranking (LR) Leiden University, since 2008

Critique of existing rankings focus on ‘whole institutions’ (ignoring internal variance) concentrate on ‘traditional’ research productivity and impact focus on ‘comprehensive research universities’ aggregate performance into composite overall indicators use constructed ‘league tables’ imply cultural and language biases imply bias against humanities and social sciences

Designing an alternative: the EC call for Tender development of concept and feasibility study of a global ranking (not only European) multi-dimensional, not only research multi-level (institutional and ‘field’) all types of higher education and research institutions multiple stakeholders

Subset of institutional activity profiles New approach to ranking Subset of institutional activity profiles

Subset of institutional activity profiles New approach to ranking Subset of institutional activity profiles

Institutional performance profiles multidimensional perspective of ‘institutional profiles’ no overall ‘league tables’ no composite institutional indicators two-level analysis (institutional and ‘field’) stakeholders driven approach & specific ‘authoritative rankings’ allowing ‘performance profiles’ in various dimensions base for benchmarking, for inter-institutional cooperation, for effective communication and profiling base for informed governmental differentiation policies at system level

Dimensions and indicators

Pilot Two levels: Institution (FIR) Field (FBR) Global sample of higher education and research institutions: 159 (target: 150), 2/3 Europe, 109 completed institutional questionnaires Two fields: Business studies Engineering (electrical and mechanical)

Pilot Self-reported institutional data by means of four online questionnaires: U-Map (Institutional) U-Multirank (Institutional level) U-Multirank (department level) Student survey International databases Bibliography Patent data

Pilot: conclusion There are issues to be resolved (like regional engagement, employability) but it may work What could it look like?

Choose the type of ranking you are interested in. Ranking of institutions Ranking of Business studies programs Ranking of Electrical engineering programs

Selection of matching institutions (1) Teaching & learning Student profile Research involvement Knowledge transfer Subjects covered Mature students Academic publications Start-up firms Degree level focus Part time students Professional publications Patent applications Orientation of degrees Distance education Other research products Cultural activities % Expenditure on teaching This is the institutional profile finder. As it does not fit onto one screen I broke it down in two parts. The user can click the various categories. Total enrolment Doctorate production Income knowledge transfer % Expenditure on research

Selection of matching institutions (2) International orientation Regional engagement Exchange stud; incoming Graduates in the region Exchange stud; sent out New entrants from region Foreign degree seeking stud Income from regional sources Non-national academic staff When the button is clicked the user will go to the sunburst page Income from internat sources Continue to ranking

Performance profiles (institutional level) Student internships in region Graduation rate bac % income from the region Graduation rate master Time to degree bac Research contracts with regional enterprise Regional joint research publications Time to degree master % exp on teaching Graduates working in the region Graduate unemployment Highly cited research publications % interdisciplinary programs Field normalised citation rate Patents awarded Post docs per academic staff Start up firms Art related research output Size of TTO % of research income from competitive sources Co-patenting This page will be further animated. The film of profiles will appear first and wil move from right to left. The user may click on nr 293, which then will appear in the full version. This is one end product. The user may also create a personalised ranking table. % income third party funding Create personalised ranking table % expenditure on research Incentives for knowledge transfer Interdisciplinary research Univ-industry joint publ. CPD courses offered International doctorate graduation rate. International joint research publications Research publication output. % programs in foreign language. % international staff % students in joint degree programs 4 98 222 196 152 148 111 293

Personalise your ranking Teaching & learning Research Knowledge transfer This is the first step in personalising the institutional ranking. I clicked the six categories Richard Thorn identified. If he clicks by mistake a different one it will not affect the rest. International orientation Regional engagement Show personalised table

Personalised institutional ranking 4 98 222 196 152 148 111 293 The personalised institutional ranking (looks still very basic ). If the first sort button is clicked the next slide is shown where the results are sorted on the first indicator sort

Personalised institutional ranking Sorted by indicator #1 4 98 222 196 152 148 111 293 Sorted by the first. If clicked the next slide shows, sorted by the third sort

Personalised institutional ranking Sorted by indicator #3 4 98 222 196 152 148 111 293 If 293 is clicked some additional information is shown (next slide) sort

Background information Name of institution: 293 Address URL Mission statement U-Map profile U-Multirank profile If you have any suggestions on additional items… If you click on the home button you get back to the starting screen

Choose the type of ranking you are interested in. Ranking of institutions Ranking of Business studies programs Ranking of Electrical engineering programs

Selection of matching institutions (1) Teaching & learning Student profile Research involvement Knowledge transfer Subjects covered Mature students Academic publications Start-up firms Degree level focus Part time students Professional publications Patent applications Orientation of degrees Distance education Other research products Cultural activities % Expenditure on teaching This is the first screen of the user who clicked business studies: the institutional profile finder Total enrolment Doctorate production Income knowledge transfer % Expenditure on research

Selection of matching institutions (2) International orientation Regional engagement Exchange stud; incoming Graduates in the region Exchange stud; sent out New entrants from region Foreign degree seeking stud Income from regional sources Non-national academic staff Income from internat sources Continue to ranking

Default business studies ranking Default field based ranking 4 98 222 196 152 148 111 293 Once the user of the FBR selected the institutions, the default table appears. We can explain here how that may be done. I have to add the sorting buttons

Default business studies ranking Default field based ranking Sorted by indicator #1 4 98 222 196 152 148 111 293 Sorted by indicator 1

Default business studies ranking Default field based ranking Sorted by indicator #12 4 98 222 196 152 148 111 293 Sorted by indicator 12. If you move the mouse button over the personalise button you move to the screen to select the indicators. I made two options: If you click on teaching and learning you go to the next screen. If you click on the personalise button you go the screen with all indicators. personalise

Personalise your ranking Teaching & learning Teaching & learning; student satisfaction Research Knowledge transfer International orientation Regional engagement Show personalised table

Personalised ranking business studies 4 98 115 The result of the personalised FBR (business studies). The only problem is that the sample of institutions does not match the pervious sample (of the default table), but is that a big problem? 138 139 144

Personalised ranking business studies Sorted by indicator #1 251 292 546 Sorted by the first 4 115

Personalised ranking business studies Sorted by indicator #3 546 613 4 Sorted by the third 144

Discussion No ‘U-Multirank league table’: less press coverage, better use? Authorative rankings: rankings in a user-defined setting? U-Multirank; a new approach? Here we may put in one or more sheets with information on the program (like the one you had from the CHEranking ** Multi-dimensional * Multi-level *** Inclusive ** Comparability check (U-Map) **** User-driveness

Thank you for your attention Here we may put in one or more sheets with information on the program (like the one you had from the CHEranking www.u-multirank.eu