1. On the feasibility of a new approach
U-Multirank
On the feasibility of a new approach
Frans Kaiser, CHEPS
Mini-seminar organised by the Transparency Tools Working Group for the BFUG
Cracow
12 October 2011

2. Rankings in higher education
instruments to compare higher education institutions
and to judge their relative position, usually in league tables
based on the actual performance of these institutions

3. The rise of global rankings
Academic Ranking of World Class Universities (ARWU) Shanghai Jiaotong University, since 2003
Times Higher Education Supplement World Rankings (THE) Times Higher Education, since 2004
Higher Education Evaluation and Accreditation Council of Taiwan Ranking (HEEACT), since 2007
The Leiden Ranking (LR) Leiden University, since 2008

4. Critique of existing rankings
focus on ‘whole institutions’ (ignoring internal variance)
concentrate on ‘traditional’ research productivity and impact
focus on ‘comprehensive research universities’
aggregate performance into composite overall indicators
use constructed ‘league tables’
imply cultural and language biases
imply bias against humanities and social sciences

5. Designing an alternative: the EC call for Tender
development of concept and feasibility study of a global ranking (not only European)
multi-dimensional, not only research
multi-level (institutional and ‘field’)
all types of higher education and research institutions
multiple stakeholders

6. Subset of institutional activity profiles
New approach to ranking
Subset of institutional activity profiles

7. Subset of institutional activity profiles
New approach to ranking
Subset of institutional activity profiles

8. Institutional performance profiles
multidimensional perspective of ‘institutional profiles’
no overall ‘league tables’
no composite institutional indicators
two-level analysis (institutional and ‘field’)
stakeholders driven approach & specific ‘authoritative rankings’
allowing ‘performance profiles’ in various dimensions
base for benchmarking, for inter-institutional cooperation, for effective communication and profiling
base for informed governmental differentiation policies at system level

9. Dimensions and indicators

10. Pilot Two levels: Institution (FIR) Field (FBR)
Global sample of higher education and research institutions: 159 (target: 150), 2/3 Europe, 109 completed institutional questionnaires
Two fields:
Business studies
Engineering (electrical and mechanical)

11. Pilot
Self-reported institutional data by means of four online questionnaires:
U-Map (Institutional)
U-Multirank (Institutional level)
U-Multirank (department level)
Student survey
International databases
Bibliography
Patent data

12. Pilot: conclusion There are issues to be resolved
(like regional engagement, employability) but it may work
What could it look like?

13. Choose the type of ranking you are interested in.
Ranking of institutions
Ranking of Business studies programs
Ranking of Electrical engineering programs

14. Selection of matching institutions (1)
Teaching & learning
Student profile
Research involvement
Knowledge transfer
Subjects covered
Mature students
Academic publications
Start-up firms
Degree level focus
Part time students
Professional publications
Patent applications
Orientation of degrees
Distance education
Other research products
Cultural activities
% Expenditure on teaching
This is the institutional profile finder. As it does not fit onto one screen I broke it down in two parts. The user can click the various categories.
Total enrolment
Doctorate production
Income knowledge transfer
% Expenditure on research

15. Selection of matching institutions (2)
International orientation
Regional engagement
Exchange stud; incoming
Graduates in the region
Exchange stud; sent out
New entrants from region
Foreign degree seeking stud
Income from regional sources
Non-national academic staff
When the button is clicked the user will go to the sunburst page
Income from internat sources
Continue to ranking

16. Performance profiles (institutional level)
Student internships in region
Graduation rate bac
% income from the region
Graduation rate master
Time to degree bac
Research contracts with regional enterprise
Regional joint research publications
Time to degree master
% exp on teaching
Graduates working in the region
Graduate unemployment
Highly cited research publications
% interdisciplinary programs
Field normalised citation rate
Patents awarded
Post docs per academic staff
Start up firms
Art related research output
Size of TTO
% of research income from competitive sources
Co-patenting
This page will be further animated. The film of profiles will appear first and wil move from right to left. The user may click on nr 293, which then will appear in the full version. This is one end product. The user may also create a personalised ranking table.
% income third party funding
Create personalised ranking table
% expenditure on research
Incentives for knowledge transfer
Interdisciplinary research
Univ-industry joint publ.
CPD courses offered
International doctorate graduation rate.
International joint research publications
Research publication output.
% programs in foreign language.
% international staff
% students in joint degree programs 4 98
222
196
152
148
111
293

17. Personalise your ranking
Teaching & learning
Research
Knowledge transfer
This is the first step in personalising the institutional ranking. I clicked the six categories Richard Thorn identified. If he clicks by mistake a different one it will not affect the rest.
International orientation
Regional engagement
Show
personalised table

18. Personalised institutional ranking4 98
222
196
152
148
111
293
The personalised institutional ranking (looks still very basic ). If the first sort button is clicked the next slide is shown where the results are sorted on the first indicator
sort

19. Personalised institutional ranking
Sorted by indicator #1 4 98
222
196
152
148
111
293
Sorted by the first. If clicked the next slide shows, sorted by the third
sort

20. Personalised institutional ranking
Sorted by indicator #3 4 98
222
196
152
148
111
293
If 293 is clicked some additional information is shown (next slide)
sort

21. Background information
Name of institution: 293
Address
URL
Mission statement
U-Map profile
U-Multirank profile
If you have any suggestions on additional items… If you click on the home button you get back to the starting screen

22. Choose the type of ranking you are interested in.
Ranking of institutions
Ranking of Business studies programs
Ranking of Electrical engineering programs

23. Selection of matching institutions (1)
Teaching & learning
Student profile
Research involvement
Knowledge transfer
Subjects covered
Mature students
Academic publications
Start-up firms
Degree level focus
Part time students
Professional publications
Patent applications
Orientation of degrees
Distance education
Other research products
Cultural activities
% Expenditure on teaching
This is the first screen of the user who clicked business studies: the institutional profile finder
Total enrolment
Doctorate production
Income knowledge transfer
% Expenditure on research

24. Selection of matching institutions (2)
International orientation
Regional engagement
Exchange stud; incoming
Graduates in the region
Exchange stud; sent out
New entrants from region
Foreign degree seeking stud
Income from regional sources
Non-national academic staff
Income from internat sources
Continue to ranking

25. Default business studies ranking
Default field based ranking 4 98
222
196
152
148
111
293
Once the user of the FBR selected the institutions, the default table appears. We can explain here how that may be done. I have to add the sorting buttons

26. Default business studies ranking
Default field based ranking
Sorted by indicator #1 4 98
222
196
152
148
111
293
Sorted by indicator 1

27. Default business studies ranking
Default field based ranking
Sorted by indicator #12 4 98
222
196
152
148
111
293
Sorted by indicator 12. If you move the mouse button over the personalise button you move to the screen to select the indicators. I made two options: If you click on teaching and learning you go to the next screen. If you click on the personalise button you go the screen with all indicators.
personalise

28. Personalise your ranking
Teaching & learning
Teaching & learning;
student satisfaction
Research
Knowledge transfer
International orientation
Regional engagement
Show
personalised table

29. Personalised ranking business studies4 98
115
The result of the personalised FBR (business studies). The only problem is that the sample of institutions does not match the pervious sample (of the default table), but is that a big problem?
138
139
144

30. Personalised ranking business studies
Sorted by indicator #1
251
292
546
Sorted by the first 4
115

31. Personalised ranking business studies
Sorted by indicator #3
546
613 4
Sorted by the third
144

32. Discussion No ‘U-Multirank league table’:
less press coverage, better use?
Authorative rankings:
rankings in a user-defined setting?
U-Multirank; a new approach?
Here we may put in one or more sheets with information on the program (like the one you had from the CHEranking
** Multi-dimensional
* Multi-level
*** Inclusive
** Comparability check (U-Map)
**** User-driveness

33. Thank you for your attention
Here we may put in one or more sheets with information on the program (like the one you had from the CHEranking