1. High fluctuations of THE-ranking results in universities with lower ranking position
Johannes Sorz, Bernard Wallner, Horst Seidler and Martin Fieder
15th ISSI 2015, Istanbul

2. Agenda Introduction Aims of the study Methodology
Results and Conclusions
Discussion and Outlook

3. Introduction Global University Rankings:
Periodical rankings of the academic performance of international universities and thematic areas/subject fields based on various indicator scores calculated from university data, publication databases, reputation surveys
Advertised by their publishers as reliable basis for decision making for university management, policy makers, students and their parents

4. Introduction Global University Rankings:
Numerous critical studies (e.g. Hazelkorn, 2007; Rauhvargers, 2011):
Aggregation of complex metrics and various dimensions into simplified score: no conclusion on actual university performance or quality in teaching/research
Comparison of universities with different size, endowment, mission and frameworks conditions
Intransparent methodology/data issues (response rates, university data, quality assurance)
WoS/SCOPUS-coverage; language and field bias (van Raan, 2005 and 2011)

5. Introduction Times Higher Education –World University Ranking (THE)
TOP-200 universities listed followed by groups of 25 from and groups of 50 from 300 to 400.
13 indicators, grouped into five areas: Teaching (30%),Research (30%),Citations (30%),Industry income (2.5%), International outlook (7.5%)
No big change in methodology since
Research output indicators based on WoS data (since 2010)
Transparency: THE does not publish scores of individual indicators, only those of all five areas combined and total scores/ranks.

6. Introduction THE Indicators and Weighing (2014-2015)
Research (30%): peer reputation (18%), research income/staff (6%), papers in WoS/staff (6%)
Teaching (30%): peer reputation (15%), staff/students (4.5%), doctorates awarded (6%), doctorates/BA (2.25%), income/staff (2.25%)
Int. Outlook (7.5%): int./domestic students (2.5%), int./domestic staff (2.5%), publications w. at least one int. co-author (2.5%)
Research influence (30%): number of citations in WoS (normalized, 4 year window)
Innovation (2.5%): industry income/staff
Peer survey: total 33%  response rates; biased by “Matthew effect” Merton (1968)

7. Introduction THE: High fluctuations in rank (2014-2015) 2010-11
Diff_SUM
TU Denmark
122
178
149
117
121
Uni Freiburg
132
189
144
152
163
Yeshiva Univ. 68
154
156
172
186
118
U Massachusetts 56 64 72 91
Unit Basel 95
111
142 74 75
116
Universities with the highest fluctuations in the THE rank (top 5)
Volatile jumps: correlation with University performance seems unlikely

8. Aims of the study
Does THE give inconclusive results for lower ranked universities due to inconsistent fluctuations, which are not related to actual academic performance?
How could THE be improved and address inconsistent fluctuations?

9. Methodology Regression analysis
Regression/plotting of scores of the ranking of the year t-1 on the scores of the year t ( )
Regression/plotting of ranks of the ranking of the year t-1 on the ranks of the year t ( )
Plotting the scores of the year t on the ranks of the year t ( )

10. Methodology Mean yearly fluctuation (%)
Definition of 8 aggregated university groups of 50 (1-50, , , , , , , ).
Mean yearly fluctuation (%) = percentage of universities that changed their rank beyond their respective ranking group, thus either moving upwards, moving downwards, being newly in the rank or respectively dropping out of the ranking (mean of the years 2012, 2013 and 2014); direction and amount of change were not considered.

11. Results and Conclusions
High year-to-year fluctuations in scores and ranks (figures 1b-d;2)
Fluctuations in scores/ranks highest /
Fluctuations in scores/ranks are especially high in lower scoring/ranked universities (below approximately a total score of 65 and a rank of 50).

12. Fig. 1: THE: Scores/ranks of the year t-1 regressing on the scores/ranks of the year t from the ranking on
Linear regression line indicates perfect association, e.g. no changes in ranks and scores between two consecutive rankings (R2 ranges from 0.71 to 0.98)

13. Fig. 2: THE: Rank of the 2014-15 regressing on the rank of the year 2013-1450 50
Linear regression line indicates perfect association, e.g. no changes in ranks and scores between two consecutive rankings (R2 ranges from 0.71 to 0.98)

14. Results and Conclusions
Top-50 universities: difference in scores considerably higher than below a rank of 50; more linear relationship between scores and ranks; ranks are much more robust to year-to-year deviations in the scores
Universities ranked below 50: very small deviations in the total score (ca. 0.5 %) lead to inconsistent year-to-year jumps in the ranks (figure 3)

15. Figure 3: THE-Score 2014-15 plotted against
Rank    

16. Results and Conclusions
Mean yearly fluctuation (%)
Year-to-year fluctuations of universities increase with increasing rank
The mean yearly fluctuation (%) increases in lower ranked groups (4% in the top group (1-50) to 64% in the lowest group ( ) – Figure 4.

17. Figure 4: Mean yearly fluctuation (%)

18. Discussion and Outlook
Below ranks of 50 very subtle inconsistent changes in scores lead to drastic changes in ranks.
 THE results do not reflect the academic performance of universities below ranks of 50
Strong emphasis on citations (30% of total score)and peer reputation survey (33% of total score)
 a few highly cited publications or a few points asserted in the survey make huge differences in total score/rank below a rank of 50 (flat tail of the curve)

19. Discussion and Outlook
THE has already addressed fluctuations to some extent by ranking universities only down to position 200, followed by groups of 25 from and groups of 50 from 300 to 400
We suggest that universities should be summarized in groups of 25 or 50 below the position of 50
THE should omit the reputation survey to minimize fluctuations

20. Future work
Comparison of THE with other rankings: ARWU (Paper submitted) and QS-Rankings
Analysis on indicator-level to further pinpoint the source of inconsistent fluctuations

21. Thank you for your attention!

22. References
Hazelkorn, E., (2007) Impact and influence of league tables and ranking systems on higher education decision-making. Higher Education Management and Policy 19 (2), 87–110
Merton, R.K. (1968) The Matthew effect in science. Science 159, 56-63
Rauhvargers, A. (2011) EUA Report on Global Rankings and their Impact – Report I (European University Association)
van Raan, T. (2005) Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143
van Raan, T., Leeuwen, T., and Visser, M. (2011) Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings. Scientometrics 88, 495–498