Positional impact of reputational surveys in global university rankings Philippe Vidal and Ghislaine Filliatreau Observatoire des Sciences et Techniques, Paris IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Reputation is an idle and most false imposition, oft got without merit, and lost without deserving Shakespeare, Othello, II, 3 IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 In Global university rankings and their impact. Report II, EUA Report on rankings 2013 2.3 Superficial descriptions of methodology and poor indicators …. Use of poor indicators also persists. In spite of widespread criticisms, reliance on reputation is becoming more and more widespread….. IREG Forum University Rankings, Warsaw 16 -17 May 2013

Facts vs reputation Rankings graphical comparison as an approach for deconvoluting ranking patterns Reputation surveys: ARWU: 0%, THE: 34.5%, QS: 40% Webometrics as a proxy for reputation IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 1 Ternary diagrams IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 2 Ranking A vs Ranking B IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 3 Absolute and relative differences Diagrams (ARWU-Y) vs ARWU and (ARWU-Y)/Y vs ARWU IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 4 Diagram ARWU/X vs ARWU IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 5 Rankings reproducibilities - years selected: 2010 - 2011 - 2012 - average deviations to the average from one year to next: ARWU: 2-3 % THE: 10-12 % QS: 6-7 % Webometrics: 25-30 % IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 6 Filtering rules 90 % of the universities (259) selected 62 % ranked in 4 rankings 6 % in 3 rankings 22 % in 2 rankings First 250 ARWU ranks: 131 universities IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Targets United States United Kingdom Northern Europe (Germany, Scandinavia, Netherlands) Southern Europe (France, Spain, Portugal, Italy) IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 1 Ternary diagrams IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 2 Ternary diagrams IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Results 3 ARWU vs THE 1 IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Results 4 ARWU vs THE 2 IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Results 5 ARWU vs QS 1 IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Results 6 ARWU vs QS 2 IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 7 ARWU vs Webometrics 1 IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 8 ARWU vs Webometrics 2 IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 9 US: Relative differences IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Results 10 Europe: Relative differences THE-QS: Continental Europe Webometrics=whole Europe IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Summary Underestimatation of universities best ranked for research Overestimation of universities less well ranked for research Clear for THE and QS, even clearer for Webometrics And for the best research universities: US: underestimation THE/QS, overestimation Webometrics UK: overestimation THE/QS, underestimation Webometrics N + S Europe: underestimation THE/QS/Webometrics IREG Forum University Rankings, Warsaw 16 -17 May 2013

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Conclusions Reputation concept still on its infancy Necessity to do research on the concept as well as its modalities of implementation IREG Forum University Rankings, Warsaw 16 -17 May 2013