Searching for Globally Feasible Indicators from Domestic Rankings Ya Lan Xie and Ying Cheng Graduate School of Education, Shanghai Jiao Tong University, China The Fourth Meeting of International Ranking Expert Group June 2009, Astana, Kazakhstan

Background The new global rankings have been published one after another in recent years Rankings have problems and global rankings have additional problems, because “Not all nations or systems share the same values and beliefs about what constitutes ‘quality’ in tertiary institutions, …” 5 th Item of Berlin Principles Consequently, very limited indicators have been used in global rankings

Aim of the study Are there some indictors used by domestic rankings that can also be used for global rankings? – Analysis of their International comparability – Examination of their relevance to institutions’ quality

Samples: 11 rankings in 8 countries Title of RankingCountry Melbourne Institute Index of the International Standing of Australian Universities 2005 Australia Sunday Times University League Tables 2008UK Guardian University Guide 2008UK U.S. News America’s Best Colleges 2008USA The Top American Research Universities 2007USA Maclean's Rankings 2007Canada Comprehensive Competitiveness Ranking of Chinese Key Universities 2007 China Netbig Ranking 2008China Perspektywy University Ranking 2002Poland Slovak Ranking 2007Slovakia Romanian University Ranking 2007Romania

Assignation of indicators into 63 categories

Classification of indicator categories according to their international comparability and universality

Group 1: Comparable at a global level Those indicators – EITHER can show universities’ global competence, e.g. Publications and citations in international indexes Globally selected top scholars, such as winners of International awards, highly cited researchers – OR are measurements of internationalization Proportion of international students Research income from international source

Group 2: Fairly comparable at a global level The actual meaning of an indicator is relatively similar in different higher education systems, e.g. – Degrees granted (Bachelor, Master’s, Doctor) – Employment rate of graduates – Spending per student – Domestic publications and patents – Total research income – Proportion of faculties with the highest degree – Students/staff ratio – Student evaluation/satisfaction – ……

Group 3: Hardly comparable at a global level The effectiveness of an indicator is heavily affected by different higher education systems, e.g. – Acceptance rate – Number of Master’s or Doctoral programs – Alumni giving rate – Graduation rate – Retention rate – Proportion of full-time faculty – Assessment by domestic administrators/scholars

Group 4: Uncomparable at a global level Not universities in all countries have such indicator, e.g. – National Academy Membership – Faculty who won major national awards – Excellent course/textbooks, nationally selected – Excellent programs, nationally selected – Excellent research labs/centers/units, nationally selected – Excellent research products, nationally selected – Honors awarded to graduates – Performance on nationally standardized tests or benchmarks – Research income, national competitive

Number of Indicator Categories in each Group No. of Indicator Categories % Total63100% Group 1 Comparable 8 59% Group 2 Fairly comparable 29 Group 3 Hardly comparable 12 41% Group 4 Uncomparable 14

Frequency of indicators by groups in 11 rankings

Weight of indicators by groups in 11 rankings

The design of indicators for global rankings: TWO PATHS Creating internationally standardized test/assessment Measuring the academic performance of students Global survey (proper methodology & large enough sample) Global research assessment Using information that can be generated by HEIs themselves Group 1: Comparable + Group 2: Fairly comparable Group 3: Hardly comparable + Group 4: Uncomparable

Can these indicators be actually used in global rankings? Group 1: Comparable at a global level – Number of international projects – Books published at international editing houses Group 2: Fairly Comparable at a global level – Number of Bachelors granted – Average age of professors

Group 1 & Group 2 indicators that are used by rankings in Three or More Countries RankingCountry Melbourne InstituteAustralia Sunday TimesUK GuardianUK U.S. NewsUSA Top American Res. U.USA Maclean'sCanada RCCSE China Netbig China Perspektywy Poland ARRA Slovakia CNCSIS Romania Number of Countries Number of Rankings Students/faculty ratio √ √ √ √ √ √ √ √ √ 7 9 Spending per student √ √ √ √ √ √ 6 6 Pub. & Cit. in int’l indexes √ √ √ √ √ 4 5 Total research income √ √ √ √ 4 4 Number of Doctors granted √ √ √ √ 4 4 E mployment rate of graduates √ √ √ √ 3 4 Proportion of postgraduates √ √ √ √ 3 4 Proportion of professors √ √ √ √ 3 4 Library resources √ √ √ √ 3 4 Proportion of Ph. D. faculty √ √ √ 3 3 Proportion of Int’l students √ √ √ 3 3

Commonly used indicators Indicator No. of Countries No. of Rankings Also used by Students/faculty ratio79THES-QS Spending per student66 Pub. & cit. in int’l indexes45ARWU; Taiwan; THES Total research income44 Number of Doctors granted44 Employment rate of graduates 34 Proportion of postgraduates34 Proportion of professors34 Library resources34Newsweek Proportion of Ph. D. faculty33 Proportion of int’l students33THES-QS

Should some indicators be normalized to be more comparable at a global level and how? Employment rate of graduates – Normalized by national employment rates or not? Spending (of an university) per student – Normalized by GDP per capita or not? Total research income – Normalized by Purchasing Power Parity instead of exchange rate?

Underlying questions Does a decision need to be made across countries? To what extent a university is a global university? Global rankings: To measure universities’ performance as comprehensive as possible OR only to measure their “globalized part”?

Thanks for your attention!