Jamil Salmi and Alenoush Saroyan CIEP, 18-20 June 2006 league tables as policy instruments: the political economy of accountability in tertiary education Jamil Salmi and Alenoush Saroyan CIEP, 18-20 June 2006

Lexus-Nexus index on rankings 1981-85 1986-90 1991-95 1996-00 2001-06 Asia/Pacific: 6 18 27 Middle East/Africa: 1 9 Europe: 2 24 68 N. & S. America: 3 17 23

The rankings business A ranking of league tables September 10, 2005

outline of the presentation typology of rankings a world of controversies do rankings measure quality? policy implications

typology of rankings: how is it done? statistical indicators produced by institutions publicly available survey of “stakeholders” employers professors students combination of both

typology of rankings: what does it apply to? entire institution or specific program gives a global score or measures several dimensions separately research or teaching / learning

cluster of indicators in league tables as measures of quality beginning characteristics learning inputs- staff learning inputs- resources learning outputs final outcomes research reputation

who prepares the ranking? A = government agency (Ministry of Higher Education, Higher Education Commission, University Grants Council, etc.) B = independent organization / professional association / university C = newspaper / magazine / media D = accreditation agency I = International ranking (IA, IB, IC and ID linking the international dimension to the type of institution conducting the ranking)

ranking systems in 2006 National and International Ranking System Region National and International Ranking System Eastern Europe and Central Asia Poland (C), Slovakia (B), Russia (B), Ukraine (B) East Asia and Pacific Australia (B), China (B, IB), Hong Kong (C), Japan (C), New Zealand (A), Thailand (A) Latin America and the Caribbean Argentina (D) Middle East and North Africa North America Canada (C), United States (C) South Asia India (D), Pakistan (A) Sub-Saharan Africa Nigeria (A) Western Europe Germany (B/C), Italy (C), Netherlands (A), Spain (B), United Kingdom (A, B, IC)

outline of the presentation typology of rankings a world of controversies

a thin line between love and hate One could assume that those who have a bad ranking don’t like it and vice-versa But not always true, there are principled people like the President of Tokyo University

Are the students interested. Definitely so Are the students interested? Definitely so. Do universities like the rankings? I am not so sure. Of course, those who do well love them. Toronto president. To attract students (especially international students), to attract new professors and researchers, to attract funding

a thin line between love and hate disagreement with principle One could assume that those who have a bad ranking don’t like it and vice-versa But not always true, there are principled people like the President of Tokyo University

a thin line between love and hate disagreement with principle criticism of methodology One could assume that those who have a bad ranking don’t like it and vice-versa But not always true, there are principled people like the President of Tokyo University

a thin line between love and hate disagreement with principle criticism of methodology boycotts One could assume that those who have a bad ranking don’t like it and vice-versa But not always true, there are principled people like the President of Tokyo University

boycotts

boycotts Asiaweek US News and World Report McLeans One could assume that those who have a bad ranking don’t like it and vice-versa But not always true, there are principled people like the President of Tokyo University

a thin line between love and hate disagreement with principle criticism of methodology boycotts court actions (New Zealand, Holland) One could assume that those who have a bad ranking don’t like it and vice-versa But not always true, there are principled people like the President of Tokyo University

outline of the presentation typology of rankings a world of controversies do rankings measure quality?

but do they measure quality? quality a moving target; Illusive definition mutlidimensional construct; unidimensional score (subjective weights to indicators) theoretical justification of measures and methodology empirical support for indicators

other shortcomings methodological flaws lesser emphasis on outcome indicators few meaningful indicators to assess teaching quality one size fits all: general disregard for non-research universities and non-university institutions encourages universities to adjust method of data reporting

and the winner is …

the Anglo-Saxon factor

the “English” factor in the 2005 rankings THES 60 out of top 100 51 31 3 12 1 2 SJTU 68 of top 100 53 11 4 US UK Canada Australia N.Z. HK Singapore India

outline of the presentation typology of rankings a world of controversies do rankings measure quality? policy implications

usefulness of rankings? for the Government? for the institutions? for the public?

government use of rankings Pakistan case promoting a culture of accurate and transparent information promoting a culture of quality

from the viewpoint of institutions sensitive to factors that affect their rankings (benchmarking) goal setting for strategic planning purposes forming strategic partnerships mergers

applying public pressure Provão

applying public pressure Provão France

applying public pressure Provão France Colombia

conclusion: divisive or helpful? We may not like them, they may be flawed, but rankings are here to stay Used for benchmarking by countries eager to revitalize their tertiary education system, and by universities as goal post to reach, especially in the US and increasingly in Europe

conclusion: divisive or helpful? We may not like them, they may be flawed, but rankings are here to stay Used for benchmarking by countries eager to revitalize their tertiary education system, and by universities as goal post to reach, especially in the US and increasingly in Europe

conclusion: divisive or helpful? rankings are here to stay useful for prospective students useful in the absence of an established evaluation and/or accreditation system useful for benchmarking, goal-setting and self-improvement purposes useful to conduct a healthy debate on issues and challenges useful to promote a culture of accountability We may not like them, they may be flawed, but rankings are here to stay Used for benchmarking by countries eager to revitalize their tertiary education system, and by universities as goal post to reach, especially in the US and increasingly in Europe

principles of an appropriate ranking instrument compare similar institutions better to focus on program than on entire institution better to rank by indicator than wholesale (Germany – Pakistan) better to focus on outcomes/outputs/results rather than inputs (labor market outcomes, publications, patents) better if used for self-improvement purposes better to advertise results publicly than to keep them secret Compare similar institutions: for example research universities with research universities