1. Jamil Salmi and Alenoush Saroyan CIEP, 18-20 June 2006
league tables as policy instruments: the political economy of accountability in tertiary education
Jamil Salmi and Alenoush Saroyan
CIEP, June 2006

2. Lexus-Nexus index on rankings
Asia/Pacific: 6 18 27
Middle East/Africa: 1 9
Europe: 2 24 68
N. & S. America: 3 17 23

3. The rankings
business
A ranking of league tables
September 10, 2005

4. outline of the presentation
typology of rankings
a world of controversies
do rankings measure quality?
policy implications

5. typology of rankings: how is it done?
statistical indicators
produced by institutions
publicly available
survey of “stakeholders”
employers
professors
students
combination of both

6. typology of rankings: what does it apply to?
entire institution or specific program
gives a global score or measures several dimensions separately
research or teaching / learning

7. cluster of indicators in league tables as measures of quality
beginning characteristics
learning inputs- staff
learning inputs- resources
learning outputs
final outcomes
research
reputation

8. who prepares the ranking?
A = government agency (Ministry of Higher Education, Higher Education Commission, University Grants Council, etc.)
B = independent organization / professional association / university
C = newspaper / magazine / media
D = accreditation agency
I = International ranking (IA, IB, IC and ID linking the international dimension to the type of institution conducting the ranking)

9. ranking systems in 2006 National and International Ranking System
Region
National and International Ranking System
Eastern Europe and Central Asia
Poland (C), Slovakia (B), Russia (B), Ukraine (B)
East Asia and Pacific
Australia (B), China (B, IB), Hong Kong (C),
Japan (C), New Zealand (A), Thailand (A)
Latin America and the Caribbean
Argentina (D)
Middle East and North Africa
North America
Canada (C), United States (C)
South Asia
India (D), Pakistan (A)
Sub-Saharan Africa
Nigeria (A)
Western Europe
Germany (B/C), Italy (C), Netherlands (A),
Spain (B), United Kingdom (A, B, IC)

10. outline of the presentation
typology of rankings
a world of controversies

11. a thin line between love and hate
One could assume that those who have a bad ranking don’t like it and vice-versa
But not always true, there are principled people like the President of Tokyo University

12. Are the students interested. Definitely so
Are the students interested? Definitely so. Do universities like the rankings? I am not so sure. Of course, those who do well love them. Toronto president. To attract students (especially international students), to attract new professors and researchers, to attract funding

13. a thin line between love and hate
disagreement with principle
One could assume that those who have a bad ranking don’t like it and vice-versa
But not always true, there are principled people like the President of Tokyo University

14. a thin line between love and hate
disagreement with principle
criticism of methodology
One could assume that those who have a bad ranking don’t like it and vice-versa
But not always true, there are principled people like the President of Tokyo University

15. a thin line between love and hate
disagreement with principle
criticism of methodology
boycotts
One could assume that those who have a bad ranking don’t like it and vice-versa
But not always true, there are principled people like the President of Tokyo University

16. boycotts

17. boycotts Asiaweek US News and World Report McLeans
One could assume that those who have a bad ranking don’t like it and vice-versa
But not always true, there are principled people like the President of Tokyo University

18. a thin line between love and hate
disagreement with principle
criticism of methodology
boycotts
court actions (New Zealand, Holland)
One could assume that those who have a bad ranking don’t like it and vice-versa
But not always true, there are principled people like the President of Tokyo University

20. outline of the presentation
typology of rankings
a world of controversies
do rankings measure quality?

21. but do they measure quality?
quality a moving target; Illusive definition
mutlidimensional construct; unidimensional score (subjective weights to indicators)
theoretical justification of measures and methodology
empirical support for indicators

22. other shortcomings methodological flaws
lesser emphasis on outcome indicators
few meaningful indicators to assess teaching quality
one size fits all: general disregard for non-research universities and non-university institutions
encourages universities to adjust method of data reporting

23. and the winner is …

24. the Anglo-Saxon factor

25. the “English” factor in the 2005 rankings
THES
60 out of top 100 51 31 3 12 1 2
SJTU
68 of top 100 53 11 4 US UK
Canada
Australia
N.Z. HK
Singapore
India

26. outline of the presentation
typology of rankings
a world of controversies
do rankings measure quality?
policy implications

27. usefulness of rankings?
for the Government?
for the institutions?
for the public?

28. government use of rankings
Pakistan case
promoting a culture of accurate and transparent information
promoting a culture of quality

29. from the viewpoint of institutions
sensitive to factors that affect their rankings (benchmarking)
goal setting for strategic planning purposes
forming strategic partnerships
mergers

30. applying public pressure
Provão

31. applying public pressure
Provão
France

33. applying public pressure
Provão
France
Colombia

35. conclusion: divisive or helpful?
We may not like them, they may be flawed, but rankings are here to stay
Used for benchmarking by countries eager to revitalize their tertiary education system, and by universities as goal post to reach, especially in the US and increasingly in Europe

36. conclusion: divisive or helpful?
We may not like them, they may be flawed, but rankings are here to stay
Used for benchmarking by countries eager to revitalize their tertiary education system, and by universities as goal post to reach, especially in the US and increasingly in Europe

37. conclusion: divisive or helpful?
rankings are here to stay
useful for prospective students
useful in the absence of an established evaluation and/or accreditation system
useful for benchmarking, goal-setting and self-improvement purposes
useful to conduct a healthy debate on issues and challenges
useful to promote a culture of accountability
We may not like them, they may be flawed, but rankings are here to stay
Used for benchmarking by countries eager to revitalize their tertiary education system, and by universities as goal post to reach, especially in the US and increasingly in Europe

38. principles of an appropriate ranking instrument
compare similar institutions
better to focus on program than on entire institution
better to rank by indicator than wholesale (Germany – Pakistan)
better to focus on outcomes/outputs/results rather than inputs (labor market outcomes, publications, patents)
better if used for self-improvement purposes
better to advertise results publicly than to keep them secret
Compare similar institutions: for example research universities with research universities