1. league tables as policy instruments: the political economy of accountability in tertiary education Jamil Salmi and Alenoush Saroyan 2 nd IREG Meeting Berlin, 18-21 May 2006

2. Lexus-Nexus index on rankings 1981- 85 1986- 90 1991- 95 19 96- 00 2001- 06 Asia/Pacific: 0061827 Middle East/Africa: 00019 Europe: 0022468 N. & S. America: 03172368

3. The rankings business A ranking of league tables September 10, 2005

4. and the winner is…

5. outline of the presentation typology of rankings typology of rankings a world of controversies a world of controversies do rankings measure quality? do rankings measure quality? policy implications policy implications

6. typology of rankings: how is it done? statistical indicators – –produced by institutions – –publicly available survey of “stakeholders” – –employers – –professors – –students combination of both

7. typology of rankings: what does it apply to? entire institution or specific program gives a global score or measures several dimensions separately research or teaching / learning

8. who prepares the ranking? A = Ranking prepared by government agency (Ministry of Higher Education, Higher Education Commission, University Grants Council, etc.) A = Ranking prepared by government agency (Ministry of Higher Education, Higher Education Commission, University Grants Council, etc.) B = Ranking prepared by independent organization / professional association / university B = Ranking prepared by independent organization / professional association / university C = Ranking prepared and published by newspaper / magazine C = Ranking prepared and published by newspaper / magazine D = Ranking prepared by accreditation agency D = Ranking prepared by accreditation agency I = International ranking (IA, IB, IC and ID linking the international dimension to the type of institution conducting the ranking) I = International ranking (IA, IB, IC and ID linking the international dimension to the type of institution conducting the ranking)

9. ranking systems in 2006 Region National and International Ranking System Eastern Europe and Central Asia Poland (C), Slovakia (B), Russia (B), Ukraine (B) East Asia and Pacific Australia (B), China (B, IB), Hong Kong (C), Japan (C), New Zealand (A), Thailand (A) Latin America and the Caribbean Argentina (D) Middle East and North Africa North America Canada (C), United States (C) South Asia India (D), Pakistan (A) Sub-Saharan Africa Nigeria (A) Western Europe Germany (B/C), Italy (C), Netherlands (A), Spain (B), United Kingdom (A, B, IC)

10. outline of the presentation typology of rankings typology of rankings a world of controversies a world of controversies

11. a thin line between love and hate

13. disagreement with principle (“Anglo-Saxon” exercise)

14. a thin line between love and hate disagreement with principle (“Anglo-Saxon” exercise) criticism of methodology

15. a thin line between love and hate disagreement with principle (“Anglo-Saxon” exercise) criticism of methodology boycotts

16. boycotts

17. boycotts Asiaweek US News and World Report

18. a thin line between love and hate disagreement with principle (“Anglo-Saxon” exercise) criticism of methodology boycotts (Asiaweek, USA) court actions (New Zealand, Holland)

20. outline of the presentation typology of rankings typology of rankings a world of controversies a world of controversies do rankings measure quality? do rankings measure quality?

21. the Anglo-Saxon factor

22. THES 60 out of top 100 51 31 3 12 1 3 2 SJTU 68 of top 100 53 11 4 US UK Canada Australi a N.Z. HK Singapo re India

23. shortcomings methodological flaws methodological flaws – –design – –choice of indicators (very few meaningful measures of quality of teaching and learning – –weight among indicators – –reliability of data no true measure of quality no true measure of quality wrongly used as “ one size fits all ” wrongly used as “ one size fits all ” encourage universities to adjust to the ranking criteria encourage universities to adjust to the ranking criteria

24. outline of the presentation typology of rankings typology of rankings a world of controversies a world of controversies do rankings measure quality? do rankings measure quality? policy implications policy implications

25. usefulness of rankings? for the Government? for the Government? for the institutions? for the institutions? for the public? for the public?

26. government use of rankings Pakistan case Pakistan case

27. government use of rankings Pakistan case Pakistan case –promoting a culture of accurate and transparent information

28. government use of rankings Pakistan case Pakistan case –promoting a culture of accurate and transparent information –promoting a culture of quality

29. from the viewpoint of institutions sensitive to factors that affect their rankings (benchmarking) sensitive to factors that affect their rankings (benchmarking) goal setting for strategic planning purposes goal setting for strategic planning purposes forming strategic alliances forming strategic alliances

30. applying public pressure Provão Provão

31. applying public pressure Provão Provão France France

33. applying public pressure Provão Provão France France Colombia Colombia

35. conclusion: divisive or helpful?

37. rankings are here to stay useful for prospective students useful in the absence of an established evaluation and/or accreditation system useful for benchmarking, goal- setting and self-improvement purposes useful to conduct a healthy debate on issues and challenges useful to promote a culture of accountability

38. principles of an appropriate ranking instrument compare similar institutions compare similar institutions better to focus on program than on entire institution better to focus on program than on entire institution better to rank by indicator than wholesale (Germany – Pakistan) better to rank by indicator than wholesale (Germany – Pakistan) better to focus on results rather than inputs (labor market outcomes, publications, patents) better to focus on results rather than inputs (labor market outcomes, publications, patents) better if used for self-improvement purposes better if used for self-improvement purposes better to advertise results publicly than to keep them secret better to advertise results publicly than to keep them secret