Impact of League Tables and Ranking Systems on Institutional Decision-making Ellen Hazelkorn Higher Education Policy Research Unit CSER Director and Dean, Faculty of Applied Arts Dublin Institute of Technology, Ireland Universities and their Markets Reseau d’Etude sur l’Enseignement Superieur (RESUP) Paris, February 2007

'The University itself is ranked among the top UK universities for the quality of its teaching' `Top of the … Student Satisfaction table' ‘Our position is clearly the second Finnish University in international rankings’ ‘The number one destination for international students studying in Australia’ ‘Institution accredited by FIMPES, Excelencia académica SEP, x Place in academic program of...’

Themes 1. Rankings: A Challenge to HEIs? 2. Position and Reputation 3. Responding to Rankings 4. Global Competitiveness of Higher Education

1. Rankings: A Challenge to HEIs?

Global and Policy Context Globalisation and knowledge society Knowledge is key ‘factor in international competitiveness’ Desire to increase presence in knowledge marketplace Battle for ‘world class excellence’ Competition between HEIs for students, faculty, finance, researchers Internationalisation of higher education Trend towards market-steering governance mechanisms Increased emphasis on accountability/quality assurance Increasing desire for comparative or benchmarking data ‘Consumer’ information for students, parents and other key stakeholders

Rise in Popularity and Notoriety Rankings part of US academic system for 100 yrs, but today increasing popularity worldwide Near-obsession with rankings Cue to consumers re: conversion potential for occupational & graduate school attainment Cue to employers what they can expect from graduates Cue to government/policymakers regarding international standards & economic credibility

What do Indicators Measure? ‘Beginning Characteristics’/Student Ability – entry scores Learning Inputs/Staff – qualifications; teaching ratios Learning Inputs/Resources – expenditure on infrastructure Learning Outputs – graduation & retention rates Final Outcomes – employment rates, further education Research – publications/citations, awards, budgets, patents Reputation – peer appraisal; opinions of other stakeholders

Three Difficulties with LTRS … 1. 1.How they are aggregated: Technical and Methodological Difficulties Indicators as proxies for quality? Ability to compare complex institutions 2. 2.Usefulness of the results as ‘consumer’, comparative or benchmarking information Interpretation that may be ascribed to the results – the uses, decisions and actions that may follow.

Challenges for HE and HEIs Are League Tables and Ranking Systems influencing and informing institutional decision-making? strategy and mission institutional priorities – academic and research resource allocation recruitment and marketing Do HEIs monitor the performance of peer institutions? Do League Tables and Ranking Systems influence collaboration or partnerships? Do League Tables and Ranking Systems influence the views or decisions of key stakeholders? Are Are League Tables and Ranking Systems influencing broader higher education objectives and priorities? Who should undertake ranking and which metrics should be used?

International Study Conducted in association with IMHE (OECD) and IAU – using their membership lists. questionnaires sent to leaders/senior administrators in June- September questionnaires sent, with some unquantifiable ‘snowballing’ 202 replies received 31.6% response rate

Respondent Profile (N=202) Age: 36% post % % pre % publicly funded Institutional type 30% teaching intensive 19% research informed 29% research intensive

Global Distribution 41 countries, N=155

2. Position and Reputation

Popularity and Purpose of Ranking Use of national rankings on the rise, but worldwide rankings have wider penetration. Over 70% respondents identified ‘providing comparative information’ as the primary purpose of LTRS However, there is a differentiation between the target audience and user of such surveys… Target audience: students and public opinion User: public opinion, government, parents and industry

Ranking Status Significant gap between current and preferred rank 93% and 82%, respectively, want to improve their national or international ranking. 58% respondents not happy with current institutional ranking Current ranking: 3% of all respondents are nationally ranked 1 st in their country, but 12% want to be so ranked; No respondents are internationally ranked 1st, but 3% want to be so ranked 70% of all respondents wish to be in top 10% nationally, and 71% want to be in top 25% internationally.

Maintaining Position and Reputation Rankings play a critical role in enabling/facilitating HEIs to maintain and build institutional position and reputation. While answers dependent upon ‘happiness with position’, almost 50% use their institutional position for publicity purposes: press releases, official presentations, website. 56% have a formal internal mechanism for reviewing their 56% by the Vice Chancellor, President or Rector 14% by the Governing Authority

Over 40% of respondents said they considered an HEI’s rank prior to entering into discussions about: international collaborations academic programmes research student exchanges 57% said they thought LTRS were influencing willingness of other HEIs to partner with them. 34% said LTRS were influencing willingness of other HEIs to support their institution’s members of academic/professional organisations. Peer-benchmarking

Influence on Key Stakeholders Examples Benefactor ‘Depends on the rank’ ‘They feel reassured supporting us’ ‘Provides international comparators’ Collaborators ‘Depends on the rank’ ‘Good for reputation’ ‘we feel an improvement’ Current Faculty ‘Increases awareness about the importance of publishing’ ‘Easier to induce improvement with the department head whose rankings are declining Employers ‘They feel reassured’ ‘Those not open to us become more receptive’ ‘Can be confusing’ Funding Agencies ‘Impact on small part of indicators’ ‘Have less pretexts to deny funding’ Future Faculty ‘Recruitment easier with good reputation’ Government ‘May believe simplistic picture’ ‘Local government included to spend additional money for an excellent university’ Industry ‘Depends on the rank: good for reputation vs. less interest’ Parents ‘Particularly in an international market where status and prestige are considered in decision- making’ Partnerships ‘Good for reputation at international level’ Students ‘High profile students apply to high profile universities’ ‘Influence at the margins’

3. Responding to Rankings

Actions Arising (1) 63% respondents have taken strategic, organisational, managerial or academic actions in response to the results Of those, Overwhelming majority took either strategic or academic decisions and actions Only 8% respondents indicated they had taken no action

Actions Arising (2) Examples Strategy ‘Indicators underlying rankings are explicit part of target agreements between rector and faculties’ ‘Have become part of a SWOT analysis’ ‘Organise benchmarking exercises’ Organisation ‘New section established to deal with indicator improvements and monitor rankings’ ‘Reorganisation of structure’ ‘Have organised investigation team’ Management ‘Rector enforces the serious and precise processing of ranking as well as control of the relevant indicators’ ‘Development of better management tools’ Academic ‘Improve teaching and learning’ ‘New academic programmes’ ‘Increase English language programmes’ ‘More scholarships and staff appointments’

Impact on Higher Education True %False % Favour Established Universities8317 Establish Hierarchy of HEIs8119 Open to Distortion and Inaccuracies8218 Provide Comparative Information7426 Emphasize Research Strengths6535 Help HEIs Set Goals for Strategic Planning6535 Provide Assessment of HEI Performance5248 Promote Accountability4852 Can Make or Break an HEI’s Reputation4258 Provide Assessment of HE Quality4159 Promote Institutional Diversity3862 Enable HEIs to Identify True Peers3367 Encourage FAIR Competition2575 Provide Full Overview of an HEI1189

Ideal ‘League Tables’ Should give fair and unbiased picture of strengths and weaknesses Provide student choice for a programme and institution Provide accountability and enhance quality Ideal metrics are: Teaching quality Employment Student-staff ratio Research, e.g. publications and income Should be developed by independent research organisations, accreditation agencies or international organisations’ Favour institutional reviews (41%) rather than at programme (29%) or departmental level (30%)’

3. Global Competitiveness of Higher Education

Enhancing Reputation Respondents strongly perceive benefits/advantages flow from high ranking. This view is borne out by: Influence on ‘traditional’ audience: students and public opinion ‘Change of use’: growing influence on government and industry Influence on policymaking, e.g. classification of institutions, allocation of research funding, accreditation LTRS have helped rather than hindered, but depends on ‘ranking’’ Institutional ‘reputation’ can be enhanced depending upon position’ ‘Matthew Effect’?

Informing Institutional Decision-making Despite criticisms of methodology or concept, HEIs taking results very seriously, and making changes: Embedding LTRS within strategic decision-making and SWOT analysis Making structural and organisational changes Integrating recruitment with strategy Ensuring senior staff are well briefed on significance of improving performance Publicising ‘rank’ information to students, parents and key stakeholders Peer-benchmarking informing strategic planning, and collaboration and other partnerships’

Impact on Higher Education Regardless of institutional type or rank, respondents are concerned about wider impact on higher education and higher education policy: Can ‘one-size’ fit all? What impact on institutional mission? Rankings can have positive impact if highly rated, but potentially harmful if the reverse is true. While there may be a distinction between perception and reality of the impact of LTRS, the ‘perception’ is very powerfully felt.

Possible Implications Institutions behaving rationally – effectively becoming what is being what is measured. Worldwide comparisons likely to become even more significant for particular institutions in the future. Development of ‘single world market’ Formation of international networks Effecting greater vertical stratification w/ growing gap between elite and mass education Balance between pursuit of excellence and mass education Impact on diversity Policy intervention is critical