The CHE ranking The multi-dimensional way of Ranking Isabel Roessler CHE – Centre for Higher Education Development International Conference “Academic Cooperation and Competitiveness – University Ranking Methodologies” Cluj-Napoca, 17 – 20 September 2009

2 Academic Cooperation and Competitiveness/ Cluj-Napoca Sept Presentation I.The CHE – Centre for Higher Education Development II.Rankings – Aims and methodology III.The CHE ranking approach IV.Towards a European ranking

3 I. CHE – Center of Higher Education Development private, not-profit organisation founded in 1994 by Bertelsmann Foundation and German Rectors Conference purpose: promotion of reforms in German higher education Ranking of German universities among founding tasks of CHE; first ranking in 1998 activities: HE policy issues consulting ranking, since 1998 staff: ~ 30 people more information: Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

4 Rankings differ by target groups, particular goals information for prospective students (US News, CHE) information about global positioning (Shanghai Jiatong, THES) Information for HE community (Germany: National Science Foundation Ranking of Research Grants, CHE Research Ranking) even: basis for accreditation (e.g. Nigeria) Rankings vary in aims and target groups as well as „in terms of what they measure, how they measure it and how they implicitly define quality“ (Usher & Savino) II. Rankings: Aims and Methodology Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

5 II. The „classical“ model: ranking orthodoxy The worldwide known Rankings (Shanghai, THES) follow the classical league table approach of ranking ranking of whole institutions aggregation indicators into a single composite overall indicator by using fixed weights league table with individual numerical positions (like soccer table) There is a growing diversity of alternative approaches: deviations from the classical model One example for an alternative approach is CHE ranking Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

III. CHE Ranking: Background  ~ 35 subjects  All three years updated  Development of results are shown (Ups and downs)  Board of experts twice a year 6 Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

7 III. CHE-Ranking: Basic Principles no ranking of whole institutions field specific ranking Berlin Principles: Rankings should be „Be clear about their purpose and their target groups. Rankings have to be designed with due regard to their purpose.“  Target groups of CHE ranking (prospective students, universities /academic staff) are interested in information about „their“ field Universities are heterogeneous units; fields differ in their performance  ranking of whole institutions gives misleading averages Berlin Principles: Rankings should be „Be clear about their purpose and their target groups. Rankings have to be designed with due regard to their purpose.“  Target groups of CHE ranking (prospective students, universities /academic staff) are interested in information about „their“ field Universities are heterogeneous units; fields differ in their performance  ranking of whole institutions gives misleading averages Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

8 III. CHE-Ranking: Basic Principles multi-dimensional ranking multi-dimensional ranking Composite indicators blur profiles and strengths & weaknesses There are neither theoretical nor empirical arguments for specific weights for single indicators Heterogeneity of preferences about indicators among target groups /users (“quality is in the eye of the beholder”)  given weights patronise users / target group of rankings Academic Cooperation and Competitiveness/ Cluj-Napoca Sept no composite overall indicator

9 III. CHE-Ranking: Basic Principles group approach (top, middle, bottom) group approach (top, middle, bottom) League tables tend to exaggerate differences between universities („7th is better than 12th“) Small differences in the numerical value of an indicator lead to big differences in league table positions (ignoring issues of statistical errors and insecurity) CHE ranking: three groups top group is significantly better than bottom group no differentiation within groups Academic Cooperation and Competitiveness/ Cluj-Napoca Sept no league table

10 city, university city, university students study outcome study outcome teaching ressources internatio- nalisation internatio- nalisation III. The CHE-Ranking: Indicators 20 – 25 indicators... Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

11 III. The CHE-Ranking: Indicators... from different data sources…  publications /citations (bibliometric analysis)  research grants (faculties/departments)  research reputation (professors survey)  relation to research (students survey) Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

12 III. The CHE-Ranking: Indicators... facts as well as judgements  student-staff-ratio (fact)  student assessment of contact between students and professors  student assessment of course organisation Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

13 III. The CHE ranking: Data sources Survey among universities / departments ~1150 faculties facts on teaching & learning, research, resources etc. Student survey ~ students up to 500 students per programme Professor survey ~ professors on reputation Bibliometric analysis Patent analysis (engineering, sciences) Graduate survey Official higher education statistics Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

14 III. CHE ranking: presentation of results Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

Compact Ranking  4-6 indicators  First ordered by name  All analysed HEI included 15 Academic Cooperation and Competitiveness/ Cluj-Napoca Sept III. CHE ranking: presentation of results

III. CHE Ranking: faculty level 16 Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

Individualised Ranking  Own priority  up to five indicators  grouped 17 Academic Cooperation and Competitiveness/ Cluj-Napoca Sept III. CHE ranking: presentation of results

18 III. CHE ranking: presentation of results Step 2: Giving priorities to indicators Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

19 III. CHE ranking: presentation of results Step 3: Individualised results Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

20 III. CHE ranking: presentation of results... looking completely different for different indicators Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

21 III. CHE ranking: Impact - Individual 2/3 of students use ranking as one source of information differences by fields /types of students:  law, medicine, engineering  humanities studies show: ranking covers needs of information of prospective students (indicators) Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

22 Ranking not used for funding decisions /allocation of money !!! Institutions use data (published data & additional analysis) as a starting point for analysis of strengths and weaknesses for internal comparison / benchmarking between faculties, incl. contracts between president - faculties for external comparison / benchmarking with other institutions ranking helps to identify deficits & asking questions, but does not give all answers III. CHE ranking: Impact - Institutional Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

23 Emergence of European Higher Education area: growing mobility of students and staff growing demand for transparency about European HEIs Internationalisation of CHE ranking: since 2004: Austrian universities 2005: Swiss universities 2006/07: EU-funded pilot project with Dutch/ Belgian (Flemish) universities 2008: Dutch universities, University Bozen/Bolzano (I) 2009: new universities in specific fields: Babes Bolyai, Semmelweis… IV. Towards a European ranking Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

24 IV. Towards a European ranking Berlin Principles: Rankings should: “Recognize the diversity of institutions and take the different missions and goals of institutions into account.” No ranking of all European HEIs who have different structures, missions and profiles Rankings within types/clusters of institutions Demand for a European Classification Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

25 CHE - European Excellence Ranking ranking of top research departments in mathematics, physics, chemistry, biology focus on information for Master, PhD students and research pre-selection of universities based on bibliometric analysis, Marie Curie programme, (active!) Nobel Price Winners disproportional to countries basic CHE approach: field specific, multi-dimensional, no league tables IV. Towards a European ranking Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

IV. Towards a European ranking 26 MGUR: Multi-Dimensional Global University Ranking, EU-Tender ranking of departments in business studies (research focus) and engineering (regional focus) pre-selection of universities disproportional to countries basic CHE approach: field specific, multi-dimensional, no league tables CHERPA Academic Cooperation and Competitiveness/ Cluj-Napoca Sept. 2009

Thank you very much ! International Conference “Academic Cooperation and Competitiveness – University Ranking Methodologies” Cluj-Napoca, 17 – 20 September 2009