Bibliometric assessment of research performance in social sciences and humanities Henk F. Moed Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands

2 Assessment of research performance: Basic assumptions - 1 The concept of research quality does have a meaning in all fields of science and scholarship An assessment methodology should take into account the nature of the field

3 Assessment of research performance: Basic assumptions - 2 Important contributions to scholarly progress are sooner or later communicated in scholarly publications

4 Research quality: Dimensions Contribution to scholarly progress Enlightenment of the general public

5 Science versus humanities (Price) - 1 ScienceHumanities OrientationQuantitativeQualitative Status of knowledge Short term permanence Permanent significance OrganisationGroups interacting at the research front Individuals

6 Science versus humanities (Price) - 2 Different substantive contents ask for different types of social organisation and information exchange Science: Citation Index Humanities: Normal archival library

7 Social sciences Social sciences constitute a heterogeneous domain, with both ‘science-like’ and ‘humanities-like’ orientations Even sub-disciplines may be heterogeneous (e.g., sociology)

8 Overall ISI coverage by main field EXCELLENT (> 80%)VERY GOOD (60-80%)GOOD(40-60%) Biochem & Mol BiolAppl Phys & ChemMathematics Biol Sci ~ HumansBiol Sci ~ Anim & Plants Economics ChemistryPsychol & PsychiatEngineering Clin MedicineGeosciencesMODERATE (<40 %) Physics & AstronSoc Sci ~ Medicine & health Other Soc Sci Humanities & Arts

9 Sub-disciplines (non-exhaustive list) Social sciences related to medicine and health Other social sciences Humanities Public environm & occupat health Nursing Sport sciences Substance abuse Sociology Anthropology Educational sciences Political science Law Literature Linguistics Historical sc Philosophy

10 Large differences in ISI coverage among (publishing) countries in humanities and ‘other’ social sciences

11 ‘Other’ social sciences and humanities ‘National publication model’ is dominant Books play an important role Basic assumptions of a journal citation index are less valid ISI Indexes have inadequate coverage They also contain national journals

12 Indicator of a journal’s national orientation (INO) The share of the papers from the country most frequently publishing in a journal A purely national journal would have an INO value of 100 per cent

13 National orientation of journals (INO) for 4 fields DisciplineNo. Journals Median INO% Journals with INO>90% Physics & astronomy Mol Biol & biochem Humanities & arts 1, Other social sci

14 Policy assumption for social sciences and humanities The extent to which research findings reach beyond a purely national or local viewpoint and are exposed to criticisms from a wide international scholarly audience is a relevant criterion of research quality

15 The problem: ISI ≠ ‘International’ Works exposed to an international audience are not necessarily included in the ISI Indexes Works included in the ISI Indexes are not necessarily exposed to an international audience

16 This book argues: It cannot be taken for granted that the ISI Citation Indexes provide such indicators in all subfields of these domains of scholarship A challenge would be to systematically explore alternative data sources and methodologies

17 4 Types of bibliometric studies Cited/TargetCiting/SourceISI coverageField/Study 1ISI Excellent – Very Good Astronomy 2ISI+non ISIISIVery Good – Good Mathematics 3ISI+non ISI Good – Moderate Computer Science 4No citation analysis at allModerateNL Law

18 Alternative approaches 1.Expand the WoS with additional sources 2.Classification of publications and sources based on scholars’ quality perceptions 3.Library collection analysis

19 1. Expand the WoS with additional sources University of Granada (Spain): Creation of a citation index with 200 Spanish social science source journals not covered by the WoS 50,000 uses per year

20 2. Classifications based on scholars’ quality perceptions Case study on Flemish Law Questionnaires; No citation analysis Publications in Dutch: 81 % Publications in journals: 59 %

21 Classification of (national) journals based on a questionnaire into: Scholarly vs. non-scholarly Outstanding (A), good (B) and less good (C)

22 The ‘best’ indicator of scholarly research performance (in Flemish Law): Count the number of Single and multi-authored books (first editions only) PhD theses Publications with a length > 5 pages

23 3. Library collection analysis Focuses on books Determine the number of academic library copies per book title Example: Use Worldcat (Linmans, CWTS, 2007)

END