Towards a Bibliometric Database for the Social Sciences and Humanities – A European Scoping Project Diana Hicks & Jian Wang School of Public Policy Georgia Institute of Technology March 2009 Consultative Stakeholder Workshop, Brighton, UK

Project tasks Analysis of literature pertinent to SSH coverage in Scopus, Web of Science, Google Scholar High level summary of known national evaluation systems with in depth comparison of US (NRC), UK (RAE) and Australia (Composite Index and ERA) Analysis of the coverage of ERIH and Norwegian lists using Ulrich’s Proposing solutions

Framework: the 4 literatures+ The literaturesProblemSolution International journal articles How good is WoS & Scopus SSH coverage? National literature Journal published outside US/UK/Netherlands OR Journal not in English Not well covered in WoS or Scopus not highly cited Extended Norwegian model However, the journals will still not achieve the highest ranking, mirroring low citation counts in WoS/Scopus Books and parts of books, monographs Not covered in WoS or Scopus highly cited in SSH Create a database of books Extended Norwegian model Non-source citations in WoS Enlightenment literatureNot covered, or respectedMight be covered in other databases Could be included in extended Norwegian model Non-textual output – performance, exhibit etc. Exclusive focus on textual output in evaluation thus far Extended Norwegian model

Abbreviations, definitions and distinctions SSH –SS – social sciences –H - humanities db = Database: –WoS = Web of Science = SCI & SSCI & AHCI = Thomson Reuters = ISI –Scopus –Scopus = Elsevier’s Scopus –GS = Google Scholar –Ulrich’s – most comprehensive worldwide LIST of periodicals, not a database of articles in periodicals All publications vs journal articles Assessing published lists of journals covered in the db or articles actually found in the db. Source and non-source/citation Fields: SSH, STEM, all... There is a small rate of error in every database.

Ulrich’s Ulrich’s is the authoritative source of bibliographic and publisher information on more than 300,000 periodicals of all types from around the world –academic and scholarly journals –Open Access publications –peer-reviewed titles –popular magazines –Newspapers –newsletters, and more. Ulrich’s has been used in bibliometric studies as the benchmark against which WoS and Scopus coverage is measured Listing all the world’s periodicals, irrespective of language or country of publication is truly ambitious. In large measure Ulrich’s succeeds. Studies have found only very small numbers of journals that are not yet indexed in Ulrich’s. –We found journals, all newer, that seemed to exist but were not yet indexed. We bought 74k records covering active, regularly appearing periodicals in SSH fields.

Norwegian model reference list The reference list of journals that are acceptable within the Norwegian evaluation system. The list covers all fields of science, social science and humanities. Covers scholarly publications, a scientific or scholarly publication must: –present new insight –in a form that allows the research findings to be verified and/or used in new research activity –in a language and with a distribution that makes the publication accessible for a relevant audience –in a publication channel with peer review Publications in local publication channels are not counted in the budgeting model. The level of a publication channel is defined by its mix of authors. –International - Authors from several countries; international language(s) –NationalMore than 2/3 of the authors are from the same country –LocalMore than 2/3 of the authors are from the same institution Source: G. Sivertsen, Bibliometrics for (or against?) the humanities, presentation 2008 G. Sivertson kindly shared with us the SSH list containing 8,165 journals. 6,103 could be matched to Ulrich’s records and we analyze those.

European Reference Index for the Humanities (ERIH) ERIH aims initially to identify, and gain more visibility for top-quality European Humanities research published in academic journals in, potentially, all European languages. It is a fully peer-reviewed, Europe-wide process, in which 15 expert panels sift and aggregate input received from funding agencies, subject associations and specialist research centres across the continent. ( areas/humanities/research-infrastructures-including-erih.html) ERIH includes good, peer-reviewed research journals in 15 broad disciplines of the Humanities. ( infrastructures-including-erih/frequently-asked-questions.html) The 15 fields are: Anthropology (Evolutionary); Anthropology (Social); Archaeology; Art, Architectural and Design History; Classical Studies; Gender Studies; History and Philosophy of Science; History; Linguistics; Literature; Music and Musicology; Pedagogical and Educational Research; Philosophy; Psychology; Religious Studies and Theology After cleaning, we believe there are 5197 journals in ERIH. 3,942 could be matched to Ulrich’s records and we analyze those.

What problems would there be constructing a database from the ERIH list? Journal status Errors Non-scholarly material included Scientific journals included ReasonERIHNorwegian list Not active & regular (frequency irregular, status ceased, suspended or unresolved) 24%7147%56 Errors in list (old ISSN, ISBN, conf proc.)49%14621%25 Exist, not in Ulrich’s13%4012%14 Science or trade fields, not included in our Ulrich’s data8%2412%14 Corrected with refined data pull4%119%11 Total100%297100%120 Why did journals on lists not match into Ulrich’s dataset?

Norwegian list coverage Ulrich's 12, % Norwegian list 4,494 36% Scopus 4,331 35% WoS 2,366 19% Norwegian list % Scopus % Ulrich's 5, % WoS, 258, 5% Journals counted are: SSH fields published in a European country or U.S. active (not ceased) & published regularly identified as academic/scholarly by Ulrich’s English languageEuropean language, not English

ERIH coverage Ulrich's 5, % ERIH, 1,980 33% Scopus 1,534 26% WoS 1,166 20% Ulrich's 3, % ERIH 1,122 26% WoS 199, 6% Scopus 250 5% English languageEuropean language, not English Journals counted are: ERIH fields active (not ceased) & published regularly identified as academic/scholarly by Ulrich’s published in a European country or U.S. Venn diagrams plotted using: Littlefield & Monroe, Venn Diagram Plotter, US Department of Energy, PNNL, Richland, WA,

Solutions

Norwegian model National research documentation system –Submission Universities submit bibliographic records and are responsible for data quality –Validation & standardization Dynamic authority file of allowed publication channels –External peer review –National scale (<2/3 of authors from 1 institution) –18,000 currently accepted Thomson Reuters and Norwegian national library data imported Authority file standardizes: –Names of publication channels –Document types –Institutional affiliations of authors –Institution names Manual work also needed Recognizes and addresses known accuracy problem in submitted data –Audit of Australian data (Butler & Visser) –Law study (Nederhof et al.)

Counting – simple and transparent All journals assigned to level 1 & 2 in field specific way, points assigned to pub types: The point system is fair to all fields because scholars in each field decide what to assign to level 1&2 Fractional counting (can be debated) Recognizes and addresses known incentive problem –Australian composite index – just counting pubs increases publication in low quality journals (Butler) Norwegian model continued Points by pub typeLevel 1Level 2 Scholarly journal articles - ISSN 13 Book chapter - ISBN0.71 Monograph - ISBN58

Norwegian model commentary Strengths –Fair to all fields in which written word predominates SSH journals and monographs incorporated and weighted –Of the “four literatures” takes care of 3: international journals, books, national journals. –The fourth, “enlightenment” can be incorporated in the model if valued by society. –Non-text output could also be incorporated in this mechanism creative writing publishers, exhibition and performance venues can be catalogued and weighted to assign points Possible issues –Fractional/whole counting can be debated, but that’s a peripheral detail –National journals likely receive low weight, replicating results of citation analysis in WoS or Scopus –Cost – full cost includes that born by universities in submission and by agency in validation –Difficulty reaching international consensus on journal rankings –No impact measures – i.e. citations, this means simplicity, but also can be seen as a limitation

Recommendations All countries implement the Norwegian model – resulting national evaluation databases are linked together –institutional submission & agency authority file, standardization and validation –ERIH could be used –Book publishers can be categorized and included –Non-text material - extend the principle of peer listing and ranking of venues/vehicles to: Dance & play performance venues Publishers of creative writing Art and craft exhibition venues building or manufacture for architects and designers Establish a book database comparable to the journal article database Support small European SSH scholarly journal publishers to enable their journals to be put online in a central infrastructure to be commissioned by ESF.

SSH online journal infrastructure recommendation ESF builds and maintains an electronic full text SSH journal infrastructure for European SSH –Metadata fields built (author, institution, journal name etc.) –Electronic full text of all articles –Provides access to read 1 page at a time with no saving or printing allowed. Sell articles cheaply for saving and download with revenue returned to publishers –The US National Research Council uses this model with its reports. –6 month delay is used in PubMed to protect publishers subscriptions. But immediacy is of little consequence in humanities, PubMed idea must be modified Requires a peer review process of journals to establish the list –Like ERIH, but with tougher standards to keep costs under control Advantages –Public money goes to small European publishers Used to overcome the obstacle to accessibility posed by fragmented publishing industryfragmented publishing industry –Makes scholarship widely available worldwide Because any full text electronic resource will be indexed by Google scholar, which also enables automatic page translation –Databases are enhanced, WoS and Scopus index the journals Because infrastructure includes article metadata and passes on a clean file to db’s every month Problems –May not be needed with WoS and Scopus expanding, further analysis needed.

Summary Norwegian model extended SSH electronic journal infrastructure for European SSH It will always be possible to criticize coverage –Set up rules and aim to include everything that meets the criteria –Must be limited in some way, for example refereed, active, regularly appearing etc.

Problem: fragmented publishing industry Number of journals from publishers of different sizes Publisher size: large – 10 or more small – 2-9 journals in Ulrich’s SSH tiny – 1 Share of journals by publisher size 380 publishers journals Herfindahl index: Journals counted are: SSH active (not ceased) & published regularly identified as refereed by Ulrich’s published in a European country or U.S.