1. Symposium on The Impact of Linguistics Journal Rankings and Citations Organized by members of the Committee of Editors of Linguistics Journals

2. Outline Brian Joseph (The Ohio State University): The editor’s perspective Tim Stowell (University of California, Los Angeles): The administrator’s perspective John Cullars (University of Illinois at Chicago): The bibliometrician’s perspective Marian Hollingsworth (Thompson Scientific): The industry perspective Keren Rice (University of Toronto): Summation Martha Ratliff (Wayne State University), Discussant

3. Rossner, M. et al. 2007. Show me the data. J. of Cell Biology. … Impact factor data … have a strong influence on the scientific community, affecting decisions on where to publish, whom to promote or hire, the success of grant applications, and even salary bonuses. Yet, members of the community seem to have little understanding of how impact factors are determined ….

4. What kind of material is there? Bare lists of journals implicitly or explicitly deemed ‘major’: DeMiller, Anna L. 2000. Linguistics. A guide to the reference literature (2nd edn.). Englewood: Libraries Unlimited. (on-line)on-line Citation indexes: Thomson Scientific, Web of Science, “impact factor” www.journal-ranking.com (on-line, interactive) www.journal-ranking.com Other efforts to evaluate, rate or rank: European Science Foundation, “European Reference Index for the Humanities”, here. [A, B, C assigned to each journal.]here Rolf A. Zwaan & Anton J. Nederhof. 1990. Some aspects of scholarly communication in linguistics: An empirical study. Language 66.3.553-57. [Survey data from Dutch linguists.] Other academic bibliometric work like Georgas and Cullars.

5. The citations approach “ Citations can be counted, and one way of measuring the quality of a publication to assess whether it appears in a high-citation outlet. … The chain of inference … runs as follows. The more papers in the journal are cited, the more impact that journal has. The more impact a journal has, the more authors will want to publish in that journal. …

6. Continued … “The more authors who want to publish in the journal, the more demanding will be the selection criteria applied in the refereeing process. The more demanding the selection criteria applied in the refereeing process, the better the average paper will be. The better the average paper in the journal, the more it will be cited. And so a virtuous circle is completed.” —British Academy Report

7. An alternative “The ERIH lists will help identify excellence in Humanities scholarship and should prove useful for the aggregate benchmarking of national research systems, for example, in determining the international standing of the research activity carried out in a given field in a particular country. As they stand, the lists are not a bibliometric tool. The ERIH Steering Committee and the Expert Panels therefore advise against using the lists as the only basis for assessment.”

8. Thomson journal-ranking.com

9. Reactions & meta-evaluation Wouter Gerritsma, Citation analysis for research evaluation. [here]here Equivalence of results from two citation analyses: Thomson ISI’s Citation Index and Google’s Scholar service. Pauly & Stergiou. Inter-Research 2005. (int-res.com) Various (negative) editorials in Nature. Peer Review: the challenges for the humanities and social sciences. A British Academy Report, Sept. 2007. Show me the data. Mike Rossner, Heather Van Epps, and Emma Hill. Journal of Cell Biology 2007. Chronicle of Higher Education, 10 October 2008: “New Ratings of Humanities Journals Do More Than Rank — They Rankle” (and earlier articles there.) Chronicle of Higher Education

10. Evaluitis — Eine neue Krankheit Bruno S. Frey, University of Zurich, 2006 "Evaluitis" - i.e. ex post assessments of organizations and persons - has become a rapidly spreading disease. In addition to the well-known costs imposed on evaluees and evaluators, additional significant costs are commonly disregarded: incentives are distorted, ossification is induced and the decision approach is wrongly conceived. As a result, evaluations are used too often and too intensively.

11. But really, a good system could … encourage rigor, establish standards, and set a tone.

12. More fundamental problems Limited or problematic data and measures Poorly defined standards Worse: Results being put to uses that weren’t intended or aren’t justified?

13. An example: Rejection rates No standard and no way to verify, yet numbers are widely quoted. Two dramatically different views: #1: Every editorial decision = submission. #2: Don’t count submissions not sent out for review, or revisions of earlier submissions. We (ELJ) are working to establish a norm for linguistics.

14. Time is tight, so let’s get going … These slides are available at … http://groups.google.com/group/editors-of-linguistics-journalshttp://groups.google.com/group/editors-of-linguistics-journals. We plan to produce and post a white paper based on this session … stay tuned.

15. Our original questions 1. Philosophical Why is ‘objective’ ranking of journals considered important? Why have organizations like Thomson ISI and Google Scholar become important within academia? What are the arguments for and against rankings and impact factors? 2. Practical Can measurement of journal impact and rank be done objectively? Is this being done within linguistics in ways that are valuable? Is this being done within linguistics in ways that might threaten the vitality of research and of journals? 3. Professional What role does the assessment of journal impact and rank actually play in grantsmanship, promotion, tenure and hiring decisions, library acquisitions, etc.?