Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill

Scholarly Communications Process Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to update analysis V7 Revision to include additional new results V8

Scholarly Communications Process Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to correct analysis V7 Revision to include additional new results V8 formulatediscussiondiscussion, revision Two peer reviews Copyproofing Criticisms, new thoughts, revision new results, revision comments Author revision

Scholarly Communications Process: What’s Produced Journal Final Revision V6

Scholarly Communications Process:What I’d like to see saved! Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to correct analysis V7 Revision to include additional new results V8 formulatediscussiondiscussion, revision Two peer reviews Copyproofing Criticisms, new thoughts, revision new results, revision comments Author revision

Peer Review Output Review (Peer) With Respect to XYZ… Accept reject revise Comments to Author Qualitative Grade Qualitative Comments Article

Generalized Review Model Review (open, peer, machine) Accept, Reject, Revise, With respect to XYZ Comments to Author Qualitative Grade Quantitative Grades Score (1-10) Qualitative Comments Article

Overview of Peer Review Review Peer, Open, Machine Accept, reject, revise with respect to XYZ standards Comments to Author Qualitative Comments Quantitative Grade Published Article Article submitted Send elsewhere Filter Reject Score (1-10) Qualitative Grade

General Review Model Parallels In general, you have sample (material) which is judged/scored quantitatively and qualitatively by an identified observer. Current Peer Review Automated Scoring Systems (lab tests) Moderated lists (announce) Moderated Eprints servers (arXiv)

Peer Review Options Human Judgement –Expert peer review (status quo) –Certified expert peer review –Open Peer Review BMJ, BioMedBMJ –Open comment review pyscprintspyscprints Computer Judgement –Computer peer reviewComputer peer review Human Usage –Citation-based (CiteSeer)CiteSeer –Usage counts (CiteSeer) ExampleCiteSeerExample –Quantity of discussion Coarse Categorization –Two Tier (grey/gold) –Moderator (current arXiv)arXiv –No review (old arXiv) Quantitative Score (1-10) #citations #hits #number of related discussions Qualitative Rel Yes/No Group Absolute Rel Yes/No Group YYYYY?YYYYY?

Judgment based on some combination of reviews/comments Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to correct analysis V7 Revision to include additional new results V8 formulatediscussiondiscussion, revision Two peer reviews Copyproofing Criticisms, new thoughts, revision new results, revision comments Author revision

What areas of improvement? Review Process Change Search, Retrieval Process Change Service Provider Process Change

Review Process Changes Include open reviews and comments to get additional feedback. Support normal scholarly discourse, allowing give and take with author responding. Add quantitative scores to allow better filtering based on quality during retrieval Add machine (automated) reviews

Search and Retrieval Changes Universal Archive: all material freely available. Universal Searching: standardized metadata (Dublin Core) for general searching.Dublin Core Automated agents to bring material of interest to your attention. Use additional review scores (public reviews, machine) to help filter search. Example: article scores > 7.0, refereed, citation count above X, type=research article, search terms = schizophrenia, geneX)

Provider Service Change What is worth paying for? –Quality review (Faculty of 1000)Faculty of 1000 –Proofing, citation linking, professional presentation (CiteSeer, Cite-base)CiteSeerCite-base –Archival (JStor)JStor Who hosts material: –Society (arXiv)arXiv –Commerical Publishers (Elesiever,BioMedCentral)BioMedCentral –University Library (MIT Dspace)Dspace

New Frameworks for Peer Review As an enabling technology: frameworks like NeoRef supports all of the above models in any combination at the same time, while eliminating many of the costs. Requirements: Based on OAI and Dublin Core, and expectation of logical universal archive, an universal unique object IDs (URL, DOIs) and person IDs.

Example Model (NeoRef) All material and metadata are author contributed to a public OAI archive (author retains ownership). All materials universally available via search engines that harvest metadata from OAI archives. OAI archives have automated or manual moderator to filter out “junk”. Everything--articles, reviews, comments, indexings, etc., are stored as digital content on archive using the same mechanism. Reviews contain quantitative score, qualitative grade, qualitative comments. Logically (although not physically), a two tier (Grey & Gold) system for materials –High quality keep forever material reviewed by known entity –Grey material (everything else)

NeoRef for Movies, Dates, DocSouth The same process used by NeoRef to support Scholarly Communication could be used for almost any purpose. All that is required is storage of Digital Content Items, and linking of reviews, comments, etc to them. Movies: Grey is everyone’s reviews; Gold is Siskel and Ebert reviews DocSouth: self cataloged and indexed items are Grey; librarian/archivist cataloged and indexed items are Gold.

Can we save the Gold and Grey? Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to correct analysis V7 Revision to include additional new results V8 formulatediscussiondiscussion, revision Two peer reviews Author revision Criticisms, new thoughts, revision new results, revision comments Copyproofing

NeoRef Storage Model Conference paper (v3) Comments on V6 Journal Submission V4 Journal Final Revision V6 Revision to include additional results and analyses V8 Auto-indexing Material expressing content Two peer reviews Local powerpoint Presentation v2 Comments on V3 Automated Author Indexing Recognized Expert Open (anyone) Top Tier (Keep Forever) Filter (Moderate) Grey Literature Author Machine Review

What do users want?

The Association of Learned and Professional Society Publishers (ALPSP) Survey Authors and Electronic Publishing Scholarly research communication has seen far-reaching developments in recent years. Most journals are now available online as well as in print, and numerous electronic-only journals have been launched; the Internet opens up new ways for journals to operate. Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.

ALPSP felt that it would be timely to discover what they actually thought and what they actually did. This survey aimed to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and nearly 9% responded; their detailed comments make thought-provoking reading. Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned and Professional Society Publishers, 2002 ).

Importance of the Peer Review Process

Importance of journal features

Importance of the peer review process

Importance of publishers’ roles FactorResponses as authorsResponses as readers Peer review8180 Gathering articles together to enable browsing of content 6449 Selection of relevant and quality-controlled content 7154 Content editing and improvement of articles 6039 Language or copy editing5034 Checking of citations/adding links 4628 Marketing (maximising visibility of journal) 4420

Importance of future dissemination channels Dissemination methodVery important plus important categories Ranking Traditional print + electronic journal911 Discipline-based electronic reprint archive782 Traditional print journal773 Traditional electronic-only journal664 Institution-based electronic reprint archive605 New forms of electronic-only journal496 Discipline-based electronic preprint archive 447 Institution-based electronic preprint archive 338

Cochrane Methodology Review Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research. There had never even been any consensus on its aims and that it would be more appropriate to refer to it as ‘competitive review’. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,” BMJ 326 (February 1, 2003): 241

Cochrane Methodology Review On the basis of the current evidence, ‘the practice of peer review is based on faith in its effects, rather than on facts,' state the authors, who call for large, government funded research programmes to test the effectiveness of the [classic peer review] system and investigate possible alternatives. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,” BMJ 326 (February 1, 2003): 241

Cochrane Methodology Review The use of peer-review is usually assumed to raise the quality of the end- product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision- making. However, these assumptions have rarely been tested. Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of Biomedical Studies. (Middle Way, Oxford: Update Software Ltd, 2003).

Cochrane Methodology Review The available research has not clearly identified or assessed the impact of peer- review on the more important outcomes (importance, usefulness, relevance, and quality of published reports) … [G]iven the widespread use of peer- review and its importance, it is surprising that so little is known of its effects Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of Biomedical Studies. (Middle Way, Oxford: Update Software Ltd, 2003).

FURTHERMORE … 16% said that the referees would no longer be anonymous 27% said that traditional peer review would be supplemented by post- publication commentary 45% expected to see some changes in the peer-review system within the next five years Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): Report version: