1. Scholarly Communications Bradley Hemminger Assistant (Associate) Professor School of Information and Library Science University of North Carolina at Chapel Hill bmh@ils.unc.edu

2. Big Questions What is scholarship? How do we measure the value of scholarship? What should we preserve? How should we design an ideal model, given that technology has freed us from the constraints of physical mediums?

3. Public Storehouse of Knowledge Multiple open digital archives, holding all the world’s knowledge. A single logical universal archive, created by dynamic federation of all public archives. Contains everything: archive holds grey literature (publicly deposited) and gold literature (refereed articles). No barriers to access. Knowledge is freely available to anyone, any time, anywhere. Access to information and knowledge correlates to society’s quality of life.

4. Archive Model (NeoRef) All material and metadata are author contributed to a public OAI archive (author retains ownership). OAI archives have automated or manual moderator to filter out “junk”. Everything--articles, reviews, comments, indexings, etc., are stored as digital content items on archive using the same mechanism. Reviews contain quantitative score, qualitative grade, qualitative comments. All materials universally available via search engines that harvest metadata from OAI archives. Retrieval is through Google like one stop shopping search interface, with dynamic filtering based on metadata and reviews to limit hits to manageable number to review.

5. Challenges are in Retrieval All material is archived (good and bad) Metrics (some new) are used to differentiate type, content, and quality. Dynamic Searching allows quickly finding material of most interest. Search on –Type article=Review AND date > 1950 –Content (schizophrenia AND GeneX) –Quality: Peer reviewed {journals}, citation rate > XYZ

6. How Peer Review might work… Author submits article to her institution’s open archive (DOI uncch:sils/0007548.pdf). Author “submits” to journal EMEDICINE by providing DOI of article. Journal Editor schedules two reviewers. Reviewers review article, and submit their reviews (cornell:0191.pdf, ucb:0084.pdf). Author revises, and places revised article (DOI uncch:sils/0007957.pdf) on archive, and submits this final version to EMEDICINE. Journal submits review (EMED:0023424.pdf) which is final statement from journal (editor), and indicates acceptance of uncch:sils/0007957.pdf as EMED article).

7. Scholarly Communications Process Today’s Example Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to update analysis V7 Revision to include additional new results V8

8. Scholarly Communications Process: What’s Captured Today Journal Final Revision V6 Only one version is captured, and the same community then pays to buy back access to article

9. Scholarly Communications Process Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to correct analysis V7 Revision to include additional new results V8 formulatediscussiondiscussion, revision Two peer reviews Copyproofing Criticisms, new thoughts, revision new results, revision comments Author revision

10. Scholarly Communications Process:What I’d like to see saved! Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to correct analysis V7 Revision to include additional new results V8 formulatediscussiondiscussion, revision Two peer reviews Copyproofing Criticisms, new thoughts, revision new results, revision comments Author revision

11. Change the Process! Think of scholarly communication as continuous process instead of single product (journal publication). Capture significant changes/versions of a work. Include all criticisms and comments about work (all stages). Support normal scholarly discourse, including authors responses as well as others comments. Add reviewer’s quantitative rating of material to allow better filtering based on absolute quality metric during retrieval. Add machine (automated) reviews. Represent and store more than just text (datasets, stats, multimedia, etc). Capture over time (Memento).

12. Challenges Searching –DC metadata to allow coarse discovery. –Specialized searching within domain after locating material (based on metadata field indicating appropriate search interface). –Interactive searching to allow refinement to most desirable set within a few seconds. Use reviews to help filter search (Facultyof1000).Facultyof1000 –Google searching on full text (covers all materials, but generates large number of hits, lower specificity). –Automated agents to bring material of interest to your attention (California digital library).California digital library Example: article scores > 7.0, refereed, citation count above 10, type=research article, search terms = schizophrenia, geneX)

13. Challenges Knowledge Representation –Extend DC to include “concepts” and “claims” to allow higher level searching compared to simple indexing. –Make OAI and DC representation more robust by always supporting DOI to uniquely identify materials. –Support unique identification of authors as well. –Making all content items submitted permanent –Use DC fields to link related items, new version of paper to old version.

14. Challenges Rights Administration –Support mechanisms to allow authors to set permissions as they desire, and enforce this. –Example is OAI recent support of rights administration using Creative Commons.

15. Challenges Peer Review –Make more public. Make available comments on articles. –Add quantitative scoring as well as qualitative.

16. Overview of Peer Review (talk about single repository, PLoS One) Review Peer, Open, Machine Accept, reject, revise with respect to XYZ standards Comments to Author Published Article Article submitted Send elsewhere Filter Reject Quantitative Grade Score (1-10) Qualitative Quantitative

17. General Review Model Parallels In general, you have sample (material) which is judged/scored quantitatively and qualitatively by an identified observer with respect to some standard.

18. NeoRef for Movies, Products,… The same process used by NeoRef to support Scholarly Communication could be used for most any communication of information purpose. All that is required is storage of Digital Content Items, and linking of reviews, comments, etc to them. DocSouth: self cataloged and indexed items are Grey; librarian/archivist cataloged and indexed items are Gold. Movies: Grey is everyone’s reviews; Gold is Siskel and Ebert reviews. Consumer Products: product reviews by Consumer Reports (gold), user reviews (grey).

19. Current Peer Review Options Human Judgement –Expert peer review (status quo) –Certified expert peer review –Open Peer Review BMJ, BioMedBMJ –Open comment review psycprintspsycprints Computer Judgement –Computer peer reviewComputer peer review Human Usage –Citation-based (CiteSeer)CiteSeer –Usage counts (CiteSeer) ExampleCiteSeerExample –Quantity of discussion Coarse Categorization –Two Tier (grey/gold) –Moderator (current arXiv)arXiv –No review (old arXiv) Quantitative √ Qualitative √ (relative) √ (absolute) √

20. What do users want? ALPSP survey was intended to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and with almost 9% responding. Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned and Professional Society Publishers, 2002 ). http://www.alpsp.org/pub5.htm

21. Importance of journal features

22. Importance of the peer review process

23. Additional Surveys (Project Romeo)Project Romeo Authors want quick and convenient dissemination of their work –Free access to others papers –Not overly concerned about copyright unless it stops them from freely distributing their work or accessing others.

24. Additional Surveys (Zhang 1999 survey) Permanence and Quality of electronically archived resources Better (faster, more accurate) searching capabilities

25. Change the Mindest: Open Commons Open Everything –Journal Articles –Datasets –Scholarly resources (think humanities, papers,maps,images, films, artwork, etc) –Reviews/comments/annotations What are the ramifications?

26. Who Holds the Materials? Publishers Universities Libraries Large private companies (like Google or Microsoft Health) Government (NLM) Ramifications?

27. What does search look like? Real time dynamic search –Text search (Google) –Faceted search (commercial webpages) –Clustering of results Main thing we’re missing is utilizing the metadata in search algorithms.

28. How do we measure value of scholarship? What is impact? For example, look at how research councils in countries like Great Britain and Sweden are trying to code this (formula for evaluating individuals and departments academic value, to tie to funding). Would this change in “open everything world”? If so, how?

29. Think bigger We are concerned because of tenure and promotion of academics in Research I institutions.

30. Think bigger We are concerned because of tenure and promotion of academics in research one institutions. There are many kinds of universities, many don’t value original research.

31. Think bigger We are concerned because of tenure and promotion of academics in research one institutions. There are many kinds of universities, many don’t value original research. Scholarship isn’t limited to universities, certainly not physical ones. –University of Phoenix –Virtual colleges –Interdisciplinary, inter-institution collaborations

32. Think bigger We are concerned because of tenure and promotion of academics in research one institutions. There are many kinds of universities, many don’t value original research. Scholarship isn’t limited to universities, certainly not physical ones. –University of Phoenix –Virtual colleges –Interdisciplinary, inter-institution collaborations Scholarship occurs in many forms –Funded patrons (Carnegie,now it’s Gates foundation) –Free tools for Creation, Storage, Dissemination can change what universities are and how teaching occurs (khanacademy, review, MIT Open courseware)khanacademy reviewMIT Open courseware

33. Think bigger We are concerned because of tenure and promotion of academics in research one institutions. There are many kinds of universities, many don’t value original research. Scholarship isn’t limited to universities, certainly not physical ones. –University of Phoenix –Virtual colleges –Interdisciplinary, inter-institution collaborations Scholarship occurs in many forms –Funded patrons (now it’s Gates foundation, Carnegie) –Free tools for Creation, Storage, Dissemination can change what universities are and how teaching occurs (khanacademy, review, MIT Open courseware)khanacademyreviewMIT Open courseware So, in this future, what scholarly value metrics will we design to measure what kinds of value? (i.e. independent of shackles to the current world like tenure and promotion ).

34. Extra Material….

35. Virginia Tech ETD arXix University of California Electronic Repository psycprints UNC Chapel Hill Harvester (NeoRef)NeoRef

36. Importance of publishers’ roles FactorResponses as authorsResponses as readers Peer review8180 Gathering articles together to enable browsing of content 6449 Selection of relevant and quality-controlled content 7154 Content editing and improvement of articles 6039 Language or copy editing5034 Checking of citations/adding links 4628 Marketing (maximising visibility of journal) 4420

37. Can we save the Gold and Grey? Idea V1 Present to colleagues V2 Present at conference V3 Submit to journal V4 Referees Revision for journal V5 Journal Final Revision V6 Revision to correct analysis V7 Revision to include additional new results V8 formulatediscussiondiscussion, revision Two peer reviews Author revision Criticisms, new thoughts, revision new results, revision comments Copyproofing

38. NeoRef Storage Model Conference paper (v3) Comments on V6 Journal Submission V4 Journal Final Revision V6 Revision to include additional results and analyses V8 Auto-indexing Material expressing content Two peer reviews Local powerpoint Presentation v2 Comments on V3 Automated Author Indexing Recognized Expert Open (anyone) Top Tier (Keep Forever) Filter (Moderate) Grey Literature Author Machine Review Content Item

39. Challenges Self Contribution –Author indexed –Author supplied metadata (Dublin Core) –Archive file(s) must be in standard open format NeoRef: PDF/A with DC elements in tags for automatic extraction of metadata

40. Challenges Archive Hosting –Off the shelf computer system with lots of disk space and public domain archiving application (DSpace, Eprints). –Who maintains the material? {Library (MIT DSpace), Grad School, University (California), Publisher (PLoS, BioMedCentral), Society (arXiv)} DSpaceCaliforniaPLoSBioMedCentralarXiv –Where are comments and reviews held (after the fact content items that reference original)?

41. Challenges Make content universally available –Export OAI items so they can be harvested –Have public domain quality harvesters that support quick and simple searching (i.e. Google for metadata).

42. http://www.update-software.com/Cochrane/MR000016.pdf

43. The Association of Learned and Professional Society Publishers (ALPSP) Survey Authors and Electronic Publishing Scholarly research communication has seen far-reaching developments in recent years. Most journals are now available online as well as in print, and numerous electronic-only journals have been launched; The Internet opens up new ways for journals to operate. Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.

44. Importance of future dissemination channels Dissemination methodVery important plus important categories Ranking Traditional print + electronic journal911 Discipline-based electronic reprint archive782 Traditional print journal773 Traditional electronic-only journal664 Institution-based electronic reprint archive605 New forms of electronic-only journal496 Discipline-based electronic preprint archive 447 Institution-based electronic preprint archive 338

45. Cochrane Methodology Review Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research. There had never even been any consensus on its aims and that it would be more appropriate to refer to it as ‘competitive review’. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,” BMJ 326 (February 1, 2003): 241 http://bmj.com/cgi/reprint/326/7383/241/a.pdf

46. Cochrane Methodology Review On the basis of the current evidence, ‘the practice of peer review is based on faith in its effects, rather than on facts,' state the authors, who call for large, government funded research programmes to test the effectiveness of the [classic peer review] system and investigate possible alternatives. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,” BMJ 326 (February 1, 2003): 241 http://bmj.com/cgi/reprint/326/7383/241/a.pdf

47. Cochrane Methodology Review The use of peer-review is usually assumed to raise the quality of the end- product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision- making. However, these assumptions have rarely been tested. Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of Biomedical Studies. (Middle Way, Oxford: Update Software Ltd, 2003). http://www.update-software.com/Cochrane/MR000016.pdf

48. Cochrane Methodology Review The available research has not clearly identified or assessed the impact of peer- review on the more important outcomes (importance, usefulness, relevance, and quality of published reports) … [G]iven the widespread use of peer- review and its importance, it is surprising that so little is known of its effects Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of Biomedical Studies. (Middle Way, Oxford: Update Software Ltd, 2003). http://www.update-software.com/Cochrane/MR000016.pdf

49. FURTHERMORE … 16% said that the referees would no longer be anonymous 27% said that traditional peer review would be supplemented by post- publication commentary 45% expected to see some changes in the peer-review system within the next five years Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258. Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdfhttp://www.jisc.ac.uk/uploaded_documents/rowland.pdf

50. Provider Service Change What is worth paying for? –Quality review (Faculty of 1000)Faculty of 1000 –Proofing, citation linking, professional presentation (CiteSeer, Cite-base)CiteSeerCite-base –Marketing –Archival (JStor)JStor Who hosts material: –Society (arXiv)arXiv –Commerical Publishers (Elesiever,BioMedCentral)BioMedCentral –University Library (MIT Dspace)Dspace