Open Access: Validating Metrics and Motivating Mandates Stevan Harnad, UQAM & U. Southampton Alma Swan, Key Persectives & U Southampton Arthur Sale, U. Tasmania OAR2008

OA Timeline and Milestones Origin of the Universe: 14 billion years Origin of Life on Earth: 4 billion years Origin of our Species: 200,000 years Origin of Language: ,000 years Origin of Writing: 10,000 years Origin of Printing: 500 years Origin of Learned journals: 340 years Origin of Internet: 40 years Origin of Web: 18 years Origin of OAI: 9 years OAR2008

What is OA? Free online access to refereed research articles OAR2008

Gold or Green? OA journals: Gold OA Self-archiving non-OA journal articles: Green OA OAR2008

Access to what? Published peer-reviewed journal articles Unrefereed preprints? Monographs? Data? OAR2008

Why OA? Maximize research uptake, usage and impact Direct benefit: Research progress Side-Benefits: Developing world access, student access, public access OAR2008

How OA? Self-archive in Institutional Repository Institutions and Funders Mandate Self- Archiving OAR2008

Refereed “Post-Print” Accepted, Certified, Published by Journal Impact cycle begins: Research is done Researchers write pre-refereeing “Pre-Print” Submitted to Journal Pre-Print reviewed by Peer Experts – “Peer- Review” Pre-Print revised by article’s Authors Researchers can access the Post-Print if their university has a subscription to the Journal Months New impact cycles: New research builds on existing research OAR2008

Refereed “Post-Print” Accepted, Certified, Published by Journal Impact cycle begins: Research is done Researchers write pre-refereeing “Pre-Print” Submitted to Journal Pre-Print reviewed by Peer Experts – “Peer- Review” Pre-Print revised by article’s Authors Researchers can access the Post-Print if their university has a subscription to the Journal Months New impact cycles: New research builds on existing research This limited subscription-based access can be supplemented by self- archiving the Postprint in the author’s own institutional repository as follows: OAR2008

New impact cycles: New research builds on existing research Researchers can access the Post-Print if their university has a subscription to the Journal Refereed “Post-Print” Accepted, Certified, Published by Journal Impact cycle begins: Research is done Researchers write pre-refereeing “Pre-Print” Submitted to Journal Pre-Print reviewed by Peer Experts – “Peer-Review” Pre-Print revised by article’s Authors Post-Print is self-archived in University’s Eprint Archive Months More impact cycles: OAR2008

What are “metrics”? Metrics are objective measures of research quality and quantity The only alternative to metrics is subjective human judgment (including peer review) “Show me a philosopher who wishes to discard metaphysics and I’ll show you a metaphysician with a rival system” (Show me someone who wishes to discard metrics, and I’ll show you a metrician with rival metrics) OAR2008

Brody et al (2007) Incentivizing the Open Access Research Web: Publication-, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3). Open Access: How? By mandating Green OA Self-Archiving OA Metrics motivate OA Mandates And OA Mandates maximize OA Metrics OAR2008

“Online or Invisible?” (Lawrence 2001) “average of 336% more citations to online articles compared to offline articles published in the same venue” Lawrence, S. (2001) Free online availability substantially increases a paper's impact Nature 411 (6837): OAR2008

Lawrence (2001) findings for computer science conference papers. More OA every year for all citation levels; higher with higher citation levels OAR2008

EA: Early Advantage QA: Quality Advantage (Seglen 80/20 effect) UA: Usage Advantage (CA: Competitive Advantage) (QB: Quality Bias) Contributors to the OA Advantage EA + QA + UA + (CA) + (QB) OAR2008

Early Access Advantage: OA is accelerating the research access/usage/citation cycle. OA articles are being cited sooner and sooner (Data from Physics Arxiv) OAR2008

Quality Advantage: Higher quality articles have greater OA Advantage

Brody, T., Harnad, S. and Carr, L. (2006) Earlier Web Usage Statistics as Predictors of Later Citation Impact. Journal of the American Association for Information Science and Technology (JASIST) 57(8): Data from arXiv Downloads (“hits”) in the first 6 months correlate with citations 2 years later Most articles are not cited at all Usage Advantage + Early Advantage: OA Articles are downloaded more and early downloads lead to later citations

Time-Course and cycle of Citations (red) and Usage (hits, green) Witten, Edward (1998) String Theory and Noncommutative Geometry Adv. Theor. Math. Phys. 2 : Preprint or Postprint appears. 2. It is downloaded (and sometimes read). 3. Next, citations may follow (for more important papers)… 4. This generates more downloads…. 5. More citations... OAR2008

(Competitive Advantage): The earlier you mandate Green OA, the sooner (and bigger) your university's competitive advantage: U. Southampton School of Electronics and Computer Science was the first in the world to adopt an OA self-archiving mandate. (Competitive Advantage vanishes at 100% OA.)

OAR2008 (Quality or Self-Selection Bias ): Better authors are more likely to self-archive, and better articles are more likely to be self-archived. (Michael Kurtz considers this in itself a sufficient rationale for self-archiving!) (Quality Bias vanishes at 100% OA.) The data below are systematically ambiguous, because they could arise from either Quality Advantage or Quality Bias

OAR2008 Some have argued that the OA Advantage might be all or mostly Just Quality (Self-Selection) Bias. So we tested this, by comparing self-selected with mandated OA:

Percentage of OA articles among the ISI-indexed articles, per institution and per year:

Ø: non-Open Access, O: Open Access, M: Mandated, N: non-Mandated Results: Five of the seven effects are statistically significant: O > Ø, OM > ON, (ON=ØN), OM > ØM, OM > Ø (ON=Ø), OM > ØN i.e., OA is more cited than non-OA, and mandated OA is cited more, not less than non-mandated OA. (The growth of the OA Advantage with more recent years may be because mandate compliance is more immediate, whereas for earlier years the deposit was made retroactively.)

OAR2008 Green OA Mandates 1: Alma Swan’s International, Multidisciplinary Survey Predictions About Researcher Compliance:

OA Mandates: Across all countries and disciplines, 95% of researchers report that they would comply with a self-archiving mandate from their funders and/or employers, and over 80% report that they would do so willingly. -- But only 15% self-archive spontaneously, if it not mandated.

OAR2008 Green OA Mandates 2: Arthur Sale’s Australian Data on Actual Researcher Compliance:

Data courtesy of Arthur Sale University of Tasmania +Repository -Incentive -Mandate Green line: total annual output Red line: proportion self-archived OAR2008

University of Queensland +Repository +Incentive -Mandate Green line: total annual output Red line: proportion self-archived Data courtesy of Arthur Sale

Queensland University of Technology +Repository +Incentive +Mandate Green line: total annual output Red line: proportion self-archived Data courtesy of Arthur Sale OAR2008

All European Universities should create institutional repositories and should mandate that all research publications must be deposited in them immediately upon publication (and made Open Access as soon as possible thereafter) as already mandated by RCUK, ERC, and NIH, and as recommended by EURAB. In addition, the EUA recommends that these (funder) self-archiving mandates should also be extended to all research results arising from EU research programme/project funding. Unanimous Recommendation by EUA, Jan universities in 46 countries OAR2008

NOW 54 & 10!

The majority of journals (63%) already endorse immediate Green Open Access Self-Archiving of the postprint ROMEO/EPRINTS (Directory of Journal Policies on author OA Self- Archiving): NOW 63% & 95% OAR2008

For the articles in the 37% of journals that have an embargo policy, the free EPrints institutional Repository-creating software has an ” Eprint Request" Button: The user who reaches the metadata for a Closed Access article puts his in a box and clicks. This sends an automatic to the author, with a URL on which the author clicks to automatically the eprint to the requester. OAR2008

Once the ID/OA mandates are universally adopted, the embargoes will soon become obsolete, under growing OA pressure worldwide. Carr & Harnad (2005) Keystroke Economy: A Study of the Time and Effort Involved in Self-Archiving. Economy: A Study of the Time and Effort Involved in Self-Archiving OAR2008

The free EPrints University Repository Software generates rich (and potentially even richer) usage metrics. It can be used for showcasing, navigating, comparing and assessing. Here is a sample of University Repository usage metrics for Southampton author Tim Berners-Lee: OAR2008

Interoperable Repository Statistics OAR2008

Interoperable Repository Statistics

Some EPrints download metrics for top deposits by Southampton author Tim Berners-Lee. OAR2008

These Local Repository Usage metrics at the individual university level can then be complemented by CITEBASE, which provides global Citation, Download, Co-citation, Hub/Authority, growth, and other metrics: OAR2008

Sample citation and download growth with time. (Downloads only start in 2005 because that is when this paper was deposited.) Early growth rate and late decay metrics for downloads and citations can also be derived.

Sample of candidate OA-era metrics: Citations (C) CiteRank Co-citations Downloads (D) C/D Correlations Hub/Authority index Chronometrics: Latency/Longevity Endogamy/Exogamy Book citation index Research funding Students Prizes h-index Co-authorships Number of articles Number of publishing years Semiometrics (latent semantic indexing, text overlap, etc.) OAR2008

These metrics can be validated in the UK Research Assessment Exercise (RAE), discipline by discipline, through multiple regression analysis: The metrics can be weighted by their ability to predict the rankings given by the evaluation by human peer panels: OAR2008

UK’s RAE 2008 will be a parallel panel/metric exercise, making it possible to develop a rich spectrum of candidate metrics and to validate each metric against the panel rankings, discipline by discipline, through multiple regression analysis, determining and calibrating the (“beta”) weights on each metric. Harnad, S. (2007) Open Access Scientometrics and the UK Research Assessment Exercise. Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1) : 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. Access Scientometrics and the UK Research Assessment Exercise OAR2008

RAE 2001 Rankings for Psychology OAR2008

Research Assessment, Research Funding, and Citation Impact “Correlation between RAE ratings and mean departmental citations (1996) (2001) (Psychology)” “RAE and citation counting measure broadly the same thing” “Citation counting is both more cost- effective and more transparent” (Eysenck & Smith 2002) OAR2008

Here is how ordinary regression (2-variable correlation works): If the correlation between Height (H) and Weight (W) is 0.6 Then if you ‘normalize’ H z and W z to the same scale, you can predict Weight from Height: predicted W z = 0.6(H z ) + unpredictable residual In multiple regression, you can improve the prediction (and reduce the unpredicatble residual) by adding more Predictor variables (e.g., gender, age, parents’ weight, activity level, etc.) in which case the equation becomes a series of predictor variables P 1, P 2 … etc. with “beta weights” b1, b 2 … All predicting a criterion variable (C): C= b 1 P 1 + b 2 P 2 +b 3 P 3 ….+b n P n + unpredictable residual OAR2008

In scientometrics, the metrics (M) are the predictors: b 1 M 1 + b 2 M 2 +b 3 M 3 ….+b n M n + unpredictable residual But what is the criterion? The natural criterion against which to validate metrics, thereby initializing the weights, is peer review! This has to be done separately, discipline by discipline so as to be sure to compare only like with like. The UK’s RAE 2008 is a unique historic opportunity to validate A rich battery of candidate metrics against the peer-panel rankings for each department of each UK university. OAR2008

Sample multiple regression analysis. The criterion here is not RAE peer rankings, but article citations, and the 7 predictors are (1) article age, (2) journal impact factor, (3) number of references, (4) number of authors, (5) science/nonscience, (6) OA/nonOA, (7) mandated/nonmandated. Note how the regression weights (in the graph) show the size Of the contribution of each predictor to the criterion (citations). Note that OA emerges as a significant contributor to citation Counts even when the other 6 variables are factored out.

4 Multiple regression analysis for the 4 journal impact factor (JIF) Quartiles, for Age, (JIF), References, Science, OA, and Mandated (OA Advantage only siginificant in to 25%). Criterion that is being predicted is number of citations.

Multiple regression analyses within each citation bracket (0, 1, 2…. 20+): OA effect grows bigger in higher citation brackets

OAR2008 Conclusions: Use RAE 3008 peer rankings to validate metrics. Use OA metrics as the incentive to motivate OA mandates Use OA mandates to generate OA content, maximising the research impact at the same time as measuring it

Author’s URLs (UQAM & Southampton): BIBLIOGRAPHY ON OA IMACT ADVANTAGE: BOAI Self-Archiving FAQ: CITEBASE (scientometric engine): EPRINTS: OA ARCHIVANGELISM: ROAR (Registry of OA Repositories): ROARMAP (Registry of OA Repository Mandates): ROMEO/EPRINTS (Directory of Journal Policies on author OA Self-Archiving): Вот - некоторые полезные вебсайты OAR2008

1995: Universal FTP Archives for Esoteric Science and Scholarship: A Subversive Proposal In: Scholarly Journals at the Crossroads. ARL : Research access, impact and assessment THES The Self-Archiving Initiative Nature Measuring and Maximising UK Research Impact THES Mandated online RAE CVs Linked to University Eprint Archives. Ariadne : Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals & Brody D-Lib The Access/Impact Problem and the Green and Gold Roads to Open Access. et al Nature Web Focus : Journal publishing and author self-archiving: Peaceful Co-Existence Berners-Lee et al Keystroke Economy: A Study of the Time and Effort Involved in Self-Archiving. Carr & Harnad Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and Research Citation Impact. Hajjem et al IEEE Data Engineering Bulletin 28 Making the case for web-based self-archiving Research Money : Self-archiving should be mandatory 2006 Research Information The Open Research Web: A Preview of the Optimal and the Inevitable Shadbolt et al in Open Access: Key Strategic, Technical and Economic Aspects : Open Access Scientometrics and the UK Research Assessment Exercise Proc 11th Ann Mtg Int Soc Scientometrics and Informetrics 11: Time to Convert to Metrics Brody et al Research Fortnight 17 Incentivizing the Open Access Research Web: Publication-, Data-Archiving and Scientometrics. Brody et al CTWatch Quarterly 3(3). OAR2008