1. RESEARCH DATA ALLIANCE BIBLIOMETRICS FOR DATA SURVEY RESULTS

2. Survey Background  Aimed to investigate what is currently used (if anything) to quantify the impact of data  Organised by the RDA Publishing Data Bibliometrics group

3. Survey Methodology  Survey drafted by the WG  Implemented in SurveyMonkey  Invitations to complete the survey sent out to various email lists and passed on from WG members to other contacts  Preliminary results presented at RDA P4, Amsterdam  Poster presented at FORCE2015, Oxford, Jan 2015

4. 1) Who Responded? n=115 Other Librarian 3 Data Scientist/data manager/data analyst 7 Student/assistant 2 Writer/Editor/publications support 3 Program Manager 1 Computer Scientist 1

5. 2) What is your field of specialty? n=115

6. 3) How long have you been working in this field? n=115

7. 4) What do you currently use to evaluate the impact of data? n=115

8. 4) What do you currently use to evaluate the impact of data? (Other responses)  “Impact factor of journal in which data has been published” / “journal articles”  “figshare stats” / “ResearchGate data”  “Mostly still citations but we are starting to use e.g. altmetrics and twitter reach/impact scores” / “altmetrics impact study being formulated for 2015”  “rich user registration meta data and their access activity”  “Talk to scholars” / “ad hoc feedback” / “informal assessment”

9. 5) If you don't use anything to evaluate the impact of data, why?  33 responses:  Majority opinion: current metrics not good enough, no standards, don’t know what to do  Other opinions: Impact metrics not important for respondee Interest in quantifying impact, but repository/policies still under development Metrics are too easily gamed, or too complicated

10. 6) Are the methods you use to evaluate impact adequate for your needs? n=111

11. 6) Are the methods you use to evaluate impact adequate for your needs? If not, why not?  75 responses:  Lack of tools and standards  Limited data citation  Available measures not good enough. Too difficult/time consuming  Need to focus on other (non-scientific) impacts (e.g. planning/educational use of data)  Hoping increased use of DOIs will help situation

12. 7) Why do you want to evaluate the impact of data?

13. 7) Why do you want to evaluate the impact of data? =>Other responses  30 responses – mostly different  Measure organisational impact  Encourage data openness/publication (and provide benefit for data producers)  Knowledge of who uses data and why – imrpove access for users - inform data retention decisions – improve user experience – justify repository investment – discover impact of data centre policies  Evaluate employees performance/institutional requirement

14. 8) In the future, what would you like to use to evaluate the impact of data?  CITATIONS! (and better tools to track them)  Downloads  Altmetrics / “anything and everything”  Peer review / community feedback  Use outside scholarly literature (e.g., in patents)  Reuse / “actual use”

15. 9) What is currently missing and/or needs to be created for bibliometrics for data to become widely used?  STANDARDS!  Data Citation  Consistent use of PIDs / DOIs  Culture Change / “A belief that they are valid.”

16. 10) What difference would it make to your work to be able to evaluate the impact of data?  91 responses. Common themes:  Promote data sharing/publication/reuse/data stewardship – provide credit for data producers  Justify funding for data activities  Other criteria for evaluation of research impact  Inform and prioritise data access systems – improve services  Influence in public-policy decisions making

17. 11) Do you know of any tools from other research areas that evaluate data impact?  Thomson Reuters DCI / Web of Science  “indicator programs in the UN”  “tools most likely lie in the field of economics and quantitative analysis (the value of decision information)”  “The numerical weather prediction community uses Observing System Simulation Experiments (OSSEs) to evaluate data impact on prediction accuracy.”

18. 12) Is there anything else you’d like to tell us about bibliometrics for data?  www.re3data.org – helps identify and cite datasets www.re3data.org  Metrics can and will be gamed – be warned!  Need for global action across all research field  DOIs and citability of data has motivated data contributions in long tail community  Include PIDs for people and institutions in the conversation  New schemes needed to peer-review data  No one solution  More to measuring impact than just citation  www.esipfed.org – doing good work www.esipfed.org  Time component important for standardising metrics – as more people use them, the notion of what a “good” number in a metric will change  Difference between valid supplementary file and unvalidated open (raw) data  Solutions to these problems should be open (and free)  Useful publications: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0092590; P. Wouters Citation culture; Schwartz, F. W., Fang, Y. C., and Parthasarathy, S., 2005: Patterns of evolution of research strands in the hydrologic sciences, Hydrologeol. J., Vol. 13, pp. 25-36; http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0092590

19. More information? Next Steps?  Collaborate with “Making Data Count” project - https://escholarship.org/uc/item/9kf081vf - working on a tool to deal with downloads and citations https://escholarship.org/uc/item/9kf081vf  NISO working on standards for their Altmetrics project – assessment of non- traditional scholarly outputs (inc. data, software, visualisations) and PIDs related to these. Recruiting WG members. http://www.niso.org/topics/tl/altmetrics_initiative/#phase2 http://www.niso.org/topics/tl/altmetrics_initiative/#phase2  Understanding what we mean by impact?  Look at what data is currently being gathered by repositories and what could be collected – output: best practices and common system to gather data metrics (in a machine actionable way)  Privacy implications – Todd planning a BoF in San Diego  SHARE (http://www.arl.org/focus-areas/shared-access-research-ecosystem- share#.VMZjQ1qUlUM), CHORUS (http://www.chorusaccess.org), OpenAIRE (http://www.openaire.eu) (systems to aggregate articles and scholarly resources from open access repositories and libraries) – interested in gathering usage data on datasets as well?http://www.arl.org/focus-areas/shared-access-research-ecosystem- share#.VMZjQ1qUlUMhttp://www.chorusaccess.orghttp://www.openaire.eu

20. Let’s discuss…