Measuring Scholarly and Public Impact: Let’s Talk Metrics Rachel Borchardt Monday, May 16th, 2017 borchard@American.edu Conference for High-Impact Research

Why metrics? Quantifiable measures of research impact Differentiate articles, journals, authors, institutions Imperfect calculations of a fuzzy concept

How are metrics used? Evaluate researchers: Hiring Tenure/promotion Merit awards Grant funding Benchmark programs and institutions Image from AU Faculty Manual, http://www.american.edu/loader.cfm?csModule=security/getfile&pageid=4304081

What are metrics Traditional Bibliometrics Based on citation counts Emerging Altmetrics Based on online tools Image from Plum Analytics

Categories of bibliometrics Article-level Journal-level Author-level Institutional-level

Traditional bibliometrics for journals Impact Factor and Citation Counts, created to measure Journals and journal articles Scholarly (journal) impact Initially created for librarians, then largely adopted by STEM Image from Journal Citation Reports (library database)

Some Impact Factor limitations Retrospective measurement Narrow measurement Confined by what is indexed as a citation Can’t be compared between disciplines Other known issues: language, co-authorship, disproportionate article citation impact Image from Essential Science Indicators (library database)

One way to correct – disciplinary context Image from Journal Citation Reports (library database)

Contextual limitations – “Political Science” category List from http://www.auspsa.org.au/publications

Other major bibliometrics Metrics from Web of Science: Eigenfactor, Immediacy Index, Cited Half-Life Metrics from Elsevier/Scopus: SJR, SNIP, IPP, CiteScore Metrics from Google Scholar: H5-index, H5-median Image from http://www.journalmetrics.com

SJR comparison – “Political Science and International Relations”

H5-index comparison – Political Science List from Google Scholar Metrics

Another alternative – citation distributions http://www.nature.com/news/beat-it-impact-factor-publishing-elite-turns-against-controversial-metric-1.20224

Why altmetrics? Measures broader scholarly outputs Books, presentations, data sets, software code, and more More measures of scholarship Views, downloads, shares, mentions in blogs, policy papers, and more More immediate metrics Measures beyond scholarly impact Policy, public, and more https://becker.wustl.edu/impact-assessment

What are altmetrics? Definition: The creation and study of new metrics based on the social web for analyzing and informing scholarship. Altmetrics.org My definition: The umbrella classification of non-citation based metrics. http://altmetrics.org/manifesto ; http://twitter.com/jasonpriem

Categories of altmetrics Article-level [individual work] Journal-level Author-level Institutional-level

Where do altmetrics come from?

How do we collect altmetrics? Directly from the individual tools From publishers (views, download data) From (some) library databases From scholarly networks Through aggregating tools Slideshare views PLOS article metrics Web of Science usage ResearchGate metrics Altmetric metrics

“Impact” and metrics Inherently fuzzy and subjective Attention Engagement Some correlate to bibliometrics; some don’t

Deciphering impact - ImpactStory

ImpactStory contextual elements

Deciphering impact from tools - Altmetric

Altmetric contextual elements https://www.altmetric.com/details/585373

Impact correlations https://arxiv.org/abs/1203.4745; http://onlinelibrary.wiley.com/doi/10.1002/asi.23309/full

Discipline-specific impact https://www.researchtrends.com/issue-30-september-2012/the-use-of-big-datasets-in-bibliometric-research/

Limitations of impact: example https://www.altmetric.com/details/2313862/policy-documents https://www.healthypeople.gov/2020/topics- objectives/topic/Adolescent-Health https://www.google.com/search?q=%22Trends+in+adolescent+and +young+adult+health+in+the+United+States%22+site%3A.gov&oq= %22Trends+in+adolescent+and+young+adult+health+in+the+United +States%22+site%3A.gov&aqs=chrome..69i57.3775j0j8&sourceid=chr ome&ie=UTF-8

Current uses of altmetrics External Applying for funding, promotion/tenure Showcasing achievements Internal Better understand engagement with research Who, why, where

More on grant funders https://www.nature.com/naturejobs/science/articles/10.1038/nj7471-397a

Grant funders: NSF Broader Impacts https://www.nsf.gov/od/oia/special/broaderimpacts/

Grant funders: Wellcome Trust http://www.hefce.ac.uk/pubs/rereports/year/2015/metrictide/

Putting it all together - suggestions Develop personal impact goals Discipline-specific, research-specific, university-specific, etc. Demonstrate achievement of those goals Tell your story Bibliometrics, altmetrics, and qualitative measures Complementary roles Track early and often!

Getting help Rachel Borchardt borchard@american.edu 202-885-3657 (I work year-round!) Metrics guide, http://tinyurl.com/AUmetrics Borrow a copy of my book

Questions?