Beethoven vs. Bieber On the meaningfulness of (alt)metrics Blaise Cronin, PhD, DSSc, DLitt (h.c.) Rudy Professor of Information Science, Indiana University,

Slides:



Advertisements
Similar presentations
A Brief Introduction to Altmetrics: Compliment or Replacement? Zohreh Zahedi ARMOOK & Shahid Beheshti University Webinar 12th March 2014.
Advertisements

Challenges of OA in the Next Frontier: ALM & Research Impact Assessment Jennifer Lin Product Manager, PLOS.
Andrea Michalek  Co-founder and CEO #altmetrics.
The complex world of science editing TWELFTH GENERAL ASSEMBLY AND CONFERENCE Split, June 2014 Session: Publishing metrics Metrics, what metrics?
Overview What is ‘Impact’, and how can it be measured? Citation Metrics Usage Metrics Altmetrics Strategies and Considerations.
Bibliometrics Toolkit Google Scholar (GS) is one of three central tools (the others being ISI and Scopus) used to generate bibliometrics for researchers.
Ronald L. Larsen May 22, Trace relationships amongst academic journal citations Determine the popularity and impact of articles, authors, and publications.
Measuring Research: Impact Factors, Citation Metrics, and "Altmetrics" Can Tweets be more important than a good Impact Factor? Thane Chambers Christina.
CURTIN UNIVERSITY LIBRARY Curtin University is a trademark of Curtin University of Technology CRICOS Provider code 00301J July 2014 Tell your impact story:
Article-Level Altmetrics: The Empirical Evidence Mike Thelwall & Kayvan Kousha Statistical Cybermetrics Research Group University of Wolverhampton.
Committed to making the world’s scientific and medical literature a public resource Leading the next frontier of OA: Filtering, Aggregating, and Evaluating.
Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie crc.ebsi.umontreal.ca/sloan.
Using Journal Citation Reports The MyRI Project Team.
What is the impact of the publications read by the different Mendeley users? Could they help to identify alternative types of impact? Zohreh Zahedi, Rodrigo.
Publication and impact in English
Are downloads and readership data a substitute for citations? The case of a scholarly journal? Christian Schlögl Institute of Information Science and Information.
The Web of Science database bibliometrics and alternative metrics
Making an impact ANU Library What is impact What (the heck) are bibliometrics Publish with impact – an overview Debate on impact How innovative are you.
Science Communication & Social Media (part 2) Karen Slides posted here:
Rajesh Singh Deputy Librarian University of Delhi Research Metrics Impact Factor & h-Index.
Making Sense of NTU 1AM: Altmetrics London by Joan WEE Who am I Altmetrics to us What did we do Next Steps.
Deep Impact [Factor] Altmetrics from a PhD perspective Jon Tennant PhD student, Imperial College London Seeking employment.
Rajesh Singh Deputy Librarian University of Delhi Measuring Research Output.
Altmetrics: a primer Where does the data come from? Can it be gamed? Buy in or build your own? Mike Taylor Research Specialist
The Web of Science, Bibliometrics and Scholarly Communication 11 December 2013
LORRIE JOHNSON U.S. DEPARTMENT OF ENERGY OFFICE OF SCIENTIFIC AND TECHNICAL INFORMATION (OSTI) ICSTI TECHNICAL ACTIVITIES COORDINATING (TACC) MEETING OCTOBER.
Altmetrics Gail McMillan, Director CDRS Services.
The Dissemination of Military Suicide Research Understood through Altmetrics Kelly Soberay, MA LPC 1,2, Megan Dwyer, MS 1,2 Jetta Hanson, MA LPC 1,2, Jeremy.
Rome, January 2014 European Commission Information Society and Media General Assembly GALA Game and Learning Alliance The European Network of Excellence.
Beyond the Impact Factor: Altmetrics A look at altmetrics and their growing significance to researchers Hardy Schwamm Research Data and Repository Manager.
Introduction to the Altmetric Explorer Your altmetricexplorer.com Your name Job title.
By Timon Oefelein Springer, Account Development Manager, North Western Europe Altmetrics for Librarians: a publisher dashboard, a university use case.
Altmetrics: a primer Where does the data come from? Can it be gamed? Buy in or build your own? Mike Taylor Research Specialist
CITATION ANALYSIS A Tool for Collection Development and Enhanced Liaison Services Christine Brown and Denis Lacroix.
AltMetrics Bruce Antelman ThinkLoud thinkloud.com Capturing the Buzz.
The Web of Science, Bibliometrics and Scholarly Communication
1 "How do I know thee? Let me count the cites..." Mike Taylor Elsevier Labs
1: From webometrics to altmetrics: one and a half decades of digital research at Wolverhampton Jonathan M Levitt Statistical Cybermetrics Research Group.
DIGGING INTO DATA A presentation to the NSF Cascades, Islands, or Streams? Time, Topic and Scholarly Activities in Humanities and Social Science Research.
Astrophysics publications on arXiv, Scopus and Mendeley – A Case Study Judit Bar-Ilan Department of Information Science, Bar-Ilan University, Israel.
Yvonne Desmond DIT Library Services
Web of Science: The Use & Abuse of Citation Data Mark Robertson & Adam Taves Scott Library Reference Dept.
Scholarly Metrics Tahirah Akbar-Williams, Mark Cyzyk, Robin Sinn The Sheridan Libraries Johns Hopkins University.
Making an impact ANU Library. Topics Research data management Open access Bibliometrics Researcher profiles Where to publish 2.
Altmetrics Sara Mitha and Avenal Finlayson. Altmetrics is … O Hot topic O Big buzz word O Trending.
Altmetrics for the humanities – Disciplines, output types, and discovery in the wild Jeremy L. McLaughlin, San Jose State University & Stacy Konkiel, altmetric.com.
Mark Patterson, Director of Publishing PIRUS 2, London: Feb 23rd, 2011 Article-level metrics at PLoS.
Understanding Altmetrics and Their Impact on Scholarly Communication Yingting Zhang, Medical Information & Education Librarian Minglu Wang, Data Services.
AN INTRODUCTION TO ALTMETRICS Sarah Goodier May 2016.
Altmetrics: Where are we and where are we going?.
Measuring Your Research Impact Citation and Altmetrics Tools University Libraries Search Savvy Seminar Series April 9 & 10, 2014 Prof. Amanda Izenstark.
Measuring Research Impact Using Bibliometrics Constance Wiebrands Manager, Library Services.
Altmetrics CAS Scholarship Day March 24, 2015 Betty Landesman Head of Technical Services and Content Management Langsdale Library.
1 QUICK REFERENCE CARDS FOR RESEARCH IMPACT METRICS.
QuicK Reference Cards for Research Impact Metrics.
Jonathan M Levitt Statistical Cybermetrics Research Group
7 steps to maximise your research profile
Journal of Sports Sciences - Measuring ‘Impact’
Altmetrics What do they measure?
Altmetrics: Analysis of Library and Information Science (LIS) Research in the Social Media Ifeanyi J. Ezema (Ph.D) Paper Presented at the 1st International.
Altmetric Indicators vs. Bibliometric Indicators
Bryan G. Cook, University of Hawaii
Altmetrics 101 LITA Altmetrics & Digital Analytics Webinar
Advanced Scientometrics Workshop
ICT trends in scientometrics and their impact on the academia
Beethoven vs. Bieber On the meaningfulness of (alt)metrics
The quality of scientific periodicals Mozilla Firefox or Google Chrome
Measuring Your Research Impact
Altmetrics: The Practical Implications Michael Taylor Head of Metrics Development Digital
Stay Savvy with Scholarly Communication Altmetrics
Presentation transcript:

Beethoven vs. Bieber On the meaningfulness of (alt)metrics Blaise Cronin, PhD, DSSc, DLitt (h.c.) Rudy Professor of Information Science, Indiana University, USA

Canonicity vs. Iconicity

Twit(ter)

Facebook

Biebermetrics

Facebook

Beethoven vs. Bieber

Apples & Oranges

Apples … and apples

The numbers game ‘Not everything that can be counted counts and not everything that counts can be counted’

Defining excellence ‘But I know it when I see it ….’ U.S. Supreme Court Justice Potter Stewart, 1964 ‘It is easier to measure activities than it is to measure quality’ NISO, 2014

Assessing excellence (Peer review … on a good day) Domain knowledge Expertise Discrimination Connoisseurship Wisdom

A culture of metrics Yesterday (c. 1955): ISI’s Citation indexes (SCI, SSCI A&HCI) Today: WoS, Scopus, Google Scholar, et al. Tomorrow: Social media monitoring & analytics (e.g., altmetric.com)

Measuring article impact Reputation of journal Journal Impact factor No. of citations Quality of citations Persistence of citations Times accessed Times downloaded Included in syllabi Media mentions etc.

Beyond bibliometrics Citations miss important traces/impacts & are lagged Online reference managers, slide-sharing services and social media capture impacts in real- time (but only partially)

Twitter reality a) All scholars b) % with Twitter accounts c) % who tweet d) % of tweets about scholarly topics e) % that are “Twitter citations”/ “tweetations”

Scholarly buzzometer

‘Researchers must ask if altmetrics really reflect impact, or just empty buzz.’ – altmetics manifesto

Citations ≠ Altmetrics Cites  authors Alts  readers Cites  few Alts  many Cites  lagged Alts  real-time Cites  mandatory Alts  optional

Effects of research Immediate vs. delayed impacts Scholarly vs. professional vs. social impacts Cited vs. read vs. used Substance vs. buzz

Immediate effects: PLOS article-level metrics

Article-level metrics Transparency Real-time Multi-dimensional Countable Ego-boosting Behavior-modifying Culturally corrosive?

Academic social capital Highly ‘liked’ Much tweeted/followed Heavily blogged about Frequently recommended Often mentioned/quoted in the media

Not to be confused! Social capital Symbolic capital

Complementary metrics Acknowledgments Data citation counts Micro-attributions for data curation Social media mentions Recommendations Downloads Mentions in extra- scientific texts Press coverage etc., etc…

Anticipating altmetrics: ‘Invoked on the Web’ (Cronin et al., 1998) ‘polymorphous mentioning’ ‘presence density’ ‘diverse ways in which academic influence is exercised and acknowledged’

Jason Priem, 2011

Genres of altmetrics Taylor & Plume (2014) Social activity (tweets, ‘likes’) Mass media (news coverage) Scholarly commentary (scientific blogs) Scholarly activity (reference managers) Altmetrics

Unequal attention Taylor & Plume (2014)

The hunt for correlations… Citations in Wikipedia and JCR data (Nielsen, 2007) Article tweets and citations (Eysenback, 2012) F1000 score and JIF (Nature Neuroscience, 2005) Inclusion in reference managers and citations (Bar-Ilan, 2012) Downloads and subsequent citations (Brody et al., 2006; Nieder, Dalhaug, Aandahl, 2013) Citations in blogs and subsequent citations (Shema, Bar-Ilan, Thelwall, 2013) Altmetrics and citations (Thelwall, Haustein, Larivière & Sugimoto, 2013; Costas, Zahedi & Wouters, 2014) Etc., etc., ….

Downloads vs. citations ScienceDirect (Moed, 2012)

Downloads & citations Nieder, Dalhaug & Aandah (2013)

Twitter mentions & arXiv downloads Shuai, Pep, Bollen (2012)

Downloading, reading, citing Schlögl et al. (2014)

1 citation = ? tweets Citations Acknowledgments Downloads Tweets ‘Likes’ etc.

(Alt)metrics issues Metrics Validity Reliability Utility Ethicality Platforms Transparency Usability Persistence Cost/benefit ratio

Mirror, mirror on the wall, who is the fairest of them all? Users, narcissism and control – tracking the impact of scholarly publications in the 21 st century Wouters & Costas (2012)

Google Scholar: Narcissism?

Scholarly Panopticon? ‘an Orwellian surveillance net’ ‘cybernating the academy’ Sosteric, 1999

The Holy Grail of holism A matrix of established & alternative metrics? A unified measure/composite score (a super h-index)?

New Age numerology? Atomization of inputs, outputs and impacts Fetishization of metrics Transparency vs. triviality Immediacy vs. canonicity Goal displacement?

Suggested readings Scholarly Metrics Under the Microscope edited by Blaise Cronin and Cassidy R. Sugimoto 2014 ITI / ASIST

And the winner is ….