1. Demonstrating journal impact: PlumX and OJS at the University of Pittsburgh
Timothy Deliyannides
Director, Office of Scholarly Communication and Publishing
University Library System
University of Pittsburgh
Library Publishing Coalition Webinar
April 26, 2016, 3:00PM EST
Thanks for asking me to share our experience with incorporating altmetrics into our publishing program at the University of Pittsburgh.
___________________________
[ABSTRACT: Academic publishers are facing increasing pressure to document the impact of the research they publish, and library publishers are no exception.  Discover how the University Library System, University of Pittsburgh is using Plum Analytics to demonstrate impact for its 40 scholarly journal titles.   We will show aggregated analytic data at the journal level as well as metrics demonstrating the impact of individual journal articles.  We will also discuss our development of an Open Journal Systems (OJS) plugin for embedding PlumX into any OJS journal, and our proof of concept with the NISO SUSHI-Lite Working Group to show how this new standard can be used with a journal publishing platform like OJS to provide article-level usage data to any altmetrics system.
This talk was presented as part of a Webinar sponsored by the Library Publishing Coalition on April 26, The Webinar also featured Andrea Michalek, President and Co-Founder of Plum Analytics, who spoke on the topic of "Altmetrics for Library Publishers: 10 Altmetrics Trends to Watch if You are a Library Publisher"]

2. PlumX at Pitt Plum Analytics’ first customer (2012)
PlumX is integrated with:
Institutional Repository (d-scholarship.pitt.edu)
4 subject-based repositories
Article-level ‘Plumprint’ appears for every digital object
Aggregated views by:
School, Program, Research Group
Researcher profiles demonstrate total impact
A few words of background on our history with PlumX-
We’ve been using PlumX for several years in a number of different ways and for several different audiences at Pitt.
Plum Analytics began by tracking every document in our institutional repository, where we are able to see aggregated views of the output of the University by school, by program, and even for small research groups.
PlumX has given us new insights, for instance on patterns of use for ETDs.
We have also begun offering researcher profiles through PlumX. We began with a small pilot for about 40 researchers, each of whom have a custom profile giving a bird’s eye view of their total impact.

3. ALTMETRICS for PUBLISHERS
But today, I’m not going to talk about any of these things. Instead, we’re focusing on altmetrics for publishers.
You may know that Pitt’s University Library System partners with scholarly societies and independent editorial teams around the world to publish 40 scholarly peer-reviewed Open Access journals. Our journal publishing program is based on PKP’s Open Journal Systems, and we are one of handful of major development partners of the Public Knowledge Project. Today I’m going to share with you how we use PlumX and OJS together to add value to our journal publishing program.

4. What’s different about these new metrics?
More comprehensive
Citations, usage, social media
Covers online behavior
Because scholars increasingly work online
Measures impact immediately
Because citation counts take years to appear in literature
I just want to recap what we see as the special value of the alternative metrics that we have through PlumX.
First, we’re able to put together a composite picture of many different measures.
Second, we’re able to track and show evidence of online behavior for the first time.
Finally, altmetrics can show impact immediately as it happens – we don’t need to wait three to five years for citations to appear before we can start to demonstrate interest and uptake in the research.

5. What can publishers do with these metrics?
Demonstrate value to stakeholders
Authors and their institutions
Editors and their sponsors (scholarly societies, universities)
Research funders
Gain insight
Improve marketing/promotion strategies
Identify top content
Learn about what’s important to your publishing partners
Improve quality of service offering
So what can we expect to do with these metrics as publishers?
First, we can demonstrate value to stakeholders. Who are these stakeholders?
Authors are under intense pressure to demonstrate that their work has impact. They can’t afford to wait five years to show this impact.
Journal editors face similar pressure. Most of our journals are Open Access, do not charge author fees, and are largely funded through scholarly societies or individual institutions of higher education. Editors need to demonstrate the value of their work and to show return on investment to their sponsors.
And of course, funders are evaluating outcomes of the research they fund, so they are always looking for ways to measure impact.
With the new information that is revealed for the first time through altmetrics, we can gain insights into our publishing portfolios. We can see what’s hot and what’s not. We can get clues to reader behaviors. My hope is that eventually, we’ll be able to predict future impact by looking at some of the early metrics. For instance, are saved bookmarks a leading indicator for citation counts later on? More research is needed in this area, but altmetrics can give us the raw data we need to answer these questions.
As publishers, if we can identify strengths and weaknesses in how our publications are being used, shared and talked about, we may be able to use this information to improve our marketing or promotion strategies. Editors may gain insights on how and where to attract quality submissions.
Lastly, as publishers, we are competing with other journals to attract the best submissions. Authors evaluate many aspects of quality when choosing a journal. Besides assessing academic quality and reputation, they want to have confidence that the publisher is well equipped to make their work visible in an online environment and has the technical infrastructure to do so. Integrating altmetrics with the publishing platform shows technological innovation and provides a richer overall service offering to potential authors.

6. PLUMX and OJS: How they work together at Pitt
PlumX harvests new articles published in OJS using OAI PMH protocol and begins tracking metrics
PlumX widget is embedded in article abstract page using the OJS PlumX plugin
The Plum Print displays current metrics each time a reader views the article abstract
PlumX Web site provides aggregated views of analytic data by journal title, volume, publication year, etc.
Before I show you how these metrics actually look in practice, I want to start with an overview of how the underlying process works.
It all begins when PlumX begins tracking metrics for all of the articles we publish. PlumX harvests new articles published in OJS using OAI Protocol for Metadata Harvesting. Once we set up each new journal, this becomes automatic.
Then, the PlumX widget is embedded in the article abstract page using the OJS PlumX plugin. More on this later.
The Plum Print displays current metrics each time a reader views the article abstract.
PlumX also offers aggregated views of analytic data with breakdowns by journal title, volume, publication year, or other parameters. This information is not embedded into OJS. To view it, we visit the PlumX Web site with proper credentials.

7. ARTICLE LEVEL METRICS
So let’s jump in a see what the article-level metrics look like when embedded into our OJS journals.

8. Here’s an example from the journal Pennsylvania Libraries: Research and Practice.
In this example, if you scroll down to the bottom of the abstract page, you see…

9. …the PlumX widget at the bottom of the abstract page
…the PlumX widget at the bottom of the abstract page. This article is about how libraries can use Wikipedia to increase the discoverability of their digital collections – interesting stuff! It hasn’t been cited yet, but it has gotten some interest in social media. Here you see Plum’s five categories of metrics in a horizontal arrangement.

10. And here is an example from our International Journal of Telerehabilitation…

11. …that also shows downloads, captures, some play in social media as well as traditional citations. One beautiful aspect of the PlumX widget is its interactivity. You can almost always drill down to see more detail. So when we click on “see details”…

12. …we now hop over to the Plum Analytics Web site for the full artifact details display. Here we can see more information than fits on the abbreviated widget display. This article happens to have been cited by seven other publications. In order for Plum to show citation counts, the article must be indexed in one of the sources tracked by Plum, like PubMed Central or Scopus. This is our only journal included in PubMed Central.
Let’s say we want to find out exactly who cited this article. We click on PubMed Central and…

13. … we are now looking at the article abstract view in PubMed Central
… we are now looking at the article abstract view in PubMed Central. We can then follow the link to the citing articles, and…

14. …we can now take a look at the seven articles that cite the work in our journal.
All of the data presented by PlumX is available through other sources, and you might have other ways of finding it. But PlumX brings it all together in one place without any extra work by the publisher. All of these paths to new information are automatically generated by PlumX for every article.

15. AGGREGATED JOURNAL METRICS
So that was a look at metrics for individual articles. Looking at these metrics rewards authors immediately with new information and insights about their own works. Article-level metrics might be of interest to readers, too.
But what about the bigger picture?
As I mentioned, we publish 40 scholarly journals.
It’s very important to us and to our publishing partners to demonstrate the impact of these publications in every way that we can. Many of our journals are small niche publications with a small readership and many of them are quite new. We’re working very hard to build up their visibility and discoverability, but it takes time to achieve high citation rates. So we find it extremely beneficial to have a broader range of metrics to draw on.

16. The PlumX interface lets us look at our entire publishing portfolio as a whole. This is an aggregate view of all of our journal publications. You can see that PlumX is analyzing all 8,292 articles across all our journal publications.

17. And if we scroll down a bit, we can see a table of these 8,000 plus articles organized by Plum’s five categories of metrics. I’ve sorted the table by ‘captures’ here. This view is highly interactive, and it’s a great way to identify your top articles for any metric across all titles and years.

18. But PlumX can also let you view your entire portfolio by journal title
But PlumX can also let you view your entire portfolio by journal title. Here you can see the top 20 journals for any of the five categories of metrics.

19. Let’s take a closer look at the ‘Usage’ category
Let’s take a closer look at the ‘Usage’ category. We can see that the journal in our portfolio that gets the most usage is Radical Teacher, followed by the University of Pittsburgh Law Review.

20. If we mouse over the Law Review, we see more detail on the breakdown of specific metrics in the ‘Usage’ category.

21. Continuing on to the ‘Captures’ category, we see a different title has the highest metrics for captures.

22. And likewise for the ‘Mentions’ category.
This can lead to insight into how reader behavior differs for each journal title. The more we drill down to the individual metrics, the more we are likely to learn.

23. Here we see that the title ‘Radical Teacher’ is the top journal for the category of Social Media metrics.

24. And what are the top journals in the ‘Citations’ category
And what are the top journals in the ‘Citations’ category? We see that the International Journal of Telerehabilitation leads the pack here. Recall that this is our only journal included in PubMed Central. This is a reminder to us of the importance of our efforts to have our journals included in major abstracting and indexing services. At the same time, it underscores the importance of alternative metrics. Many of our small niche journals in the humanities and social sciences will face an uphill battle to be included in the top indexes. This makes it more difficult to compile citation data, but they can still demonstrate their impact using other metrics.

25. We’ve been looking at summary analytics for our entire portfolio of journals.
But what kind of data might a journal editor want to see? Journal editors will surely want to know what impact their journal is having, what topics or authors are drawing the most interest, and whether the impact of the journal is growing over time.
We can filter the analytics to focus on a single journal title and then breakout the view by year of publication, as shown in this example from “Biblios: Journal of Librarianship and Information Science”.
Here we can see the metrics trending up each publication year. This curve is very different from the traditional pattern we see for citation counts, where the curve typically tapers off over the most recent three years due to the nature of citation counts as a lagging indicator.

26. An editor can also group the metrics by individual journal issue
An editor can also group the metrics by individual journal issue. So here we are comparing all the individual issues of the same journal and we see that issue #52 had the highest usage.

27. Here is Radical Teacher once again
Here is Radical Teacher once again. We are looking at Social Media metrics, this time, grouped by journal volume. The volume with the highest numbers is volume 102.
What was it about volume 102 that captured so much attention?

28. Let hop over to a summary view of Volume 102 of Radical Teacher.
We can see here that Plum is tracking the 12 articles in Volume A bar chart shows the relative metrics for these articles broken out by the five categories again. There are actually only four categories showing here, since no citations have been clocked yet.

29. If we scroll down the screen, we can actually see the 12 individual articles in volume 102.
This happens to be a special issue devoted to teaching about climate change, which may have something to do with its high visibility on social media.
I have sorted the artifacts here by social media impact (the column in pink). So let’s take a look at the top article with 268 metrics on social media, by clicking on the title.

30. Here we see the full artifact display in PlumX, where we can get details on all of the metrics categories and drill down to learn more.
If I’m an editor, I might want to know what people are actually saying about this article. By clicking on the “TWEETS” tab, I can see…

31. …this detail display on twitter interactions
…this detail display on twitter interactions. I can read each and every tweet.
This is a way for editors to stay informed about which articles are being talked about, by whom, and what is being said.

32. OJS and PLUMX Under the hood
I want to close by telling you about our work on integrating PlumX with Open Journal Systems, and how you can actually set this up in OJS.

33. Pitt’s work on integrating OJS & PlumX
PlumX plugin for OJS
Embeds PlumX widget in any OJS journal
Developed by Clinton Graham, University of Pittsburgh
SUSHI-Lite compliance for OJS
Improves harvesting of OJS usage data at the article level
Can be used by any external system (such as Plum Analytics)
We’ve done work in two areas specific areas. Both of these are the result of programming written by my colleague Clinton Graham, a software Developer at the University of Pittsburgh.
First, we’ve created an OJS plugin to make it very easy to add Plum Analytics to any OJS journal.
We’ve created a second OJS plugin to improve the way OJS shares usage data with external systems like PlumX – The SUSHI-Lite Plugin.

34. The PlumX plugin for OJS
Requirements:
OJS 2.4 or a later release of OJS 2.x
Articles must have DOIs
Subscription to PlumX
Setup: follow the instructions at
Install as a ‘generic plugin’ in OJS
Log in as Journal Manager and enable the plugin
Configure the widget display – highly customizable!
Integrating PlumX with any OJS journal is just a matter of installing the free, Open Source plugin for PlumX into your OJS system. From there, any journal manager can configure the way the PlumX widget appears in your OJS interface.
Keep in mind that you first need to be a PlumX customer in order to use the PlumX widget in OJS. You also need to assign DOIs to your articles. The DOI is the hook needed to display PlumX metrics in OJS.

35. Configuring the Plugin
Home > User > Journal Manager > System Plugins > Generic Plugins > Plum Analytics Artifact Widget Settings
Plum Analytics Artifact Widget Settings
With this plugin enabled, the Plum Analytics Artifact widget will be added to your articles. The article's DOI will be used to link to the PlumX artifact metrics. For details and examples of the different widget settings, please see the PlumX Widget Page.
Widget Type *
Summary
Horizontal
Below the article’s abstract ✓
Hide When Empty
Show Border
This is an example of the plugin setup that the Journal Manager will use to configure the display of the PlumX widget within OJS. Dropdowns and checkboxes provide a simple interface to set the HTML options for the standard widget.
Hide the PlumPrint
Popup Alignment
Display Orientation
Width
Display Widget *

36. Positioning the widget
OJS Header
Left Sidebar
Article Title
Plum Analytics Widget Block
Above the article’s title
Right Sidebar
Article Abstract
There’s a lot of flexibility as to where you place the widget on the OJS abstract page. You can locate it in a number of the OJS template hooks, or in a widget block, which can be manipulated like any of the other block plugins.
Below the article’s abstract
Article Footer
In the article’s page footer
Plum Analytics Widget Block
OJS Footer

37. OJS and SUSHI-Lite
Finally, our work on a better way to share OJS usage data out to PlumX or to any other external system using the new SUSHI-Lite standard.

38. SUSHI-Lite compliance in OJS
Alternative to the OJS COUNTER interface
SUSHI-Lite focuses on article-level usage
Uses standard Web services (REST, JSON, etc.)
Ideal for on-demand and just-in-time usage harvesting
A better way to share usage data from OJS journals
Some of you may be familiar with the COUNTER plugin for OJS. The SUSHI-Lite plugin is an alternative with some advantages for article-level usage that uses more flexible Web Services to allow on-demand and just-in-time harvesting of usage data.

39. SUSHI-Lite Plugin for OJS
Developed by Pitt, but can be used by any OJS host
Used by PlumX to harvest OJS usage data
Based on NISO standard
Can be used by any external system to harvest OJS usage data
Setup: follow the instructions at
The OJS plugin we contributed to the OJS community is now being used by PlumX. But it could also be used by any other service provider that wants to follow the emerging NISO standard. The plugin and instructions are given on our github site at the URL on your screen.

40. More about PlumX at Pitt
URL for this presentation:
Collister, Lauren Brittany and Deliyannides, Timothy S.  (2016) Altmetrics: Documenting the Story of Research. Against the Grain, 28 (1). pp ISSN
Collister, Lauren Brittany and Graham, Clinton T. (2016) Building a Bridge for Metrics with SUSHI-Lite. Poster presented at SPARC Meeting on Openness in Research and Education, 3/7/2016-3/8/2016, San Antonio, TX.
Deliyannides, Timothy S. (2015) Practical uses of Altmetrics at the University of Pittsburgh. Webcast presented by Library Journal and Plum Analytics, February 11,
So, I hope I’ve given you some ideas about how altmetrics might be integrated into your service offerings as a library publisher, and how this might benefit your various stakeholders.
This presentation is available our institutional repository, at the address shown.
To learn more about PlumX at Pitt, including some of the areas not covered in this presentation, please see the other links on this page.

41. PlumX at Pitt: http://plu.mx/pitt
THANK YOU
Tim Deliyannides
@deliyannides
PlumX at Pitt:
Thank you, and please contact me at any time if you have follow-up questions.
Office of Scholarly Communication and Publishing @OSCP_Pitt