1. Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact John Barnett Scholarly Communications Librarian Jennifer Chan Assistant Scholarly Communications Librarian Office of Scholarly Communication and Publishing University Library System University of Pittsburgh

2. Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact

3. University of Pittsburgh  Pittsburgh campus + regional campuses in Bradford, Greensburg, Johnstown, and Titusville  16 undergraduate, graduate, and professional schools  456+ degree programs  2012: conferred 8,949 degrees

4. University of Pittsburgh  Top 10 American higher ed. in federal funding (NSF)  Top 5 in annual research support (NIH)  5,369 faculty; 4,470 full-time faculty  Research conducted: more than 300 centers, institutes, laboratories, clinics

5. University Library System  ARL  22 nd largest academic library system in North America  25 libraries; 6.6 million volumes  279,000 current serials

6. Sum of the Parts Liaison Librarians ULS Department of Information Technology Office of Scholarly Communication and Publishing

7. Why Pitt?  Strategic goal: Innovation in scholarly communication  Providing services that scholars understand, need, and value  Putting ourselves in faculty “spaces”  Re-envisioning our librarian liaison program  Deepening our understanding of scholarly communications issues

8. Why PlumX?  Making research “more assessable and accessible” –Gathering information in one place –Making it intelligible and useful  Measuring and visualizing research impact  Correlating metrics from traditional and new forms of scholarly communication  Allowing researchers, labs, departments, institutions to track real-time scholarly impact  Promoting research, comparing with peers, connecting with new research

9. Altmetrics Project Timeline Spring 2012: First meeting with Plum Analytics Summer 2012: Announcement of Pitt as Plum Analytics’ first partner Fall 2012 Gathered data from pilot participants Winter 2013 PlumX pilot system made public Spring 2013 Faculty surveyed; enhancements made Fall 2013 IR widget launched; rollout preparations

10. Pilot project aims  Develop a tool for measuring and visualizing research impact  Gathering information in one place  Intelligible and useful  Impact in social media and other scholarly communication methods  Traditional measures counted as well  See where to disseminate works to increase impact

11. Traditional vs. new Traditional measures are also counted Findings are complementary to conventional methods of measuring research impact (e.g., H-Index) Not intended to replace them

12. New measures  More comprehensive: Altmetrics = ALL METRICS –Citations –Usage –Captures –Mentions –Social Media  Covers impact of online behavior –Because scholars increasingly work online  Measures impact immediately –Because citation counts take years to appear in literature

13. Pilot Process Created Altmetrics Task Force Built Sharepoint Site Engaged Liaison Librarians Pilot Selection CV receipt and records creation

14. Pilot Project Participants 32 researchers, various disciplines 9 schools 18 departments 1 complete research group Others joined as they learned about the project

15. Pilot Project Participants disciplineschool/department online behavior level of career advancement Selected faculty participants, diversified by:

16. Technologies  Internal –IR built on Eprints Platform –Sharepoint –Microsoft Office Suite –PMID/DOI data import tool  External –PlumX –DOIs –PMID

17. Data collection for pilot project Created records in D-Scholarship@Pitt, our institutional repository Focused on articles, books, book chapters, proceedings Scholarly output with standard identifiers DOI, ISBN, PubMed ID, official URL, etc. Scholarship produced since 2000

18. Other Library work Developed guidelines to standardize record creation Data entry from faculty CVs into IR (2 to 3 student workers with QA by librarians) Librarian liaisons and other staff trained in record creation SharePoint site used to track work completed Coordination with pilot faculty Gathered feedback and administered online survey

19. Sharepoint  Altmetrics Meetings Minutes  Faculty CVs  Excel spreadsheets  Word docs

20. External Data Sources

21. Metadata sources  Faculty CVs... But verify metadata!  Books: PittCat, WorldCat, Books in Print, publisher sites, online retailers  Journals: Serials Solutions list, journal websites, JournalSeek, UlrichsWeb, DOAJ, PubMed  Conference presentations: Websites, PittCat, indexes, WorldCat

22. PMID Import Tool  Custom build by SysAdmin for Eprints Platform  Utilizing PMIDs from PubMed, able to import records that prepopulate metadata fields –Item Type, Title, Abstract, Creators, Publication Title, ISSN, Volume/Issue, Page ranges, Date and Date type, DOI, MeSH Headings, Grant Information, Keywords, etc.

23. Data Ingestion

24. Full-text sources  DOAJ  ERIC  PLOS  SSRN*  Other repositories*  Federal government websites*  Conference websites* * Use with caution

25. Plum Analytics processing activities Harvest records from Pitt IR for each participant Build profile for each researcher in PlumX Harvest additional online artifacts NOT in Pitt IR Use data mining to harvest publically available metrics from hundreds of sites on the Web Create visualizations to display metrics on PlumX interface

26. Key features  Faculty profiles  Online ‘artifacts’ –Article –Book –Book chapter –Video –Etc.  Impact graph  Sunburst

27. Faculty profile

28. Online ‘artifact’ display

29. Impact graph

30. Sunburst

31. Feedback Solicited via email and online survey Generally positive in most cases Data corrections Errors in profiles Links to wrong data Quickly corrected by Plum staff Requests for results from additional online sources (Google Scholar, SlideShare, Reddit, etc.) PlumX collects data from these but did not gather information in advance for profiles

32. The survey says  Surveyed pilot project faculty in spring 2013  @ 1/3 rd responded to the survey  Meaning 13 out of 32 participants responded

33. Accurate and useful data

34. The bar graph

35. The sunburst 0

36. Traditional & new measures 0

37. Usefulness of altmetrics

38. Learning something new

39. Comments  Affiliations/bio inaccurate or has missing information  “Mentions” by whom & when?  Publications misclassified –Books vs. conference proceedings  Data not collected –Google Scholar –Slideshare

40. Comments  Filter out unwanted information  Data are wrong—and not useful  Overabundance of information in sunburst  “I only care what a select group of scholars thinks of my work”  “I did not find this useful for my discipline”

41. Observations  Lacked information about faculty practices  Are the results useful to all faculty, all disciplines?  May appeal more to faculty who are early in their careers or whose work is more contemporary  Will the data be used against faculty or programs?  Labor-intensive strategy  When it comes down to it... Does anyone care?

42. Embeddable widgets (in development) For researchers, to add to: their own Web pages department directories IR researcher profile page For individual artifacts, to build article level metrics for imbedding in: IR document abstract page Article abstract page for journals we publish

43. Roll-out challenges  Who creates profiles? Who edits?  What information should be included in profiles? Who can view them?  Separate data gathering from D-Scholarship deposits?  Who promotes the service? Who trains?  Timing...

44. Future plans  Data checking  Additional data gathering  Record merging/deduping  Ability to edit user profiles and artifact records locally  Open API  To allow integration with other online systems  More exhaustive scholarly practices survey for all faculty  Rollout to all Pitt Researchers  Will use automatic feed from Pitt IR to PlumX

45. Discussion  How would you “sell” PlumX to additional faculty?