Bibliometric Analysis Tools for Research Portfolio Analysis and Management Ann Kushmerick Manager, Research Evaluation and Bibliometric Data May 3, 2013

2 LARGEST PROVIDER OF INTELLIGENT INFORMATION Thomson Reuters is the largest provider of intelligent information to business and professional customers in the world. GLOBAL OBJECTIVEEXPERIENCED We operate in over 90 countries around the world. We support over 5,600 universities, governments and research institutions from more than100 countries. We offer unbiased metrics based on citation activity of the most impactful global and regional journals to the scholarly community, and remain free from proprietary involvement. Our roots run deeper than any other information partner today. And our research evaluation experts have demonstrated experience in science policy, research administration and project management. Thomson Reuters

Thomson Reuters: Solutions for the Scientific Research Community

4 Connecting research to impact for the world’s leading research funding organizations, governments, publishers, associations and institutions… RESEARCH PRODUCING ORGANIZATIONS INTELLIGENT INFORMATION ADVANCED TOOLS AND ANALYTICS PROFESSIONAL SERVICES Benchmarking and comparison products Journal evaluation and comparisons tools Customized grant management reporting Customized research portfolio analysis Customized research impact evaluations Expert services Solutions for Research Funders

Benchmark and compare performance and output to peers Identify and promote partnerships or collaborations both inside and outside the organization Identify and promote areas of strength or specializations tied to impact Decision support for policy and program development Comprehend the research landscape Provide your teams with flexible, customized systems Position with board members and key stakeholders Identify emerging subject areas or researchers Demonstrate results and impact of research activities Identify new trends and key indicators to enable policy development 5 Critical Decision Support for Research Funders

INTELLIGENT INFORMATION 6

7 Selectivity and control of content: consistent, transparent standards 12,000 prestigious journals and 716 million+ Cited References Multidisciplinary: Sciences, Social Sciences, Arts & Humanities Depth of more than 100 years, including cited references Consistent and reliable data: Cover to cover indexing Cited references for all years of coverage Indexing of all author names and addresses Unmatched expertise: 50+ yrs. of citation indexing and analysis Conferences and Scholarly Books: complete the research picture Funding acknowledgments: link funding source to outputs The gold standard used by 5,000+ institutions in over 100 countries Web Of Science: The foundational data for research analytics solutions

Datasets to enable your analysis We offer customized datasets and studies from any of our data sources: – Web of Science- (data can be customized or full database) – Medline – Journal Citation Reports – Essential Science Indicators – InCites – ScienceWire: Our data warehouse where funding inputs (grants) and outputs (journal articles, conference proceedings, patents, etc.) are linked.

EXPERT STAFF 9

10 ABOUT OUR RESEARCH ANALYTICS CAPABILITIES Highly Trained: Our global product managers, consultants, account managers and customer training teams are credentialed in all major fields and are experts in evaluation, bibliometrics, and management in Library Science, Bioinformatics, Chemistry, Neuroscience, Physics, Statistics, Mathematics, Science Policy and Evaluation. Highly Experienced: We have demonstrated expertise in science policy, research administration and project management with an understanding and appreciation of the scientific discovery process. And because we have teams around the world, we are able to provide evaluations and objective reporting in global and regional contexts. Scalable, global team of experts with strong research backgrounds

TAILORED SOLUTIONS 11

Providing a clear picture of research activities and offering decision support to customers around the world - from answering a single question to creating an enterprise-wide decision support system 12 PROVIDING UNMATCHED CONTENT, TOOLS, AND SERVICES CONSULT PROFILE BENCHMARK

13 InCites 13 Bibliometric analysis of research output and impact with peer comparisons.

14 Questions InCites answers for funders  What papers resulted from our funding? Which have the highest impact in their field?  Which of our funded authors are producing high impact research? What fields of research are they publishing in? What journals?  What institutions are we funding? How do they compare to each other?  What authors, institutions, and countries are citing our funded works?

InCites offers different types of analysis to answer a variety of research evaluation questions 15 Global Comparisons: Compare research impact of countries and universities. Analyze trends back to Institutional Profiles: Compare institutions using a 360 o view. Includes bibliometric indicators, academic reputation survey, and institution submitted data, Research Performance Profile: Run custom reports on journal and conference literature you have funded. Measure researcher impact, collaboration success, and field strengths.

How “good” is this? What is the context? Additional metrics are needed to understand research performance cites/paper H-index: 13 Citation rates vary among fields. What is good or average in mathematics is very different from what is good or average in biochemistry. Normalized citation metrics put citation information in context 14.5 cites/paper H-index: cites/paper H-index: cites/paper H-index: 3 16

A VARIETY OF INDICATORS 17 MetricCalculationEvaluator Questions Productivity # papers share of papers in field # papers # papers in field/ papers in field What is the research output of X? (a country, subject, researcher, etc.) Impact # citations h-index # citations Number of papers (N) with at least N citations each. What is the overall impact and/or productivity of a body of work? Influence Average citation rate (CPP) Percent of papers cited Total citations/Total papers # papers with at least one citation/ Total # papers in population What is the rate at which a body of work is cited? How many papers get cited? Never get cited? Relative Impact Journal performance ratio Category performance ratio Percentile in category and mean percentile % papers in top x% of their field Sum of citations/sum of journal or category expected citation rates Percentile placement of article within a journal category Has this body of work performed better than average vis-à-vis the journals or scientific disciplines represented? How has this body of work performed compared to the disciplines represented? What proportion of a body of work achieves a specific level of performance?

18 Compare like with like – The Golden Rule Use relative measures, not just absolute counts More applicable to hard sciences than arts/humanities Know your data parameters: –journal categories –author names –author addresses –time periods –document types Obtain multiple measures Recognize skewed nature of citation data Ask whether the results are reasonable Understand your data source Guidelines for citation analysis For further guidance on citation metrics, download the white papers at:

InCites: Global Comparisons Analyze global trends in publication 19 Contains data on: -countries and regions -9 category schemes -thousands of institutions

Analyze Research Activity by Country 20 Public, Environmental, and Occupational Health is a growing field in these Asian countries, particularly China.

21 Analyze Research Activity by Country For these countries, relative citation impact in this field is close to the world average. 1.0 represents the world average (field baseline)

22 Identify which institutions are contributing to China’s growth These Chinese institutions are producing the most research in this field.

InCites: Research Performance Profile custom built for detailed analysis of your publications 23

Options for building a Research Performance Profile for a Funder  Web of Science funding acknowledgements  Medline funding acknowledgments for records shared by Medline and Web of Science  Matching data provided by the customer: Pubmed IDs, UTs (Web of Science unique identifier), bibliographic records supplied by funder 24

InCites includes a detailed dataset built to your specifications 25 Compare a citation count to journal and category norms. This paper clearly outperforms the average. This paper ranks in the top 0.23% of all neuroscience reviews from Examples taken from Mt Sinai School of Medicine sample dataset View detailed metrics on each article from your dataset

Analyze researcher performance: Neurosciences Sort by number of documents to identify the most prolific authors These metrics are unique to InCites. They normalize citation performance by research field. Hof, PR received over 2 times the expected rate of citation in neuroscience, and on average his papers rank in the top 18% of neuroscience papers.

How can we identify and promote outstanding or up- and-coming researchers? Examples of how InCites users are applying the data Identify high-impact researchers in each department and utilize them as mentors for other faculty. Identify researchers with strong citation performance who are less successful at obtaining funding. Help them to demonstrate their research success to funders using citation metrics. Identify researchers will strong funding success but low citation metrics. Help them to identify and submit to high-impact journals.

Generate dashboards for a variety of citation metrics, e.g. Hof, PR Mean percentile: on average, Hof’s papers rank in the top 20% of their respective fields Self citation analysis: 13% of Hof’s citations are self citations. H index without self citations is 58. H index: 65 papers have been cited at least 65 times each Journal actual/expected ratio: Hof gets cited 1.4 times the average for the journals he publishes in

Hof, PR: analyze citation growth year on year 29

Identify potential new areas of excellence: analyze data by research field 30 Medical ethics comprises a small portion of Mt Sinai’s total output, but the category normalized impact is over 2 times the expected rate.

Understand research collaborations among your funded institutions 31 Mt Sinai doesn’t collaborate often with McGill Univ, but the research has high impact in its field.

What is the global impact of your research? 32 Identify government bodies, global universities, and corporations who are building on your work.

Understand the co-funders of your research Where else are your fundees getting support? 33 Drill down to papers funded by Pfizer. View the topics funded by Dept of Veterans Affairs Look for specific grant numbers.

Save your custom reports and share with other users Export reports to PDF and Excel Save and share your report parameters with other users Save time when creating reports; efficiently support evidence-based decision making

CUSTOM PROJECTS 35 Our experts can help you answer your key evaluation questions.

36 Electronic Scientific Portfolio Assistant (eSPA): analyzes linkages from grants to their measurable outputs 36

The Global Landscape of Autism Research HHS IACC/OARC Autism Spectrum Disorder: The Global Landscape of Autism Research How does the volume of autism publications align with a strategic plan advancing autism research? Automated classification of >10,000 publications with 85% accuracy.

The Global Landscape of Autism Research The US accounts for the largest number of ASD publications each year The contribution from other countries is growing over time Share of ASD publications globally. “All Other Countries” category is led by China, Taiwan, South Korea, India, Brazil, and Poland

Case Study: Alzheimer’s Research UK A UK-based biomedical charity was concerned about research capacity in dementia and wanted to use publication data to investigate the statement “research capacity in dementia is lower than in other diseases such as cancer or heart disease” Dementia_-_Building_capacity_to_capitalise_on_the_UKs_research_strengths.pdf

Global Research Report: Neglected Tropical Diseases Download free Global Research Reports:

Real-time, Dynamic Progress Report Evaluation System (NCI-iTRAQR) Center A, June 2012 Capture Progress Reports

 Build grant progress report data model, database, and data entry tool  Improved data collection on progress report forms  Use bibliometrics to identify research areas, collaboration, and merging of fields.  Examine research collaborations using co- authorships and network graphs Capture Progress Reports Real-time, Dynamic Progress Report Evaluation System (NCI-iTRAQR)

Tracking non-publication outcomes: How has the National Cancer Institute supported development of breast cancer biomarkers? Patent Indication Biomarker Acknowledgement of NCI Support Publication Validity Status Integrity Biomarkers Data NCI Grant Integrity Biomarker data includes references to Web of Science publications and US PTO patents. Patents and Publications acknowledge funding from NCI.

Tracking non-publication outcomes: How has the National Cancer Institute supported development of FDA approved Drugs? FDA products linked to NCI funding through acknowledgment in patents listed in New Drug Applications (NDA). Direct patent acknowledgements represent direct links to the NCI projects. Additional candidate links established through non- patent references found in the patents listed in the NDA. Publication Acknowledgement of NCI Support Non-Patent Reference List NCI Grant Patent Acknowledgement of NCI Support NDA-Listed Patent [USPTO Patent] NDA-Listed Patent [USPTO Patent] Approved Drugs [FDA Orange Book] Approved Drugs [FDA Orange Book]

CUSTOM INDICATORS AND EVALUATIONS Breakthrough Indicator- Our goal is to create a Breakthrough Paper Indicator as a multidimensional vector Cumulative Citations Score Citations Velocity Score Subject Diversity Score Geographical diversity score Prestige Citation Score Media Coverage Score Key words, coin words usage score BP Indicator Multi-dimensional vector for predicting breakthroughs based on publication characteristics such as: –Citation rates –Geographical patterns of references and citations –Disciplinary patterns of references and citations

Innovative Medicines Initiative Joint Undertaking (IMI) Evaluation The Innovative Medicines Initiative Joint Undertaking (IMI) is a public private partnership between the European Union and the European Federations of Pharmaceutical Industries and Associations (EFPIA). With a €2 billion euro budget, IMI supports collaborative research projects and builds networks of industrial and academic experts in Europe that will boost innovation in healthcare. The first funding call to researchers was announced in 2008

IMI wanted to evaluated its funding by: Identifying publications Citation metrics and analysis Innovation and relevance to new technologies Network analysis Journal and patent analysis First report (October2012): 0Report%201_FINAL.pdf Second report (March 2013): al.pdf Innovative Medicines Initiative Joint Undertaking (IMI) Evaluation

PAST PERFORMANCE 49

Thomson Reuters Custom Analytics: Used Across The Globe For Evaluation And Policy

FUTURE VISION 51

OUR LONG TERM STRATEGIC VISION

Thank You