Performance Measurement: a Publisher’s Perspective 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services Performance Measurement: a Publisher’s Perspective Dr. Michiel Kolman Senior Vice President Global Academic Relations August 2011

Intro – research metrics “exciting times are here!” August 2011 Outsell* report: Google Scholar and Microsoft Academic Search have both just announced new functionalities, currently in beta, that cut across some of the features presented by commercial offerings such as Thomson Reuters’ Web of Knowledge and Elsevier’s Scopus. This presentation: Research Performance Metrics at Country level Institutional level (if time allows) Metrics on Access. First: the underlying database is Scopus..... *By David Bousfield, Vice President & Lead Analyst, Outsell, August 10, 2011

Scopus - background Scopus is the world’s largest multidisciplinary abstract and citation database with peer-reviewed research literature, quality web sources, patents and more. Scopus covers 18,000 + titles from more than 5,000 global publishers. Health Sciences (100% Medline coverage): 6,800 Physical Sciences: 7,200 Life Sciences: 4,300 Social Sciences & Humanities: 5,300 Scopus contains over 20 million researcher profiles with author and affiliation disambiguation, and affiliation identification for all institutions that a researcher is associated with. Scopus is used by many research institutions globally to analyse research outputs, e.g. Scopus is the world’s largest abstracting and citations database, and contains over 20 million researcher profiles that are disambiguated for author name and affiliation identifications over the entire course of a researcher’s working life. 3

Scopus – background (continued) Scopus is the world’s largest abstracting and citations database, and contains over 20 million researcher profiles that are disambiguated for author name and affiliation identifications over the entire course of a researcher’s working life. 4

Metrics at a Country Level

Country share of global published journal articles Share of total global articles published (annual increase in total number of articles of 5.3%) (Bold indicates in global top 10) 32 30 28 Publications CAGR 26 24 22 20 USA 1.6% China* 19.5% 12 How is this changing the global research landscape? Take China for example: in 1996 China had a 3% share of global published articles. By 2009 this was 14% and because it continues to increase R&D spending by over 20% per year it is expected that China will become the world’s leading publisher of scientific articles (ahead of the USA) possibly as early as 2013. India, who has grown its publication output by an average of 8% per year since 1996, became a global top 10 publisher of scientific information in the last 5 years. South Africa has also managed to increase its publications by over 6% per year since 1996, against a global total growth rate of 5.3%. (South Africa is current 34th in share of publications, just after Romania.) UK 3.2% 10 Germany 3.5% 8 Japan 1.8% 6 India* 8.1% 4 Brazil* 12.5% Russia 0.7% 2 South Africa* 6.1% 6 Source: Scopus

Research collaboration map: e.g. for South Africa However, we also see that collaboration between developing countries is still minimal. For example, a recent study showed that out of all African biomedical research papers published from 2004-2008, 77% were produced with international partners, but just 5% were the result of collaborations with another African country. This collaboration map shows the major collaboration partners of South Africa, at a country level, based on co-authored articles. The thicker the line connecting the nodes, the higher the volume of co-authored articles. Not surprisingly, the US accounts for the largest number of co-authored articles reflecting the size of its research base. We also see that the UK, Germany, Australia, Canada are all important collaboration partners. But, although further out on the collaboration map, we also see increased collaboration with other developing countries. Note: Collaboration relationships are shown for papers authored by South African researchers counted in a 5-year window 2005-09 and are represented as variable-thickness lines (edges) between nodes. Line thickness represents the share of collaboration to or from the connected Visualisation is by the Force Atlas algorithm, which treats the network of edges as a system of interconnected springs and seeks to satisfy the tension of all edges simultaneously in a 2D rendering; hence, countries sharing a collaborative relationship tend to group together, while those that do not are placed further apart. 7 Source: Scopus

Peer-reviewed materials from past 5 years Scival Spotlight: Publication leadership at a country level: leadership in article 1. output, 2. citations and 3. currency (State of the Art) Peer-reviewed materials from past 5 years ‘Highly-cited’ A B C D E Competency No. 1 Competency No. 2 Paper 1 Ref. Paper A Ref. PaperB Paper 2 Paper 3 Ref. Paper B Co-Citation Analyses on ‘highly-cited’ references Papers from the country in question A B C D E References ~2 million references ~15-16 million references To generate a map (e.g. 2009), we take all the peer-reviewed articles that have been published and indexed in Scopus for the past 5 years (i.e. 2005-2009).; and we reviewed all the references in the articles (about 15-16 million references). We identify only ‘highly-cited’ references. This whittles down the relevant references to about 2 million. We than analyse the co-citations made in these references and identify natural clusters that form. Next we identify the papers from India within the clusters and clusters become competencies when the number of papers within them meet the appropriate threshold. Ref: Ref: ‘Co-Citation Analysis: Methodology of SciVal Spotlight’ 8

Multi-disciplinary strengths Finland counts 25 competencies at global level, rather covering the whole spectrum of research MATH & PHYSICS COMPUTER SCIENCE 25 competencies of which 14 distinctive competency and 11 emerging competencies Strong position in Biology but also in Computer Science and Medical Specialties Several multidisciplinary competencies already exist  could be expanded SOCIAL SCIENCES CHEMISTRY HUMANITIES circles away from circumference circles with multi-coloured bars Multi-disciplinary strengths BRAIN RESEARCH ENGINEERING HEALTH SCIENCE EARTH SCIENCES BIOLOGY MEDICAL SPECIALTIES BIOTECHNOLOGY 9 INFECTIOUS DISEASES Source: SciVal Spotlight 2009 (November 2010)

Scival Spotlight Country Maps:Different countries have different “maps” Total 401 Competencies Total 1,707 Competencies Total 398 Competencies Source: SciVal Spotlight 2009 (November 2010) 10

Sample 1 for a list of basic indices by countries Key metrics per country: article output, citations per article (CPA), State of the Art (SotA) Source: SciVal Spotlight 2009 (November 2010) 11

Brain Drain: where do the brains from Japan end up? A prototype.... http://els001.vmware.server.biz/maps/jpnmapall.html

Brain Drain prototype for Japan: in which discipline are most brains exported from Japan? Note: Number of researchers can be double counted across discipline. This is caused by papers in multiple disciplines

Exporting Institutions of brains from Japan

Selected ‘brains’ that moved out of Japan

Metrics at an Institutional Level

University of York, share of research outputs: 2003-2008 Size of bubble proportional to 5-year article output 2008 2007 2006 2005 2004 2003 York has steadily increased its share of article outputs in the UK from 2003 to 2008 Its share of UK articles increased from 1.05% in 2003 to 1.39% in 2008 Its share of global articles increased from 0.074% in 2003 to 0.090% in 2008 York has achieved this against a backdrop of declining UK article share The UK’s share of global research articles fell from 7.1% in 2003 to 6.4% in 2008 The UK’s share decline is common to many scientifically developed nations (e.g. US, Germany, France) and reflects the rapidly rising share of countries in Asia, the Middle East and South America Note: Data shows York’s article outputs (research articles, reviews and conference papers) and shares using 5 year periods, e.g. 2008 corresponds to 2004-2008 publications . 17 Source: Scopus 17

University of York, impact of research outputs, 2003-2008 Size of bubble proportional to field-weighted impact relative to world York’s 5-year citation impact is high and rising in absolute terms. Its mean unweighted citation impact for the 5 years ending 2008 was 6.165 compared to 5.817 for the UK average and 4.618 for the global average York’s field-weighted citation impact is also high, standing at 1.461 in 2008 (compared to a world average of 1.000 and UK average of 1.338) Note: Data shows unweighted citation impact and relative field-weighted impact using 5-year periods, e.g. 2008 corresponds to 2004-2008 publications and 2004-2008 citations. Field weightings account for shifting research portfolios in fields with differing citation profiles. Source: Scopus 18

University of York research outputs: rates of collaboration 37.5% of York’s articles are co-authored with non-UK researchers in 2008, (vs. 41.3% for the UK average in 2008, and 33.3% for the global average in 2008) Yorks’s rate of international collaboration has been steadily increasing, as has the average for the UK and globally International scientific collaboration is generally acknowledged as a positive force driving institutional and national impact and prestige Note: Data shows proportion of article outputs representing international collaboration (where one or more other countries are listed in the author address) in 5-year periods, e.g. 2008 corresponds to 2004-2008 publications and 2004-2008 citations. 19 Source: Scopus 19

International collaboration rates correlates strongly with publication impact Field-weighted relative impact 1 2 3 4 5 Number of collaborating countries (where 1 = domestic) International scientific collaboration is generally acknowledged as a positive force driving national impact and prestige Domestic articles (‘1’) have no collaboration partners have around 3 times fewer citations per article than those with four collaborating countries (‘5’) 20 Source: Scopus 20 20

University of York – map of competencies This is a circle map of York’s research competencies, which number 105 in total. These range from large “distinctive competencies” to smaller “emerging competencies,” as determined by fractional article counts in the relevant interdisciplinary field. As can be seen from an initial examination, York has particular strength in a number of social scientific, computer science, and biology and biotechnology disciplines. The diagram shows that York has 105 distinctive research competencies as defined by Elsevier’s SciVal Spotlight tool These range from large “distinctive competencies” to smaller “emerging competencies,” as determined by fractional article counts in the relevant interdisciplinary field. York has particular strength in a number of social scientific, computer science, and biology and biotechnology disciplines. Source: SciVal Spotlight 21

University of York DC #1 – Pharmaco Economics and Public Health York’s largest research competency in terms of number of fractionalized article counts is an interdisciplinary competency related to pharmacoeconomics and public health. York is a leader in this area, with the highest number of articles and citations globally. York’s largest research competency in terms of number of fractionalized article counts is an interdisciplinary competency related to Pharmaco Economics and public health. York is a leader in this area, with the highest number of articles and citations globally. Source: SciVal Spotlight 22

University of York DC #1 – Pharmaco Economics and Public Health York’s largest research competency is distinctive in all respects: it is a large area of research in which York researchers are prolific, highly cited and are citing the most recent research relative to peers Source: SciVal Spotlight

University of York DC #1 -- Leading Institutions Similarly, York’s primacy in this field can be seen on the institutional table as well, with York authors contributing more than double the number of fractionalized articles than #2 Harvard does. York authors contribute more than double the number of fractionalized articles than #2 Harvard. Source: SciVal Spotlight 24

University of York DC #1 – Leading Authors If we look at the list of leading authors globally, it is notable that the UK has major strengths in this area, with top authors coming from 4 UK institutions. Dr. Sculpher, a faculty member at York, has by far the largest citation counts of the group. The UK has major strengths in this area, with top authors coming from 4 UK institutions. Dr. Sculpher has by far the largest citation counts of the group Source: SciVal Spotlight 25

University of York – Matrix View EC #49 DC #1 This is the matrix view of York’s 105 competencies—using the same data as displayed in the circle map, but plotting it in a different manner. The Y axis indicates the rapidity of the interdisciplinary field’s year-over-year growth, while the x-axis indicates University of York’s share of that competency. Each bubble on the right-hand side of the vertical line in the middle of the graph is a competency in which York is #1 globally. Let’s take a look at EC #49, which is a small competency where York is currently #1 and a field which is growing extremely rapidly. This chart shows York’s 105 competencies in a different way. The Y axis indicates the rapidity of the interdisciplinary field’s year-over-year growth, while the x-axis indicates University of York’s share of that competency. Each bubble on the right-hand side of the vertical line is a competency in which York is #1 globally. Emerging Competency (EC) #49 is a one where York is currently #1 and the sub-field is growing extremely rapidly. It pertains to Magnetic Resonance and Catalytic Chemistry Source: SciVal Spotlight 26

Bristol and Musketeers: distinctive competencies, 2009 University of York’s competencies vs. some comparators York: 105 competencies Bristol: 171 competencies Source: SciVal Spotlight Bath: 84 competencies UCL: 226 competencies

University of York: collaboration within its 105 competencies Again, the picture from the list of leading institutions is a mixed one. York has clearly published the most articles in this interdisciplinary field, but in terms of citation counts, Max Planck, Liverpool, and UCLA are as productive or more productive on a per-article basis. This has been a very brief exploration of a few of the distinctive and emerging research competencies at the University of York. We hope that it gives the sense of the specificity and detail of the analysis that can be provided for an individual competency. For each competency, there are different competing authors, different competing and collaborating institutions, and a different research dynamic that Spotlight can illuminate. In conjunction with existing performance, planning and funding tools, Spotlight can help to guide institutional research strategies for senior research executives. This analysis shows the extent of collaboration (defined as co-authorship) within York’s 105 competencies among comparator institutions. For example, York and Imperial College London share 30 competencies, but its researchers are only collaborating within 9 of them. Within its competencies, York collaborates most with Leeds of its stated comparators. Note: this chart shows only collaboration within SciVal Spotlight competencies: it is not a measure of overall collaboration. 28

Metrics on Access Level

Access by content type, by region Access to research articles by region n=3759 n=2940 n=1262 n=1653 n=2989 n=2118 n=1294 n=2565 n=1868 n=2273 n=841 n=2362 Eastern Europe 84% Western Europe 94% North America 97% Middle East 85% APAC 91% Since we care deeply about these three things - access, quality and sustainability – it is essential that we can measure where we are with respect to each so that we can calibrate the impact of any innovations on the publishing system. On Access, we have done extensive research to see how satisfied are researchers with access today: a recent study has shown from 4,000 people that were surveyed around the world, of all the different kinds of content they use in their research, access to journal articles is the one they are the most satisfied with. 93% of the researchers globally find access to journals either easy or very easy. The area where researchers have the most difficultly with access is data sets. As publishers, we will work to find ways to make experimental data available. As you can see, there are some variations geographically as you would expect. North America has the highest and Africa has the lowest, but generally, very high levels of satisfaction globally. Latin America 88% Africa 78% 30

Access to scientific research data sets: major challenge today Very high importance , very high satisfaction High importance , low satisfaction Elsevier is taking steps to facilitate access to experimental data sets Link data sets to journal articles, e.g. Pangaea, CCDC Support and drive guidelines with key partners, e.g. Wellcome Trust NSF Bill and Melinda Gates Foundation Note: 4,109 researcher respondents, global study

I am willing to allow other researchers to access my raw research data Why scientists don’t share data Having access to other researchers’ data benefits/would benefit my own research I am willing to allow other researchers to access my raw research data Total Chemistry Computer Science Earth & Env Sciences Engineering Life Sciences Materials Science Maths Medicine & Allied Health Physics & Astronomy SocSci + Arts Hum + Economics  Author Reader dichotomy more want access than are willing to share Other survey results show that scientists are happy to have access to others’ data, but are distinctly less willing to allow access to their own data by others—this is really consistent and quite striking across fields.  Significant difference between subset and total Significant difference between 2005 and 2010 data

Where is open access today? % of STM articles 1. Author pays journals 1% Comments 95%+: BioMed Central, PLoS, Hindawi 2005 2% 1% 10% 7% 2010 2011 2% 2. Sponsored articles <1% 25% of articles published in journals with the option 1% 3. Delayed Access 2% at <6 months; 4% at 12 months; 1% at 18 months+ Primarily life and health sciences 7% 7% So, where is open access today? We have been tracking these different kinds of open access for a number of years. Author pays journals in 2005 were around about 1%. Sponsored articles of course were brand new at the time so were less than 1% in 2005. Delayed access: around 7% of the articles were made available on a delayed access basis and around 5% of authors archived their manuscripts. There has been an awful lot of talk about open access and there continues to be. But if you actually look at the uptake, in 2010 we have seen that the percentages of uptake are actually largely the same. The only type that is really showing any movement, in terms of percentage of articles published, is manuscript archiving. There’s an increasing number of institutions and funding bodies that mandate that manuscripts are posted so now that has grown by 7%. 4. Manuscript posting 5% Most journals allow preprint and/or manuscript posting Increasing number of posting policies 10%

In conclusion: research performance evaluation from all angles....... + Standard tools New tools Publications Competencies Interdisciplinary, new areas Citations Usage Different perspective; Social Sciences & Humanities more visible Patents Collaboration networks Brain circulation Researcher mobility and attraction London │17 March 2011