1. Performance Measurement: a Publisher’s Perspective
9th Northumbria International Conference on Performance Measurement in Libraries and Information Services
Performance Measurement: a Publisher’s Perspective
Dr. Michiel Kolman
Senior Vice President
Global Academic Relations August 2011

2. Intro – research metrics “exciting times are here!”
August 2011 Outsell* report:
Google Scholar and Microsoft Academic Search have both just announced new functionalities, currently in beta, that cut across some of the features presented by commercial offerings such as Thomson Reuters’ Web of Knowledge and Elsevier’s Scopus.
This presentation:
Research Performance Metrics at
Country level
Institutional level
(if time allows)
Metrics on Access.
First: the underlying database is Scopus.....
*By David Bousfield, Vice President & Lead Analyst, Outsell, August 10, 2011

3. Scopus - background
Scopus is the world’s largest multidisciplinary abstract
and citation database with peer-reviewed research
literature, quality web sources, patents and more.
Scopus covers 18,000 + titles from more than 5,000
global publishers.
Health Sciences (100% Medline coverage): 6,800
Physical Sciences: 7,200
Life Sciences: 4,300
Social Sciences & Humanities: 5,300
Scopus contains over 20 million researcher profiles with author and affiliation disambiguation, and affiliation identification for all institutions that a researcher is associated with.
Scopus is used by many research institutions globally to analyse research outputs, e.g.
Scopus is the world’s largest abstracting and citations database, and contains over 20 million researcher profiles that are disambiguated for author name and affiliation identifications over the entire course of a researcher’s working life. 3

4. Scopus – background (continued)
Scopus is the world’s largest abstracting and citations database, and contains over 20 million researcher profiles that are disambiguated for author name and affiliation identifications over the entire course of a researcher’s working life. 4

5. Metrics at a Country Level

6. Country share of global published journal articles
Share of total global articles published
(annual increase in total number of articles of 5.3%)
(Bold indicates in global top 10) 32 30 28
Publications CAGR 26 24 22 20
USA
1.6%
China*
19.5% 12
How is this changing the global research landscape? Take China for example: in 1996 China had a 3% share of global published articles. By 2009 this was 14% and because it continues to increase R&D spending by over 20% per year it is expected that China will become the world’s leading publisher of scientific articles (ahead of the USA) possibly as early as 2013.
India, who has grown its publication output by an average of 8% per year since 1996, became a global top 10 publisher of scientific information in the last 5 years. South Africa has also managed to increase its publications by over 6% per year since 1996, against a global total growth rate of 5.3%. (South Africa is current 34th in share of publications, just after Romania.) UK
3.2% 10
Germany
3.5% 8
Japan
1.8% 6
India*
8.1% 4
Brazil*
12.5%
Russia
0.7% 2
South Africa*
6.1% 6
Source: Scopus

7. Research collaboration map: e.g. for South Africa
However, we also see that collaboration between developing countries is still minimal. For example, a recent study showed that out of all African biomedical research papers published from , 77% were produced with international partners, but just 5% were the result of collaborations with another African country.
This collaboration map shows the major collaboration partners of South Africa, at a country level, based on co-authored articles. The thicker the line connecting the nodes, the higher the volume of co-authored articles.
Not surprisingly, the US accounts for the largest number of co-authored articles reflecting the size of its research base. We also see that the UK, Germany, Australia, Canada are all important collaboration partners. But, although further out on the collaboration map, we also see increased collaboration with other developing countries.
Note: Collaboration relationships are shown for papers authored by South African researchers counted in a 5-year window and are represented as variable-thickness lines (edges) between nodes. Line thickness represents the share of collaboration to or from the connected Visualisation is by the Force Atlas algorithm, which treats the network of edges as a system of interconnected springs and seeks to satisfy the tension of all edges simultaneously in a 2D rendering; hence, countries sharing a collaborative relationship tend to group together, while those that do not are placed further apart. 7
Source: Scopus

8. Peer-reviewed materials from past 5 years
Scival Spotlight: Publication leadership at a country level: leadership in article 1. output, 2. citations and 3. currency (State of the Art)
Peer-reviewed materials from past 5 years
‘Highly-cited’ A B C D E
Competency No. 1
Competency No. 2
Paper 1
Ref. Paper A
Ref. PaperB
Paper 2
Paper 3
Ref. Paper B
Co-Citation Analyses on ‘highly-cited’ references
Papers from the country in question A B C D E
References
~2 million references
~15-16 million references
To generate a map (e.g. 2009), we take all the peer-reviewed articles that have been published and indexed in Scopus for the past 5 years (i.e ).; and we reviewed all the references in the articles (about million references). We identify only ‘highly-cited’ references. This whittles down the relevant references to about 2 million. We than analyse the co-citations made in these references and identify natural clusters that form. Next we identify the papers from India within the clusters and clusters become competencies when the number of papers within them meet the appropriate threshold.
Ref: Ref: ‘Co-Citation Analysis: Methodology of SciVal Spotlight’ 8

9. Multi-disciplinary strengths
Finland counts 25 competencies at global level, rather covering the whole spectrum of research
MATH & PHYSICS
COMPUTER SCIENCE
25 competencies of which 14 distinctive competency and 11 emerging competencies
Strong position in Biology but also in Computer Science and Medical Specialties
Several multidisciplinary competencies already exist  could be expanded
SOCIAL
SCIENCES
CHEMISTRY
HUMANITIES
circles away from circumference
circles with multi-coloured bars
Multi-disciplinary strengths
BRAIN
RESEARCH
ENGINEERING
HEALTH
SCIENCE
EARTH
SCIENCES
BIOLOGY
MEDICAL
SPECIALTIES
BIOTECHNOLOGY 9
INFECTIOUS DISEASES
Source: SciVal Spotlight 2009 (November 2010)

10. Scival Spotlight Country Maps:Different countries have different “maps”
Total 401 Competencies
Total 1,707 Competencies
Total 398 Competencies
Source: SciVal Spotlight 2009 (November 2010) 10

11. Sample 1 for a list of basic indices by countries
Key metrics per country: article output, citations per article (CPA), State of the Art (SotA)
Source: SciVal Spotlight 2009 (November 2010) 11

12. Brain Drain: where do the brains from Japan end up? A prototype....

13. Brain Drain prototype for Japan: in which discipline are most brains exported from Japan?
Note: Number of researchers can be double counted across discipline. This is caused by papers in multiple disciplines

14. Exporting Institutions of brains from Japan

15. Selected ‘brains’ that moved out of Japan

16. Metrics at an Institutional Level

17. University of York, share of research outputs: 2003-2008
Size of bubble proportional to 5-year article output
2008
2007
2006
2005
2004
2003
York has steadily increased its share of article outputs in the UK from 2003 to 2008
Its share of UK articles increased from 1.05% in 2003 to 1.39% in 2008
Its share of global articles increased from 0.074% in 2003 to 0.090% in 2008
York has achieved this against a backdrop of declining UK article share
The UK’s share of global research articles fell from 7.1% in 2003 to 6.4% in 2008
The UK’s share decline is common to many scientifically developed nations (e.g. US, Germany, France) and reflects the rapidly rising share of countries in Asia, the Middle East and South America
Note: Data shows York’s article outputs (research articles, reviews and conference papers) and shares using 5 year periods, e.g. 2008
corresponds to publications . 17
Source: Scopus 17

18. University of York, impact of research outputs, 2003-2008
Size of bubble proportional to field-weighted impact relative to world
York’s 5-year citation impact is high and rising in absolute terms. Its mean unweighted citation impact for the 5 years ending 2008 was compared to for the UK average and for the global average
York’s field-weighted citation impact is also high, standing at in 2008 (compared to a world average of and UK average of 1.338)
Note: Data shows unweighted citation impact and relative field-weighted impact using 5-year periods, e.g corresponds to publications and citations. Field weightings account for shifting research portfolios in fields with differing citation profiles.
Source: Scopus 18

19. University of York research outputs: rates of collaboration
37.5% of York’s articles are co-authored with non-UK researchers in 2008, (vs. 41.3% for the UK average in 2008, and 33.3% for the global average in 2008)
Yorks’s rate of international collaboration has been steadily increasing, as has the average for the UK and globally
International scientific collaboration is generally acknowledged as a positive force driving institutional and national impact and prestige
Note: Data shows proportion of article outputs representing international collaboration (where one or more other countries are listed in the author address) in 5-year periods, e.g corresponds to publications and citations. 19
Source: Scopus 19

20. International collaboration rates correlates strongly with publication impact
Field-weighted relative impact 1 2 3 4 5
Number of collaborating countries (where 1 = domestic)
International scientific collaboration is generally acknowledged as a positive force driving national impact and prestige
Domestic articles (‘1’) have no collaboration partners have around 3 times fewer citations per article than those with four collaborating countries (‘5’) 20
Source: Scopus 20 20

21. University of York – map of competencies
This is a circle map of York’s research competencies, which number 105 in total. These range from large “distinctive competencies” to smaller “emerging competencies,” as determined by fractional article counts in the relevant interdisciplinary field. As can be seen from an initial examination, York has particular strength in a number of social scientific, computer science, and biology and biotechnology disciplines.
The diagram shows that York has 105 distinctive research competencies as defined by Elsevier’s SciVal Spotlight tool
These range from large “distinctive competencies” to smaller “emerging competencies,” as determined by fractional article counts in the relevant interdisciplinary field.
York has particular strength in a number of social scientific, computer science, and biology and biotechnology disciplines.
Source: SciVal Spotlight 21

22. University of York DC #1 – Pharmaco Economics and Public Health
York’s largest research competency in terms of number of fractionalized article counts is an interdisciplinary competency related to pharmacoeconomics and public health. York is a leader in this area, with the highest number of articles and citations globally.
York’s largest research competency in terms of number of fractionalized article counts is an interdisciplinary competency related to Pharmaco Economics and public health.
York is a leader in this area, with the highest number of articles and citations globally.
Source: SciVal Spotlight 22

23. University of York DC #1 – Pharmaco Economics and Public Health
York’s largest research competency is distinctive in all respects: it is a large area of research in which York researchers are prolific, highly cited and are citing the most recent research relative to peers
Source: SciVal Spotlight

24. University of York DC #1 -- Leading Institutions
Similarly, York’s primacy in this field can be seen on the institutional table as well, with York authors contributing more than double the number of fractionalized articles than #2 Harvard does.
York authors contribute more than double the number of fractionalized articles than #2 Harvard.
Source: SciVal Spotlight 24

25. University of York DC #1 – Leading Authors
If we look at the list of leading authors globally, it is notable that the UK has major strengths in this area, with top authors coming from 4 UK institutions. Dr. Sculpher, a faculty member at York, has by far the largest citation counts of the group.
The UK has major strengths in this area, with top authors coming from 4 UK institutions.
Dr. Sculpher has by far the largest citation counts of the group
Source: SciVal Spotlight 25

26. University of York – Matrix View
EC #49
DC #1
This is the matrix view of York’s 105 competencies—using the same data as displayed in the circle map, but plotting it in a different manner. The Y axis indicates the rapidity of the interdisciplinary field’s year-over-year growth, while the x-axis indicates University of York’s share of that competency. Each bubble on the right-hand side of the vertical line in the middle of the graph is a competency in which York is #1 globally. Let’s take a look at EC #49, which is a small competency where York is currently #1 and a field which is growing extremely rapidly.
This chart shows York’s 105 competencies in a different way. The Y axis indicates the rapidity of the interdisciplinary field’s year-over-year growth, while the x-axis indicates University of York’s share of that competency.
Each bubble on the right-hand side of the vertical line is a competency in which York is #1 globally.
Emerging Competency (EC) #49 is a one where York is currently #1 and the sub-field is growing extremely rapidly. It pertains to Magnetic Resonance and Catalytic Chemistry
Source: SciVal Spotlight 26

27. Bristol and Musketeers: distinctive competencies, 2009
University of York’s competencies vs. some comparators
York: 105 competencies
Bristol: 171 competencies
Source: SciVal Spotlight
Bath: 84 competencies
UCL: 226 competencies

28. University of York: collaboration within its 105 competencies
Again, the picture from the list of leading institutions is a mixed one. York has clearly published the most articles in this interdisciplinary field, but in terms of citation counts, Max Planck, Liverpool, and UCLA are as productive or more productive on a per-article basis.
This has been a very brief exploration of a few of the distinctive and emerging research competencies at the University of York. We hope that it gives the sense of the specificity and detail of the analysis that can be provided for an individual competency. For each competency, there are different competing authors, different competing and collaborating institutions, and a different research dynamic that Spotlight can illuminate. In conjunction with existing performance, planning and funding tools, Spotlight can help to guide institutional research strategies for senior research executives.
This analysis shows the extent of collaboration (defined as co-authorship) within York’s 105 competencies among comparator institutions. For example, York and Imperial College London share 30 competencies, but its researchers are only collaborating within 9 of them.
Within its competencies, York collaborates most with Leeds of its stated comparators.
Note: this chart shows only collaboration within SciVal Spotlight competencies: it is not a measure of overall collaboration. 28

29. Metrics on Access Level

30. Access by content type, by region
Access to research articles by region
n=3759
n=2940
n=1262
n=1653
n=2989
n=2118
n=1294
n=2565
n=1868
n=2273
n=841
n=2362
Eastern
Europe
84%
Western
Europe
94%
North
America
97%
Middle
East
85%
APAC
91%
Since we care deeply about these three things - access, quality and sustainability – it is essential that we can measure where we are with respect to each so that we can calibrate the impact of any innovations on the publishing system. On Access, we have done extensive research to see how satisfied are researchers with access today: a recent study has shown from 4,000 people that were surveyed around the world, of all the different kinds of content they use in their research, access to journal articles is the one they are the most satisfied with. 93% of the researchers globally find access to journals either easy or very easy. The area where researchers have the most difficultly with access is data sets. As publishers, we will work to find ways to make experimental data available. As you can see, there are some variations geographically as you would expect. North America has the highest and Africa has the lowest, but generally, very high levels of satisfaction globally.
Latin
America
88%
Africa
78% 30

31. Access to scientific research data sets: major challenge today
Very high importance , very high satisfaction
High importance , low satisfaction
Elsevier is taking steps to facilitate access to experimental data sets
Link data sets to journal articles, e.g. Pangaea, CCDC
Support and drive guidelines with key partners, e.g.
Wellcome Trust
NSF
Bill and Melinda Gates Foundation
Note: 4,109 researcher respondents, global study

32. I am willing to allow other researchers to access my raw research data
Why scientists don’t share data
Having access to other researchers’ data benefits/would benefit my own research
I am willing to allow other researchers to access my raw research data
Total
Chemistry
Computer Science
Earth & Env Sciences
Engineering
Life Sciences
Materials Science
Maths
Medicine & Allied Health
Physics & Astronomy
SocSci + Arts Hum + Economics 
Author Reader dichotomy more want access than are willing to share
Other survey results show that scientists are happy to have access to others’ data, but are distinctly less willing to allow access to their own data by others—this is really consistent and quite striking across fields. 
Significant difference between subset and total
Significant difference between 2005 and 2010 data

33. Where is open access today?
% of STM articles
1. Author pays journals 1%
Comments
95%+: BioMed Central, PLoS, Hindawi
2005 2% 1%
10% 7%
2010
2011 2%
2. Sponsored articles
<1%
25% of articles published in journals with the option 1%
3. Delayed Access
2% at <6 months; 4% at 12 months; 1% at 18 months+
Primarily life and health sciences 7% 7%
So, where is open access today? We have been tracking these different kinds of open access for a number of years. Author pays journals in 2005 were around about 1%. Sponsored articles of course were brand new at the time so were less than 1% in Delayed access: around 7% of the articles were made available on a delayed access basis and around 5% of authors archived their manuscripts. There has been an awful lot of talk about open access and there continues to be. But if you actually look at the uptake, in 2010 we have seen that the percentages of uptake are actually largely the same. The only type that is really showing any movement, in terms of percentage of articles published, is manuscript archiving. There’s an increasing number of institutions and funding bodies that mandate that manuscripts are posted so now that has grown by 7%.
4. Manuscript posting 5%
Most journals allow preprint and/or manuscript posting
Increasing number of posting policies
10%

34. In conclusion: research performance evaluation from all angles.......+
Standard tools
New tools
Publications
Competencies
Interdisciplinary, new areas
Citations
Usage
Different perspective; Social Sciences & Humanities more visible
Patents
Collaboration networks
Brain circulation
Researcher mobility and attraction
London │17 March 2011