Download presentation
Presentation is loading. Please wait.
Published byあいぞう おおはし Modified over 5 years ago
1
How good is our research? New approaches to research indicators
June 2006
2
Average is a metric; distribution is a picture
‘Average impact’ is a good bibliometric index but not sufficient A tool for reporting but not for action Data are skewed, so average is not central Many papers are uncited and a few papers are very highly cited New approach looks at where the spread of performance falls Activity is located within distribution by more than a single metric Thresholds help in describing peak of performance. This improves descriptive power, information content and management value June 2006
3
Traditional impact indicators
June 2006
4
Distribution of “research performance”
June 2006
5
Distribution of “research performance”
A good ‘indicator’ should capture and reflect this in some meaningful way Do current metrics do this? June 2006
6
Distribution of data values - income
Minimum Maximum June 2006
7
Distribution of data values - impact
The variables for which we have metrics are skewed and therefore difficult to picture in a simple way June 2006
8
Simplifying the data picture
Scale data relative to a benchmark, then categorise Could do this for any data set All journal articles Uncited articles (take out the zeroes) Cited articles Cited less often than benchmark Cited more often than benchmark Cited more often but less than twice as often Cited more than twice as often Cited less than four times as often Cited more than four times as often June 2006
9
Categorising the impact data
This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale. June 2006
10
UK ten-year profile 680,000 papers
MODE (cited) MODE AVERAGE RBI = 1.24 MEDIAN THRESHOLD OF EXCELLENCE? June 2006
11
Implications Is the UK as good as we thought? New metrics are needed
YES - the average is unchanged What lies beneath just became apparent The effective peak is very concentrated Other countries would probably look similar New metrics are needed Average impact not indicative of distribution Need to add median, mode Proportion of activity at thresholds of excellence Above world average, More than 4 x world average, etc Evaluate methodology Does it work by year and by subject? How can we apply it? June 2006
12
Time profile June 2006
13
Subject based curves June 2006
14
Subject & site profiles – molecular biology
June 2006
15
HEIs – 10 year totals - 1 June 2006
16
HEIs – 10 year totals - 2 June 2006
17
HEIs – 10 year totals – 4.1 Smoothing the lines would reveal the shape of the profile June 2006
18
HEIs – 10 year totals – 4.2 Absolute volume would add a further element for comparisons June 2006
19
HEIs – 10 year totals – 4.3 June 2006
20
What next? Profiles Applications
Create a view of the distribution of performance Provide more information useful to management Require a change in metrics Applications Disaggregate the components of the research base Track institutional profiles against benchmark Evaluate the link between platform and peak Track papers through time: e.g. leaders vs. climbers June 2006
21
How good is our research? New approaches to research indicators
June 2006
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.