1. How good is our research? New approaches to research indicators
June 2006

2. Average is a metric; distribution is a picture
‘Average impact’ is a good bibliometric index but not sufficient
A tool for reporting but not for action
Data are skewed, so average is not central
Many papers are uncited and a few papers are very highly cited
New approach looks at where the spread of performance falls
Activity is located within distribution by more than a single metric
Thresholds help in describing peak of performance.
This improves descriptive power, information content and management value
June 2006

3. Traditional impact indicators
June 2006

4. Distribution of “research performance”
June 2006

5. Distribution of “research performance”
A good ‘indicator’ should capture and reflect this in some meaningful way
Do current metrics do this?
June 2006

6. Distribution of data values - income
Minimum
Maximum
June 2006

7. Distribution of data values - impact
The variables for which we have metrics are skewed and therefore difficult to picture in a simple way
June 2006

8. Simplifying the data picture
Scale data relative to a benchmark, then categorise
Could do this for any data set
All journal articles
Uncited articles (take out the zeroes)
Cited articles
Cited less often than benchmark
Cited more often than benchmark
Cited more often but less than twice as often
Cited more than twice as often
Cited less than four times as often
Cited more than four times as often
June 2006

9. Categorising the impact data
This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale.
June 2006

10. UK ten-year profile 680,000 papers
MODE (cited)
MODE
AVERAGE
RBI = 1.24
MEDIAN
THRESHOLD OF EXCELLENCE?
June 2006

11. Implications Is the UK as good as we thought? New metrics are needed
YES - the average is unchanged
What lies beneath just became apparent
The effective peak is very concentrated
Other countries would probably look similar
New metrics are needed
Average impact not indicative of distribution
Need to add median, mode
Proportion of activity at thresholds of excellence
Above world average, More than 4 x world average, etc
Evaluate methodology
Does it work by year and by subject?
How can we apply it?
June 2006

12. Time profile
June 2006

13. Subject based curves
June 2006

14. Subject & site profiles – molecular biology
June 2006

15. HEIs – 10 year totals - 1
June 2006

16. HEIs – 10 year totals - 2
June 2006

17. HEIs – 10 year totals – 4.1
Smoothing the lines would reveal the shape of the profile
June 2006

18. HEIs – 10 year totals – 4.2
Absolute volume would add a further element for comparisons
June 2006

19. HEIs – 10 year totals – 4.3
June 2006

20. What next? Profiles Applications
Create a view of the distribution of performance
Provide more information useful to management
Require a change in metrics
Applications
Disaggregate the components of the research base
Track institutional profiles against benchmark
Evaluate the link between platform and peak
Track papers through time: e.g. leaders vs. climbers
June 2006

21. How good is our research? New approaches to research indicators
June 2006