June 2006 How good is our research? New approaches to research indicators

June 2006 Average is a metric; distribution is a picture Average impact is a good bibliometric index but not sufficient –A tool for reporting but not for action Data are skewed, so average is not central –Many papers are uncited and a few papers are very highly cited New approach looks at where the spread of performance falls –Activity is located within distribution by more than a single metric –Thresholds help in describing peak of performance. This improves descriptive power, information content and management value

June 2006 Traditional impact indicators

June 2006 Distribution of research performance

June 2006 Distribution of research performance A good indicator should capture and reflect this in some meaningful way Do current metrics do this?

June 2006 Distribution of data values - income MaximumMinimum

June 2006 Distribution of data values - impact The variables for which we have metrics are skewed and therefore difficult to picture in a simple way

June 2006 Simplifying the data picture Scale data relative to a benchmark, then categorise –Could do this for any data set All journal articles –Uncited articles (take out the zeroes) –Cited articles Cited less often than benchmark Cited more often than benchmark –Cited more often but less than twice as often –Cited more than twice as often »Cited less than four times as often »Cited more than four times as often

June 2006 Categorising the impact data This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale.

June 2006 UK ten-year profile 680,000 papers AVERAGE RBI = 1.24 MODE (cited) MEDIAN THRESHOLD OF EXCELLENCE? MODE

June 2006 Implications Is the UK as good as we thought? –YES - the average is unchanged –What lies beneath just became apparent The effective peak is very concentrated –Other countries would probably look similar New metrics are needed –Average impact not indicative of distribution –Need to add median, mode –Proportion of activity at thresholds of excellence Above world average, More than 4 x world average, etc Evaluate methodology –Does it work by year and by subject? –How can we apply it?

June 2006 Time profile

June 2006 Subject based curves

June 2006 Subject & site profiles – molecular biology

June 2006 HEIs – 10 year totals - 1

June 2006 HEIs – 10 year totals - 2

June 2006 HEIs – 10 year totals – 4.1 Smoothing the lines would reveal the shape of the profile

June 2006 HEIs – 10 year totals – 4.2 Absolute volume would add a further element for comparisons

June 2006 HEIs – 10 year totals – 4.3

June 2006 What next? Profiles –Create a view of the distribution of performance –Provide more information useful to management –Require a change in metrics Applications –Disaggregate the components of the research base –Track institutional profiles against benchmark –Evaluate the link between platform and peak –Track papers through time: e.g. leaders vs. climbers

June 2006 How good is our research? New approaches to research indicators