Paris, May 2007 How good is the research base? New approaches to research indicators Colloque de l’Académie des sciences "Évolution des publications scientifiques" mai 2007

Paris, May 2007 What is Evidence? Research performance analysis and interpretation –Founded 2000, grew from government and HE research management Studies for e.g. UUK, HEFCE, OST, Defra, EC, universities –Annual OSI PSA target indicators for UK science and engineering –Research funding and impact studies across research base –Current work for Austria, New Zealand, Sweden Quantitative research analysis products –Overview of complete research process Funding, activity and outputs in detailed, structured and mapped databases Data reconciled to subject areas and institutions –Higher Education Research Yearbook (5 th ed’n) –Indicator applications, Publication databases, Research profiling

Paris, May 2007 Linking indicators to management information ‘Average impact’ is a good bibliometric index but not sufficient –A great tool for reporting but not for action –Average is a metric; distribution is a picture Data are skewed, so average is not central –Many papers are uncited and a few papers are very highly cited New approach looks at where the spread of performance falls –Activity is located within distribution by more than a single metric –Thresholds help in describing peak of performance This improves descriptive power, information content and management value

Paris, May 2007 Traditional impact indicators are excellent for international reports

Paris, May 2007 Bibliometrics track increase in UK share of world citations in response to research assessment

Paris, May 2007 Impact index is coherent across UK grade levels - data are for core science disciplines, grade at RAE96

Paris, May 2007 Chemistry – alternative bibliometric indices, both correlate with our mapping to RAE grade Each data point is an institution

Paris, May 2007 Bibliometric impact ( ) is related to RAE2001 grade for UoA14 Biology

Paris, May 2007 Assumed distribution of “research performance”

Paris, May 2007 Actual distribution of data values The variables for which we have metrics are skewed and therefore difficult to picture in a simple way

Paris, May 2007 Simplify the data picture Scale data relative to a benchmark, then categorise –Could do this for any data set All journal articles –Uncited articles (take out the zeroes) –Cited articles Cited less often than benchmark Cited more often than benchmark –Cited more often but less than twice as often –Cited more than twice as often »Cited more often but less than four times as often »Cited more than four times as often

Paris, May 2007 Categorise the impact data This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale.

Paris, May 2007 UK Impact Profile TM [10 years; 680,000 papers] AVERAGE RBI = 1.24 MODE (cited) MEDIAN THRESHOLD OF EXCELLENCE? MODE

Paris, May 2007 Implications Is the UK research base as good as we thought? –YES - the average is unchanged –What lies beneath just became apparent The ‘peak’ of high impact is very concentrated Evaluate Impact Profile TM methodology –Do other countries look similar? Yes, we profiled the USA as well –Does it work by year and by subject? See Scientometrics, Vol. 72, No. 2 (2007) 325–344 –How can we apply it?

Paris, May 2007 Impact Profiles TM for subjects & sites – molecular biology

Paris, May 2007 Impact Profiles TM for international institutes Location USA EMBL UK France Japan

Paris, May 2007 Where does this take us? New metrics would advance utility and application –‘Average impact’ is not indicative of distribution –Should we also use e.g. median, mode? –Index proportion of activity at thresholds of excellence? Above world average, More than 4 x world average, etc Weight the categories to produce a single metric –A single metric is still useful for reporting –How do we treat uncited? –How much do we value the truly exceptional? A picture has value in itself –It has descriptive power beyond a simple index –It enables rapid and transparent comparisons for the less expert –It helps us locate specific activity within a profile

Paris, May How good is the research base? New approaches to research indicators