iREG Forum on University Rankings May 2013 Dr Lisa Colledge Snowball Metrics Program Director Snowball Metrics 1

Snowball Metrics are… Endorsed by a group of distinguished UK universities to support their strategic decision making Tried and tested methodologies that are available free-of-charge to the higher education sector Absolutely clear, unambiguous definitions enable apples-to-apples comparisons so universities can benchmark themselves against their peers to judge the excellence of their performance Snowball Metrics are unique because: Universities drive this bottom up Academia – industry collaboration 2

Trends in Research Management 3 Growing recognition of the value of data/metrics to inform and monitor research strategies, to complement but not replace existing methods Unless you have [data] you cannot make informed decisions; you would be acting based on opinions and hearsay. Frustration over the lack of a manageable set of standard metrics for sensible measurements [There is little] thought leadership and knowledge development around best practice. Frequent similar data requests from external bodies looking at performance in a way that is not necessarily of most value to universities themselves The principle drivers for our systems are often external… but they shouldnt be… a research strategy should… be developed… to respond to our strengths and the external environment, our systems should be defined to run our business.

University-driven (bottom-up) benchmarking is very important 4 This report recommended that universities and funders should work more collaboratively, and develop stronger relationships with suppliers Universities need to benchmark themselves to know their position relative to their peers, so they can strategically align resources to their strengths and weaknesses Universities should work together more to make their collective voice heard by external agencies. The lack of a long-term vision makes it hard to… co-operate within a university let alone across the sector. Suppliers do not know what research offices do on a daily basis. How educated are we at asking suppliers the right questions?

Snowball Metrics addresses university-driven benchmarking Someone needs to take ownership of the process: it is impossible to please all of the people all of the time so somebody needs to be strong enough to stand behind decisions and follow through. 5 Snowball Metrics Project Partners It would be great if the top five [universities] could collaborate

The project partners… Agree a pragmatic approach from the point of view of the research office Endorse metrics to generate a dashboard that supports university strategy Draw on and combine university, proprietary and third party / public data Ensure that the metrics can be calculated, and in the same way by universities with different systems and data structures 6

Main roles and responsibilities Everyone is responsible for covering their own costs University project partners – Agree the metrics to be endorsed as Snowball Metrics – Determine methodologies to generate the metrics in a commonly understood manner to enable benchmarking, regardless of systems Elsevier – Ensures that the methodologies are feasible when applied to real data, prior to publication of the recipes to share with the sector – Distribute the recipes using our communications networks – Day-to-day project management of the global program Outside the remit of the Snowball Metrics program – Nature and quality of data sources used to generate Snowball Metrics – Provision of tools to enable the global sector to generate and use Snowball Metrics 7

Snowball Metrics are feasibile 8

Metrics can be size-normalised 9

Metrics can be sliced and diced 10

Recipe Book shares the methods with the sector free of charge 11 Input Metrics - Applications Volume - Awards Volume Process Metrics - Income Volume - Market Share Output Metrics - Scholarly Output - Citation Count - h-index - Field-Weighted Citation Impact - Publications in Top Percentiles - Collaboration University and discipline levels only First set of Snowball Metrics

Elsevier and Snowball Metrics 12 Declaration from the project partners Agreed and tested methodologies… are and will continue to be shared free-of- charge None of the project partners will at any stage apply any charges for the methodologies Any organisation can use these methodologies for their own purposes, public service or commercial (Extracts from Statement of intent, October 2012) Universities are also requesting the provision of calculated metrics Some organisations do not want to use the recipe book themselves, and are approaching Elsevier for help to implement and use Snowball Metrics Elsevier charges for our support in this, and can offer the metrics in a Custom Report, and in Pure (Current Research Information System) We plan to continue to build commercial tools to help any universities who want to adopt Snowball Metrics but prefer not to generate them in house

Global benchmarking 13 Research InputsResearch Processes Research Outputs and Outcomes Research Post-Graduate Education Enterprise Activities Research applications Research awards Research income Publications & citations Collaboration (co- authorship) Impact / Esteem Post-graduate research Post-graduate experience Industrial income and engagement Contract turnaround times Industry research income Patenting Licensing income Spin-out generation / income Completion rates People Organisations Themes / Schemes Researchers Role Institution Institutional unit External groupings Funder type Award type Subject area / keywords Denominators Slice and dice Normalise for size Numerators Denom. Vision: Snowball Metrics drive quality and efficiency across higher educations research and enterprise activities, regardless of system and supplier

Achieving the vision Continue to agree, test and share new metrics to illuminate research and enterprise activities Facilitate adoption by the sector by translating the metrics into standard data formats that can be easily understood by systems Ensure that Snowball Metrics support global benchmarking as far as possible 14 Vision: Snowball Metrics drive quality and efficiency across higher educations research and enterprise activities, regardless of system and supplier

Global vs national standards for benchmarking Snowball Metrics start life with a national perspective – currently UK The aim is to promote all aspects of Snowball Metrics as far as possible to a global standard 15 UK metrics Country 2 metrics Country 1metrics Illustrative only, testing underway Common core where benchmarking against global peers can be conducted. Aim is to make this as big as possible Shared features where benchmarking between Countries 1 and 2, but not UK, can be conducted e.g. regional benchmarking National peculiarity can support benchmarking within Country 1, but not globally i.e. national benchmarking

Possible end point of a metric 16 Version enabling global benchmarking Multiple versions enabling regional benchmarking Multiple versions enabling national benchmarking e.g. Discipline represented by a universal journal classification e.g. Discipline represented by subject mapping of a regional body e.g. Discipline represented by the UKs HESA Cost Centres Increasing circle of peers for benchmarking Illustrative only, testing underway

THANK YOU FOR YOUR ATTENTION! Contact Dr Lisa Colledge or