1. Evaluation of impact
Understanding impact first, deciding on indicators last
The case of largescale research infrastructures
Eu-SPRI Paris – 6 June 2018 – Leonie van Drooge, Isabelle van Elzakker, Jasper Deuten

2. Impacts of an RI / ERIC – the case of ACCELERATE (anecdotal)
Content
Context of research
What is an RI / European RI consortium (ERIC) – the case of ACCELERATE (anecdotal)
Impacts of an RI / ERIC – the case of ACCELERATE (anecdotal)
Consequences for the definition / identification of impacts
Consequences for monitoring (ex durante), evaluation (ex ante and ex post) and for the identification of indicators
Towards a general approach – use of Theory of Change
What is evident for Eu-SPRI members, is not for RI management. What is evident for RI management, is not for sci-pol resesearcher.
Understanding impact - the case of largescale research infrastructures

3. Understanding impact and largescale research infrastructures
What a research infrastructure (RI) is, is not evident
What an “appropriate” impact is (societal, economic or otherwise), depends on goals, expectations, missions, funders, stakeholders, phase
Many of the impacts expected do not at all relate to the specific research the RI enables; some impacts are politically defined
It is not possible or useful to define a fixed set of impacts, pathways and indicators for practical use by RIs for monitoring and evaluation
It is possible to develop a general approach for the definition of impacts and the development of indicators for RIs
Understanding impact - the case of largescale research infrastructures

4. Vision
A Research Infrastructure and its funders / stakeholders share a common and realistic view on impacts expected and intended; they have agreed on understanding, monitoring and evaluation of impacts; they have indentified and agreed on indications for these impacts
They jointly discuss the development of impacts timely, informed by the indications; this provides – and they seize – the opportunity to reflect and learn and if needed to adapt a goal, an indication, a practice, or otherwise
The RI knows what it is responible / accountable for and what not; knows if and when it shares a responsibility with others; knows these actors and collaborates / coordinates / is in contact if, when and how necessary.
The promises and expectations regarding impact are not unrealistic, overheated or nonsensical
Understanding impact - the case of largescale research infrastructures

5. Context of research I
Dutch Rathenau Institute “stimulates public and political opinion forming on social aspects of science and technology. We perform research and organise debate relating to science, innovation and new technologies”
H2020 ACCELERATE “to help secure [large scale] research infrastructures’ sustainability, relevance and effectiveness, the project …. [defines] common protocols for monitoring and assessing RIs’ socio-economic impact.” ACCELERATE is funded by the European Union Framework Programme for Research and Innovation Horizon 2020, under grant agreement
Understanding impact - the case of largescale research infrastructures

6. Context of research II – sustainability of RIs
In Europe, new large scale research infrastructures (RIs) and new organisations of research infrastructures are being established
The sustainable operation of these infrastructures is on the agenda
Including on the agenda of H2020 / excellence pillar
One “sustainable operation issue” is the impact of RIs
There is political and social pressure to demonstrate impact
Impact is an increasingly important factor in decisions to establish or fund an RI (ex ante)
There are scientific and consultancy studies on impact of RIs, or of a specific RI (Technopolis, OECD), making use of logical frameworks or impact pathways
Understanding impact - the case of largescale research infrastructures

7. Understanding impact - the case of largescale research infrastructures

8. Context of research III – needs of RIs regarding impact (assessment)
There are potentially many diverse types of impacts possible; this is due to the variety of RIs / ERICs
It is not evident what impact is important or relevant for a RI / ERIC
“There is a need for funders such as regional, national and supranational authorities to articulate what impact they expect”
“There is a need to set up effective means of determining the economic and wider social value of RIs, and incorporate these benefits into science-policy-society dialogues”
Indicators of impact need to relate to goals
Understanding impact - the case of largescale research infrastructures

9. Present situation (from Eu-SPRI 2017)
RIs receive funding from a variety of sources, including European structural funds, regional funds, research councils (and of member states, 2018), with different requirements
RIs promise a wide range of far going impacts in all aspects of science and society, yet they are not specific
In general, RIs have very little responsibility regarding the impact to become manifest
There are many implicit assumptions
There is hardly any inclusion of stakeholders (in the definition of impact and of proper evidence / indications)
Understanding impact - the case of largescale research infrastructures

10. Understanding a research infrastructure (accelerator)
Understanding impact - the case of largescale research infrastructures

11. Members of ACCELERATE I
All members represent largescale research infrastructures
They enable materials research: characterisation of matter, from subatomair to supramoleculair scale
The research is done with equipment (beamlines, instruments) that is connected to a powerful source (accelerator, spallation source, laser)
Members are:
CERIC-ERIC Central European RI Consortium – IT
ESS European Spallation Source ERIC – SE
ELI-DC Extreme Light Infrastructure Delivery Consortium
HZG Helmoltz-Zentrum Geesthacht – DE
FRM II Research Neutron Source Heinz Maier-Leibnitz Munich Technical University – DE
Understanding impact - the case of largescale research infrastructures

12. Members of ACCELERATE II
2 European Research Infrastructure Consortiums (ERICs); 1 in the process of becoming an ERIC; 2 national facilities
2 are in construction phase; 3 are operational
1 partner governs a single sited accelerator 1 partner governs formally a distributed infrastructure, but the accelerator is single sited (the datacentre is in a neighbouring country) 1 partner governs 3 accelerators, each in a different country 1 partner owns equipment at distributed facilities (including other consortium partners) 1 partner only distributes beamtime (time to use equipment) at distributed facilities
Understanding impact - the case of largescale research infrastructures

13. Understanding impact - the case of largescale research infrastructures

14. Scientific excellence, capacity building, tech transfer,…. CERIC:
Impacts I
Scientific excellence, capacity building, tech transfer,….
CERIC:
ERIC – hence ERA (ERA priorities are politically defined and change)
Member states Italy – Romania – Eastern EU member states
coordination
ELI:
building phase funded by structural and regional development funds
Eastern EU member states
Understanding impact - the case of largescale research infrastructures

15. Member states France – the Netherlands
Impacts II
ESS:
Member states France – the Netherlands
Region of Skane (southern Sweden): (what) economic impact
Business development
FRM II:
Is a neutron source
Delivers radioisotopes for clinical use
HZG / GEMS:
Not a seperate organisational entity
“What other impacts do you think we have?”
Understanding impact - the case of largescale research infrastructures

16. Consequences for the definition / identification of impact
Impact of a RI / ERIC is highly context dependent
Context relates to phase, aim, scope, responsibilities, funders, stakeholders, intentions, expectations
RIs have options regarding impacts:
To spend time and effort, to dedicate activities, policies and practices aiming for a certain type of impact – or not (some impacts will become manifest anyway; some impacts are marginal anyway)
To identify other actors responsible for an impact and collaborate / coordinate – or not
To understand choices, activities, policies and practices, outputs and outcomes as contributing to a certain impact – or not
To report on these impacts – or not
Understanding impact - the case of largescale research infrastructures

17. Consequences for the definition / identification of impact
Some impacts relate to statutes, goals, aims, primary or secondary missions
Some impacts relate to enabling research(ers) or to building a high tech facility
Some impacts expected are regardless of the type of research the RI enables / the fact that it is a RI (even: scientific aims are not allowed)
For the sustainable operation of a RI, it is important to understand what impacts a stakeholder or funder expects (ex post and ex ante)
Director of ACCELERATE member: “we need to discuss the expectations with our funders” (see also ESFRI)
Impact is governance
Understanding impact - the case of largescale research infrastructures

18. Consequences for monitoring, evaluation, indicators
For the sustainable operation of a RI, it is important to understand what evidence of impact a stakeholder or funder accepts / expects
It is important to understand the evaluation context / practice / requirements / question / who is responsible / who evaluates (and this is not obvious)
Understanding impact - the case of largescale research infrastructures

19. A general approach for impact assessment of RIs
Identify relevant actors (funders, stakeholders) (RI)
Identify impacts relevant for these actors (RI and funders/stakeholders)
Identify other impacts relating to mission / primary goal (jointly)
Understand and unpack the impact journey that leads towards these impacts / promises - and identify indications or illustrations of progress (including cases)
Identify preconditions and assumptions
Identify who and what contributes to the realisation of the goal, and decide on whether this is within the influence of the RI
Shed light on how impact can be (or is!) organised and embedded in the RI (and what serves as evidence of the RI taking responsibility)
Understanding impact - the case of largescale research infrastructures

20. Understanding impact - the case of largescale research infrastructures

21. case
case
case
results
activities
practices
procedures
policies
strategies
preconditions
assumptions
target values
choices
inputs

22. Leonie van Drooge l.vandrooge@rathenau.nl @LeonievanDrooge
Thank you!
Leonie van
Isabelle van Elzakker
Jasper Deuten
Understanding impact - the case of largescale research infrastructures