Towards a framework for the evaluation of Transformative Innovation Policies EU-SPRI 2018. Governance and relevance: Towards a new generation of research and innovation policies Alejandra Boni, Jordi Molas-Gallart and Sergio Belda Miquel ESIEE, Paris 6 June 2018 INGENIO [CSIC-UPV] Ciutat Politècnica de la Innovació | Edif 8E | Camí de Vera s/n | 46022 València tel +34 963 877 048 fax +34 963 877 991

Such new generation approaches pose new evaluation challenges Defining the issue Transformative Innovation Policy as laid out by Schot and Steinmueller (2016) can be considered a “new generation of “research and innovation policy” Such new generation approaches pose new evaluation challenges

Key assumptions of “Transformative Innovation Policies”… To address key societal problems (e.g. as reflected in the UN Sustainable Development Goals) requires changes in socio-technical systems Systemic changes with directionality To achieve such socio-technical system the starting point is a set of “policy experiments” in “protected niches” If successful, policies are implemented to deepen and scale up initial (niche) changes

…generate evaluation problems Amplify the “attribution” problem common in Research Impact Assessment Project outcomes are a contribution (and not a determining cause) of socio-economic impacts Long “distance” between a niche and the change in socio-technical system Complexity: TIPs rely on a set of interrelated experiments operating at different levels Different projects defining local experiments operating in local niches, strategies to deepen and scale up… The processes, timelines and immediate objectives of each initiative are different Should take into account the interactions among this diverse set of policy initiatives Governance changes (including participatory processes) need to extend to evaluation methods and practices

Evaluation strategies “Outcome mapping” as a common response… Assumes that impact processes cannot be controlled by the policy There is no “theory of change” Focus on behavioural change in the communities involved in an intervention Based on participative techniques …but how can we know a specific TIP project is being successful?

An alternative approach: using a ToC in TIP evaluation TIP is based on socio-technical transitions theory and therefore embodies a generic ToC TIP evaluations should describe in detail the ToC of each specific policy consistent with the generic ToC Clearly define the different levels of policy intervention Projects in protected niches set up as experiments to stimulate local socio-technical changes Emphasizes changes in governance to support broad participation A Programme links several to enhance synergies and learning, and to protect and nurture them Seeks to deepen the scope and widen the reach of the project niches A Transformative Innovation Policy is constituted by several programmes targeting a specific socio-technical system Identify key evaluative dimensions to be assessed at every level Main actors involved Resources, processes, outcomes (and impacts where relevant)

Transformative Innovation Policies: evaluating a multi-level policy mix

Work in Progress: Evaluating a small experiment Factoría de Política de Ciencia, Tecnología e Innovación para la Transformación (Medellín, Colombia) Small initiative launched by a group of academics (6) from 2 Medellín universities Challenge: Existing HEIs practices shift the focus away from social and environmental challenges that threaten the sustainability of the current development model Local level project The initiative is supported by Colciencias and a mentoring activity organised as part of a broader Transformative Innovation Policy Consortium Forty attendants over 1 week

The “Factory”: comparing results Outcome mapping would focus on behavioural changes among those attending Too early to assess but Clarification of constitutive elements of transformative innovation policies Active debates among participants … Likely to generate changes in practices in the participating group A ToC approach focuses on expected Learning: About different innovation frames, and the distinguishing characteristics of transformative innovation Networks and relationships: new networks created among academics, firms and government agencies but lack of participation from civil society Commitments: consensus about the need for change but misgivings about the notion of “experiments” with many feeling a top-down approach would be more adequate. No clear commitments emerge Organisational arrangements: linkages with other “niche experiments” lacking, plans for future activities not laid out..

INGENIO [CSIC-UPV] Ciudad Politécnica de la Innovación | Edif 8E 4º Camino de Vera s/n 46022 Valencia