1. Towards a framework for the evaluation of Transformative Innovation Policies
EU-SPRI Governance and relevance: Towards a new generation of research and innovation policies
Alejandra Boni, Jordi Molas-Gallart and Sergio Belda Miquel
ESIEE, Paris 6 June 2018
INGENIO [CSIC-UPV] Ciutat Politècnica de la Innovació | Edif 8E | Camí de Vera s/n | València
tel
fax

2. Such new generation approaches pose new evaluation challenges
Defining the issue
Transformative Innovation Policy as laid out by Schot and Steinmueller (2016) can be considered a “new generation of “research and innovation policy”
Such new generation approaches pose new evaluation challenges

3. Key assumptions of “Transformative Innovation Policies”…
To address key societal problems (e.g. as reflected in the UN Sustainable Development Goals) requires changes in socio-technical systems
Systemic changes with directionality
To achieve such socio-technical system the starting point is a set of “policy experiments” in “protected niches”
If successful, policies are implemented to deepen and scale up initial (niche) changes

4. …generate evaluation problems
Amplify the “attribution” problem common in Research Impact Assessment
Project outcomes are a contribution (and not a determining cause) of socio-economic impacts
Long “distance” between a niche and the change in socio-technical system
Complexity: TIPs rely on a set of interrelated experiments operating at different levels
Different projects defining local experiments operating in local niches, strategies to deepen and scale up…
The processes, timelines and immediate objectives of each initiative are different
Should take into account the interactions among this diverse set of policy initiatives
Governance changes (including participatory processes) need to extend to evaluation methods and practices

5. Evaluation strategies
“Outcome mapping” as a common response…
Assumes that impact processes cannot be controlled by the policy
There is no “theory of change”
Focus on behavioural change in the communities involved in an intervention
Based on participative techniques
…but how can we know a specific TIP project is being successful?

6. An alternative approach: using a ToC in TIP evaluation
TIP is based on socio-technical transitions theory and therefore embodies a generic ToC
TIP evaluations should describe in detail the ToC of each specific policy consistent with the generic ToC
Clearly define the different levels of policy intervention
Projects in protected niches set up as experiments to stimulate local socio-technical changes
Emphasizes changes in governance to support broad participation
A Programme links several to enhance synergies and learning, and to protect and nurture them
Seeks to deepen the scope and widen the reach of the project niches
A Transformative Innovation Policy is constituted by several programmes targeting a specific socio-technical system
Identify key evaluative dimensions to be assessed at every level
Main actors involved
Resources, processes, outcomes (and impacts where relevant)

7. Transformative Innovation Policies: evaluating a multi-level policy mix

8. Work in Progress: Evaluating a small experiment
Factoría de Política de Ciencia, Tecnología e Innovación para la Transformación (Medellín, Colombia)
Small initiative launched by a group of academics (6) from 2 Medellín universities
Challenge: Existing HEIs practices shift the focus away from social and environmental challenges that threaten the sustainability of the current development model
Local level project
The initiative is supported by Colciencias and a mentoring activity organised as part of a broader Transformative Innovation Policy Consortium
Forty attendants over 1 week

9. The “Factory”: comparing results
Outcome mapping would focus on behavioural changes among those attending
Too early to assess but
Clarification of constitutive elements of transformative innovation policies
Active debates among participants …
Likely to generate changes in practices in the participating group
A ToC approach focuses on expected
Learning: About different innovation frames, and the distinguishing characteristics of transformative innovation
Networks and relationships: new networks created among academics, firms and government agencies but lack of participation from civil society
Commitments: consensus about the need for change but misgivings about the notion of “experiments” with many feeling a top-down approach would be more adequate. No clear commitments emerge
Organisational arrangements: linkages with other “niche experiments” lacking, plans for future activities not laid out..

10. INGENIO [CSIC-UPV] Ciudad Politécnica de la Innovación | Edif 8E 4º Camino de Vera s/n
46022 Valencia