1. ECCENTRIC Project-level Evaluation
Helber López – PEM
CIVITAS Forum 2018
September Umeå
On behalf of our working group:
Prof. Dr. Wulfhorst (Work Package leader)
Local Evaluation Managers
Montserrat Miramontes

2. ECCENTRIC Project-level Evaluation Motivation
Why going the extra mile?
ECCENTRIC has planned a deeper project-level evaluation, based on the exahustive activities performed at local level and particular emphasis in process evaluation.
Limitations of traditional comparative analisis (CBA)
Costs:
Pilots are not representative (innovation and scale issues)
competition in new markets (e-vehicles, shared mobility, etc.) pose several restrictions to costs and bussiness information avaliability.
Benefits:
Measures not comparable
Ramp-up within ECC lifespan
Transferability within partner cities is limited if local conditions are not observed
Europe-wide policy recomendations are not possible if the mobility relevant „landscape“ is not considered
Complexities and relationships with other aspects of the urban realm
Local specific dynamics (economy-social-governance-urban/natural enviroment)
Initial conditions

3. ECCENTRIC Project-level Evaluation Evaluation framework
8.1 Common Evaluation Framework
System framework
ECCENTRIC Key Performance Indicators (KPI)
8.2 Evaluation Plans
Impact, process, and project level
8.3 Process Evaluation
During planning and implementation phases mainly
8.5 Project Evaluation
System analysis based
Comparative and integrated eval.
Upscaling
D8.3 Process Evaluation Report
D8.5 Project Evaluation Report
Baseline + BaU
Two measurements during demonstration
8.4 Impact Evaluation
D8.4 Impact Evaluation Report
D8.1 Common Evaluation Framework for ECCENTRIC
D8.2 Evaluation Plans
Talk about SATELLITE and the evaluation framework.
Impact evaluation focus heavily in awareness, acceptance and satisfaction as precondition/drivers for behavioral change
Implementation of detailed process evaluation for several measures, up to the end of ECC.
Both efforts feed the system analysis

4. ECCENTRIC Project-level Evaluation Project-level evaluation

5. ECCENTRIC Project-level Evaluation The system Analysis approach
Example of a “simple” system
To determine the outcome of such a system is a gigantic, and probably useless task.
Which variables describe the system? Hint: there might be several ways to do it!
How are these variables related. Is there a clear direct relationship? Has it been quantified?
The system analysis doesn’t help to determine quantitatively the outcome of the system (who wins the game). Instead, it supports the development of actions and strategies that support the achievement of given objectives.
A good and “simple” example of a system is a chess game. Such a simple system has 4’ possible outcomes after only five moves. To deterministically estimate the outcome of such a system requires effort and resources hardly available in most cases. Trying to do the same for an urban system is nowadays infeasible.
In the chess example, if each piece had its own will, each piece would be a stakeholder, whose actions affects those of others. The system can be explained by variables: linear variables such as number of moves (time), input variables such as aggressiveness, and output variables (indicators) such as number of remaining pieces, or exposure of the king. The interrelationship between those variables is sometimes easy to explain and demonstrate (e.g. between number of remaining pieces and number of available squares). In many cases, the relationship is not even apparent or hardly demonstrable.
The system analysis aims to raise the level of qualitative understanding of the system where decisions are made, and actions take place, not to determine quantitatively the outcome of the system: in chess, who wins the game. The system analysis can help to identify trends and patterns, enabling the development of actions and strategies that support the achievement of given objectives; in chess, this might be to capture the opponent’s king.

6. ECCENTRIC Project-level Evaluation ECCENTRIC system model
Vester’s Sensitivity Model: Tool of choice for ECCENTRIC
2017
2018
2019
2020
PER2
MER3
PER3
Local evaluation
Interaction with WPL
Interaction with WPL
M18
MUC Meeting
M30
MAD Meeting
M42
MUC Meeting
M12
System analysis
Development of the system model (PEM+WPL)
M35
M36
Cross-site evaluation
Exploitation of the system model (PEM+WPL)
M43
M48
M45
WP support

7. ECCENTRIC Project-level Evaluation ECCENTRIC system model
Main activities
System analysis
Development (PEM+WPL)
Cross-site evaluation
Exploitation (PEM+WPL) 1 6 2 5
The system model team doesn’t have to worry about the complexity of the process.
PEM will facilitate the process and make it more streamlined for the team 3 4

8. ECCENTRIC Project-level Evaluation ECCENTRIC system model
Main activities 1 6 2 5 3 4

9. ECCENTRIC Project-level Evaluation System analysis progress
Hands-on work…
Two interactions and 4 stages in progress: 1 2 4
October 2017
Until January 2018
January 2018

10. Stockholm PMG3 Meeting • 2017 September 13-14th • <XXX>

13. Thank you! Helber López Contact Details Technische Universität München
Arcisstr. 21
80333 Munich