1. Izaro Etxebarria, ISDEFE, Spain 7th April, Brussels, EUROCONTROL
CLASSICAL AND COMPLEX NETWORK PERFORMANCE METRICS SESAR WP-E TREE, NEWO projects
Complexity and Data Science, Challenges and Opportunities in SESAR 2020
Izaro Etxebarria, ISDEFE, Spain
7th April, Brussels, EUROCONTROL

2. Outline COMPLEXITY SCIENCE & Network Performance Assessment;
CLASSICAL METRICS;
COMPLEX NETWORK PERFORMANCE METRICS;
KEY CHALLENGES AND OPPORTUNITIES;

3. COMPLEXITY SCIENCE & NETWORK PERFORMANCE ASSESSMENT
Air Transport Network Structure:
Irregular, complex and dynamically evolving in time
Propagation, Congestion phenomena.
Complex Networks
Multiple components and interactions
LEPA
LEBL
Two examples of projects where complex systems theory is applied for better understanding of the ATM performance, focusing on different spatial and temporal scales.
Let’s focus on TREE project where relevant progress has been made on non-classical complex metrics
According to several studies performed in the last decades, the ATN presents, among others, the following characteristics:
non-trivial topological features;
irregular structure, complex and dynamically evolving in time;
propagation phenomena…
All these characteristics and properties and more, are common in Complex Networks.
Complex network approaches are used for analysing ATN behaviour and performance,
A couple of examples: TREE/NEWO
The exploration of non-classical and complex performance metrics needs to be set in the context of Complex Networks;
Reactionary delays have a large impact in the air transportation system both at operational and economical point of view. However, research efforts to understand their origin and propagation patterns in Europe have been limited. The TREE project (data-driven modeling of network-wide extension of the tree of reactionary delays in ECAC area) is focused on characterizing and forecasting the propagation of reactionary delays through the European Network. The best approach to tackle this problem passes through the use of Complex Systems theory. This theory analyzes systems formed by a large number of components interacting between them by means of networks and attempts at predicting their meso-scale and global behaviors. In this project the nodes are the airports and the links are created by direct flights, the delays appear as malfunctions in the schedule planned everyday that can and do propagate to an important fraction of the airports in the network. In this work, an agent-based approach is introduced, able to simulate the delay propagation process. The first results show a promising similarity with the real delay propagation patterns, being able to describe the cluster of congested airports and its evolution along the day.
Complex Systems Theory is used for analysing and predicting meso-scale and global network behaviours.

4. COMPLEXITY SCIENCE & NETWORK PERFORMANCE ASSESSMENT
SESAR WP-E TREE
TREE: characterize and predict delay propagation through the European Air Transport;
SESAR WP-E NEWO
NEWO: analyze emerging network-wide effects of new local operational approaches.
Two examples of projects where complex systems theory is applied for better understanding of the ATM performance, focusing on different spatial and temporal scales.
Let’s focus on TREE project where relevant progress has been made on non-classical complex metrics
According to several studies performed in the last decades, the ATN presents, among others, the following characteristics:
non-trivial topological features;
irregular structure, complex and dynamically evolving in time;
propagation phenomena…
All these characteristics and properties and more, are common in Complex Networks.
Complex network approaches are used for analysing ATN behaviour and performance,
A couple of examples: TREE/NEWO
The exploration of non-classical and complex performance metrics needs to be set in the context of Complex Networks;
Reactionary delays have a large impact in the air transportation system both at operational and economical point of view. However, research efforts to understand their origin and propagation patterns in Europe have been limited. The TREE project (data-driven modeling of network-wide extension of the tree of reactionary delays in ECAC area) is focused on characterizing and forecasting the propagation of reactionary delays through the European Network. The best approach to tackle this problem passes through the use of Complex Systems theory. This theory analyzes systems formed by a large number of components interacting between them by means of networks and attempts at predicting their meso-scale and global behaviors. In this project the nodes are the airports and the links are created by direct flights, the delays appear as malfunctions in the schedule planned everyday that can and do propagate to an important fraction of the airports in the network. In this work, an agent-based approach is introduced, able to simulate the delay propagation process. The first results show a promising similarity with the real delay propagation patterns, being able to describe the cluster of congested airports and its evolution along the day.

5. CLASSICAL METRICS FOR ANALYSING AIR TRANSPORT NETWORK PERFORMANCE
The metrics used to assess any system:
determined by the way in which the system is interpreted;
driven by the framework used to formalise the system.
A classical formulation of the air transport system generates classical metrics;
Classical or Traditional metrics:
describe most of the indicators that are currently in use in air transport performance assessment;
they are predefined and;
do not come from complexity science techniques.
The metrics we use to assess any system are, to a large degree, determined by the way in which we think about that system.
They are driven by the framework we use to formalise it.
A classical formulation of an air transport system generates classical metrics,
Classical metrics describe most of the metrics currently in use in air transport performance assessment (they are predefined (average aircraft delay), are univariate (drawn on one variable in the data), the do not come from complexity science techniques
So, since they are common to most of you I will not stop here explaining each metric. However I would like you to focus on the distinction of the scales (spatial/temporal)
The initial measure is the total minutes or hours of arrival or departure delay in the full system during a certain time period. This measure has been extensively used when performance is studied from the Network Manager perspective and when an economic value is associated to each delay. This was, for example, the case of the 2008 US Congress report on economic impact of air traffic delays [6].
A next metric is the average arrival/departure delay per flight. This metric provides better insights on the system performance. It is typically used when the delays are considered by type such as reactionary, airport operational, weather-caused, etc. Some examples can be found in the CODA digest.
The average delay per flight has the inconvenience that since the percentage of flights arriving or departing on time is typically high this metric is not very informative when the focus is on the seriousness of the delays of delayed flights. To avoid this problem, other metric commonly used is the average delay of delayed flights (ADD), also commonly used in CODA digest.
The percentage of flights arriving on time is quite generalized and used to rank airports and airlines (also used in CODA and by the same airlines in their own reports).
Another set of indicators such as DDI-F and BTO is based on the idea of time compartments. The DDI-F (Delay Difference indicator -Flight) represents the difference between arrival and departure punctuality expressed in minutes. The Block Time Overshoot (BTO) is defined as the share of flights with a longer real than scheduled block time.
congestion.

6. CLASSICAL METRICS FOR ANALYSING AIR TRANSPORT NETWORK PERFORMANCE
CLASSICAL or TRADITIONAL METRICS
SCALE (spatial/temporal)
Total minutes of arrival/departure delay in the full system in a certain time period
Network
Average arrival/departure delay per flight
Average delay of delayed flights
Network/Airport
Percentage of flights arriving on time
Airlines/Airports
Delay Difference Indicator _Flight (DDI_F)
Time Compartments
Block Time Overshoot (BTO)
The metrics we use to assess any system are, to a large degree, determined by the way in which we think about that system.
They are driven by the framework we use to formalise it.
A classical formulation of an air transport system generates classical metrics,
Classical metrics describe most of the metrics currently in use in air transport performance assessment (they are predefined (average aircraft delay), are univariate (drawn on one variable in the data), the do not come from complexity science techniques
So, since they are common to most of you I will not stop here explaining each metric. However I would like you to focus on the distinction of the scales (spatial/temporal)
The initial measure is the total minutes or hours of arrival or departure delay in the full system during a certain time period. This measure has been extensively used when performance is studied from the Network Manager perspective and when an economic value is associated to each delay. This was, for example, the case of the 2008 US Congress report on economic impact of air traffic delays [6].
A next metric is the average arrival/departure delay per flight. This metric provides better insights on the system performance. It is typically used when the delays are considered by type such as reactionary, airport operational, weather-caused, etc. Some examples can be found in the CODA digest.
The average delay per flight has the inconvenience that since the percentage of flights arriving or departing on time is typically high this metric is not very informative when the focus is on the seriousness of the delays of delayed flights. To avoid this problem, other metric commonly used is the average delay of delayed flights (ADD), also commonly used in CODA digest.
The percentage of flights arriving on time is quite generalized and used to rank airports and airlines (also used in CODA and by the same airlines in their own reports).
Another set of indicators such as DDI-F and BTO is based on the idea of time compartments. The DDI-F (Delay Difference indicator -Flight) represents the difference between arrival and departure punctuality expressed in minutes. The Block Time Overshoot (BTO) is defined as the share of flights with a longer real than scheduled block time.
congestion.

7. COMPLEX NETWORK PERFORMANCE METRICS
The exploration of Complex performance metrics needs to be set in the context of Complex Networks;
Challenge in SESAR WP-E TREE (propagation of reactionary delays):
to explore the development of new metrics in ATM to improve the understanding of the Network Performance by extending the scope of the network topology;
In TREE the network topology is extensively analyzed by the study of delay spreading phenomena;
First simulations results show a promising similarity with the real delay propagation patterns, being able to describe the cluster of congested airports in its evolution along the day.
The exploration of non-classical and complex performance metrics needs to be set in the context of Complex Networks;
Challenge in SESAR WP-E TREE: to explore the development of new metrics in ATM to improve the understanding of the Network Performance by extending the scope of the network topology;
TREE is focused on characterizing and forecasting the propagation of reactionary delays through the European Network;
The network topology is extensively analyzed by the study of delay spreading phenomena;
Flight links related to flights using the same aircraft/crew/passengers are the skeleton through wich the delays are propagated.
An agent-based approach is introduced, able to simulate the delay propagation process.
First results show a promising similarity with the real delay propagation patterns, being able to describe the cluster of congested airports in its evolution along the day.
One of the main challenges in TREE is :…
Reactionary delays have a large impact in the air transportation system both at operational and economical point of view. However, research efforts to understand their origin and propagation patterns in Europe have been limited. The TREE project (data-driven modeling of network-wide extension of the tree of reactionary delays in ECAC area) is focused on characterizing and forecasting the propagation of reactionary delays through the European Network. The best approach to tackle this problem passes through the use of Complex Systems theory. This theory analyzes systems formed by a large number of components interacting between them by means of networks and attempts at predicting their meso-scale and global behaviors. In this project the nodes are the airports and the links are created by direct flights, the delays appear as malfunctions in the schedule planned everyday that can and do propagate to an important fraction of the airports in the network. In this work, an agent-based approach is introduced, able to simulate the delay propagation process. The first results show a promising similarity with the real delay propagation patterns, being able to describe the cluster of congested airports and its evolution along the day.

8. COMPLEX NETWORK PERFORMANCE METRICS
METRICS BASED ON TREES OF DELAYS
‘Trees’ are graphs without loops and have been studied in Mathematics and Physics for many years.
SIZE OF THE TREE
The number of affected flights by the delay;
Large trees used to be consequence of either tight schedules or very connected networks.
The framework in which the metrics must be understood
The nodes that are part of a tree can be typically organized in generations if there is a natural hierarchy among them. Time is the variable that introduces a hierarchy in the case of air transport. A tree of delays is formed by flights. The root is occupied by the flight suffering the primary delay and then the flights that incur in reactionary delays induced by the root are connected to it. These second generation flights affect in turn to other flights that are connected to each of them forming thus the third generation in the tree and so on.
The trees can be characterized mathematically by a few parameters.
Contenido anterior de la diapo
Extension of the classical framework :
By first better characterising the European air transport system through a better exposition of the topological properties of the network we can design better metrics in the context of this topology
Trees are graphs without loops and have been studied in Mathematics and Physics for many years.
Nodes that are part of a tree can be typically organized in generations if there is a natural hierarchy among them.
Time is the variable that introduces a hierarchy in the case of air transport.
A tree of delays is formed by flights.
The root is occupied by the flight suffering the primary delay and then the flights that incur in reactionary delays induced by the root are connected to it;

9. COMPLEX NETWORK PERFORMANCE METRICS
METRICS BASED ON TREES OF DELAYS
NUMBER OF GENERATIONS
A tree of delays is formed by flights.
Nodes that are part of a tree can be typically organized in generations if there is a natural hierarchy among them (i.e. Time);
The root is the flight suffering the primary delay and the flights connected to it incur in reactionary delays;
1st generation
The framework in which the metrics must be understood
The nodes that are part of a tree can be typically organized in generations if there is a natural hierarchy among them. Time is the variable that introduces a hierarchy in the case of air transport. A tree of delays is formed by flights. The root is occupied by the flight suffering the primary delay and then the flights that incur in reactionary delays induced by the root are connected to it. These second generation flights affect in turn to other flights that are connected to each of them forming thus the third generation in the tree and so on.
The trees can be characterized mathematically by a few parameters.
Contenido anterior de la diapo
Extension of the classical framework :
By first better characterising the European air transport system through a better exposition of the topological properties of the network we can design better metrics in the context of this topology
Trees are graphs without loops and have been studied in Mathematics and Physics for many years.
Nodes that are part of a tree can be typically organized in generations if there is a natural hierarchy among them.
Time is the variable that introduces a hierarchy in the case of air transport.
A tree of delays is formed by flights.
The root is occupied by the flight suffering the primary delay and then the flights that incur in reactionary delays induced by the root are connected to it;
3rd Generation…
2nd generation

10. COMPLEX NETWORK PERFORMANCE METRICS
METRICS BASED ON TREES OF DELAYS
MULTIPLICATION FACTOR BETWEEN GENERATIONS
The number of new affected flights in the generation t+1 on average per each delayed flight in the previous generation t.
This metric is dependent of the airport in which each flight is operated.
DELAY MULTIPLICATOR FACTOR PER AIRPORT t
1st generation
6/2 multiplication factor
The trees can be characterized mathematically by a few parameters. The total number of affected flights, the so-called size of the tree or number of nodes is the first of them. Obviously, large trees are consequence of very tight schedules and very connected networks. Añadir animaciones al segundo diagrama para distinguir generaciones
2nd generation
t+1

11. COMPLEX NETWORK PERFORMANCE METRICS
METRICS DESCRIBING NETWORK-WIDE CONGESTION
The approach used in TREE:
An airport is considered congested when the average delay of departing flights in an hour is over ‘x’ minutes.
SIZE OF THE LARGEST CLUSTER OF AIRPORTS SHOWING CONGESTION
The delay in a cluster of connected airports showing congestion in similar or close time intervals can be seen as correlated and likely to come from a common source (NETWORK CONGESTION)
Most of the metrics described so far focus on the performance of flights, airports or airlines. They are useful thus to assess performance from the perspective of airport managers and airline practitioners. Metrics can be also developed, of course, to estimate the effect of delays on passengers. However, a point of view that has been often forgotten is that of the global Network Manager. For example, a metric to estimate network congestion was introduced in [9]. The idea behind it is quite simple. The first step is to define a measure for the performance of the airports. For instance, in [9] an airport was considered congested when the average delay of the departing flights in a one-hour interval was over 29 mins. The operations in the airports are connected by the flights that travel between them. A network was, therefore, constructed with the daily flights. Then the delay in a cluster of connected airports showing congestion in similar or close time intervals can be seen as correlated and likely to come from a common source. The size of the largest connected cluster can then be used as a measure of the level of network-wide congestion.

12. Key Challenges and Opportunities
Further research on a better Alignment of Models and Metrics with:
Actual & Future ATM System Behaviour
Further investigating how the properies of the system change over different temporal and spatial scales;
SESAR Concepts
New operational objectives (policy, those driven from high-level European political agendas related to mobility).
White paper- non-classical metrics
The current state of metrics in use in air transport is not fully adequate to the task of measuring new operational objectives, most notably those driven from high-level European political agendas related to improved service delivery to the passenger.
How are we to measure the effectiveness of new passenger-driven performance initiatives in air transport in general, and in ATM in particular if we do not have the corresponding set of passenger-oriented metrics?
Characterisation of the air transport network to be informative in terms of offering new insights into performance.
Temporal scales: maybe far more granular for a tactical application than a strategic one.
Complex Performance metrics: temporal and spatial granularity of the KPI.
KPIs must be tailored to the user needs and the objetives of the measurement;
Relevant spatial scales depend on wether the stakeholder is an airline, an ANSP or the NM.