1. Traffic Engineering with Forward Fault Correction (FFC)
Hongqiang “Harry” Liu,
Srikanth Kandula, Ratul Mahajan, Ming Zhang,
David Gelernter (Yale University)
Today we will take a look at Forward Fault Correction – FFC in traffic engineering,
which proactively makes a network eliminate congestion automatically when faults happen.
This is a joint work with the MSR folks: Srikanth, Ratul and Ming, and my advisor David at Yale.

2. Cloud services require large network capacity
Growing traffic
Cloud Networks
Nowadays, cloud services are rapidly growing. This growth gives cloud infrastructure a huge pressure.
Especially for cloud networks, it is a big challenge to face the growing traffic, since the networks are expensive, so that it is difficult to upgrade the capacity or expand the network.
Expensive
(e.g. cost of WAN: $100M/year)

3. TE is critical to effectively utilizing networks
Traffic Engineering
Microsoft SWAN
Google B4 ……
Devoflow
MicroTE ……
This is also why in recent years, cloud providers are taking great efforts to deploy traffic engineering systems in both Wide Are Networks and Datacenter networks to highly utilize their network resources.
WAN Network
Datacenter Network 3

4. But, TE is also vulnerable to faults
TE controller
Network configuration
Network view
Frequent updates for high utilization
Control-plane faults
While TE has been shown to be very helpful for highly utilizing the network resources, its vulnerability to faults is an essential dark side.
This is because TE monitors and controls the network remotely, and it also has to frequently update the network for a continuous high utilization.
Remote and highly dynamic system could be undermined by various of faults that are common in reality.
These faults fall into two major categories, control-plane faults and data-plane faults.
Data-plane faults
Network

5. Control plan faults
Failures or long delays to configure a network device
TE Controller
Switch
TE configurations
RPC failure
Firmware bugs
Overloaded CPU
Memory shortage

6. Congestion due to control plane faults
s2 s4 (10)
New Flows (traffic demands):
s s2 (10)
s s3 (10)
s s4 (10)
s s4 (10)
Configurations might causes severe congestions. For example, in this TE update from left to right, link s1-s4 will be overloaded if s2 fails to implement the new configuration which moves the traffic on tunnel s2-s1-s4 onto tunnel s2-s4.

7. Rescaling: Sending traffic proportionally to residual paths
Data plane faults
Link and switch failures
Rescaling: Sending traffic proportionally to residual paths
s2 s4 (10)
The second category….
Combining 7 & 8
Because of the importance of TE, many network providers deploy TE systems in their network.
In such a system, a conceptually centralized TE controller is the core to run TE algorithms.
It relies on monitoring systems to get the currently traffic demands and network topology.
After obtaining a new TE solution, it configures the traffic rate limiters and ingress routers in the network via a controlling system.
Nevertheless, the input and the output of the TE controller are not 100% reliable.
For example, link and router failures can change the network topology, which might not be reported timely by the monitoring system.
For another example, configurations to some routers might be rejected or lost, which makes the controlling system only partially implement a new TE.
s s4 (10)

8. Control and data plane faults in practice
In production networks:
Faults are common.
Faults cause severe congestion.
Control plane:
fault rate = 0.1% -- 1% per TE update.
Data plane:
fault rate = 25% per 5 minutes.
Congestion due to faults like what are shown in the previous examples usually happen in practice.
Our measurement in a production network shows that both control and data plane faults are very common.
For instance, the fault rate of control plane fault can be as high as one percent, which means if we have 100 switches in the network, there is very likely to be one control plane fault each time we update the network. And for data plane faults, even within 5 minutes, it is also very likely to see one or more than one faults like link failures.
And our evaluation shows that even a single control plane can be sufficient to cause large congestion. For example, this figures shows the CDF of maximum link over-subscription rate if each network update has a control plane fault. This point shows that there is a 10% change to make some link overloaded by 10% of its capacity or more.
And for data plane faults, even a single link failure could result in a much worse situation than control plane faults.

9. State of the art for handling faults
Heavy over-provisioning
Reactive handling of faults
Control plane faults: retry
Data plane faults: re-compute TE and update networks
Big loss in throughput
Just because the faults are so common and their consequences are so bad, we must handle them.
However, in the state of art, people have very limited ways to handle the faults.
Typically they use reactive approach, which means they detect and handle faults.
For example, when control plane faults happen, they retry switches that are not responding.
When data plane faults happen, they try to compute a new TE and update the network.
Nevertheless, the reactive approaches are far from satisfaction. The major problem is that..
Cannot prevent congestion
Slow
(seconds -- minutes)
Blocked by control plane faults

10. How about handling congestion proactively?
Small font

11. Forward fault correction (FFC) in TE
[Bad News] Individual faults are unpredictable.
[Good News] Simultaneous #faults is small.
FEC guarantees no information loss under up to k arbitrary packet drops.
Packet loss
with careful data encoding
Bound the congestion, or bound the level for over-provisioning.
FFC guarantees no congestion under up to k arbitrary faults.
Network faults
with careful traffic distribution

12. Example: FFC for control plane faults
Non-FFC
When looking into control plane faults, we already know that configuration failures on s2 (or s3) will cause congestions.
So that the new TE on the right is non-FFC or we say k=0.

13. Example: FFC for control plane faults
Control Plane FFC (k=1)
Missing: FFC can have overhead in terms of throughput
Just like FEC has overhead. (important)

14. Trade-off: network utilization vs. robustness
(Control Plane FFC)
K=2
(Control Plane FFC)
Missing: FFC can have overhead in terms of throughput
Just like FEC has overhead. (important)
Non-FFC
Throughput: 50
Throughput: 47
Throughput: 44

15. Systematically realizing FFC in TE
Formulation:
How to merge FFC into existing TE framework?
Computation:
How to find FFC-TE efficiently?

16. Basic TE linear programming formulations
LP formulations
Sizes of flows
𝑏 𝑓
TE decisions:
Traffic on paths
𝑙 𝑓, 𝑡
TE objective:
Maximizing throughput
max ∀𝑓 𝑏 𝑓
Deliver all granted flows
s.t. ∀𝑓: ∀𝑡 𝑙 𝑓,𝑡 ≥ 𝑏 𝑓
Basic TE constraints:
No overloaded link
∀𝑒: ∀𝑓 ∀𝑡∋𝑒 𝑙 𝑓,𝑡 ≤ 𝑐 𝑒 … …
Titles
Transfer from previous slide.
Simplify this.
Before this slide:
How to formulation in the framework of existing te.
Solving Efficiently.
FFC constraints:
𝑘 𝑐 control plane faults
No overloaded link up to 𝑘 𝑒 link failures
𝑘 𝑣 switch failures

17. Formulating control plane FFC
𝑓 1
𝑓 2
𝑓 3
Total load under faults?
𝑓 1 ’s load in old TE
𝑓 2 ’s load in new TE
Fault on 𝑓 1 :
𝑙 1 𝑜𝑙𝑑 + 𝑙 2 𝑛𝑒𝑤 + 𝑙 3 𝑛𝑒𝑤 ≤link cap
𝟑 𝟏
Fault on 𝑓 2 :
𝑙 1 𝑛𝑒𝑤 + 𝑙 2 𝑜𝑙𝑑 + 𝑙 3 𝑛𝑒𝑤 ≤link cap
Trying English first and simplify the formulation.
Jump
Fault on 𝑓 3 :
𝑙 1 𝑛𝑒𝑤 + 𝑙 2 𝑛𝑒𝑤 + 𝑙 3 𝑜𝑙𝑑 ≤link cap
Challenge: too many constraints
With n flows and FFC protection k:
#constraints = 𝒏 𝟏 + … + 𝒏 𝒌 for each link.

18. An efficient and precise solution to FFC
Our approach:
A lossless compression from O( 𝑛 𝑘 ) constraints to O(kn) constraints.
Total load under faults ≤ link capacity
Total additional load due to faults ≤ link spare capacity
𝑥 𝑖 : additional load due to fault-i
Given 𝑋={ 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 }, FFC requires that
the sum of arbitrary k elements in 𝑋 is ≤ link spare capacity
O( 𝑛 𝑘 )
Simplify  abstract
Paper both data and control
optimity
Define 𝑦 𝑚 as the mth largest element in 𝑋:
𝑚=1 𝑘 𝑦 𝑚 ≤ link spare capacity
O(1)
Expressing 𝑦 𝑚 with 𝑋?

19. Sorting network 𝑥 1 𝑦 1 (1st largest) 𝑧 4 𝑧 5 𝑥 2 𝑦 2 (2nd largest)
𝑧 2
𝑧 3
𝑧 7
𝑧 8
𝑥 3
𝑧 1
𝑧 6
A comparison
𝑥 4
𝑥 1
𝑧 1 =max{ 𝑥 1 , 𝑥 2 }
𝑥 2
𝑧 2 =min{ 𝑥 1 , 𝑥 2 }
1st round
2nd round
𝑦 𝑦 2 ≤ link spare capacity
Complexity: O(kn) additional variables and constraints.
Throughput: optimal in control-plane and data plane if paths are disjoint.

20. FFC extensions
Differential protection for different traffic priorities
Minimizing congestion risks without rate limiters
Control plane faults on rate limiters
Uncertainty in current TE
Different TE objectives (e.g. max-min fairness) …
We show the extensions
One sentence for each item
My formulation is general.

21. Evaluation overview Testbed experiment Large-scale simulation
FFC can be implemented in commodity switches
FFC has no data loss due to congestion under faults
Large-scale simulation
A WAN network with O(100) switches and O(1000) links
Injecting faults based on real failure reports
Single priority traffic in a
well-provisioned network
Multiple priority traffic in a
well-utilized network

22. FFC prevents congestion with negligible throughput loss
Single priority
160%
High priority (High FFC protection)
Medium priority (Low FFC protection)
Low priority (No FFC protection)
100 80
From the figures, we can also observe that only using priority queuing to protect higher priority traffic is not sufficient.
Because if the traffic layout is not robust, high priority traffic could also suffer from congestion because high priority flows congestion each other. However, if we use both priority queuing and FFC to protect higher priority traffic, we can indeed realize the vision that when problems happen, the network always sacrifices low priority traffic first. 60
Ratio (%) 40 20
<0.01%
FFC Data-loss / Non-FFC Data-loss
FFC Throughput / Optimal Throughput

23. Conclusions
Centralized TE is critical to high network utilization but is vulnerable to control and data plane faults.
FFC proactively handle these faults.
Guarantee: no congestion when #faults ≤ k.
Efficiently computable with low throughput overhead in practice.
FFC
At the end this talk, I want to mention that FFC is essentially offering a new knob for network operators to better balance the trade-off between the risk of congestion and network over-provisioning. FFC provides a simple interface--- the FFC protection level k ---- to the operators, and achieves an efficient balance in the trade-off.
High risk of congestion
Heavy network over-provisioning