1. PMD Auto Load Balancing
PMD Auto Load Balancing
Nitin Katiyar Pradeep Venkatesan Jan Scheurich
© Ericsson AB 2018

2. PMD Auto Load Balancing
Agenda
Background
Auto Load Balancing – Proposal
Example
Summary
Questions
© Ericsson AB 2018

3. PMD Auto Load Balancing
Background
Port Rx queues assignment to PMDs is realized
statically by configuration
dynamically - on demand or when certain events occur
Static port queue assignment:
Simple
Works best if traffic patterns are well understood and unchanging
© Ericsson AB 2018

4. PMD Auto Load Balancing
Background(contd…)
Dynamic port queue assignment:
triggered by port addition, deletion etc.
triggered by CLI pmd-rxq-rebalance command
configurable assignment algorithm i.e. cycles or roundrobin
can better handle unpredictable traffic patterns but still not adaptive
© Ericsson AB 2018

5. PMD Auto Load Balancing
Background(contd…)
Round Robin Assignment
Simple algorithm
Load distribution not uniform
Reassignment may or may not result in better load distribution
Cycles based Assignment
Assignment aims to keep load distribution uniform across PMDs
Each PMD keeps track of the processing cycles spent for each Rx Queue
Six snapshots of the queue processing cycles each spanning 10 seconds are maintained
When reassignment requested, the queue statistics are used to get a uniform distribution
© Ericsson AB 2018

6. Automatic Load Balancing - Proposal
PMD Auto Load Balancing
Automatic Load Balancing - Proposal
Trigger cycles based assignment automatically (disabled by default)
Maintain additional per-queue statistics to help detect if any PMD is overloaded
Additional configuration parameters to fine tune the behavior
Reassignment will be triggered only if such a reassignment will result in better load distribution
© Ericsson AB 2018

7. Automatic Load Balancing - Proposal Details
PMD Auto Load Balancing
Automatic Load Balancing - Proposal Details
Additional per-queue statistics to be maintained
Number of Rx drops for physical port queues
Number of times vhost ports exceeded the qfill threshold
New configuration parameters
Enable/Disable automatic load balancing
PMD Overload Threshold (utilization %)
Whether Q Drops should factor into PMD Overload determination
Minimum improvement in load distribution expected as a result of reassignment
© Ericsson AB 2018

8. Automatic Load Balancing -Proposal Details (contd…)
PMD Auto Load Balancing
Automatic Load Balancing -Proposal Details (contd…)
Modifications to Cycles based reassignment function
A new option termed "dry-run"
Dry-run option will return the poll list resulting from running the algorithm without actually performing the reassignment.
© Ericsson AB 2018

9. Automatic Load Balancing - Details (contd…)
PMD Auto Load Balancing
Automatic Load Balancing - Details (contd…)
A PMD will consider itself overloaded, if for six consecutive 10-second intervals:
the PMD utilization is more than the configured Overload threshold AND
if Q drops are to be factored, number of Rx queue drops or number of times qfill-threshold exceeded is non-zero
© Ericsson AB 2018

10. PMD Auto Load Balancing
Automatic Load Balancing - Details (contd…)
OVS Main thread changes:
if any PMD considers itself overloaded, perform a dry-run of the reassignment
if dry-run poll list identical to current poll list, nothing more to do
if dry-run poll list different from current poll list, reassignment may offer benefits
© Ericsson AB 2018

11. PMD Auto Load Balancing
Automatic Load Balancing - Details (contd…)
OVS Main thread changes:
Evaluate if the reassignment will meet the configured minimum improvement as follows
Calculate the total processing cycles for each PMD based on current poll list of RX queue
Calculate the average variance in processing cycles for each PMD
Repeat above calculations using dry-run poll list
If dry-run average variance is less, the reassignment will likely result in a more uniform load distribution
Calculate the percentage improvement of the new average variance over the old average variance
Trigger reassignment if the difference is more the configured improvement threshold
© Ericsson AB 2018

12. PMD Auto Load Balancing
Automatic Load Balancing - Details (contd…)
PMD Thread
PMD Thread
Record processing cycles and checks if PMD is overloaded
Record processing cycles and checks if PMD is overloaded
Monitor PMD load
Monitor PMD load
Main Thread
Continue monitoring PMD load
Is any of the PMD thread overloaded? No
Yes
Reload PMD with new “RX queue mapping”
Reload PMD with new “RX queue mapping”
Initiate PMD dry run
Does dry run indicate improvement? No
Yes
Initiate data path reconfiguration
© Ericsson AB 2018

13. PMD Auto Load Balancing
Automatic Load Balancing -Example
Initial Configuration
Configured pmd-auto-lb-thresh = 80%
Processing load of PMD1 = 90% (Q0=25%, Q1=30%, Q2=35%)
Processing load of PMD2 = 35% (Q3=10%, Q4=20%, Q5=5%)
Mean load = 62% and Variance = 756
© Ericsson AB 2018

14. PMD Auto Load Balancing
Automatic Load Balancing -Example(contd…)
PMD Dry Run
Processing load of PMD1 = 60% (Q0=25%, Q1=30%, Q5=5%)
Processing load of PMD2 = 65% (Q2=35%, Q3=10%, Q4=20%)
Mean load = 62% and Variance = 6
Improvement = ((756 – 6) * 100) / 756 = 99%
© Ericsson AB 2018

15. PMD Auto Load Balancing
OVS CLI changes
We propose to have following knobs for Automatic PMD load balancing.
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb="true"
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-thresh=“<>"
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-min-improvement=“<>"
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-drop-check=“<>“
© Ericsson AB 2018

16. PMD Auto Load Balancing
RFC Patch
RFC Patch v1 is available in ML:
© Ericsson AB 2018

17. PMD Auto Load Balancing
Future Enhancements
Frequent change in RX queue to PMD assignment can be avoided by adding a dampening logic in main thread.
Time interval and frequency for PMD auto load balancing can be made configurable via CLI.
The threshold for RX Drops for physical ports and RX qfill for vhost user ports can be made configurable via CLI.
© Ericsson AB 2018

18. PMD Auto Load Balancing
Summary
PMD auto load balancing feature provides a mechanism to efficiently utilize the PMDs if traffic pattern changes after RX queue assignment to PMDs.
This feature can be enabled/disabled dynamically via OVS CLI and no additional cost in processing when this feature is disabled.
Provides various tunable parameters via OVS CLI to setup automatic load balancing according to the needs.
© Ericsson AB 2018

19. PMD Auto Load Balancing
Questions
© Ericsson AB 2018

20. PMD Auto Load Balancing
© Ericsson AB 2018