1. QoS Support in High-Speed, Wormhole Routing Networks Mario Gerla, B. Kannan, Bruce Kwan, Prasasth Palanti,Simon Walton

2. Overview Introduction QoS via separate subnets QoS via synchronous framework QoS via virtual channels Conclusions

3. Introduction Wormhole routing offers low latency, high speed interconnection for supercomputers and clusters. It’s a modification of virtual cut-through: -A packet is forwarded to output port once its head is received at the switch -If channel is busy, whole packet is buffered at input port -Wormhole: packet composed of several flits is stored across several switches Used in high speed LANs like Myrinet: -Asynchronous LAN -uses wormhole routing, source routing, backpressure flow control to achieve low latency and high bandwidth

4. Supercomputer SuperNet Two level architecture: –Optical backbone based on physical optical star topology –High speed wormhole routing are Myrinet LANs –Optical Channel Interface connects electronic LANs to optical backbone Provide support for distributed supercomputing. –Scientific visualization, video display, parallel applications –Different types of traffic (low latency datagram, high bandwidth connection oriented) –Different types of QoS

5. Objective Want to provide connection oriented traffic with QoS parameters: -reliable support: no worm loss -scalable and deadlock free network Assumption: -Traffic with QoS is connection oriented -QoS parameters specified at connection setup -Connection can be refused if no guarantee for Qos parameters -QoS parameters: average bandwidth, end-to-end delay or jitter

6. QoS support via Separate Subnet Create two subnets –One carries QoS traffic –Another carries non QoS traffic –Routing is independent for the subnets (Myrinet has support) Issues: –Call admission and control –Source host behavior –Number of interfaces at host

7. Call Admission & Control Admission agent maintains state of QoS subnet Request for QoS traffic connection comes in Upon receiving request, agent decides a suitable route If route not available, host can retry or use other subnet If route exists, connection is accepted, host can send Once completed, host informs admission agent Admission agent update its view or state of subnet

8. Host Behavior Host must be responsible for amount of traffic injected in subnet according to QoS parameters it required Solution: host uses pacing mechanism –Allow only predetermined number of flits to be transmitted per time period

9. Number of Interfaces Suppose host has only one interface Sender side: –Host can schedule transmission into the network Receiver side: –Possible non-QoS worm may block QoS worm –QoS worm encounters delay if non-QoS worm is large Solutions: –Two interfaces: this double cost of network –Account for the worst case non-QoS traffic delay on single host interface at call setup time

10. Alternative Subnets: –difficult to provide delay bounds due to delay dynamics from blocking at different cross points Alternative: –Impose synchronous structure on top of the asynchronous network –Enables control over the blocking –Delay bounds and message priorities may be implemented Trade off: –Network is no longer asynchronous –Under low traffic load, messages suffer delay due to synchronous protocol overhead

11. QoS support via Synchronous framework Similar to dedicated traffic channels Use timed-token to control traffic streams Provides tighter delay bounds and bandwidth guarantees Target Token Rotation Time TTRT limit the amount of transmission Average delay = TTRT Worst case delay = 2 * TTRT

12. How to support timed token Dedicated unidirectional ring is embedded in the network Attributes of Token scheme: -fair -deadlock free

13. Issues Number of host interfaces –Caused by interaction of QoS & non-QoS traffic –QoS traffic travel on core ring while non-QoS travel on other links not on ring –If host with one interface is busy receiving non- QoS message, a QoS message will suffer delay –QoS message must have preemptive priority To increase non-QoS throughput, embedded ring may be used if bandwidth is not completely taken by QoS traffic

14. Continued Scalability: –throughput performance maintained by increasing TTRT parameter –Allows nodes to transmit for longer time when they have the token –Causes less capability to provide tight delay bounds

15. Virtual Channel Based QoS Each link is split into two different sets of virtual channels used for datagram and QoS traffic Each input port buffer of switch is split into several disjoint buffers Link between node and input port of switch is a collection of virtual channels Allows worms to be interleaved Give QoS traffic priority in the network

16. Non-preemptive priority A worm arriving at QoS virtual channel does not get transmitted right away Current worm (datagram or QoS) being transmitted on outgoing link must either complete or get blocked Then, scan QoS virtual channels before datagram channels to schedule the worm for transmission on outgoing link Easy to apply preemptive priority by making arrived worm preempt datagram worm at the QoS virtual channel

17. Implementation Preemptive and non-preemptive implementation require intelligence switch At a switch: –monitor all traffic passing –Schedule QoS & non-QoS traffic according to protocol Harder to implement preemtive: –Switch must check arrival of QoS traffic at any input port before transmitting non-QoS flit from output port

18. Advantage of virtual channels Network appears the same for both traffic Intelligent switches allocate bandwidth as required to support QoS Can provide delay jitter bounds Bandwidth guarantee is provided by employing call admission agent

19. conclusions Wormhole routing networks provide low latency, high bandwidth support for datagram traffic To support QoS traffic is a challenge Dedicated QoS subnet with pacing and call admission control can support QoS Synchronous framework on top of asynchronous network provides guaranteed bandwidth and delay Virtual channels with priority mechanism also is effective way to support QoS