1. A Differentiated Services Implementation for High- Performance TCP Flows Volker Sander, Ian Foster, Alain Roy and Linda Winkler Forschungszentrum Jülich GmbH Argonne National Laboratory The University of Chicago

2. the GARA project 23-May-2000, TNC 2000 2 Outline l The GARA quality of service architecture l GARA and network quality of service l Experimental results l Summary and future directions

3. the GARA project 23-May-2000, TNC 2000 3 GARA l General-purpose Architecture for Reservation and Allocation l Developed as part of the Globus Project –www.globus.org –Globus Toolkit provides enabling security and directory services –Otherwise independent of Globus l Goal: Provide end-to-end Quality of Service (QoS) to high-end applications

4. the GARA project 23-May-2000, TNC 2000 4 Example High-end Applications l Teleimmersion: Virtual reality collaboration l Distributed Scientific Computing –Analysis of large data sets –Real-time simulation steering –Remote control of scientific instruments –Real-time visualization –Collaboration: Multiple people visualizing & steering l We have to address UDP and TCP flows

5. the GARA project 23-May-2000, TNC 2000 5 How does GARA Help? l Resource discovery –Network –Computers –Disks l Advance reservations l Security: control over who gets reservations l Monitoring of reservations l Coordination of multiple reservations } Not just Network QoS!

6. the GARA project 23-May-2000, TNC 2000 6 Resource Manager l The Resource Manager is the core of GARA –Admission control –Configures resource –Monitors resource l Assumes exclusive access to the resource –Otherwise no guarantees can be made GARA’s network resource manager is a Bandwidth Broker

7. the GARA project 23-May-2000, TNC 2000 7 Network Resource Manager l Admission control –Has knowledge of topology within a domain –Tracks reservations on path through domain l Configures routers –Differentiated services (more detail soon) –Currently telnet with command-line interface –COPS/SNMP in the future l Monitors resource –Query edge router for flow statistics >Conforming and exceeding (dropped) packets

8. the GARA project 23-May-2000, TNC 2000 8 Network QoS Implementation l We use differentiated services l Expedited forwarding to implement “premium service” –We use Cisco 7507 routers (thanks Cisco!) –Edge routers controlled with resource manager, as described above –Packets that exceed the reservation are dropped

9. Experiments

10. the GARA project 23-May-2000, TNC 2000 10 The GARNET Testbed

11. the GARA project 23-May-2000, TNC 2000 11 Evaluation Tools l UDP Traffic Generator –Modified Version of Andy Adamson’s gen_send and gen_recv >Evaluate admission control >Creating competing traffic –MGEN/Drec >Evaluate Delay and Jitter for Premium UDP Flow –IPERF l Modified Version of ttcp –GARA-enabled (wait for reservation) –Support for a desired application rate –Consecutive bandwidth reporting –Bulk transfer ttcp

12. the GARA project 23-May-2000, TNC 2000 12 Basic Experiment I UDP Sender GARA Diffserv Resource Manager UDP Receiver

13. the GARA project 23-May-2000, TNC 2000 13 Basic Experiment I l Goal: Proof of Admission Control

14. the GARA project 23-May-2000, TNC 2000 14 Basic Experiment II GARA Diffserv Resource Manager UDP Sender UDP Reveiver UDP Sender UDP- Realtime- Receiver

15. the GARA project 23-May-2000, TNC 2000 15 Basic Experiment II l Goal: Demonstrate Low-Latency for UDP flows

16. the GARA project 23-May-2000, TNC 2000 16 0 5 10 15 20 25 30 35 40 45 0 Slow Start Exponential Growth TCP Issue: Exceeding the Reservation TCP Congestion Window Size time Congestion Control Linear Growth

17. the GARA project 23-May-2000, TNC 2000 17 Basic Experiment III GARA Diffserv Resource Manager TCP Sender UDP Sender UDP Receiver TCP Receiver

18. the GARA project 23-May-2000, TNC 2000 18

19. the GARA project 23-May-2000, TNC 2000 19 l Avoid any drops if you care about short-term impact l Instead use feedback mechanisms to inform the application / the agent to adapt –its transmission rate –its reservation Conclusions for Implementing a Bandwidth Broker

20. the GARA project 23-May-2000, TNC 2000 20 TCP Issue II l TCP’s Flow Control –Traffic might become bursty if the actual window size is large –Bandwidth*Latency product as minimum window size

21. the GARA project 23-May-2000, TNC 2000 21 Demonstration of TCP’s Burstiness

22. the GARA project 23-May-2000, TNC 2000 22 Conclusions for Implementing a Bandwidth Broker l Be aware of burstiness l Token bucket depth should allow a full window burst T = Reserved_BW * Estimated_RTT Or implement signaling from the application l How does this interfere with UDP low-latency flows in one aggregate behavior?

23. the GARA project 23-May-2000, TNC 2000 23 Basic Experiment IV GARA Diffserv Resource Manager TCP Sender TCP Receiver UDP Sender UDP Receiver UDP-RT Receiver

24. the GARA project 23-May-2000, TNC 2000 24 99% WFQ, No Traffic Shaping

25. the GARA project 23-May-2000, TNC 2000 25 Conclusions for Implementing a Bandwidth Broker l If Traffic Shaping is not possible, guarantee as much bandwidth to the premium flow as possible. l Admission Control is performed at the edge BUT: Be aware of default queue limits 99% WFQ BW results in a maximum increase of RTT by RTT/2 (assuming 33% EF-Traffic)

26. the GARA project 23-May-2000, TNC 2000 26 Standard BE Behavior

27. the GARA project 23-May-2000, TNC 2000 27 WFQ: Default Buffer - Just BE

28. the GARA project 23-May-2000, TNC 2000 28 WFQ: Modified Buffer - Just BE

29. the GARA project 23-May-2000, TNC 2000 29 TCP & Low-Delay Flows l TCP can interfere with UDP flows that want low-delay: –We want traffic shaping to smooth out premium bursts –TCP can be very bursty l Solution: –Traffic Shaping (but…)

30. the GARA project 23-May-2000, TNC 2000 30 Traffic Shaping for Entire Premium Class

31. the GARA project 23-May-2000, TNC 2000 31 TCP & Low-Delay Flows l Aggregate-based traffic shaping adds delays for low-delay UDP traffic l Solution: –Don’t use a single traffic-shaping configuration for the entire premium class

32. the GARA project 23-May-2000, TNC 2000 32 QoS Mechanisms: Inside the Ingress Router

33. the GARA project 23-May-2000, TNC 2000 33 Dynamic Traffic Shaping

34. the GARA project 23-May-2000, TNC 2000 34 Bulk Transfer (LAN) When a reservation begins, the bulk-transfer backs off When a reservation ends, the bulk-transfer speeds up The competitive UDP traffic never interferes

35. the GARA project 23-May-2000, TNC 2000 35 Bulk Transfer (WAN)

36. the GARA project 23-May-2000, TNC 2000 36 Network + CPU Reservations Initially easy to get 80 Mbps Until competition begins We reserve net Then CPU becomes loaded We reserve CPU Finally, we turn off network reservation

37. the GARA project 23-May-2000, TNC 2000 37 Current Status l A working GARA prototype exists –Differentiated Services –Real-Time CPU Scheduling (DSRT) –DPSS Disk Access –Integrated security, resource discovery, etc. l Many experiments have been performed –Expedited Forwarding is working l Work with early adopters has started –E.g., DOE labs, MREN universities, NASA

38. the GARA project 23-May-2000, TNC 2000 38 Future Work l Technology development –More work on Co-reservations –Policy issues: Who has access when; costs, accounting; priorities, preemption –COPS interfaces, inter-domain issues –Experimentation with more real applications l Higher bandwidth flows –MPLS, wavelength allocation

39. the GARA project 23-May-2000, TNC 2000 39 Questions? l Feel free to email us: –Volker Sander: v.sander@fz-juelich.de –Ian Foster: foster@mcs.anl.gov –Alain Roy: roy@mcs.anl.gov –Linda Winkler: winkler@mcs.anl.gov l Check our web site: –http://www.mcs.anl.gov/qos/ –Numerous technical papers available