1. INDIANAUNIVERSITYINDIANAUNIVERSITY TransPAC QBSS “Scavenger Service” Chris Robb Indiana University TransPAC Network Engineer chrobb@iu.edu

2. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Topics QBSS on TransPAC Scavenger Availability Juniper Implementation Juniper Testing Learnings Future Directions Questions and Comments

3. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Scavenger on TransPAC The need to implement QBSS on the TransPAC circuit came about after discussions with a group of researchers from Indiana University and the University of Tokyo. A GRAPE-6 machine at Tokyo will be generating terabytes of data Indiana University is supplying an HPSS archive to store the data for analysis by other US researches. This data will need to be sent across the TransPAC OC-3 in a fashion that wouldn’t cause any significant network impact Researchers agreed to tag their IP datagrams as QBSS so they wouldn’t interrupt the other Tokyo-Chicago traffic. This project is slated to begin transfers very soon.

4. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Where is Scavenger Available? TransPAC circuit from Tokyo to Chicago – We will focus on this link because it focuses on a Juniper implementation

5. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Juniper Implementation Both sides of the TransPAC link were serviced by Juniper routers, so a Juniper implementation of Scavenger Service needed to be created and tested. Juniper proposed an implementation and the Global NOC volunteered to test its functionality.

6. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Juniper Test Setup Juniper donated an M5 for general Global NOC testing in June of 2001 The test setup was as follows: Both workstations were Linux-based. JunOS 4.3 was loaded on the M5

7. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Juniper Test Setup (cont.) Juniper doesn’t yet have DSCP support, so TOS bits were used instead. The I2 QBSS implementation accounts for this and Juniper has said that DSCP support is forthcoming. A precedence map was created to place the tagged packets into their appropriate queues. This remained fairly constant throughout the testing: precedence-map QBSS-test { bits 000 output-queue 0; <- Best Effort Traffic bits 001 output-queue 1; <- QBSS tagged traffic bits 101 output-queue 0; bits 010 output-queue 0; bits 011 output-queue 2; <- Premium traffic bits 100 output-queue 0; bits 110 output-queue 3; <- Network Management Traffic. Juniper advises bits 111 output-queue 3; <- not to change the queue assignment for these }

8. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Juniper Test Setup (cont.) After the traffic is placed in the queue, a WRR profile is created for the individual interfaces. The following WRR profile was created to service the outgoing ATM interface: weighted-round-robin { output-queue 0 weight 98; <- best-effort traffic output-queue 1 weight 1; <- QBSS gets a minimum of 1% of the circuit output-queue 2 weight 0; <- Premium service receives no servicing output-queue 3 weight 1; <- Network Management traffic must have some servicing } This configuration would change, depending on what we wanted to test. In the above configuration, we weren’t testing premium service

9. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Juniper Test Setup (cont.) A patched version of iPerf that allowed for TOS tagging was used to generate multiple streams from the GigE NIC towards the ATM NIC. Most of the tests involved running 200Mbit UDP streams of one type, and then introducing another stream of the other type The results were logged by iPerf at one second intervals, providing a picture of overall traffic behavior during the particular test.

10. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Test results The overall results were positive! The following graph shows the results of one particular test: At +0s, a 200Mb QBSS stream was run. At +10s, a best-effort stream was introduced, pushing down the QBSS stream to about 1 percent of the available bandwidth. At + 40s, the BE stream was removed and the QBSS stream reclaimed the pipe.

11. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Test Results (cont.) Here’s another graph showing a 200Mb QBSS stream before and after the introduction of a 10Mbit best-effort stream: The full results of these tests can be viewed at: http://www.transpac.org/qbss-html

12. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Learnings Juniper will only use the WRR queuing profile when a circuit is saturated. Because you cannot configure WRR queuing on a per-VC basis, this has wide implications for small VCs. – For example, a policed 10Mbit VC will never fill an entire OC-3 ATM PIC. – QBSS traffic on that VC will not be serviced any differently than best-effort traffic unless the circuit is filled up with traffic from other configured VCs One way to get around this is to generate traffic on the other VCs to artificially fill up the circuit, although this is highly undesirable.

13. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Learnings (cont.) Fortunately, Juniper does have a way around this! – There is a “hidden” command in the JunOS that will force all traffic on an interface, regardless of the link saturation, to be subject to the WRR profile: set chassis fpc pic transmit-buffers It is also important to set queue lengths for the individual VCs on the circuit to make sure that a small VC doesn’t eat up all the buffer space when delaying QBSS traffic.

14. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Learnings (cont.) Expedited traffic must be placed in queue 0 – When swapping the best effort and QBSS queues around (i.e. QBSS in queue 0 and best-effort in queue 1), QBSS traffic did not yield to best-effort traffic – Instead, they evenly split the circuit, with both flows getting half of the bandwidth – We spoke at length with Juniper about this problem. Unfortunately, we had to return the M5 before this was resolved. – It isn’t crippling, but something to keep in mind

15. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu TransPAC Implementation Thanks to the APAN NOC, TransPAC is now enabled for QBSS transmissions, using the WRR profile shown earlier. Initial tests have proven positive, but more testing is needed before the GRAPE transfers are initiated.

16. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Future Directions Results must be shared with QoS working group QBSS mailing list More complete testing of the “queue 0” problem is needed We also need to test TCP performance IU has recently reacquired the hardware to complete such tests. Results will be shared.

17. INDIANAUNIVERSITYINDIANAUNIVERSITY 4 October 2001 TransPAC Juniper QBSS Implementation Chris Robb – Indiana University Global NOC chrobb@iu.edu Questions? Comments?