1. Ben Teitelbaum January 25th, 2001 TEQUILA Workshop on Internet Design for SLS Delivery QBone Service Specification and TE Early DiffServ Deployment Experiences and Reduced Expectations

2. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)2 The Holy Grail of computer networking is to design a network that has the flexibility and low cost of the Internet, yet offers the end-to- end quality of service guarantees of the telephone network. - S. Keshav

3. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)3 The Paradox of Internet QoS Internets success largely due to its lack of QoS! Best-Efforts: scalability on steroids Routers: complexity of forwarding very low L2: minimal requirements Providers: magic-free provisioning and low user expectations Developers: E2E transport and simplicity of Berkeley socket interface QoS complicates everything (costs), but opens many opportunities (benefits) Developers: new apps exploit the ubiquity of the Internet Providers: may make more efficient use of the network Big Question: How can we build upon DiffServ without compromising scalability?

4. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)4 architecture deployment How We Got Here (short version) Began chanting: enable advanced applications,… Assessed requirements What do you want? What can you give us? Recommended DiffServ Selected Premium service to meet demands of loss/jitter sensitive apps Charted QBone initiative Specified QBone architecture Now proceeding to implement it and tweak the architecture

5. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)5 QBone Architecture A Service: QBone Premium Service Built on Expedited Forwarding (EF) (RFC 2598) Assurance: near-zero loss & low, bounded jitter for marked traffic conforming to a specified peak rate –a.k.a. virtual leased line, virtual wire Reservation Setup Protocol Now: long-lived, manual setup Proposed: SIBBS protocol between QBone domains; RSVP end-to-end between hosts QBone Measurement Architecture Uniform collection of QoS metrics Uniform dissemination interface

6. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)6 QBone Measurement Architecture 1/2 Collection metrics, EF and BE... Active metrics (paths) –One-way delay-variation –One-way loss –Traceroutes –e.g. IPPM Surveyors Passive metrics (interfaces) –Load –EF reservation load –Discards (suggested) –Link bandwidths (suggested) –e.g. OCxMon, RTFM, MIBs Boundary Router Intra-Domain Premium Path Inter-Domain Premium Path Active Measurements MIB-based statistics Passive Measurements PM node AM node QBone Domain2 QBone Domain1 QBone Domain3

7. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)7 QBone Measurement Architecture 2/2 Dissemination Standard URL query syntax: whois server to learn canonical names for QBone domains, routers, sniffers, etc label ::= { } router ::= label"-ROUTER probe ::= label"-PROBE sniffer ::= label "-SNIFFER host ::= router | probe | sniffer path ::= host "/" host PHB ::= "BE" | "EF metric ::= "LOSS" | "ONEWAY" | "PING" | "IPDV" | "LOAD" | "TRACEROUTE" | "COMMITMENT" | "RESERVATION" year ::= digit digit digit digit month ::= "01" | "02" |... | "12 day ::= "01" | "02" |... | "31 YYYYMMDD ::= year month day aggregation ::= prefix ::= http | ftp | query ::= prefix ://" path "/" PHB "/" metric "/ YYYYMMDD "/" metric "/" aggregation

8. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)8 Simple Interdomain Bandwidth Broker Signaling (SIBBS) Basic Idea Simple domain-to-domain reservation request/response protocol to signal changes in the aggregate reservations of globally well-known services Design Goals Simple & extensible Bootstrap wide-area experimentation with QoS-needy apps, while providing hooks for clouds to experiment with various QoS TE approaches Integrate with end-to-end signaling capabilities of hosts Status Protocol draft nearing completion (see: http://qbone.internet2.edu/bb/) New protocol for now; looking at mapping to COPS

9. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)9 SIBBS: Basic Reservation Setup Web interface RSVP DIAMETER COPS RYO CLI SNMP COPS

10. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)10 Virtual Peerings & Core Tunnels BB SIBBS Src RTR Dest Virtual SIBBS peering BB Core Tunnel

11. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)11 SIBBS Approach to SLSs Basic approach: Define services and reservations, but punt on SLSs Services Globally specified, standardized, and named Include many parameters from draft-tequila-sls (scope, traffic envelope, performance guarantees, etc) Reservations Identify a service ID Specify service parameters SLSs Each SLS along a reservation path should be regarded as a black box that answers admissions questions and extends reservations across the peerings

12. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)12 Simple Interdomain Bandwidth Broker Signalling (SIBBS) BB HH DS Domain1DS Domain3DS Domain2 S LS 1-2 S LS 2-3 RRRR Control Data

13. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)13 QBone E2E Picture GigaPoP A Campus A Campus B GigaPoP B Campus C Campus D Backbone X Kbps of QPS from hither to………..yon Key

14. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)14 Abilene Topology January, 2001 47 connectors 183 participants 34 connections to 20 peer networks

15. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)15 Miami NYC (60 Hudson) LA Abilene Singaren SEATTLE NYC (Telehouse 25 Broadway) STARTAP Argentina Brazil Chile Columbia Brazil Developing International Peerings Courtesy: Linda Winkler, STAR TAP CERN SINET Abilene ESnet vBNS JAnet SURFnet NORDUnet Abilene ESnet CA*net3 AARnet CERNET CUDI HARNET SINET Abilene vBNS ESnet DREN NREN NISN CA*net3 SURFnet MIRnet NORDUnet REUNA IUCC APAN CERN SINET Singaren Tanet GEMNET Renater DFN INFN DANTE CA*net3

16. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)16 Abilene Load Snapshot

17. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)17 APS Participation Goal: Make APS a reference implementation of the QBone architecture Current Participants MAGPI (U. Penn) iCAIR PSC (Penn State) OARNet (Ohio State) NASA EOS Others in the wings TF-NGN (through DANTE) MIRnet Various other international ANL UIUC DOE Science GRID (peering transit network)

18. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)18 Initial Engineering Plan (obsolete) Measurement Edge Policing Manual Setup EF Core Forwarding Automated Setup EF Edge Forwarding Shaping SweetwaterMidlandOdessaPecos (GTS) (Surveyor + SNMP + HTTP) (Firehose CAR) (Whiteboard + CLI) (MDRR) (BB)

19. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)19 APS Measurements Status: Collecting EF/BE loads and CAR conform/exceed stats Not currently monitoring IPDV, but Abilene Surveyor nodes now OC-3 connected and operational Ohio-ITEC hosting APS measurements and QBone- wide whois server Near Future: IPDV along edge-to-edge QBone paths Abilene Surveyor timing improvements –Better NTP –New CDMA timing sources (can't get GPS in Qwest POPs) Collection of AS-level traffic matrices

20. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)20 Committed Access Rate (CAR) Classifies traffic based on certain matching criteria (interface, DSCP, or ACL) and meters it to a leaky bucket traffic profile Depending on metering result, different actions applied (drop, transmit, set DSCP,…) Syntax: rate-limit {input | output} [access-group [rate-limit] acl-index] bps burst-normal burst-max conform-action action exceed-action action

21. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)21 CAR Experience For the most part, CAR is exactly what the DiffServ doctor ordered However, there are some limitations… PPS performance cost Quirky constraints on token bucket depths Not easy to do "virtual trunk" style classification Problem: we want... "virtual trunk" "firehose" But what we get is…

22. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)22 Abilene Architecture Limitation: Porous Edge Problem DoQoS problem with current architecture MDRR (EF forwarding) on interior interfaces easily subverted by unpoliced connectors

23. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)23 How to Crisp the Edge? Problem: EF requires that all connectors be policed Solution: Short term: Stochastically detect illegal EF traffic with NetFlow and/or OCxMon passive monitoring; gigaPoP would be asked to police Longer term: Wait for E3 edge cards, deploy them aggressively, and police everywhere Road not taken:Re-write all non-participant traffic with DSCP 000000 (using PIRC hack); need to pass some DSCP values (reason why coming up…)

24. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)24 Looking Forward: Resource Accounting and Admissions Problem: how to account for link EF capacity and commitment in routed network? Potential solution: DiffServ-Aware MPLS-TE Basic idea: –Edge-to-edge MPLS tunnels –LSP setup constrained by available EF bandwidths –OSPF augmented to carry QoS link state attributes –See draft-lefaucheur-diff-te-reqts Solves DiffServ admissions problem Does not solve general DiffServ provisioning problem Status: completed extensive lab test of Ciscos implementation of this idea; considering field trial

25. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)25 Current Engineering Plan Measurement Edge Policing Manual Setup EF Core Forwarding Automated Setup EF Edge Forwarding Shaping SweetwaterMidlandOdessaPecos (GTS) (Surveyor + SNMP + HTTP + whois + traffic matrices) (CAR + QPPB) (Whiteboard + CLI) (MDRR) (SIBBS? + DSTE?)

26. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)26 QPS Status (Good News) The Good News: With enough elbow grease, E2E services can be built DiffServ functionality in most modern routers Many hosts now support QoS signaling (RSVP) Numerous testbed trials Partial implementations in ESnet and Abilene will form nucleus of QBone

27. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)27 SC2000 Interdomain QoS Demo November 6-9, 2000 Premium service over two wide-area paths LBNL-ESnet-Abilene-SCinet-Internet2 booth Stanford-CalREN2-Abilene-SCinet-Internet2 booth Congestion induced at multiple points CD-quality interactive audio application shown with/without QoS SC2000 Network Challenge Winner: Award for "Most Captivating and Best Tuned" Demo

28. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)28

29. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)29 QPS Status (Bad News) The Bad News: Router vendor hype/reality mismatch Deploying QPS requires upgrading the world Low demand for QPS (app/net chicken/egg dynamic) Elbow grease in short supply in most campus IT shops

30. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)30 Bottom Line Progress on QPS will continue to be slow Internet2 will continue with a QPS work program in 2001… Deploy incrementally where it counts Revise/complete architecture (joint work with TF-NGN) Work to build QoS-sensitive application user communities who understand their needs.However, QPS will no longer be the only Internet2 QoS activity Need to reap lower hanging fruit from DS (I2SS) Much stronger push on measurements/monitoring Big Internet2 E2E performance initiative ramping up

31. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)31 E2E Performance and Measurements Internet2 E2E Performance Initiative Typical BE TCP throughputs often much less than one would expect Common problems –Broken TCP stacks –Ethernet auto-negotiation failures Evolving attributes of initiative –Performance Emergency Response Teams (PERTs) –Sharper tools for measurement, monitoring, and analysis Measurement arsenal WEB100 (Mathis et al.) Open Internet2 measurement architecture (à la QBone) Open source and protocols for one-way delay measurement Application-level performance fault-isolation (reflector infrastructure)

32. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)32 Internet2 Scavenger Service Basic idea A lower priority class of best-effort Voluntary marking hints to network that degraded service is OK (think of a "nice" for the network) Intended uses Non-time-critical traffic (e.g. server-to-server NNTP, anonymous FTP, network backups) Bulk data transfers using TCP Non-mission applications (e.g. Napster, games, etc) New kinds of distributed applications that attempt to use idle network capacity

33. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)33 I2SS Service Specification Rigorously defining the E2E I2SS is difficult! Hoping to define relative to best effort I2SS traffic indicated by DSCP 001000 Modification of class selector PHB Note that the I2SS codepoint has global significance I2SS domain requirements Traffic leaving must be marked I2SS, if it entered so Router requirements (all SHOULDs) Forward I2SS independently giving it a lower probability of timely forwarding OR forward in same manner as BE Offer I2SS a very small minimum departure rate (unlike BH PDB) Offer I2SS all un-used bandwidth

34. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)34 For more information... Internet2 Home: http://www.internet2.edu/ Internet2 QoS Working Group Home: http://www.internet2.edu/wg/qos/ QBone Home: http://qbone.internet2.edu/ Abilene Premium Service Home: http://www.internet2.edu/abilene/qos/

35. Service Specification and TE for the QBoneAmsterdam (January 25th, 2001)35