Sequin Technical Summary Mauro Campanella INFN-GARR

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st, Synergy with - A joint and task force on advanced networking research

M. Campanella - Sequin Workshop - February 1st, Approach to the task (end to end QoS across multiple domains) QoS Definition Top - down Qualitative through user’s questionnaire Bottom - up Quantitative, using a minimum and sufficient set of QoS parameters QoS service(s) definition, architecture, testing and implementation in NREN networks

M. Campanella - Sequin Workshop - February 1st, Basic components of QoS the applicationthe operating system and the transport protocols the User the network

M. Campanella - Sequin Workshop - February 1st, Intuitive definition of QoS The network offers a QoS service when it’s capable of handling selected packets in such a way to fulfill application’s requirements. “some packets are more equal than others…”

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st, Users’ Interview A questionnaire has been developed to asses user’s needs of QoS. The questionnaire was articulated in 26 questions in 4 sections: - geography; - qualitative perception of QoS; - quantitative perception of QoS; - network options and expectation. It was sent to 20 groups of pan-European and large users, out of which 11 responded.

M. Campanella - Sequin Workshop - February 1st, Interview results outside Europe to which connectivity is needed used

M. Campanella - Sequin Workshop - February 1st, Interview results (continued) QoS (QoS need)

M. Campanella - Sequin Workshop - February 1st, Interview results (continued) Overall the users showed medium to low knowledge of their QoS needs and QoS techniques, but unanimously requested it, as a way to have a better service from network for their work. Present difficulties are mainly due to congestion. Willingness to pay is proportional to the real benefits, granularity of the service, provisioning time and flexibility and behaviour of Best Effort. Need for simple, fast access to the QoS service.

M. Campanella - Sequin Workshop - February 1st, Questionnaire summary

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st, QoS parameters From users’ requirements and technical considerations :  - one-way delay;  - IP packet delay variation;  - capacity (rate);  - one-way packet loss. The set is common to IETF and ITU-T. Naming and definitions are chosen to be comply to RFC 2330 (Framework for IP Performance metrics) and follow the ongoing IPPM IETF working group work.

M. Campanella - Sequin Workshop - February 1st, QoS parameters sample value ranges

M. Campanella - Sequin Workshop - February 1st, QoS parameters (continued) Memento To build a QoS service based on the previous listed parameters, some basic requirements on the network should be fulfilled: - physical and data link stability; - exhibit a Bit Error Rate better than ; - overall network hardware performance. The minimum MTU size should be chosen large enough to avoid fragmentation. Duplicate and out-of-order packets at the physiological level (which is not null, but very small)

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st, Which QoS service Start with the “very good” service and call it “Premium IP”: - it satisfies all the users’ requests - it is “the best” achievable service possible - it maps to very high priority scheduling techniques available now - it is similar to a “virtual wire”

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st,

M. Campanella - Sequin Workshop - February 1st, Provision QoS for the European research users in the form of an end to end network service offering the equivalent of a leased line. The service has to be implemented by combining border to border services provided by the NRENs and networks The architecture has to be simple, modular, scalable, adapt to network changes easily, based on IP and independent from the transport technology. The implementation and Service Level Agreements have to match the current status of hardware availability and network topology Premium IP goals

M. Campanella - Sequin Workshop - February 1st, Increasing complexity QoS Protocols and Architectures Overview StatelessStateful Stateless Stateful Overprovisioning Diffserv 802.1p RSVP (aggregates) Diffserv p RSVP (per flow) Diffserv p RSVP - Intserv MPLS -Traffic engineering ATM - ATM signaling

M. Campanella - Sequin Workshop - February 1st, Which QoS framework to use ? Differentiated Services - RFC Integrated Services - RFC Overprovisioning  

M. Campanella - Sequin Workshop - February 1st, Premium IP Specification   Differentiated Services Architecture and use the expedited forwarding per hop behavior (EF PHB)   interface definition between domains that behaves as an EF PHB   do not starve best effort traffic (limited percentage of link capacity devoted to Premium IP, about 5%)   initial provisioning structure: static, no dynamic signaling   IETF IPPM QoS parameters measurement framework   QoS parameters monitoring system is a key element

M. Campanella - Sequin Workshop - February 1st, Premium IP Specification   minimize number of action per node   modular approach that allows different implementation schemes at every hop or domain and allows domain to join the service when readyand do not try to solve the most general problem, but rather develop a model that can be implemented in parallel with the start of GÉANT, using available tools

M. Campanella - Sequin Workshop - February 1st, IP v4 IP v6

M. Campanella - Sequin Workshop - February 1st, Simplifying the actions for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling In principle, each node might perform an awful lot of tasks: - marking- policing - shaping

M. Campanella - Sequin Workshop - February 1st, Use the information in the IP - IP source and destination (prefixes) as near to the source as possible - the DSCP (or IP precedence equivalent value) along the path - perform an optional, suggested, admission control based on AS source and destination at inter-domain links (safety measure) - rules might be based on additional parameters, as time-of-day Admission control

M. Campanella - Sequin Workshop - February 1st, The consequences are: - allowing the computation of total requested Premium IP capacity at each network node in the default case (and for main backup cases too) - short access list near users’ premise (few users) - simple control at backbones (IP addresses are not propagated) - choosing destination-aware service (next slide) Admission control (continued)

M. Campanella - Sequin Workshop - February 1st, Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling In principle, each node might perform an awful lot of tasks: - marking- policing - shaping always

M. Campanella - Sequin Workshop - February 1st, Mark each “EF” legal packet at first classification point - Use the same DSCP value on all domains (Class selector 5 - decimal 40 [RFC 2474] to have interoperability with ToS-only capable hardware) - strongly suggested - - valid DSCP coupled to invalid IP addresses implies discard to allow easy debugging - packets with other DSCP values are left untouched Marking is mandatory at the first classification point, remarking is optional. Marking

M. Campanella - Sequin Workshop - February 1st, Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling- marking- policing - shaping always Selected locations

M. Campanella - Sequin Workshop - February 1st, Microflow policing should be done as close as possible to the source according to agreed (through SLA) Premium IP capacity. This step is mandatory Policing will be done using a token bucket. The depth of the token bucket will be two MTU close to the source and increase to 5 or more along the path if additional policing is required It is suggested to perform only one additional policing stage at the ingress to GÉANT from an NREN, with a larger aggregated capacity value than the sum of the agreements. “Avoid unwanted packet loss” is the motto. Policing

M. Campanella - Sequin Workshop - February 1st, The additional policing stage at the ingress to GÉANT from an NREN serves the purpose of protecting Premium IP traffic from misconfiguration/DoS coming from a single source. It creates virtual “pipes” for the aggregated Premium flows from each NREN to each other (when needed). The failure of one “pipe” does not influence the others. Policing (continued)

M. Campanella - Sequin Workshop - February 1st, L1L2 N1 N2 N3 CORE L1, L2 : end user domain (for example LANs) N1, N2, N3 : intermediate transport domains (for example NRENs backbones) CORE : interconnection domain (for example GÉANT) : router/switch Sample multidomain network Classification on IP addresses Strict policing Policing can be avoided at ingress when receiving from a trusted backbone Classify by DSCP Police by (AS source,dest) aggregate capacity on all border nodes Policing not needed

M. Campanella - Sequin Workshop - February 1st, Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling- marking- policing - shaping always Selected locations Selected locations

M. Campanella - Sequin Workshop - February 1st, Use the highest priority queueing mechanism (PQ or WRR). Limit total Premium capacity when assigning service to users at about 5% of each core link. Assigned Premium capacity can be larger near users’ sites. Scheduling Total link capacity Premium traffic Best Effort traffic Suggested Premium limit This should never happen, but it works..

M. Campanella - Sequin Workshop - February 1st, Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling- marking- policing - shaping always Selected locations Selected locations always

M. Campanella - Sequin Workshop - February 1st, Shaping The compliance of the Premium user flow to the contracted capacity is the key for the result of the service. Shaping is intended here as limiting the rate of transmission to a specific value. The speed of the core link and the highest priority in scheduling for the packets of the Premium IP service make delay variation small even at aggregation points. At 2.5 Gb/s the transmission time of a 1500 bytes packet is about 5 microseconds. The consideration suggests to start the service without enabling shaping in the core and it shaping may be optional also at the border, provided the sources produce a well shaped flow.

M. Campanella - Sequin Workshop - February 1st, Shaping The sending source is hence required to shape the traffic it produces. Shaping inside the sending host itself is the preferred way, shaping by the network will in most case lead to packet losses TCP Application Scheduling IP Network Interface No Packet/Data losses host

M. Campanella - Sequin Workshop - February 1st, Examining the tasks for each node - monitoring and accounting- QoS rules propagation - congestion control - admission control and classification - scheduling- marking- policing - shaping always Selected locations Selected locations always NO Done by source not needed Selected locations Selected locations

M. Campanella - Sequin Workshop - February 1st, Shape ONLY here Classify (IP pair prefixes) Police - Strict, Capacity Mark Classification (DSCP) and High priority scheduling on all nodes Do not police on egress Do not shape anywhere Policing can be avoided at ingress when receiving from a trusted backbone Police by (AS source,dest) aggregate capacity on all border nodes Premium IP Summary

M. Campanella - Sequin Workshop - February 1st, Summary Innovations: - interdomain behaviour specification - end to end service level agreement The architecture allows: - different implementation strategy in each domain - asynchronicity in implementation - sub-domain implementation - explicit rate limitation only near sending source - enabling a user just adding few lines of access control

M. Campanella - Sequin Workshop - February 1st, Example (one direction) Domain p VLAN Or dedicated wire Classification (IP) Policing (strict 2 MTU) Marking - scheduling Domain p VLAN Or dedicated wire Domain 2 ATM Dedicated PVC Domain 3 Backbone Classification (DSCP) Policing (AS aggregate) Classification (DSCP) scheduling Domain 4

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st, SLA/SLS Basic implementation In the first phase the SLS negotiation will be performed manually (no bandwidth broker). The analytical computation of the QoS metric in a IP based network is extremely complex and the SLA specification will require extensive testing of the available infrastructure. Usually only QoS parameter ranges can be specified and assurances as percentages of total time. There are always two SLA, one for each direction. The contracted values might be different (asymmetric capacity for example)

M. Campanella - Sequin Workshop - February 1st, The service must be defined both as an end to end service level agreement and be accepted as a modification in the chain of service level agreements between all involved domains. The SLA/SLS is in reality a chain of SLA/SLS between neighbour domains and a final end-to-end one. Fundamental concepts (continued) User 1 User 2 User 1 NREN1 GÉANT NREN2 User 2 +

M. Campanella - Sequin Workshop - February 1st, Fundamental concepts (continued) Users must understand the application requirements in term of the QoS parameter, at least the requi- rement for the maximum sending/receiving rate of the application. There is the need of a central database to keep up to date track of allocate resources and check their availability. Debugging can be assigned to just one specific entity.

M. Campanella - Sequin Workshop - February 1st,

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st, Monitoring Highly distributed measurement of QoS parameters that can measure the end to end and single hops performance. Use a mixture of active (in-band) and passive methodologies In-house developed tool for GÉANT (Taksometro)

M. Campanella - Sequin Workshop - February 1st, Active injects measurement traffic at small capacity use low cost dedicated hardware like RIPE TT boxes, surveyor, chariot suitable for loss, delay, jitter can be implemented in key locations Passive used also for regular (BE) traffic known tools based on SNMP, like netflow, that read counters on nodes suitable for packet loss, policing, queue depth… can access every node Methodologies

M. Campanella - Sequin Workshop - February 1st, Per-domain measurement taksometro Web interface router NREN A NREN B NREN C Ripe ttm

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st, Premium IP road map QoS Definition Deliverable D2.1 Sequin Premium IP Architecture Deliverable D9.1 QoS Testbed Definition Deliverable D3.1Sequin Premium IP SLA/SLS Deliverable D9.1- Addendum 2 Premium IP Implementation Deliverable D9.1- Addendum 1 QoS Monitoring Deliverable D9.4 Testing Activity in GÉANT and Testbeds - Pilot users

M. Campanella - Sequin Workshop - February 1st, Premium IP status Currently tests are running between Switzerland, Italy, Germany and Greece. Goal is to validate the model, measure the network performance (end to end) and measure the effect using a videoconferencing application based on H.323 Premium IP is configured on GÉANT routers.

M. Campanella - Sequin Workshop - February 1st, Premium IP in progress Fine tuning of buffering and token bucket depth in routers. As a rule of thumb the token bucket depth can be assumed to be 1.2 * (number of Diffserv active interfaces on router) Scalability - the maximum amount of aggregated Premium IP capacity the network can offer - hardware capabilities Fast provisioning of the service Widespread availability and tuning of “last mile” (LANs)

M. Campanella - Sequin Workshop - February 1st, Agenda - Approach to the problem - Users’ requirements - Quantitative definition of QoS - Which QoS service - Premium IP service - Service Level Agreements - Monitoring - Premium IP status - Other QoS services

M. Campanella - Sequin Workshop - February 1st, Other QoS services ? Less than Best Effort - Less than Best Effort (Scavenger) Already working in Internet2, requires queuing and marking, not access control Assured Forwarding - Assured Forwarding based services. Sequin could not find a implementation scenario for NRENs The limit is the number of hardware queues in the routers and the hardware performance.

M. Campanella - Sequin Workshop - February 1st, References All the deliverables, presentation and relevant documentation can be found on the web in: and

M. Campanella - Sequin Workshop - February 1st, Thank you and user QoS is even tougher

M. Campanella - Sequin Workshop - February 1st, Overprovisioning Two possible definitions: - istantaneous link load never greater than 30% - no packet losses (weaker) It works for 99.9 % of the cases, but capacity is far from being overprovisioned all over Europe (yet). Even many LANs have not enough capacity. It’s not perfect (yet) though...

M. Campanella - Sequin Workshop - February 1st, % clean Tier 1 US backbone [ From Nanog 22]

M. Campanella - Sequin Workshop - February 1st, Tier 1 US backbone (continued) 99.99% clean [ From Nanog 22] 99.99%

M. Campanella - Sequin Workshop - February 1st, Overprovisioning (continued) Deviation from % of delay variation constant value due to : - routing problems; - routing timers set-up; - ARP cache timeouts; -... It’s mostly instability/misbehaviour of the software layer on routers/switches.