Network Measurement and Control WG BOF Jeff Boote, Martin Swany, Verena Venus

OGF IPR Policies Apply “I acknowledge that participation in this meeting is subject to the OGF Intellectual Property Policy.” Intellectual Property Notices Note Well: All statements related to the activities of the OGF and addressed to the OGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the OGF and its participants certain licenses and rights in such statements. Such statements include verbal statements in OGF meetings, as well as written and electronic communications made at any time or place, which are addressed to: the OGF plenary session, any OGF working group or portion thereof, the OGF Board of Directors, the GFSG, or any member thereof on behalf of the OGF, the ADCOM, or any member thereof on behalf of the ADCOM, any OGF mailing list, including any group list, or any other list functioning under OGF auspices, the OGF Editor or the document authoring and review process Statements made outside of a OGF meeting, mailing list or other function, that are clearly not intended to be input to an OGF activity, group or function, are not subject to these provisions. Excerpt from Appendix B of GFD-C.1: ”Where the OGF knows of rights, or claimed rights, the OGF secretariat shall attempt to obtain from the claimant of such rights, a written assurance that upon approval by the GFSG of the relevant OGF document(s), any party will be able to obtain the right to implement, use and distribute the technology or works when implementing, using or distributing technology based upon the specific specification(s) under openly specified, reasonable, non- discriminatory terms. The working group or research group proposing the use of the technology with respect to which the proprietary rights are claimed may assist the OGF secretariat in this effort. The results of this procedure shall not affect advancement of document, except that the GFSG may defer approval where a delay may facilitate the obtaining of such assurances. The results will, however, be recorded by the OGF Secretariat, and made available. The GFSG may also direct that a summary of the results be included in any GFD published containing the specification.” OGF Intellectual Property Policies are adapted from the IETF Intellectual Property Policies that support the Internet Standards Process.

Working Group Summary Generation and exchange of network measurements is critical for all networked environments, and in particular advanced environments like the Grid. The schemata for network metrics that have been defined in the OGF’s Network Measurement Working Group (NM-WG) have spawned a vibrant community among R&E network operators who have been developing and deploying an infrastructure called perfSONAR. The perfSONAR effort began in September 2004 in a series of meetings in and around GGF-12 in Brussels, Belgium. The perfSONAR consortium has begun to produce a series of protocol documents describing the messaging functionality in that system and the NMC group will house the formalization of those standards within the OGF.

Purpose and Scope The purpose of the Network Measurement and Control Working Group is to standardize the XML-based protocols that are currently in use in the perfSONAR project to control network measurement infrastructure and to share the results of the measurements and metrics that are generated. These protocols are already in widespread use and are described across a number of documents with various degrees of formality. The scope of the Network Measurement and Control Working Group is to define base protocols and extension frameworks for those protocols, as well as to define extensions that are already in common use.

Relationships with other WGs

Deliverables and Milestones D1. Base perfSONAR Protocol. This document describes the basic message formats and exchange models. This document also describes the extension mechanism and simple extensions that are common to all perfSONAR services as well as the “result codes” and their extension mechanism. This document will also discuss the AA framework and describe more complicated message patterns and the extension points to allow definition of new ones.

Deliverables and Milestones D2. Measurement Archive Protocol. This document describes the extension and application of the base protocol to support communication with a Measurement Archive (MA). The particular message types that MAs support are defined here.

Deliverables and Milestones D3. Information Service Protocol. This document describes the extension and application of the base protocol to support communication with an Information Service. The particular message types that ISs support are defined here.

Deliverables and Milestones D4. Measurement Point Protocol. This document describes the extension and application of the base protocol to support communication with a Measurement Point (MP). The particular message types that MPs support are defined here.

Deliverables and Milestones D5. Transformation Service Protocol. This document describes the specific messages and interactions required to provide a generic transformation service. The concept is to allow a mechanism for data transformation using a pipeline of web services.

Deliverables and Milestones OGF 23 (June 2008): – official start of working group – first drafts of deliverables D1, D2, D3 ready for discussion August 2008: – second draft of deliverables D1, D2, D3 ready for WG review on mailing list – First draft of deliverable D4 OGF 24 (September 2008): – WGLC for D1, D2, D3 – Second draft of deliverable D4 November 2008 – WGLC for deliverable D4 September 2009: – official closure of working group

What is perfSONAR A collaboration – Production network operators focused on designing and building tools that they will deploy and use on their networks to provide monitoring and diagnostic capabilites to themselves and their user comunities. An architecture & a set of protocols – Web Services Architecture – Protocols based on the Open Grid Forum Network Measurement Working Group (NM-WG) Schemas – Emerging standards in the Network Markup Language WG (NML-WG) Several interoperable software implementations – Java & Perl A Deployed Measurement infrastructure

perfSONAR Goals Increase network awareness – Set user expectations accurately Reduce diagnostic costs – Performance problems noticed early – Performance problems addressed efficiently – Network engineers can see & act outside their “turf” Transform application design – Incorporate network intuition into application behavior

Vision: Network Performance Information is … Available – People can find it (Discovery) – “Community of trust” allows access across administrative domain boundaries Ubiquitous – Widely deployed (Paths of interest covered) – Reliable (Consistently configured correctly) Valuable – Actionable (Analysis suggests course of action) – Automatable (Applications act on data)

perfSONAR Collaborators GRNET HEAnet Indiana University Internet2 ISTF POZNAN UNINETT University of Delaware Renater RedIRIS SLAC SWITCH SURFnet GRNET HEAnet Indiana University Internet2 ISTF POZNAN UNINETT University of Delaware Renater RedIRIS SLAC SWITCH SURFnet RNP ARNES BELNET CARNET CESNET CYNET DANTE DFN ESnet FCCN FERMI GARR GEANT Georiga Tech RNP ARNES BELNET CARNET CESNET CYNET DANTE DFN ESnet FCCN FERMI GARR GEANT Georiga Tech And anybody else we missed

perfSONAR Architecture Interoperable network measurement middleware: – Modular – Web services-based – Decentralized – Locally controlled Integrates: Network measurement tools Network measurement archives Discovery Authentication and authorization Data manipulation Resource protection Topology Based on: Open Grid Forum Network Measurement Working Group schema.

perfSONAR: System Description Domains represented by a set of services Each domain can deploy services important to the domain Analysis clients interact with service across multiple domains

perfSONAR: Services (1) Lookup Service – Allows the client to discover the existing services and other LS services. – Dynamic: services registration themselves to the LS and mention their capabilities, they can also leave or be removed if a service gets down. AuthN/Z Service – Internet2 Middleware Group, GN2-JRA5 (eduGAIN) – Authorization functionality for the framework – Users can have several roles, the authorisation is done based on the user role. – Trust relationships defined between users affiliated with different administrative domains.

perfSONAR Services (2) Transformation Service – Transform the data (aggregation, concatenation, correlation, translation, etc). Topology Service – Make the network topology information available to the framework. – Find the closest MP, provide topology information for visualisation tools Resource protector – Arbitrate the consumption of limited resources between multiple services.

Here is who I am, I’d like to access MA B Where Link utilisation along - Path a,b,c,d,e,f? a,b,c: Network A – LS A, c,d,e,f : Network B, MA B, AA B Inter-domain perfSONAR example interaction Client Network ANetwork B LS ALS B MA A MA B AA A AA B ab cd e f Where Link utilisation along - Path a,b,c? a,b,c : Network A, MA A, AA A Token MB Here is who I am, I’d like to access MA A Get link utilisation c,d,e,f Here you go Token MA Get Link utilisation a,b,c Here you go Useful graph

Schema Key Goals: Extensibility, Normalization, Readability Break representation of performance measurements down into basic elements Data and Metadata Measurement Data – A set of of measurement events that have some value or values at a particular time Measurement Metadata – The details about the set of measurement data

Schema Normalization Can simply the database representation for many types of measurement data – While optimizations are certainly possible, many measurement types can be viewed as one value over time Assists Combination/Concatenation of metrics – Creating derived metrics Normalization helps with inferring relationships between types of metrics

Schema Basic Elements - Metadata Subject – The measured/tested entity EventType (Verb) – What type of measurement, value, or event occurred – Characteristic, tool output, or generic event Parameters (Adjectives and Adverbs) – How, or under what conditions, did this event occur?

Schema Basic Elements - Data Some sort of value - Datum – Existence of an event might point to the case where there no additional value As in “Link up/down” or threshold events Time – Must be extensible since even agreement about the right structure is not easy E.g. UNIX timestamp vs NTP time

A Message Message MetadataData

An Object Store Store MetadataData

A Data is Linked to a Metadata Metadata someId Data someId

A Metadata may be linked to another Metadata someId Metadata someOtherId someId

Schema Namespaces All measurements have some sort of Data and Time All measurements can be described by the Metadata identifying who, what and how The specific structures of the Data and Metadata elements depend on the measurement Approach: Consistently use Data and Metadata elements and vary the namespaces of the specific elements

Schema Namespaces - 2 We encode the measurement/event type in the namespace – And as a standalone element Some components of the system can pass Data and Metadata elements through without understanding their specific structure Allows and implementation to decide whether it supports a particular type of data or not Allows validation based on extended (namespace- specific) schemata

Schema Namespaces and Extensibility One key to extensibility is the use of hierarchy with delegation – Similar to OIDs in the IETF management world The NM-WG has a hierarchy of network characteristics – Good starting point However, not all tools are cleanly mapped onto the Characteristic space – Often a matter of some debate

Schema Namespaces and Extensibility - 2 Organization-rooted tools namespace addresses this Some top-level tools ping, traceroute Easy to add new tools in organization-specific namespaces Performance Event Repository – Add a schema and get a URI – Add Java classes

perfSONAR-PS Motivation Create separate implementation of perfSONAR standard – Use same protocol/standards – Proof of interoperability (strengthens the standard) Targeted for NOC deployments – Lightweight – Easy to deploy/manage – (We were unable to convince our primary users to deploy Java services due to the complexity of dependencies)

perfSONAR-PS Beta Release (0.09) (May/08) Focus on development of major perfSONAR components – LS - perfSONAR_PS::Services::LS::LS – SNMP MA - perfSONAR_PS::Services::MA::SNMP – Status MA - perfSONAR_PS::Services::MA::Status – CircuitStatus MA - perfSONAR_PS::Services::MA::CircuitStatus – Topology MA - perfSONAR_PS::Services::MA::Topology – PingER (SLAC) * Recently released – OWAMP/BWCTL archive (perfSONARBUOY) Not released via CPAN

SNMP Measurement Archive Provide access to network performance data – Utilization – Errors – Discards Numerous tools exist to collect passive measurements (via SNMP): – MRTG – Cacti – Cricket Expose archives from RRD files

SNMP Measurement Archive Current Deployment: – Internet2 Network – ESnet – Georgia Tech/SOX – Fermilab

Pinger Based MP/MA Joint effort between Fermi Lab and SLAC – Present views of historic Pinger data – Expose interface to schedule live tests Built with perfSONAR-PS infrastructure pingER is used in the LHCOPN

Link Status Measurement Archive Provide access to up/down status information about layer2 links – Data stored in a SQL database – Database schema allows for storing time ranges during which a link had a certain status Minimizes storage costs for rarely changing links Communication/Configuration via XML Target audience is network operators and users interested in obtaining the status of the links over which their data flows

Link Status Measurement Archive Collector – Allows for the periodic collection of the status of one or more links – Can use SNMP, Scripts or simply Constants – Can store results directly into a database or into a remote Measurement Archive – Future Plans: TL1 Collection

Link Status Measurement Archive Visualization – A perfSONAR-UI Plugin is available that can display a network and the status of its links Current Deployment – Internet2 Network – Internet2 Dynamic Circuit Network (limited)

Circuit Status Measurement Archive An e2emon-compatible service – Integrates with the Link Status MA to provide the information stored in MAs Can work with local MAs directly or with remote MAs – Can use the Topology MA to obtain necessary information about nodes – Can use a Lookup Service to lookup the MA containing information on each link Target audience is administrators who want to publish circuit status information to e2emon clients

Circuit Status Measurement Archive Visualization – Any tool that is compatible with e2emon will work with this service Current Deployment – Internet2 Network

Topology Service Provides a queryable repository for obtaining topology information about a domain – Can obtain the entire network – Xquery interface allows the construction of complex queries about the network Topology is specified according to the schema in development in the OGF

Topology Service Current Deployments – Internet2 – SLAC (PingER Topology Information) Planned Deployments – DICE Dynamic Circuit Service Sites – ESnet

perfSONAR Lookup Service Directory service of perfSONAR deployments – Accept service registrations – Handles queries for service location and capabilities and location of available data – Manage the lifetimes of data and services to keep information up to date Web Service interface to XML Database – Sleepycat XML Database – Service Info/Data kept in native formats Draw away the complex query tasks from otherwise 'busy' services

Lookup Service Also XML based configuration/protocol – Native storage/query mechanisms [Xpath/XQuery] – Message format to exchange the data Targeted at single domain deployment – Single instance to manage multiple services Client components and applications use the LS to find services – perfSONAR-UI – perfAdmin

Lookup Service Current Deployment: – Internet2 (Ann Arbor)‏ – University of Delaware Planned Deployment: – IU for Internet2 network and regionals – DICE Dynamic Circuit Network sites – International Partners

Distributed Lookup Service Federation of individual LS instances into a global system “Meta”-lookup phase allows a query to find the specific LS that has relevant information – Or perhaps the relevant LSes that have said info The specific query is sent directly to the LS in question Recent active design and development

Distributed Lookup Service Service and measurement metadata is “summarized” for propagation to distant domains – IP addresses in service and measurement metadata are compressed into network/netmask pairs in the same way that routes are advertised (CIDR-style) These summarized metadata elements are advertised to external “scopes” – A “scope” is a set of LSes that are related by e.g. being in the same administrative domain (although multiple scopes within a single domain are possible)

Weather Maps - Internet2

Gmaps from SLAC

CNM from DFN

perfSONARUI from acad.bg

PerfsonarUI 1

PerfsonarUI 2

PerfsonarUI 3

Oscars Circuit plugin - Internet2

Oscars circuit plugin

E2Emon - Monitoring Circuits

E2Emon: Status of E2E link CERN-LHCOPN-FNAL-001 E2Emon generated view of the data for one OPN link [E2EMON]

Traceroute Visualizer Forward direction bandwidth utilization on application path from LBNL to INFN-Frascati (Italy) – traffic shown as bars on those network device interfaces that have an associated MP services (the first 4 graphs are normalized to 2000 Mb/s, the last to 500 Mb/s) 1 ir1000gw ( ) 2 er1kgw 3 lbl2-ge-lbnl.es.net 4 slacmr1-sdn-lblmr1.es.net (GRAPH OMITTED) 5 snv2mr1-slacmr1.es.net (GRAPH OMITTED) 6 snv2sdn1-snv2mr1.es.net 7 chislsdn1-oc192-snv2sdn1.es.net (GRAPH OMITTED) 8 chiccr1-chislsdn1.es.net 9 aofacr1-chicsdn1.es.net (GRAPH OMITTED) 10 esnet.rt1.nyc.us.geant2.net (NO DATA) 11 so rt1.ams.nl.geant2.net (NO DATA) 12 so rt1.fra.de.geant2.net (NO DATA) 13 so rt1.gen.ch.geant2.net (NO DATA) 14 so rt1.mil.it.geant2.net (NO DATA) 15 garr-gw.rt1.mil.it.geant2.net (NO DATA) 16 rt1-mi1-rt-mi2.mi2.garr.net 17 rt-mi2-rt-rm2.rm2.garr.net (GRAPH OMITTED) 18 rt-rm2-rc-fra.fra.garr.net (GRAPH OMITTED) 19 rc-fra-ru-lnf.fra.garr.net (GRAPH OMITTED) www6.lnf.infn.it ( ) ms ms ms link capacity is also provided