Data Plane Testing on USN, ESnet, and Internet2 Networks ESCC/Internet2 Joint Techs Summer Meeting July 16, 2007 Fermi Lab Batavia, Illinois Tom Lehman (USC/ISI) Nasir Ghani (Tennessee Tech) Chin Guok (ESnet) Nagi Rao (ORNL) John Vollbrecht (Internet2) John Moore (MCNC) Hybrid MLN US Dept. of Energy Office of Science

Outline Dataplane testing configuration Dataplane test data Dataplane simulation Dataplane test conclusions Future Work

Hybrid Network Data Planes Basic premise of hybrid networks is the availability of both best effort routed service and deterministic dedicated resource paths, i.e., circuits There are many technologies available over which to construct these circuits IP router-based Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) “circuits” Ethernet VLAN based “circuits” SONET/SDH TDM “circuits” Wavelength Division Multiplexing (WDM) “circuits”

What Data Plane Technologies to Use? What do you want to do with your circuits? Dedicated bandwidth connections for deterministic file transfers? Dedicated bandwidth & low jitter for instrument control or interactive applications? Connector backhaul to your IP Network? Traffic engineering of your IP Network? Dynamic router-to-router circuits for traffic cut thru? Computer to Computer communications? Processor to memory? block data storage system access? Setting up application specific topologies to create & optimize distributed application or data storage systems?

Data Plane Testing Test characteristics/performances of “circuits” constructed via different technologies; and also “end-to-end paths” constructed via concatenation of individual circuits Questions What is difference between the different technologies? How well does the concatenation/stitching work? How well does policing/shaping work at the edge? What happens to a flow that is policed/shaped at the ingress edge by the time it exits the egress edge?

Data Plane Testing Data Paths Across: ESnet USN Abilene Internet2 Network DRAGON http://hybrid.east.isi.edu

Test Equipment Spirent AX4000 - Hardware based Traffic Source and Sink External CDMA Clock allows for synchronized timestamps Spirent AX4000 10 Gbps with OC192 POS / BERT / 10GbE Two Gigabit Ethernet

Data Collection Approximately 75 individual tests (generally a unique path) tests were sourced from 1 Gbps interfaces, some from 10 Gbps some of the measured flows had cross traffic introduced Tests generally included 9 measured data collection runs 64, 500, 8000 byte MTU 100, 500, 800 Mbps for 1 Gbps paths 1, 5, 8 Gbps for 10 Gbps paths For each test run, the following data was collected: average datarate total packet loss average latency jitter profile (histogram) transfer delay (histogram)

HOPI-Abilene-UltraScience Net-ESnet Test Histograms packet inter-arrival packet end-to-end delay Spirent source  Washington-Force 10  Washington-Juniper T640  Chicago-Juniper T640 Chicago-Force 10 Chicago-Glimmer Glass  Chicago-Force 10 Chicago-Cisco 6509  Seattle-Juniper T640  Sunnyvale-Juniper T640 Sunnyvale-Force 10  Sunnyvale-CDCI  Seattle-CDCI  Chicago-CDCI  Chicago-Force 10  Chicago-Juniper T640  Washington-Juniper T640 Washington-Force 10  Spirent receiver

Circuit Description Example The formal description of this extended inter-network path: Circuit type: usn [ethernet:tdm:ethernet]:i2dsn [ethernet:tdm:ethernet]:esnet [ethernet:pscq:ethernet]:usn[ethernet:tdm:ethernet] Circuit path: usn [ORNL:CHIN]:i2dsn [CHIN:WASH]:esnet [WASH:CHIN]:usn [CHIN:STTL:SUNV]

OPNET ModelerTM Environment “In-House” Development Modeling & Simulation OPNET ModelerTM Environment Overview Discrete event simulation GUI interface, high re-use Full C/C++ interface Hierarchical modeling: Subnet-node-link-process “In-House” Development MPLS/GMPLS control: RSVP-TE, OSPF-TE, PCE Layer 2/3 data plane: IP/MPLS, VLAN Full Layer 1 support: DWDM, SONET,GFP Model any networks

Test Plans, Reports, and Data Repositories Test Report and Plans http://hybrid.east.isi.edu  DataPlane Testing and Analysis Raw data repositories http://www.csm.ornl.gov/ultranet/SpirentMeasurements/ http://hpn.east.isi.edu/dataplane/sprint-test-data/

Summary/Conclusions All of the tested networking technologies (PSC, L2SC, TDM, LSC) and networks (ESnet, USN, Abilene, HOPI, DRAGON) performed well both individually and when concatenated together There are some key differences observed between the various networking layer technologies when driven at or close to bottleneck capacity QoS techniques applied to router MPLS or Ethernet switched paths exhibited notably different delay behaviors versus dedicated circuit-paths (TDM) TDM-based infrastructures is most germane for applications requiring stringent guarantees on latency, jitter, and bandwidth protection Inter-layer cross-connections can be achieved in a reasonable manner by “stitching” together different network layer technologies. Ethernet VLANs presents the least problematic demarc (automated techniques needed to coordinate VLAN tag space) Future Work impact of ingress traffic "burstiness” on end-to-end delay and loss profiles, i.e., both for reference and interfering cross-traffic streams. best techniques for ingress policing and transit node QoS vendor interoperability testing additional network testing

Questions & Comments ? Tom Lehman tlehman@east.isi.edu Thank-You Questions & Comments ? Tom Lehman tlehman@east.isi.edu