1. Inter-Domain Services Presented to Quilt Spring Workshop Salt Lake City, UT By Linda Winkler winkler@mcs.anl.gov April 4, 2007

2. Motivation Support applications that demand capabilities that are hard to support in a shared packet infrastructure Support applications that demand capabilities that are hard to support in a shared packet infrastructure Large bandwidth applicationsLarge bandwidth applications Applications don’t want to worry about their impact on other network users, or vice versaApplications don’t want to worry about their impact on other network users, or vice versa Applications that benefit from circuit characteristics, and that may be low bandwidth in natureApplications that benefit from circuit characteristics, and that may be low bandwidth in nature Dedicated network resources Dedicated network resources Deterministic performance Deterministic performance Repeatable and predictableRepeatable and predictable Very high performanceVery high performance Multi gigabit flows, low latency, low loss, minimal jitterMulti gigabit flows, low latency, low loss, minimal jitter User community expectations User community expectations Extreme bandwidth, dedicated and on-demandExtreme bandwidth, dedicated and on-demand Inter-regional and international connectivity requirementsInter-regional and international connectivity requirements Reserve and schedule network resources in advance Reserve and schedule network resources in advance In conjunction with availability of non-network resources (e.g. instruments, clusters, viz devices)In conjunction with availability of non-network resources (e.g. instruments, clusters, viz devices) Flexible and dynamic Flexible and dynamic Capability to acquire resources on short notice from many potential service/resource providersCapability to acquire resources on short notice from many potential service/resource providers E-Science services growth E-Science services growth Broad scope of applicationsBroad scope of applications Petabyte demandsPetabyte demands Globally distributed applications, resources and collaboratorsGlobally distributed applications, resources and collaborators Emerging large-scale, globally distributed applications require more sophisticated network services than have previously been delivered.

3. Trends=>Set the Stage Applications community desire to treat network resources as an integral Grid resource Applications community desire to treat network resources as an integral Grid resource R&E Optical Networks R&E Optical Networks Locally-managed fiber termination pointsLocally-managed fiber termination points Locally-organized peering relationshipsLocally-organized peering relationships Locally-controlled layer-0/1/2/3 servicesLocally-controlled layer-0/1/2/3 services Community ownership and control of network infrastructureCommunity ownership and control of network infrastructure Ubiquitous Ethernet Ubiquitous Ethernet Most-requested client service interfaceMost-requested client service interface Both point-to-point and virtual-LAN topologiesBoth point-to-point and virtual-LAN topologies Apps consuming bandwidth in Ethernet-sized incrementsApps consuming bandwidth in Ethernet-sized increments 10G has become a commodity10G has become a commodity Application expectations for error-free connections keep rising. Apps with minimal error checking running on pipes with almost no error correction, across inter-RON pipes with no error reporting  a recipe for failureApplication expectations for error-free connections keep rising. Apps with minimal error checking running on pipes with almost no error correction, across inter-RON pipes with no error reporting  a recipe for failure Convergence Optical + Ethernet Layers Convergence Optical + Ethernet Layers Ethernet client interface, Optical transportEthernet client interface, Optical transport Optical service established via GMPLSOptical service established via GMPLS Control plane technologies emerging to help meet user expectations in the current environment Control plane technologies emerging to help meet user expectations in the current environment FPGA technology FPGA technology Allowing WDM transport hardware to be built with software defined functionalityAllowing WDM transport hardware to be built with software defined functionality Flexibility, fewer components, configurations, sparesFlexibility, fewer components, configurations, spares Investment protection and future-proofingInvestment protection and future-proofing Pluggable optics Pluggable optics Flexibility, lower costsFlexibility, lower costs Newer capabilitiesNewer capabilities

4. Technology Choices Reasons you may want/need build circuits Dedicated bandwidth connections for deterministic file transfers Dedicated bandwidth & low jitter for instrument control or interactive applications Connector backhaul to your IP Network Traffic engineering of your IP Network Dynamic router-to-router circuits for traffic cut through Computer to Computer communications Processor to memory or block data storage system access Setting up application specific topologies to create & optimize distributed application or data storage systems There are many technologies available over which to construct these circuits IP router-based Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) “circuits” Ethernet VLAN based “circuits” SONET/SDH TDM “circuits” Wavelength Division Multiplexing (WDM) “circuits”

5. IP Ethernet SONET DM encap GFP Lambda Fiber Network TechnologyService Dark Fiber Wavelength Lightpath (sub-lambda) Ethernet VLAN IP Layer 3 Layer 2 Layer 1 Layer 0 Mapping Technology to Services

6. Starlight iCAIR UvA NetherLight, Amsterdam Nortel Research Labs Ottawa Glimmerglass Topology for L1 Dynamic Optical Multicast Testbed 750-~980+ Mbps Hi-Def Video 3x1:2 Optical Multicast: StarLight OME Nortel HDXc OC192/STM64 10GE LAN 750-~980 Mbps Nortel HDXc UvA Nortel Nortel DX Fujitsu UvA Nortel iCAIR UvA iCAIRNortel 3x750 M ONS 3x750 M 3x1:2 Optical multicast 2 1 OMNInet Nortel DX Nortel 8600 10 G WAN Nortel 8600 1 2 ONS 1 2 CA*Net5 Nortel OME 6500 iCAIR Nortel OME 6500

7. Nortel Brno LSU Starlight Netherlight SC NOC Baton Rouge (LSU Class) McLEAN A-Wave CAT 6506 3x1.5G LSU Topology for L2 Multicast Demo 1.5 Gbps Hi-Def video 3x1:2 Ethernet multicast: StarLight & SCinet Layer1 and Layer2 networking Nortel HDXc Nortel HDXc Raptor CaveWave Catalyst 65xx e300 BrnoTampa F10 E1200 Calient CAT 6509 Brno ONS Brno LONI TampaBrno LONITampa 3x1.5G OC-192 GFP/VCAT 10GE LAN 10G WAN 1.5 Gbps 3 x 1.5G 10GE switch OPTERON 1 2 1 4 2 1 2 1 3 1 2 1 2 CAT 6504 SL E1200 2 2 Calient Tampa LSU Nortel MERS 8600 OPTERON AMD PC ONS OMNInet EnLIGHTened Wave CAT 6506

8. IP NetworkCircuit Switched VLAN over MPLS tunnel Layer-1/2 connection VLAN over Ethernet or SONET Alignment of VLANS Unification of Data Plane Technology Across Layers 1/2/3—One Method

9. Dynamic Circuit Setup/Allocation Lots of work in the community on dynamic resource allocation (lightpaths) and how we’re going to get there from here, eg GLIF forum. Lots of work in the community on dynamic resource allocation (lightpaths) and how we’re going to get there from here, eg GLIF forum. Soon we will be to able to rely on multiprovider/multivendor control plane signaling for circuit establishment. Soon we will be to able to rely on multiprovider/multivendor control plane signaling for circuit establishment. Simplify or add complexity? Simplify or add complexity? Dynamic resource mechanisms will increase our capabilities and enable new capabilites, like timesharing resources between large short-duration flows. Dynamic resource mechanisms will increase our capabilities and enable new capabilites, like timesharing resources between large short-duration flows. Cost TBD Cost TBD

10. Connection Brokering per Domain Network Descriptions Network Descriptions Paths through networkPaths through network Information about pathsInformation about paths Common Services Definition Common Services Definition Description of connectivity serviceDescription of connectivity service Predictable Predictable Verifiable Verifiable Repeatable Repeatable End-to-end End-to-end Agreement between user and provider(s)Agreement between user and provider(s)

11. Control Plane Functions Bandwidth required Bandwidth required Allow for reservation of future bandwidthAllow for reservation of future bandwidth User authentication User authentication Report status of infrastructure Report status of infrastructure Retain and share information on topology resources and reachability Retain and share information on topology resources and reachability Routing- topology discovery and path computation Routing- topology discovery and path computation Signaling- standard communications protocols between network elements for the establishment and maintenance of connections Signaling- standard communications protocols between network elements for the establishment and maintenance of connections Neighbor discovery- cross-domain sharing of connectivity details among neighbors Neighbor discovery- cross-domain sharing of connectivity details among neighbors Domain (local) resource management- accounting of available resources Domain (local) resource management- accounting of available resources

12. Interdomain Connection Setup TDM (SONET/SDH Layer) RON / NREN Data Plane TDM or L2 substrate Interdomain Control Plane

13. Challenges Engineering/configuration/setup Engineering/configuration/setup O&M complexity O&M complexity TroubleshootingTroubleshooting Fault managmentFault managment MonitoringMonitoring PerformancePerformance Debugging tools Debugging tools End-to-end measurement for monitoring and performance analysis End-to-end measurement for monitoring and performance analysis Policies Policies a way to enforce policies and filters at the domain boundaries to process the incoming inter- domain requests based on certain agreed trust and service levels/contracts between domainsa way to enforce policies and filters at the domain boundaries to process the incoming inter- domain requests based on certain agreed trust and service levels/contracts between domains Authentication, Authorization, Accounting (AAA) Authentication, Authorization, Accounting (AAA) Security Security Scheduling Scheduling Service Level Agreements (SLAs) Service Level Agreements (SLAs) User interface User interface Capacity planning Capacity planning Many lightpaths traverse a number of transport boxes and domains Many lightpaths traverse a number of transport boxes and domains Number of domains participating on the rise (backbone providers, RONs, campuses). Are they prepared for the challenge? Number of domains participating on the rise (backbone providers, RONs, campuses). Are they prepared for the challenge?

14. Dynamic Resource Allocation Research MAX DRAGON MAX DRAGON Internet2 Circuit Services- based on extension of the DRAGON control plane architecture Internet2 Circuit Services- based on extension of the DRAGON control plane architecture Internet2 BRUW (Bandwidth Reservation for User Work) Internet2 BRUW (Bandwidth Reservation for User Work) ESNet Science Data Network and the OSCARS (On-demand Secure Circuits and Advance Reservation System) project ESNet Science Data Network and the OSCARS (On-demand Secure Circuits and Advance Reservation System) project DANTE/GEANT JRA3 project DANTE/GEANT JRA3 project SURFnet and collaboration with Nortel on the DRAC project SURFnet and collaboration with Nortel on the DRAC project University of Amsterdam, Network Description Language University of Amsterdam, Network Description Language European Union PHOSPHORUS Project European Union PHOSPHORUS Project G-Lambda project (Japan) G-Lambda project (Japan) CANARIE- UCLP CANARIE- UCLP DOE UltraScience Network DOE UltraScience Network DOE Hybrid Multi-Layer Network Control Project (ESnet, ORNL, I2, ISI East, Tenn Tech) DOE Hybrid Multi-Layer Network Control Project (ESnet, ORNL, I2, ISI East, Tenn Tech) TeraPaths Brookhaven National Lab TeraPaths Brookhaven National Lab Enlightened Computing Project (NCREN, MCNC, LSU, NCSU, RCI) Enlightened Computing Project (NCREN, MCNC, LSU, NCSU, RCI) Cheetah (Univ of Virginia) Cheetah (Univ of Virginia) UltraLight UltraLight DICE (DANTE, Internet2, CANARIE, ESnet) DICE (DANTE, Internet2, CANARIE, ESnet) OptIPuter/iCAIR OptIPuter/iCAIR

15. Cisco/UltraLight wave EnLIGHTened wave (Cisco/NLR) LONI wave Members: - MCNC GCNS - LSU CCT -NCSU -(Subcontract) RENCI Official Partners: - AT&T Research - SURA - NRL - Cisco Systems - Calient Networks - IBM NSF Project Partners - OptIPuter - UltraLight - WAN-in-LAB - DRAGON - Cheetah International Partners LUCIFER - EC G-Lambda - Japan -GLIF CHI HOU DAL TUL KAN PIT WDC OGD BOI CLE POR DEN SVL SEA Baton RougeRaleigh To Asia To CanadaTo Europe L.A.San Diego CAVE wave Chicago Enlightened Computing Testbed

16. HD Media Testbed to support class on Computational Science

17. Enlightened Architecture

18. Internet2 Circuit Service Attributes Physical connection Physical connection 1G or 10G Ethernet1G or 10G Ethernet OC-48 or OC-192 SONETOC-48 or OC-192 SONET Guaranteed bandwidth (STS-1 to STS-192) Guaranteed bandwidth (STS-1 to STS-192) Circuit Service Type Circuit Service Type Point-to-point Ethernet framed SONET CircuitPoint-to-point Ethernet framed SONET Circuit Point-to-point SONET CircuitPoint-to-point SONET Circuit Deterministic behavior (bounded jitter, latency and packet loss) Deterministic behavior (bounded jitter, latency and packet loss) Static or dynamic Static or dynamic

19. NLR FrameNet FrameNet Ethernet-based transport services over the nationwide NLR optical infrastructureEthernet-based transport services over the nationwide NLR optical infrastructure National Exchange FabricNational Exchange Fabric Dedicated FrameNetDedicated FrameNet Non-Dedicated FrameNetNon-Dedicated FrameNet Research with MCNC Research with MCNC Calient Optical SwitchCalient Optical Switch

20. References dragon.maxgigapop.net dragon.maxgigapop.net dragon.maxgigapop.net www.es.net/oscars www.es.net/oscars www.es.net/oscars www.internet2.edu/network/library/internet2-network- service-descriptions-02272007-v1.1a.pdf www.internet2.edu/network/library/internet2-network- service-descriptions-02272007-v1.1a.pdf www.ist-phosphorus.eu/ www.ist-phosphorus.eu/ www.ist-phosphorus.eu/ www.mfaforum.org/interop/GMPLSwhitepaper_Final100902 1.pdf www.mfaforum.org/interop/GMPLSwhitepaper_Final100902 1.pdf www.mfaforum.org/interop/GMPLSwhitepaper_Final100902 1.pdf www.mfaforum.org/interop/GMPLSwhitepaper_Final100902 1.pdf www.glif.is www.glif.is www.glif.is enlightenedcomputing.org enlightenedcomputing.org www.oiforum.com www.oiforum.com www.oiforum.com IETF IETF ITU ITU