1 Proposed future direction for CHEETAH Outline Strategy discussion: What's our goal for the CHEETAH network:  eScience network or a scalable GP network? Bandwidth sharing mode:  Book-Ahead (BA) or Immediate-Request (IR)? Tactical aspects: Network evolution Networking software modules Application software modules Interconnection to HOPI/DRAGON Malathi Veeraraghavan University of Virginia August 23, 2006

2 Observation "Many e-science experiments are unique applications that involve collaboration among a handful of facilities. As a result, networks supporting these experiments are optimized to provide maximum throughput to a few facilities, as opposed to moderate throughput to millions of users, which is the raison d'etre for commercial networks."

3 eScience networks eScience network requirements Number of users small Hard to achieve high utilization; also not impt. Overprovision network to keep call blocking rate low We can then focus on creating software to allow scientists to automatically create high- speed application-specific topologies: AST, UCLP, OSCARS, USN scheduler, BRUW Bandwidth-sharing algorithms of less concern

4 General-purpose commercial networks Has to be scalable: large number of users Metcalfe's statement: Value of a network increases exponentially with the number of users High utilization is an important goal Low call blocking probability or low waiting time for resources Focus on efficient bandwidth-sharing algorithms

5 Circuit/VC service on GP commercial networks Just for ISPs/enterprise admins: needs similar to eScience router-to-router circuits limited number of users high-bandwidth, long-held circuits low price not a high priority need BA mode of bandwidth sharing For end users large number of users can only offer moderate BW and limited call holding times IR mode of sharing becomes feasible

6 BW sharing modes in circuit/VC networks Mean waiting time is proportional to mean call holding time Can afford to have a queueing based solution if calls are short Large m Moderate throughput Small m Short callsLong calls Bank tellerDoctor's office High throughput immediate-request with call blocking + retries ("call queueing") (video, gaming) immediate-request with delayed-start times (file transfers) book-ahead m is the link capacity expressed in channels e.g., if 1Gbps circuits are assigned on a 10Gbps link, m = 10

7 Impact of increasing m at different values of link utilization U d m=10 P q =41% Prob. of arriving job finding all m circuits busy Offered load: call arrival rate/call departure rate Low-rate per-call circuitsHigh-rate per-call circuits Link capacity expressed in channels

8 Impact of mean call holding time Number of ports aggregating traffic on to the link Mean waiting time for delayed calls : per host call-generation rate U d : 90%

9 Main findings of analysis Two key parameters: If m is small (per-circuit BW is high) and mean call holding time is large  then need BA to avoid long waiting times and mean call holding is small (file transfers)  then use "call queueing" If m is large, switch hardware costs increase N, number of aggregation ports, high level of demultiplexing high Moderate m: best choice

10 Book-Ahead (BA) or Immediate-Request (IR)? Bandwidth- sharing mechanisms Book-Ahead (BA)Immediate-Request (IR) eScience networks very large file transfers need high-BW and long holding time + remote viz. need to reserve other resources such as displays None? general-purpose networks circuit service to only ISPs/enterprise admins - router-to-router circuits circuit service for end users - host-to-host + router-to- router (end-to-end) - partial-path router-to-router circuits on congested links (called in by end user)

11 Support for the BA mechanism of bandwidth sharing Since RSVP-TE does not have parameters for BA calls (call duration, start time), this mode is not implemented in switch controllers Need an external scheduler to manage bandwidth into the future Easiest to make it centralized - one per domain Cannot utilize the BW management software implemented in switch controllers as part of GMPLS control-plane software The BA mode is necessary for high-BW, long-held calls

12 Support for the IR mechanism of bandwidth sharing Switches have built-in (G)MPLS control- plane software (RSVP-TE/OSPF-TE) Bandwidth management is part of RSVP-TE switch controller software Hence it is distributed bandwidth management Need to limit call holding time - reminders for renewals and automatic release Moderate-to-high per-call bandwidth

13 To implement BA, IR, or both? Implement only BA Develop and "standardize" protocols for scheduler-to- scheduler signaling for interdomain circuits (one centralized scheduler per domain) Implement scheduler and test with other networks Create software tools to enable scientists and ISP/enterprise admins to visualize network topologies and request appropriate circuits/VCs High-BW, long-held: Therefore AAA is a must Path being pursued by DRAGON, USN, OSCARS, UCLP

14 Opportunity missed if the whole optical testbed community only experiments with BA What opportunity? Enable the creation of large-scale circuit/VC networks with moderate-rate circuits that can support a brand new class of applications economic value for the telcom industry A "reservations-oriented" mode of networking to complement today's connectionless Internet ala airlines that complement roadways Could prove useful to FIND, GENI, net-neutrality Alternative pricing models for bandwidth

15 What "brand new class of applications?" Video, video, video Gaming Remote software access + Sync. storage Async storage Multimedia (large) files in web sites

16 Video applications Improve quality of conferencing, telephony, surveillance, entertainment and distance- learning by a significant degree Expend bandwidth for a higher-quality, lower latency, multi-camera, auto-movement, auto- mixing experience Make the "flat world" flatter Energy savings/environmental benefits Moderate bandwidth - IR with call blocking/retries

17 Gaming applications Current gamers buy personal graphics cards Players talk of "lag" caused by differences in graphics processing speeds Moderate-speed circuits can enable a new class of games in which rapidly-changing scenes are possible compare movies in which multiple story lines keep scenes changing vs. gaming scenes Players connect to graphics servers Data transferred is not GL commands, but rather rendered bits (doable?) Moderate bandwidth - IR with call blocking/retries

18 Remote software access/sync storage Remote software access Reduce computer administration cost Personal computers vs. machine rooms I loaded 22 new applications on my new laptop Instead: connect and run! Virtual Computing Laboratory: Mladen Vouk, NCSU Synchronous storage access Disaster recovery Moderate bandwidth - IR with call blocking/retries

19 Asynchronous storage Asynchronous storage depots will lower costs for backups disaster recovery Need for increased storage grows with multimedia files High bandwidth, short calls IR with delayed start

20 Larger files in web sites Multimedia files in web sites Imagine the use of video/audio files in all sorts of web sites instead of ASCII My own course PPT files: I use audio sparingly because of bandwidth Think assembly instructions for electric fans, furniture  Kinesthetic learning - show me a video Think hotel web pages  Show me exactly where the beach is relative to my room; do I have a balcony - saying it in text format is one thing; seeing it in a video format quite another! Content distribution network & web caching High bandwidth, short calls IR with delayed start

21 Are all these "high"-BW apps just a matter of increasing BW of links in the current Internet? No The socialistic mode of bandwidth sharing on the Internet discourages individual investment in network bandwidth Age-old question: should we pay for bandwidth with tax dollars - "free" for the whole community? "Tragedy of the commons" (Tanenbaum) should we create a network where individuals can pay for bandwidth on congested links more directly? - think higher-toll HOV lanes

22 What does all this mean? Let's build a scalable circuit/VC network in which bandwidth is shared in IR mode Scalability will create "Metcalfe's value" Provides an opportunity to finally recoup our investment in (G)MPLS technologies standards creation effort implementation: Cisco, Juniper, Sycamore, Movaz Assign at least a few of the optical testbeds that we are investing in now to study whether this IR mode of bandwidth sharing can help with our understanding of net-neutrality, economic growth, FIND questions IR more natural in data world unlike in airlines (BA)

23 Argument: IR is just a "now" in BA BA and IR cannot coexist without some form of bandwidth partitioning BA allows for high-BW, long-duration calls IR calls will suffer a high call blocking rate if supported through BA scheduler (the "add- now-as-an-option-in-scheduler" solution) Should you admit an IR call if it arrives a few seconds before start time of a BA call and hope it completes before the BA call start time, or reject the call and waste bandwidth?

24 CHEETAH and TSI The CHEETAH network solves only part of the TSI problem Other problems Cray computer I/O problem Local-area connectivity within NCSU If the CHEETAH project was a production solution to support TSI, we should spend money to solve these two problems for TSI But as an experimental short-lived networking project, where should we focus?

25 Outline Strategy discussion: What's our goal for the CHEETAH network: eScience network or a scalable GP network? Bandwidth sharing: Book-Ahead (BA) or Immediate-Request (IR)? Tactical aspects: CHEETAH network evolution Networking software modules Application software modules Interconnection to HOPI/DRAGON

26 Network evolution to support IR Current CHEETAH network only supports 10 circuits per OC192 link remember IR mode does not work well when m, the link capacity in channels, is small (i.e., 10) Recoup OC1-crossconnect capability of the SN16Ks from its current 1Gbps use Has three advantages supports higher m; better for IR GMPLS standards based signaling Call setup delay: 166ms for two-hop instead of 1.5sec!

27 Network evolution options Four options: VLAN-enabled NICs + VLSR for SN16K with VLSR IP router with VLSR Ethernet switch with VLSR

28 Example: web caching application CHEETAH zelda4 Xiuduan Fang, Bob Gisiger, ORNL zelda3 Atlanta, GA wukong Raleigh, NC mvsut6C'ville, VA UTK UGa Gatech duke NCSU UN C UVa VT VLANs

29 VLAN-enabled NICs + SN16K VLSR SN16K has data-plane support to map a sub-Gb/s VLAN on an Ethernet port to a corresponding number of OC1s on a SONET port But, it does not have control-plane support for this type of circuit Even GMPLS support for the GbE port mapping to a 21-OC1 VCAT signal is an experimental release just for CHEETAH usage Because GMPLS support for such hybrid circuits is non- standard Can implement our own (non-standard) solution as a VLSR But, goal is to use off-the-shelf switches with GMPLS support to demonstrate IR mode

/VLSR at each PoP Make the serve as the intermediary between Ethernet NICs in hosts and SONET based SN16Ks at CHEETAH PoPs A VLSR could be useful for other projects, UCLP, Ultralight Cisco has no plans to implement a GMPLS control-plane engine for the Two problems: Non-standard solution for hybrid circuits VLAN ID continuity requirement Cannot support partial-path circuits

31 IP router/VLSR at each PoP Use channelized OCxx SONET interfaces to connect IP router to SN16K Connect web caches to router Have routers initiate pure SONET circuit setup Use PBR or just ordinary routing table update to map flows to different OCxx circuits; support multiple circuits from one web cache

32 CHEETAH wide-area network Raleigh PoP (MCNC) Control card GbE/ 10GbE card ORNL PoP Control card GbE/ 10GbE card SN16000 Control card OC192 card OC-192 GbE/ 10GbE card End hosts Atlanta PoP (SOX/SLR) SN16000 GaTech End hosts ORNL OC192 card NCSU OC192 card OC-192 (via NLR/SLR/NCREN) via NCREN UVa CUNY Via Nysernet/HOPI Via Vortex/HOPI

33 CHEETAH evolution to support sub-Gb/s circuits Raleigh PoP Control card OC192 card ORNL PoP Control card OC192 card SN16000 Control card OC192 card OC-192 End hosts Atlanta PoP SN16000 GaTech End hosts ORNL OC192 card NCSU OC192 card OC-192 (via NLR/SLR/NCREN) UVa CUNY OC192 card GbE

34 IP router/VLSR at each PoP Can support end-to-end circuits web caching CDN servers video apps at 10-15Mbps - map to one OC1 storage depots Has the potential to support PPCs (partial-path circuits) Place router with VLSR in enterprises at edge of GbE cheetah access link

35 Ethernet switch/VLSR at each PoP Does not help with the problems noted in today's Gb/s circuit use of the SN16K long call setup delays: 1.5sec non-standard solution high per-circuit BW Using an Ethernet switch/VLSR at an enterprise (e.g. CUNY) requires all VLANs sharing 1Gbps CHEETAH access link to be switched to the same exit SN16K. Even worse, m=1 if whole 1Gb/s link used for a circuit

36 Software modules required Networking software: CVLSR for IP router CTCP code to support multiple simultaneous flows Application software: Add CHEETAH API to web caching squid software Write software for video apps CDN and storage software

37 Upcoming year goals specified in special report Work itemMilestones and deliverable Responsible individuals Item IStabilize the CHEETAH network; increase user base; complete doc. MV and Xuan/Tao Item IIExtend CHEETAH network by adding routers/VLAN switches/MSPPs MV and Tao Item IIIInterconnect to testbeds, such as HOPI, USN, DRAGON MV and Tao

38 Work itemMilestones and deliverable Responsible individuals Item IVDevelop software, both apps and networking s/w, such as CVLSRs Router CVLSR: Mark CTCP: Helali Apps: Xiuduan Item VSupport ORNL and NCSI in TSI apps Mark and Tao/MV Item VIEnhance theoretical understanding of sharing modes IR blocking/retries: XF BA: Xiangfei IR delayed start: Mark Upcoming year goals specified in special report

39 Equipment required IP routers with channelized SONET cards with GA GMPLS UNI implementation need one for ORNL PoP if we can partner with SOX in ATL, NCREN in Raleigh, MAX in McLean, purchase channelized OC192 cards IP routers with GbE blades for V. Tech and UVA If NC 454 transponders are unavailable, purchase transponders for DC-Raleigh NLR link - since HOPI doesn't have this link Colocation costs at NLR McLean and Raleigh

40 Interconnect CHEETAH and HOPI Through IP routers With our IP router/VLSR combo, setup router-to-route SONET OC1 circuits via cheetah and router-to-router VLAN virtual circuits through HOPI. At routers, do PBR mapping for flows or just update routing tables This means packets go back to IP layer between two networks

41 CHEETAH-HOPI interconnection McLean, VA 10GbE CHEETAH HOPI network: courtesy of Rick Summerhill NC GATN Web cache Web cache Web cache Web cache Web cache Web cache Web cache Web cache VLAN MPLS SONET

42 HOPI and web caching Seems like a good match Rick's black cloud experiment - same as web caching Exercises "hybrid" goal of HOPI Small per-circuit BW possible with VLANs

43 Connection to DRAGON Spoke with Jerry Sobieski, Aug. 12, 2006 He said DRAGON PoPs have Ethernet VLAN switches Therefore, can use similar IP router demarcation points to interconnect CHEETAH/HOPI to DRAGON

44 Conclusions We'd like to enable and demonstrate general-purpose apps using circuit/VC service with scalability as a key goal support IR mode of bandwidth sharing with limited per-call bandwidth and limited call holding times call blocking with retries delayed start