1. WAN in Lab NSF Site Visit John Doyle, CDS/EE/BE Steven Low (PI), CS/EE Harvey Newman, Physics Demetri Psaltis, EE/CNS Steven Yip, Cisco March 5, 2003

2. netlab.caltech.edu Reviewer concerns  Narrow focus on TCP/AQM A range of IST research at Caltech Spanning theory, implementation, experiment, deployment WAN in Lab a critical component  Alternatives not discussed Use spectrum of tools at different stages  How to manage and share WAN in Lab Part of Federated Emulab Both demand and excellent support for global sharing Experience in global collaboration, e.g. Newman’s VRVS

3. netlab.caltech.edu Agenda  EAS, IST Initiative, Theory program, FAST Intellectual environment in which WAN in Lab fits (Murray, Doyle, Low)  WAN in Lab Design, capabilities, alternatives, management (Low) Cisco example & collaboration (Yip)  Education, outreach, poster session  Research talks Projects that will use WAN in Lab International collaboration, leverage & impact on HENP & Grids

4. WAN in Lab Steven Low netlab.CALTECH.edu NSF Site Visit March 5, 2003

5. Why Testbed in IST “A lack of wide-area testbeds would contribute to a growing tendency towards paper solutions to thesis- factory problems, leaving the real networking world short of new ideas and technologies” “Prototypes & testbeds are required to gain acceptance of new concepts with potential user communities” A value of testbeds is “… building and maintaining research collaborations and communities” - NSF Workshop on Network Research Testbeds (Nov 2002)

6. netlab.caltech.edu Outline  Proposal summary Basic design, equipment, costs  Unique features  Alternatives Spectrum of tools Emulated delay  Community resource Demand Management software  Why Caltech Leverage on Abilene, HENP, CalREN, TeraGrid  Summary Reviewer concerns Review criteria

7. netlab.caltech.edu Goal State-of-the-art WAN  High speed 2.5G  10G  Large distance 50 – 200ms  Controlled & repeatable experiments  Reconfigurable & evolvable

8. 1 20 1 fiber spool OPM Max path length = 10,000 km Max one-way delay = 50ms S S S S R R H R : server : router electronic crossconnect S S S S R R EDFA 500 km

9. netlab.caltech.edu Equipment  26 Servers GbE cards (  10GbE cards)  12 routers 10 Cisco 15454 with router blades  2-port GbE, 8-channel OC48 2 Force10 E600  24-port GbE, 2-port OC48  DWDM gears 500km fiber 6 EDFA 2 Dispersion compensation modules 2 optical mux/demux  Tektronix TDS7404 Oscilloscope  Integration with global network

10. netlab.caltech.edu Costs  26 Servers: $104K  12 routers: $1.03M 2 Force10 E600: $280K ($340K if OC192) 10 Cisco 15454 with router blades: $750K ($810K if OC192)  DWDM gears: $148K 500km fiber: $8K 6 EDFA: $60K 2 Dispersion compensation modules: $40K 2 optical mux/demux: $40K  Tektronix TDS7404 Oscilloscope: $50K  Integration with global network: $110K  Personnel, software, service & maintenance  Total: $2M (NSF) + $0.67M (cost sharing)

11. netlab.caltech.edu Yearly costs  Year 1: $1.128K 10 servers, 5 routers, 2.5Gbps  Year 2: $564K 20 servers, 8 routers, 2.5Gbps  Year 3: $124K Software development  Year 4: $733K 26 servers, 10 routers, 2.5Gbps  Year 5: $120K Software development  Total: $2M (NSF) + $0.67M (cost sharing)

12. netlab.caltech.edu Networking Lab Jorgensen Lab NetLab  WAN in Lab 3 racks, 2 consoles  Networking Lab 424 sq ft Next to CACR Easy connection to global network Renovation (cost sharing)  New IST Building

13. netlab.caltech.edu Unique capabilities  WAN in Lab Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip  Configurable & evolvable Topology, rate, delays, routing Always at cutting edge  Risky research MPLS, AQM, routing, …  Integral part of R&A networks Transition from theory, implementation, demonstration, deployment Transition from lab to marketplace  Global resource (a) Physical network R1 R2 R 10 1 3 20 2 18 19 1 3 20 2 4 19

14. netlab.caltech.edu Unique capabilities  WAN in Lab Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip  Configurable & evolvable Topology, rate, delays, routing Always at cutting edge  Risky research MPLS, AQM, routing, …  Integral part of R&A networks Transition from theory, implementation, demonstration, deployment Transition from lab to marketplace  Global resource R1 R2 R3 R 10 (b) Logical network 1 2 3 4 19 20

15. netlab.caltech.edu  WAN in Lab Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip  Configurable & evolvable Topology, rate, delays, routing Always at cutting edge  Risky research Dynamic recovery, AQM, MPLS, routing, …  Integral part of R&A networks Transition from theory, implementation, demonstration, deployment Transition from lab to marketplace  Global resource Federated Netlab (Emulab) Unique capabilities Calren2/Abilene Chicago Amsterdam CERN Geneva SURFNet StarLight WAN in Lab Caltech research & production networks Multi-Gbps 50-200ms delay Experiment

16. netlab.caltech.edu Outline  Proposal summary Basic design, equipment, costs  Unique features  Alternatives Spectrum of tools Emulated delay  Community resource Demand Management software  Why Caltech Leverage on Abilene, HENP, CalREN, TeraGrid  Summary Reviewer concerns Review criteria

17. netlab.caltech.edu Spectrum of tools log(cost) log(abstraction) mathsimulationemulationlive nkWANiLab NS SSFNet QualNet JavaSim Mathis formula Optimization Linear model Nonlinear model Stocahstic model DummyNet EmuLab ModelNet WAIL HENP Abilene CalREN WAIL PlanetLab CAIRN NLR ? …we use them all

18. netlab.caltech.edu Spectrum of tools mathsimulationemulationlive nk WANiLab DistanceHigh SpeedHigh Low RealismHigh Low TrafficHighLow ConfigurableLowMediumHigh MonitoringLowMediumHigh CostHighMediumLow Critical in development e.g. Web100

19. netlab.caltech.edu Emulated delay  Available technology inadequate Spirent SX/14 Link Simulator: 1ms (155Mbps) – 10s (100bps)  Adequate technology too expensive 2.5Gbps, 100ms delay: IC expert at least 2 man-years & $200K  Less realistic 1 20 1 S S S S R R S S S S R R High speed electronic memory

20. netlab.caltech.edu HENP testbed Ins. RTT CWND: 5801-5815 Sylvain Ravot (Caltech/CERN)

21. netlab.caltech.edu Example1: end-to-end delay X: ACK # Y: RTT (us) Ins. RTT Avg RTT CWND: 5873-5887 Delay between Geneva & Chicago

22. netlab.caltech.edu Example1: end-to-end delay X: ACK # Y: RTT (us) Ins. RTT Avg RTT CWND: 5801-5815 instantaneous RTT

23. netlab.caltech.edu Example1: end-to-end delay Y: RTT (us) Ins. RTT CWND: 5801-5815 X:Real Time (us) instantaneous RTT Delay between Geneva & Chicago average RTT

24. netlab.caltech.edu Example1: end-to-end delay Y: RTT (us) Ins. RTT CWND: 5801-5815 X:Real Time (us) instantaneous RTT RTT=270ms 12450 pkts!? Delay between Geneva & Chicago 1500 pkt time without buildup!?

25. netlab.caltech.edu Example1: end-to-end delay X:Real Time Y: RTT (us) Ins. RTT Avg RTT Y: RTT (us) Ins. RTT CWND: 8700->4000 X:Real Time (us) RTT=980ms!? Passive monitoring in WANiLab can help debug

26. netlab.caltech.edu Example1: end-to-end delay X:Real Time Y: RTT (us) Ins. RTT Avg RTT X:Real Time (us) Y: ACK Seq#CWND: 8700->4000

27. netlab.caltech.edu Example2: 10G Expt inst RTT avg RTT Loss & retransmission

28. netlab.caltech.edu Example2: 10G Expt inst RTT avg RTT Losses & retransmissions Real time s ss Delay between Geneva & Sunnyvale

29. netlab.caltech.edu Example2: 10G Expt inst RTT avg RTT Retransmission without loss!? Losses & retransmissions Real time s ss

30. netlab.caltech.edu Example2: 10G Expt inst RTT avg RTT Retransmission without loss!? Passive monitoring in WANiLab can help debug Real time s ss

31. netlab.caltech.edu Network debugging  Performance problems in real network Simulation will miss Emulation might miss Live network hard to debug  Enable or speed up FAST development 10GExpt: 20 people in 8 organizations for 3 months Complete facility available only for a week Many mysteries unresolved  WAN in Lab Passive monitoring inside network Active debugging possible

32. netlab.caltech.edu Passive monitoring David Wei (Caltech) Fiber splitter DAG RAID Timestamp Header GPS Monitor  No overhead on system  Can capture full info at OC48 UofWaikato’s DAG card captures at OC48 speed Can filter if necessary Disk speed = 2.5Gbps*40/1500 = 66Mbps  Monitors synchronized by GPS or cheaper alternatives  Data stored for offline analysis

33. netlab.caltech.edu Passive monitoring David Wei (Caltech) Fiber splitter DAG RAID Timestamp Header GPS Monitor Server router monitor Web100, FAST monitor

34. netlab.caltech.edu Outline  Proposal summary Basic design, equipment, costs  Unique features  Alternatives Spectrum of tools Emulated delay  Community resource Demand Management software  Why Caltech Leverage on Abilene, HENP, CalREN, TeraGrid  Summary Reviewer concerns Review criteria

35. netlab.caltech.edu DataTAG link Ins. RTT CWND: 5801-5815 Sylvain Ravot (Caltech/CERN)

36. netlab.caltech.edu DataTAG Link (Sylvain Ravot, Caltech/CERN)

37. netlab.caltech.edu HEP Network (DataTAG) NL SURFnet GENEVA UK SuperJANET4 ABILEN E ESNET CALRE N It GARR-B GEANT NewYork Fr Renater STAR-TAP STARLIGHT Wave Triangle  2.5 Gbps Wavelength Triangle 2002  10 Gbps Triangle in 2003 Newman (Caltech)

38. netlab.caltech.edu DataTAG link  Funded by EU (CERN), USA (DoE, NSF, Caltech)  OC48 circuit StarLight-CERN Upgrade to OC192 by August 2003  Linux farms StarLight: 20 CPU (P4), 20 Syskonnect CERN: 12 CPU (P4), 12 Syskonnect  50 users, 13 institutes, 7 countries (Feb 2003)  Heavy utilization European hours: 100% reservation US hours: 25% reservation, but busy

39. netlab.caltech.edu Netbed (Emulab)  Funded by NSF with Cisco donations  Integrates simulation, emulation, live Internet Dummynet & VLAN  Emulab Classic University Utah: 168 PC, 5 100M Ethernet cards Connected by 4 Cisco 6409 Testbed backplane limited to 2Gbps University of Kentucky: 48 PC, similar setup  Netbed: Federated Emulab 32 nodes in 25 sites  Heavy utilization July2002  65 user accounts (40 external)  54 projects Feb 2003  400 user accounts  94 projects (10 Utah, 78 US, 6 Int’l) University of Utah

40. netlab.caltech.edu Management software  Part of Federated Emulab Tailor Emulab management software Jay Lepreau’s team consult on setup  Complementary to existing federated Emulab WANiLab  High speed large distance (Gbps WAN)  Small network (30 nodes) Emulab  Low speed (100Mbps LAN, 10M WAN)  Large network (200+ nodes)  Instantly available to Emulab community Web accessible anywhere any time Virtual machine for network experimentation

41. netlab.caltech.edu Experiment life cycle (White et al)  Experiment creation Web based sign-up form by project lead Approved by Emulab team  Experiment specification ns script or Java GUI Can download own OS, host algorithms, etc Links emulated by Dummynet nodes with specified rate, delay, loss  Experiment realization Map target configuration to physical resources Reserve resources for each experiment Oversubscription dynamic reallocation, swap in, swap out

42. netlab.caltech.edu Why Caltech: synergies  Caltech networking research FAST project: the missing experimental facility (Doyle, Low) IST Initiatives: testbed tied to rich theory program (Murray, Psaltis) Combination of theory, implementation, experiment & deployment  Synergy in research Caltech’s leadership role in IT for global HENP (Newman) Vibrant research in HENP, astronomy, geological sci, biology, visualization, CACR Early testing ground & adopter of FAST (Newman) Availability of real data for ultrascale networks  Synergy in facility Integration with HENP networks, Abilene, CalREN XD, TeraGrid (see Newman’s talk)  Synergy with Cisco See Yip’s talk

43. netlab.caltech.edu Why Caltech: experience  Hardware Cisco’s testbed Psaltis, Yip, Hajimiri, DeHon  Software Netbed management software  Operation Newman’s group  Testbed driven by networking research IST, Theory Program, FAST, optics, scientific computing, network coding, …

44. netlab.caltech.edu Team  Hardware Yip’s team: Doraiswami (Cisco) Psaltis (EE/CNS), DeHon (CS), Hajimiri (EE) Caltech Information Tech Services, CACR  Software Lepreau’s team Low’s team: Almsberger (CS), Jin (CS), Wei (CS), Hu (CS)  Operation Newman’s team: Bunn (Physics), Ravot (Physics/CERN), Suresh (CACR)  Testbed driven by networking research Caltech IST Institute

45. netlab.caltech.edu Global research network NL SURFnet GENEVA UK SuperJANET4 ABILEN E ESNET CALRE N It GARR-B GEANT NewYork Fr Renater STAR-TAP STARLIGHT Wave Triangle Newman (Caltech) WAN in Lab Caltech

46. netlab.caltech.edu Outline  Proposal summary Basic design, equipment, costs  Unique features  Alternatives Spectrum of tools Emulated delay  Community resource Demand Management software  Why Caltech Leverage on Abilene, HENP, CalREN, TeraGrid  Summary Reviewer concerns Review criteria

47. netlab.caltech.edu Reviewer concerns  Narrow focus on TCP/AQM A range of IST research at Caltech (Murray, Doyle)  Spanning theory, implementation, experiment, deployment  WAN in Lab a critical component External projects in HENP, Grid & Emulab communities  Alternatives not discussed Use spectrum of tools at different stages Each complementary but not replaceable DWDM gears more realistic and cheaper  How to manage and share WAN in Lab Part of Federated Emulab Both demand and excellent support for global sharing Experience in global collaboration, e.g. VRVS  How much hardware development needed Mostly off-the-shelf (Yip) Sample system & experience from Cisco Local expertise: Psaltis (Optics), Yip (Cisc), Hajimiri (high speed IC), DeHon (VLSI)

48. netlab.caltech.edu Review Criteria  Intellectual merit Theory, implementation, experiment, deployment  Must inform and influence each other intimately  Approach validated by pilot project Experimental facility tied to rich theory program  Broader impacts HENP’s global collaborations a model for future corporations & society  FAST protocols enabling technology Shared by & stimulate external research that need high speed large distance  HSTCP, Scalable TCP, TCP Westwood, AVQ, REM/PI, … Internet as simplest complex system

49. netlab.caltech.edu Review Criteria  Integration of research & education Excellent projects for undergraduates and graduates  During & after development Unique teaching platform for advanced networking, distributed systems, complex systems, optics course  Bruck, Chandy, Doyle, Hickey, Low, Psaltis  Diversity 33% women grad students in Netlab 50% women postdocs and grad students in Doyle’s group  Synergy among projects Bring together 4 CISE projects (1 ITR, 1 STI, 2 pending) Leverage for additional funding and industry collaborations

50. netlab.caltech.edu NSF Workshop Criteria  Tested driven by research agenda Rich and strong networking effort “A network that can break”  Multi-user experimental facility With a clear research focus and foreseeable impact  Federated testbed Leverage on Netbed’s management software  Integrated monitoring & measurement facility Fiber splitter passive monitors  Technology transfer Strong leadership in FAST user community (Newman)

51. netlab.caltech.edu Some potential projects  TCP: FAST, HSTCP(Floyd, ICIR), TCP Westwood(Gerla, UCLA), Scalable TCP(Kelly, Cambridge/CERN), XCP (Dina, MIT)  AQM: REM(Low), PI(Misra/Towsley), AVQ(Srikant, UIUC)  Protocol decomposition (Doyle, Low, Caltech)  Network self-management (Yemini, Columbia)  Content distribution (Bruck, Low, Caltech, Xu, Washington U)  Optical switching (Low, Psaltis, Caltech)  Network separation theory (Doyle, Low, Caltech Paganini, UCLA)  Real-time control over high performance networks (Dolye, Low, Murray, Caltech)  Simple Optics Smart Router (SOSR) (Yates, AT&T Research)  Optical protection, recovery (Yates, AT&T Research; Nirmalathas, Melbourne U)  Dynamic lightpath configuration & provisioning (Tucker, Melbourne U)  Active probing (Veitch, CUBIN)  Passive monitoring (Veitch, CUBIN)  Building & testing firewalls (Hoffman, U of Victoria)  High performance active network node (Turner, Washington)