1. A WAN-in-LAB for Protocol Development Netlab, Caltech Lachlan Andrew, George Lee, Steven Low(PI), John Doyle, Harvey Newman

2. Outline What and why is WAN-in-Lab? What and why is WAN-in-Lab? What can I do with WiL? What can I do with WiL? Why would I use WiL? Why would I use WiL? How do I use WiL? How do I use WiL? Future plans Future plans

3. What is WAN-in-Lab? “Wide Area” Network in a laboratory “Wide Area” Network in a laboratory Real fibre delaysReal fibre delays Carrier-class routers, switches, …Carrier-class routers, switches, …

4. Why -- Spectrum of tools cost abstraction mathssimulationemulationlive netwkWANinLab NS2 SSFNet QualNet JavaSim Mathis formula Optimization Control theory Nonlinear model Stocahstic model DummyNet EmuLab ModelNet WAIL UltraLight PlanetLab Abilene NLR LHCNet CENIC etc ? All scales are important– WAN-in-Lab fills a gap

5. What can I do with WAN-in-Lab?

6. Other groups’ interests Protocol development Protocol development FAST, delay-basedFAST, delay-based MaxNet, explicit signallingMaxNet, explicit signalling ADPM, single-bit explicit signallingADPM, single-bit explicit signalling Impact of small buffers (U. Pittsburgh) Impact of small buffers (U. Pittsburgh) Test automatic configuration of routers (MonALISA, Ultralight) Test automatic configuration of routers (MonALISA, Ultralight) Test distributed file-system (MojaveFS) Test distributed file-system (MojaveFS)

7. TCP Benchmarking Our current main direction Our current main direction Evaluating others’ protocols Evaluating others’ protocols Web interface Web interface Submit kernel patchSubmit kernel patch Standard tests automatically performedStandard tests automatically performed Results mailed backResults mailed back Explicit or implicit signalling protocols Explicit or implicit signalling protocols

8. Capabilities: Topology Four Cisco 7609 OC-48 routers Four Cisco 7609 OC-48 routers Line cards for four physical linksLine cards for four physical links More “virtual” links using IP routingMore “virtual” links using IP routing 18 GbE servers 18 GbE servers Two wired as software routers for AQMTwo wired as software routers for AQM 2 standalone 10GbE servers 2 standalone 10GbE servers Library of standard topologies Library of standard topologies Users can also create their ownUsers can also create their own

9. Physical topology

10. Capabilities: Delay 24 spools of 100km fibre, many loopbacks 24 spools of 100km fibre, many loopbacks Set delay by MEMS switching loops in/outSet delay by MEMS switching loops in/out 125ms uses IP loopback 125ms uses IP loopback 4-hop: pairs of hops have equal delay4-hop: pairs of hops have equal delay 2 Dummynets: long delay for cross-traffic 2 Dummynets: long delay for cross-traffic 125 ms, 1.8ms steps

11. Why use WAN-in-Lab?

12. Why use WiL? Complement other levels of abstraction, not replace them Complement other levels of abstraction, not replace them Different ways to use it: reasons for each Different ways to use it: reasons for each Standard platform for TCP benchmarking Standard platform for TCP benchmarking Easier to compare with others’ resultsEasier to compare with others’ results No need to write your own test suiteNo need to write your own test suite

13. Artifacts of software delays Packets sent on 1ms “ticks” Packets sent on 1ms “ticks” 1Gbps = 83,333 pk/s 1Gbps = 83,333 pk/s 1ms 83 packets

14. How can I use WAN-in-Lab?

15. Management structure

16. Accounts Mail wil at cs.caltech.edu Mail wil at cs.caltech.edu Sudo access to “network” commands Sudo access to “network” commands Ifconfig/…/Ifconfig/…/ Custom commands to set topologiesCustom commands to set topologies Login to routers if required Login to routers if required Separate accounts for “benchmark only” Separate accounts for “benchmark only”

17. Time sharing Coarse switching between projects Coarse switching between projects Servers rebooted, routers reconfiguredServers rebooted, routers reconfigured Switchover takes ~5 minutes Switchover takes ~5 minutes Book in advance Book in advance For longer bookings, book further in advanceFor longer bookings, book further in advance Also “ad hoc” bookings for individual hostsAlso “ad hoc” bookings for individual hosts Can log in while others have booked Can log in while others have booked

18. Future plans

19. Benchmarking infrastructure Benchmarking infrastructure Better control over capacities and buffers Better control over capacities and buffers Saturating links Saturating links Current servers 1Gbps, links 2.5GbpsCurrent servers 1Gbps, links 2.5Gbps Investigate cross-traffic generation Investigate cross-traffic generation Harpoon currently installedHarpoon currently installed Creative ways to obtain more topologies Creative ways to obtain more topologies Better monitoring Better monitoring Obtaining DAG cardsObtaining DAG cards

20. External connections Will link to Ultralight, 10Gbps Physics WAN Will link to Ultralight, 10Gbps Physics WAN Smooth migration testing -> deployment Smooth migration testing -> deployment Delay Delay longerlonger jitterjitter Cross traffic Cross traffic Monitor data routed through WiL Monitor data routed through WiL

21. Conclusion WAN-in-Lab fills the gap between emulation and live network experiments WAN-in-Lab fills the gap between emulation and live network experiments Seeks to be as realistic as possible Seeks to be as realistic as possible Long links, simple topologyLong links, simple topology Focus will be on TCP benchmarking Focus will be on TCP benchmarking We encourage people to use it We encourage people to use it<http://wil.cs.caltech.edu>

22. Case Study: MaxNet

23. Spare Slides

24. WAN in Lab WAN in Lab Capacity: 2.5 – 10 GbpsCapacity: 2.5 – 10 Gbps Delay: 0 – 120 ms round tripDelay: 0 – 120 ms round trip Breakable Breakable Won’t take down live networkWon’t take down live network Flexible, active debugging Flexible, active debugging Passive monitoring, AQMPassive monitoring, AQM Configurable & evolvable Configurable & evolvable Topology, rate, delays, routeTopology, rate, delays, route Modular design stays up to dateModular design stays up to date Integral part of R&A networks Integral part of R&A networks Transition from theory, implementation, demonstration, deploymentTransition from theory, implementation, demonstration, deployment Transition from lab to marketplaceTransition from lab to marketplace Global resource Global resource Part of global infrastructure UltraLight led by Harvey NewmanPart of global infrastructure UltraLight led by Harvey Newman Aim: Wind Tunnel of Networking

25. Equipment 4 Cisco 7609 routers with OC48 line cards 4 Cisco 7609 routers with OC48 line cards 6 Cisco ONS 15454 switches 6 Cisco ONS 15454 switches A few dozen high speed servers A few dozen high speed servers 1G switch to routers/servers 1G switch to routers/servers Calient switch for OC48 Calient switch for OC48 2,400 kilometres of fibre, optical amplifiers, dispersion compensation modules 2,400 kilometres of fibre, optical amplifiers, dispersion compensation modules 63ms aggregate RTT delay, in two hops 63ms aggregate RTT delay, in two hops 120ms using IP loopbacks120ms using IP loopbacks

26. Topology – up to 4 virtual hops

27. Configuration -- Delays Want maximum delay from limited fibre Want maximum delay from limited fibre Signals traverse fibre 16 timesSignals traverse fibre 16 times 4 WDM wavelengths 4 WDM wavelengths 4 OC48 (2.5G) MUXed onto OC192 (10G) 4 OC48 (2.5G) MUXed onto OC192 (10G) Lots of transponders Lots of transponders WDM amplifier joins 100km spools  200kmWDM amplifier joins 100km spools  200km

28. Configuration – delays OC48 slot Amp -------WDM Wavelength-------- Bidirectional 100km 16x200km

29. Configuration – delays Delay varied by adjusting the number of OC48 hops traversed Delay varied by adjusting the number of OC48 hops traversed Calient optical switch selects required hops Calient optical switch selects required hops Hop lengths 200km up to 1600km Hop lengths 200km up to 1600km Maximise granularity given limited switch portsMaximise granularity given limited switch ports Switch

30. Projects TCP benchmarking TCP benchmarking FAST FAST Delay-based congestion controlDelay-based congestion control MaxNet MaxNet Explicit signalling congestion controlExplicit signalling congestion control MojaveFS MojaveFS New distributed file systemNew distributed file system University of Pittsburgh University of Pittsburgh TCP with small buffersTCP with small buffers University of Melbourne University of Melbourne Single-bit congestion markingSingle-bit congestion marking

31. WAN-in-Lab testbed Dummynet and simulation introduce artifacts Dummynet and simulation introduce artifacts Also need to test on real equipment Also need to test on real equipment WAN with real delays, located in a single room WAN with real delays, located in a single room Connected to an external WAN (Ultralight)Connected to an external WAN (Ultralight) Open for the community to use for benchmarking Open for the community to use for benchmarking OC-48

32. WAN-in-Lab capabilities CurrentPlanned Two 2.5G bottlenecks Multiple 1G bottlenecks Six 2.5G bottlenecks Two “real” delays (Emulate cross traffic delay) Up to six “real” delays End-to-end RTT, drop Per-router delay, drop (movable DAG cards)

33. Configuration -- delays OC48 slot Amp -------WDM Wavelength-------- Bidirectional 100km

34. Using WAN-in-Lab Contact me – lachlan at caltech. Edu Contact me – lachlan at caltech. Edu Coarse timesharing Coarse timesharing Some users set up experiments while others run experimentsSome users set up experiments while others run experiments Software setup still being developed Software setup still being developed Your chance to influence our directions to tailor it to your needsYour chance to influence our directions to tailor it to your needs

35. Sample MaxNet results Achieves realistic delay at 1Gbit/s Achieves realistic delay at 1Gbit/s