A WAN-in-LAB for Protocol Development Netlab, Caltech Lachlan Andrew, George Lee, Steven Low(PI), John Doyle, Harvey Newman

Outline What and why is WAN-in-Lab? What and why is WAN-in-Lab? What can I do with WiL? What can I do with WiL? Why would I use WiL? Why would I use WiL? How do I use WiL? How do I use WiL? Future plans Future plans

What is WAN-in-Lab? “Wide Area” Network in a laboratory “Wide Area” Network in a laboratory Real fibre delaysReal fibre delays Carrier-class routers, switches, …Carrier-class routers, switches, …

Why -- Spectrum of tools cost abstraction mathssimulationemulationlive netwkWANinLab NS2 SSFNet QualNet JavaSim Mathis formula Optimization Control theory Nonlinear model Stocahstic model DummyNet EmuLab ModelNet WAIL UltraLight PlanetLab Abilene NLR LHCNet CENIC etc ? All scales are important– WAN-in-Lab fills a gap

What can I do with WAN-in-Lab?

Other groups’ interests Protocol development Protocol development FAST, delay-basedFAST, delay-based MaxNet, explicit signallingMaxNet, explicit signalling ADPM, single-bit explicit signallingADPM, single-bit explicit signalling Impact of small buffers (U. Pittsburgh) Impact of small buffers (U. Pittsburgh) Test automatic configuration of routers (MonALISA, Ultralight) Test automatic configuration of routers (MonALISA, Ultralight) Test distributed file-system (MojaveFS) Test distributed file-system (MojaveFS)

TCP Benchmarking Our current main direction Our current main direction Evaluating others’ protocols Evaluating others’ protocols Web interface Web interface Submit kernel patchSubmit kernel patch Standard tests automatically performedStandard tests automatically performed Results mailed backResults mailed back Explicit or implicit signalling protocols Explicit or implicit signalling protocols

Capabilities: Topology Four Cisco 7609 OC-48 routers Four Cisco 7609 OC-48 routers Line cards for four physical linksLine cards for four physical links More “virtual” links using IP routingMore “virtual” links using IP routing 18 GbE servers 18 GbE servers Two wired as software routers for AQMTwo wired as software routers for AQM 2 standalone 10GbE servers 2 standalone 10GbE servers Library of standard topologies Library of standard topologies Users can also create their ownUsers can also create their own

Physical topology

Capabilities: Delay 24 spools of 100km fibre, many loopbacks 24 spools of 100km fibre, many loopbacks Set delay by MEMS switching loops in/outSet delay by MEMS switching loops in/out 125ms uses IP loopback 125ms uses IP loopback 4-hop: pairs of hops have equal delay4-hop: pairs of hops have equal delay 2 Dummynets: long delay for cross-traffic 2 Dummynets: long delay for cross-traffic 125 ms, 1.8ms steps

Why use WAN-in-Lab?

Why use WiL? Complement other levels of abstraction, not replace them Complement other levels of abstraction, not replace them Different ways to use it: reasons for each Different ways to use it: reasons for each Standard platform for TCP benchmarking Standard platform for TCP benchmarking Easier to compare with others’ resultsEasier to compare with others’ results No need to write your own test suiteNo need to write your own test suite

Artifacts of software delays Packets sent on 1ms “ticks” Packets sent on 1ms “ticks” 1Gbps = 83,333 pk/s 1Gbps = 83,333 pk/s 1ms 83 packets

How can I use WAN-in-Lab?

Management structure

Accounts Mail wil at cs.caltech.edu Mail wil at cs.caltech.edu Sudo access to “network” commands Sudo access to “network” commands Ifconfig/…/Ifconfig/…/ Custom commands to set topologiesCustom commands to set topologies Login to routers if required Login to routers if required Separate accounts for “benchmark only” Separate accounts for “benchmark only”

Time sharing Coarse switching between projects Coarse switching between projects Servers rebooted, routers reconfiguredServers rebooted, routers reconfigured Switchover takes ~5 minutes Switchover takes ~5 minutes Book in advance Book in advance For longer bookings, book further in advanceFor longer bookings, book further in advance Also “ad hoc” bookings for individual hostsAlso “ad hoc” bookings for individual hosts Can log in while others have booked Can log in while others have booked

Future plans

Benchmarking infrastructure Benchmarking infrastructure Better control over capacities and buffers Better control over capacities and buffers Saturating links Saturating links Current servers 1Gbps, links 2.5GbpsCurrent servers 1Gbps, links 2.5Gbps Investigate cross-traffic generation Investigate cross-traffic generation Harpoon currently installedHarpoon currently installed Creative ways to obtain more topologies Creative ways to obtain more topologies Better monitoring Better monitoring Obtaining DAG cardsObtaining DAG cards

External connections Will link to Ultralight, 10Gbps Physics WAN Will link to Ultralight, 10Gbps Physics WAN Smooth migration testing -> deployment Smooth migration testing -> deployment Delay Delay longerlonger jitterjitter Cross traffic Cross traffic Monitor data routed through WiL Monitor data routed through WiL

Conclusion WAN-in-Lab fills the gap between emulation and live network experiments WAN-in-Lab fills the gap between emulation and live network experiments Seeks to be as realistic as possible Seeks to be as realistic as possible Long links, simple topologyLong links, simple topology Focus will be on TCP benchmarking Focus will be on TCP benchmarking We encourage people to use it We encourage people to use it<

Case Study: MaxNet

Spare Slides

WAN in Lab WAN in Lab Capacity: 2.5 – 10 GbpsCapacity: 2.5 – 10 Gbps Delay: 0 – 120 ms round tripDelay: 0 – 120 ms round trip Breakable Breakable Won’t take down live networkWon’t take down live network Flexible, active debugging Flexible, active debugging Passive monitoring, AQMPassive monitoring, AQM Configurable & evolvable Configurable & evolvable Topology, rate, delays, routeTopology, rate, delays, route Modular design stays up to dateModular design stays up to date Integral part of R&A networks Integral part of R&A networks Transition from theory, implementation, demonstration, deploymentTransition from theory, implementation, demonstration, deployment Transition from lab to marketplaceTransition from lab to marketplace Global resource Global resource Part of global infrastructure UltraLight led by Harvey NewmanPart of global infrastructure UltraLight led by Harvey Newman Aim: Wind Tunnel of Networking

Equipment 4 Cisco 7609 routers with OC48 line cards 4 Cisco 7609 routers with OC48 line cards 6 Cisco ONS switches 6 Cisco ONS switches A few dozen high speed servers A few dozen high speed servers 1G switch to routers/servers 1G switch to routers/servers Calient switch for OC48 Calient switch for OC48 2,400 kilometres of fibre, optical amplifiers, dispersion compensation modules 2,400 kilometres of fibre, optical amplifiers, dispersion compensation modules 63ms aggregate RTT delay, in two hops 63ms aggregate RTT delay, in two hops 120ms using IP loopbacks120ms using IP loopbacks

Topology – up to 4 virtual hops

Configuration -- Delays Want maximum delay from limited fibre Want maximum delay from limited fibre Signals traverse fibre 16 timesSignals traverse fibre 16 times 4 WDM wavelengths 4 WDM wavelengths 4 OC48 (2.5G) MUXed onto OC192 (10G) 4 OC48 (2.5G) MUXed onto OC192 (10G) Lots of transponders Lots of transponders WDM amplifier joins 100km spools  200kmWDM amplifier joins 100km spools  200km

Configuration – delays OC48 slot Amp WDM Wavelength Bidirectional 100km 16x200km

Configuration – delays Delay varied by adjusting the number of OC48 hops traversed Delay varied by adjusting the number of OC48 hops traversed Calient optical switch selects required hops Calient optical switch selects required hops Hop lengths 200km up to 1600km Hop lengths 200km up to 1600km Maximise granularity given limited switch portsMaximise granularity given limited switch ports Switch

Projects TCP benchmarking TCP benchmarking FAST FAST Delay-based congestion controlDelay-based congestion control MaxNet MaxNet Explicit signalling congestion controlExplicit signalling congestion control MojaveFS MojaveFS New distributed file systemNew distributed file system University of Pittsburgh University of Pittsburgh TCP with small buffersTCP with small buffers University of Melbourne University of Melbourne Single-bit congestion markingSingle-bit congestion marking

WAN-in-Lab testbed Dummynet and simulation introduce artifacts Dummynet and simulation introduce artifacts Also need to test on real equipment Also need to test on real equipment WAN with real delays, located in a single room WAN with real delays, located in a single room Connected to an external WAN (Ultralight)Connected to an external WAN (Ultralight) Open for the community to use for benchmarking Open for the community to use for benchmarking OC-48

WAN-in-Lab capabilities CurrentPlanned Two 2.5G bottlenecks Multiple 1G bottlenecks Six 2.5G bottlenecks Two “real” delays (Emulate cross traffic delay) Up to six “real” delays End-to-end RTT, drop Per-router delay, drop (movable DAG cards)

Configuration -- delays OC48 slot Amp WDM Wavelength Bidirectional 100km

Using WAN-in-Lab Contact me – lachlan at caltech. Edu Contact me – lachlan at caltech. Edu Coarse timesharing Coarse timesharing Some users set up experiments while others run experimentsSome users set up experiments while others run experiments Software setup still being developed Software setup still being developed Your chance to influence our directions to tailor it to your needsYour chance to influence our directions to tailor it to your needs

Sample MaxNet results Achieves realistic delay at 1Gbit/s Achieves realistic delay at 1Gbit/s