winter 2008 Evaluation Tools1 Brief Overview of Networking Evaluation Methods and Tools

winter 2008 Evaluation Tools2 Outline Tools –Simulation ns2 –Cluster based network emulation Emulab –Live Distributed Testbed PlanetLab Questions –What is the right tool(s) for my research? –What does it take to get up and running?

winter 2008 Evaluation Tools3 Outline For each tool/testbed –How does it work? –A “hello world” in ___ –What kind of research is it good for? Tradeoffs between tools

winter 2008 Evaluation Tools4 Simulation (ns2)

winter 2008 Evaluation Tools5 What is ns? Network simulator a discrete event simulator focused on modeling network protocols –wired, wireless, satellite –TCP, UDP, multicast, unicast –Web, telnet, ftp –Ad hoc routing; sensor networks –Infrastructure: stats, tracing, error models etc.

winter 2008 Evaluation Tools6 ns --- what is it good for? Evaluate performance of existing network protocols. Prototyping and evaluation of new protocols. Large-scale simulations not possible in real experiments. Used to:

winter 2008 Evaluation Tools7 ns Event-driven simulator –Model world as events –Simulator has list of events –Process: take next one, run it, until done –Each event happens in instant of virtual time, but takes arbitrary real time Single thread of control Packet level How does it work:

winter 2008 Evaluation Tools8 Ns models Traffic/applications –CBR, FTP, telnet, web Routing/Queuing –Drop-tail, FQ, SFQ, RED, DRR –Wired routing, adhoc routing etc Transport –TCP (variants), UDP, multicast (SRM)

winter 2008 Evaluation Tools9 ns - software structure Object oriented (C++, OTcl) – code reuse Scalability + Extensibility –Control/”data” separation –Split C++/OTcl object C++ for packet-processing (fast to run) OTcl for control - (fast to write) –Simulation setup and configuration

winter 2008 Evaluation Tools10 otcl and C++: The Duality OTcl C++ Pure OTcl objects Pure C++ objects C++/OTcl split objects ns Your ns-script

winter 2008 Evaluation Tools11 Outline Overview Tcl, OTcl basics ns basicsns basics Extending ns ns internals

winter 2008 Evaluation Tools12 Basic structure of ns-scripts Creating the event scheduler [Tracing] Creating network topology Creating Transport Layer - Agents Creating Applications - Applications Events!

winter 2008 Evaluation Tools13 Creating Event Scheduler Create scheduler –set ns [new Simulator] Schedule event –$ns at – : any legitimate ns/tcl commands Start scheduler –$ns run

winter 2008 Evaluation Tools14 “Hello World” in ns simple.tcl set ns [new Simulator] $ns at 1 “puts \“Hello World!\”” $ns at 1.5 “exit” $ns run ns simple.tcl Hello World!

winter 2008 Evaluation Tools15 Creating Network Nodes –set n0 [$ns node] –set n1 [$ns node] Links & Queuing –$ns duplex-link $n0 $n1 –Queue type: DropTail, RED, CBQ, FQ, SFQ, DRR

winter 2008 Evaluation Tools16 Routing + traffic Unicast –$ns rtproto – : Static, Session, DV Multicast support also. Traffic –Simple two layers: transport and application. –Transport: TCP, UDP etc. –Applications: web, ftp, telnet etc.

winter 2008 Evaluation Tools17 The transport layer: UDP UDP –set udp [new Agent/UDP] –set null [new Agent/NULL] –$ns attach-agent $n0 $udp –$ns attach-agent $n1 $null –$ns connect $udp $null

winter 2008 Evaluation Tools18 The transport layer: TCP TCP –set tcp [new Agent/TCP] –set tcpsink [new Agent/TCPSink] –$ns attach-agent $n0 $tcp –$ns attach-agent $n1 $tcpsink –$ns connect $tcp $tcpsink

winter 2008 Evaluation Tools19 Creating Traffic: On Top of TCP FTP –set ftp [new Application/FTP] –$ftp attach-agent $tcp –$ns at “$ftp start” Telnet –set telnet [new Application/Telnet] –$telnet attach-agent $tcp

winter 2008 Evaluation Tools20 Creating Traffic: On Top of UDP CBR –set src [new Application/Traffic/CBR] Exponential or Pareto on-off –set src [new Application/Traffic/Exponential] –set src [new Application/Traffic/Pareto] Trace driven traffic –Inter-packet time and packet-size

winter 2008 Evaluation Tools21 Attaching a traffic source set cbr [new Application/Traffic/CBR] $cbr attach-agent $udp $ns at “$cbr start”

winter 2008 Evaluation Tools22 Tracing Trace packets on all links: –set f[open out.tr w] –$ns trace-all $f –$ns flush-trace –close $f cbr cbr r cbr Is tracing all links always the best thing to do?

winter 2008 Evaluation Tools23 More Tracing Tracing specific links –$ns trace-queue $n0 $n1 $f Tracing variables –set cwnd_chan_ [open all.cwnd w] –$tcp trace cwnd_ –$tcp attach $cwnd_chan_

winter 2008 Evaluation Tools24 Controlling object parameters Almost all ns objects have parameters –ex. Application/Traffic/Exponential has rate and packetSize –set parameters in OTcl set etraf [new Application/Traffic/Exponential] $etraf set rate_ 1Mb $etraf set packetSize_ 1024

winter 2008 Evaluation Tools25 Putting it all together set ns [new Simulator] set n0 [$ns node] set n1 [$ns node] $ns duplex-link $n0 $n1 1.5Mb 10ms DropTail Creating Topology set tcp [$ns create-connection TCP $n0 TCPSink $n1 0] Creating Transport layer set ftp [new Application/FTP] $ftp attach-agent $tcp Creating Applications $ns trace-queue $n0 $n1 $f Schedule Events $ns at 0.2 "$ftp start“ $ns at 1.2 ”exit“ $ns run n0n1 1.5Mb 10ms FTP/TCP

winter 2008 Evaluation Tools26 Example: XCP vs CSFQ Compare utilization of links Compare throughput of flows

winter 2008 Evaluation Tools27 nam – the network animator n0n1 set nf [open out.nam w] $ns namtrace-all $nf exec nam out.nam & …

winter 2008 Evaluation Tools28 ns “components” ns,ns, the simulator itself namnam, the Network AniMator –Visualize ns output –GUI input simple ns scenarios Pre-processing: –Traffic and topology generators Post-processing: –Simple trace analysis, often in Awk, Perl, or Tcl

winter 2008 Evaluation Tools29 Network Dynamics: Link failures $ns rtmodel-at $n0 $n1 $ns rtmodel Trace $n0 $n1 $ns rtmodel $n0 $n1 : Deterministic, Exponential

winter 2008 Evaluation Tools30 Issues in Simulations Suppose you want to study the way TCP sources share a bottleneck link… Agent/FTP Which topology? When to start sources? What else affects results? Which traffic sources? Background Traffic ?

winter 2008 Evaluation Tools31 Cluster Based Emulation (Emulab) 368 nodes 3 big Ciscos Slides based on SOSP poster by Jay Lepreau et. al. University of Utah

winter 2008 Evaluation Tools32 Why? “We evaluated our system on five nodes.” -job talk from university with 300-node cluster “We evaluated our Web proxy design with 10 clients on 100Mbit ethernet.” “Simulation results indicate...” “Memory and CPU demands on the individual nodes were not measured, but we believe will be modest.” “The authors ignore interrupt handling overhead in their evaluation, which likely dominates all other costs.” “You have to know the right people to use the cluster.” “The cluster is hard to use.” “ runs FreeBSD 2.2.x.” “February’s schedule for is…” “ is tunneled through the Internet”

winter 2008 Evaluation Tools33 What? An instrument for experimental CS research A completely configurable “Internet emulator” in a room –At its core, it’s bare hardware, with… –… complete remote access and control But, also simple to use –Lots of fast tools for common case Automatic topology, node and link configuration Automatic traffic generation –Universally available Universities, research labs, companies –Zero-penalty for remote research

winter 2008 Evaluation Tools34 Key Design Aspects Allow experimenter complete control –Configurable network properties link bandwidth, Latency, and loss rates via transparently interposed “traffic shaping” nodes that provide WAN emulation –Configurable OS image Linux, FreeBSD, Windows Virtualization –of all experimenter-visible resources –node names, network interface names, network addrs e.g., node-0.esmexp1.esm.emulab.net

winter 2008 Evaluation Tools35 Emulab Architecture Programmable “Patch Panel” PC Web/DB/SNMP Switch Mgmt Users Internet Control Switch/Router Serial Sharks PowerCntl

winter 2008 Evaluation Tools36 Experiment Creation Process

winter 2008 Evaluation Tools37 Using Emulab Submit ns script via web form –Specify number of nodes –Specify link properties Relax while emulab … –Generates config from script & stores in DB –Maps specified virtual topology to physical nodes –Provides user accounts for node access –Assigns IP addresses and host names –Configures VLANs –Loads disks, reboots nodes, configures OSs Two ways to run experiments –Interactive Works only if there are enough free nodes right now –Batch Runs experiment when enough free nodes available

winter 2008 Evaluation Tools38 Using Emulab Time sharing model for nodes –If you check out 50 nodes, they are exclusively yours until you release them (or forced to release them!)

winter 2008 Evaluation Tools39 Using Emulab set ns [new Simulator] source tb_compat.tcl ##setup the core nodes for {set i 0} {$i <= 4} {incr i} { set node($i) [$ns node] tb-set-node-os $node($i) FC4-STD } ## setup the core links for {set i 0} {$i < 4} {incr i} { set j [expr $i + 1] $ns duplex-link $node($i) $node($j) 100Mb 0ms DropTail } $ns duplex-link $node(4) $node(0) 100Mb 0ms DropTail ## setup the edge nodes and links for {set i 5} {$i <= 10} {incr i} { set node($i) [$ns node] tb-set-node-os $node($i) FC4-STD set j [expr $i % 5] set link($i) [$ns duplex-link $node($i) $node($j) 1000Kb 0ms DropTail] tb-set-link-simplex-params $link($i) $node($i) 0ms 500Kb 0.0 } # Go! $ns rtproto Static $ns run D D D D D D

winter 2008 Evaluation Tools40 What is Emulab Good For? (creators words) Simulation –Fast prototyping, easy to use, but less realistic Small static testbeds –Real hardware and software, but hard to configure and maintain, and lacks scale Live networks –Realistic, but hard to control, measure, or reproduce results Emulab complements and also helps validate these environments

winter 2008 Evaluation Tools41 What is Emulab Good For? Studying routing protocols –Difficult to create richly connected nodes –Experiments that need access to the network core Studying peer-to-peer protocols –But, lacks scale Scaling experiments with “raw” machines with no link emulation Good controlled environment for testing before going to PlanetLab

winter 2008 Evaluation Tools42 Live Distributed Testbed (PlanetLab) 644 nodes at 304 sites

winter 2008 Evaluation Tools43 Resource Sharing Statistical sharing of resources –Each application has a “slice” of the overlay resources Bandwidth, CPU, memory –No guarantees on bandwidth, CPU or memory fairness –Slices guaranteed 1Mbps of bandwidth, rest shared by all slices

winter 2008 Evaluation Tools44 Resource Sharing Emulab PlanetLab

winter 2008 Evaluation Tools45 Slices

winter 2008 Evaluation Tools46 Slices

winter 2008 Evaluation Tools47 Slices

winter 2008 Evaluation Tools48 Per-Node View Virtual Machine Monitor (VMM) Node Mgr Local Admin VM 1 VM 2 VM n …

winter 2008 Evaluation Tools49 PlanetLab Virtualization: VServers Kernel patch to mainstream OS (Linux) Gives appearance of separate kernel for each virtual machine –Root privileges restricted to activities that do not affect other vservers Some modification: resource control (e.g., File handles, port numbers) and protection facilities added

winter 2008 Evaluation Tools50 How to Use PlanetLab Get a slice –Request local PI –Setup SSH keys (no passwords on PlanetLab) Add nodes to your slice through the web interface Write (or borrow) scripts to deploy and run your system on PlanetLab and gather data from PlanetLab –Pssh/pscp (parallel versions of ssh/scp) Remember, you’re running on a live network! –Test your code before you deploy

winter 2008 Evaluation Tools51 Not Just a Testing Infrastructure PlanetLab started by Intel … “…to support seamless migration of an application from an early prototype, through multiple design iterations, to a popular service that continues to evolve.”

winter 2008 Evaluation Tools52 Testbed Comparison ns2EmulabPlanetLab Event-driven simulated network Scale of 100s depending of configuration Simulated topology Controlled environment Emulated network Scale of 10s most of the time Real controllable topology Controlled environment Real overlay network Scale of 100s (goal is over 1000) Real fixed topology Live network, uncontrolled environment