Patrick Crowley and Jon Turner and John DeHart, Mart Haitjema Fred Kuhns, Jyoti Parwatikar, Ritun Patney, Charlie Wiseman, Mike Wilson, Ken Wong, Dave Zar Computer Science & Engineering Washington University Prototype Deployment of Internet Scale Overlay Hosting Status Report – 11/2009

Project Objectives Deploy five experimental overlay hosting platforms »located at Internet 2 PoPs »compatible with PlanetLab, moving to GENI control framework »performance characteristics suitable for service deployment integrated system architecture with multiple server blades shared NP-based server blades for fast-path packet processing Demonstrate multiple applications Chassis Switch 10x1 GbE CP External Switch net FPGA GPE NPE Line Card Control Processor General Purpose Processing Engines Network Processing Engine Line Card Chassis Switch External Switch

Target Internet 2 Deployment

Status First three nodes deployed and operational »installation went smoothly (mostly) »two hardware issues replaced KC linecard, still need to replace one GPE at SALT Demonstration this afternoon Ready for a few friendly users »still need to complete flow stats »version 1 of NPE software »not yet ready to add fastpath code options but happy to start working with interested groups Tutorial planned for next GEC »let me know if you’re interested

Current Deployment I2 Optical Connections ProtoGENI SPP SPPLC SALT KANS WASH drn06.arl. wustl.edu I2 Internet Service I2 Router

Photo Gallery (thanks Chris)

Hosting Platform Details Chassis Switch 10x1 GbE CP External Switch net FPGA GPE NPE Line Card PLOS VM... General Purpose Processing Engine filter... filter Line Card lookupparseheader format... queues... Network Processing Engine

Application Framework Fastpath/slowpath »fastpath mapped onto NPE »slice manager in vServer on GPE Configurable elements »code option – determines how packets processed by parse, header format »logical interfaces may be public or tunnel guaranteed bandwidth »TCAM filters »Queues length, bandwidth Slice manager can configure fastpath using provided library »or manually, using command line interface ParseLookup Filters Control Interface Header Format Queue Manager Fast Path... output interfaces input interfaces Slice Manager (in vServer) GPE Remote Login Interface exception packets & in-band control

Demonstrations this Afternoon IPv4 network using NP-based fastpaths »traffic generated from PlanetLab nodes »demonstrates dynamic changes to installed filters script in GPE slice issues control messages to modify filters »handling of exception cases by GPE slice »use of traffic monitoring Forest overlay network – GPE-only implementation »multiple dynamic multicast groups over per-session, tree-structured communication channels »traffic generated from PlanetLab nodes »traffic monitoring using statistics data generated by GPE slice

IPv4 Demo Planetlab traffic generators Line Card SALT-R2 NPECPGPE WASH-R3 NPECPGPE Line Card KANS-R1 NPECPGPE Line Card experiment control SPPLC CTL FPCTL FP CTL FP

Forest Overlay Network Focus on real-time distributed applications »large online virtual worlds »distributed cyber-physical systems Large distributed sessions »endpoints issue periodic status reports »and subscribe to dynamically changing sets of reports »requires real-time, non-stop data delivery, even as communication pattern changes Per-session communication channels (comtrees) »unicast data delivery with route learning »dynamic multicast subscriptions using multicast core overlay router client server access connection comtree

Forest Demo SALT-R2 NPECPGPE Line Card WASH-R3 NPECPGPE Line Card KANS-R1 NPECPGPE Line Card fRtr R2 R3 R1 comtree2 R1 R3 R2 comtree1 multicast sender multicast subscribers Planetlab nodes SPPLC experiment control

Developing a new SPP Application Create reference model as foundation »work out functional issues »develop and debug control code »build with separate control and datapath processes »implement in Open Network Lab or Emulab Port reference model to SPPs using GPEs only »initially, just reproduce reference model functionality »extend to use multiple external interfaces, reserved capacity Design new NPE code option »we’ll work with you and do NPE code development »need working GPE version and fastpath design spec

Working with SPPs Define new slice using SPPLC »much as with PlanetLab Login to GPE slice to install application code Reserve resources needed for your experiment »includes interface bandwidth on external ports and NPE fastpath resources To run experiment during reserved period »configure reserved resources as needed »setup logical endpoints (aka meta-interfaces) »configure fastpath (filters, queues) »setup real-time monitoring »run application and start traffic generators

Web Site and SPPLC drn06.arl.wustl.edu /Internet_Scale_Overlay_Hosting

Steps in Setting Running Experiment Reserve resources in advance At start of session »configure logical interfaces (aka meta-interfaces) associate with physical interfaces and port numbers assign bandwidth »setup fastpath assign queues to logical interfaces, set queue parameters install fastpath filters »setup traffic monitoring At end of session »release resources in use »cancel any remaining reservation time

Command Line Tools resrv – reserve resources in advance »make_resrv,update_resrv,cancel_resrv,get_resrv,... client – perform variety of operations »get_ifaces,get_ifattrs »get_queue_params, set_queue_params, bind_queue »alloc_endpoint, alloc_tunnel »write_fltr, update_result, get_fltr_bykey »read_stats, create_periodic, get_queue_len create-fp – instantiate fastpath fltr – manipulate fastpath filters for IPv4 code opt sliced – monitor traffic and display remotely

Reserving Resources in Advance resrv --cmd make_resrv --xfile resFile reserved time period aggregate fastpath bandwidth # of TCAM filters, queues, packet buffers, traffic counters bandwidth on external interfaces, specified by IP address external interface bw for GPE slice

Configuring Logical Interfaces For use by GPE slice »client --cmd alloc_endpoint --bw ipaddr proto 6 --port 3551 For use by fastpath »client --cmd alloc_udp_tunnel --fpid 0 --bw ipaddr port setup externally visible port in Kb/s address of interface 6 for TCP, 17 for UDP port number identifier of fastpath within this slice

Configuring Fastpath Binding queues to logical interfaces »client --cmd bind_queue --fpid 0 --miid 1 --qid_list_type 0 --qid_list 4 --qid_list 6 Setting queue parameters »client --cmd set_queue_params --fpid 0 --qid 3 --threshold bw 8000 Setting up filters (for IPv4 fastpaths) »fltr --cmd write_fltr --fpid 0 --fid 10 --key_type 0 --key_rxmi 2 --key_daddr xx --mask_daddr xx --key_dport xx –-txdaddr xx --txdport xx --qid 5 meta-interface id assigned queues

Displaying Real-Time Data Fastpath maintains traffic counters and queue lengths To display traffic data »configure an external TCP port »run sliced within your vServer, using configured port /sliced --ip port 3552 & »on remote machine, run SPPmon.jar use provided GUI to setup monitoring displays SPPmon configures sliced, which polls the NPE running the fastpath SCD-N reads and returns counter values to sliced can also display data written to file within your vServer NPECPGPE Line Card SCD-N sliced SPPmon

Summary SPPs ready for use »start with a few friendly users, but will open up soon »tutorial at next GEC To do list »complete flow stats subsystem »version 2 of NPE software higher performance and multicast support »resource allocation mechanisms enable coordinated reservation across all nodes »work with user groups on new code options