1. June 2000 MSR Design 1 Washington WASHINGTON UNIVERSITY IN ST LOUIS The Washington University Active Network Software Framework John DeHart Washington University jdd@arl.wustl.edu http://www.arl.wustl.edu/~jdd http://www.arl.wustl.edu/arl/projects/msr/ann_sw.pdf

2. June 2000 MSR Design 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS Overview Some thoughts on active networking Software Architecture Router Plugins Active Router Plugins Multi-Port Router Architecture

3. June 2000 MSR Design 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS Who Writes the Active Code End User - “classic” active networking –“Its my data, here’s how I want to manipulate it …” Application/Service Developer –“Its my application, I understand the format of it best, here’s some capabilities for manipulating it …” –e.g. Vendor X provides Active congestion control for its MPEG video conferencing system. Network Provider/Developer –“Its my router/network, I’ll give you these capabilities for manipulating your data …” –e.g. Alternate Routing and Topology based decisions.

4. June 2000 MSR Design 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS Greater Flexibility Greater Performance Performance: Executing the Active Code User Space Execution Environments –ANTS from MIT –CANES from Georgia Tech –PLAN from UPenn –... Kernel Space Execution Environments –Scout from Arizona –ANN (Active Network Node) from WU –... Hardware (e.g. FPGAs) –P4 from UPenn –FPX from WU

5. June 2000 MSR Design 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS Requirements for WU AN Platform Code should be deployed automatically and on- demand to router –this should happen rarely! Simple security model we understand –secure plugins and code servers –do not verify each packet’s secureness Fast enough for data-path applications –155 Mbits/s in software –1.2 Gbits/s with hardware support (FPGA)

6. June 2000 MSR Design 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS Software Architecture (NetBSD) Kernel Plugins Plugin Ctl. Unit Active Plugin Loader Plugin Requestor Plugin DB Controller Policy Controller Security Gateway ANN Manager Key DB Plugin DB Policy Rules Code Server Active Code DB... anetd Java VM ANTS NOSI Packet Classification and Routing IPv4/6 Header Processing IP Packets Driver Packet Scheduler Driver Packet Scheduler TCP/UDP PE Kernel Space RSVP/SSP Routing PE User Space, Switch Controller, & Remote Servers SAPF Packet Selector/ Dispatcher... Active Function Dispatcher... Resource Controller

7. June 2000 MSR Design 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS Software Architecture (NetBSD) Kernel Plugins Plugin Ctl. Unit Packet Classification and Routing IPv4/6 Header Processing Driver Packet Scheduler TCP/UDP SAPF Packets Selector/ Dispatcher... Active Function Dispatcher... Resource Controller IP Packets Std. proc. for “plain” IP packets. –classification & routing, header processing, output queueing Active packets move through configured kernel plugins. »active function dispatcher passes packets to instances of plugin objects »instantiates objects or triggers download of plugin class, as needed streamlined processing of SAPF packets using pre-established state

8. June 2000 MSR Design 8 Washington WASHINGTON UNIVERSITY IN ST LOUIS Router Plugins (Crossbow: where we started) Modular kernel software architecture developed at Washington U. and ETH Zurich Allows for –dynamically adding code modules called Plugins at run- time into the IP forwarding loop –binding plugins to individual IP flows Provides fast packet classification

9. June 2000 MSR Design 9 Washington WASHINGTON UNIVERSITY IN ST LOUIS Router Plugins (con’t) Packet classification engine greatly simplifies implementation of plugins Filters to bind plugins to flows Gate 3 IP Security DES IP Security MD5 Filter

10. June 2000 MSR Design 10 Washington WASHINGTON UNIVERSITY IN ST LOUIS Plugin Identifiers for Crossbow Code identified by protocol numbers, port numbers or hardware Plugins applied to flows selected by filters.

11. June 2000 MSR Design 11 Washington WASHINGTON UNIVERSITY IN ST LOUIS Plugin Identifiers Abstract view: Datagram = Sequence of Code identifiers (pi n ) and parameters (P n ) F Implement code as plugins! Call identifiers “Plugin Identifiers” Plugin Identifier (PI) determines which plugin has to be called pi 2 pi 3...

12. June 2000 MSR Design 12 Washington WASHINGTON UNIVERSITY IN ST LOUIS Active Packet 0x8002001 (Plugin ID) Plugin ID in packet ==> NO FILTERS!!

13. June 2000 MSR Design 13 Washington WASHINGTON UNIVERSITY IN ST LOUIS Plugin Invocation based on Plugin IDs Active Function Dispatcher (AFD) Plugin Control Unit IP Security Video Congestion Control IP which plugin? return plugin to gate call plugin lookup plugin identifier which plugin? return plugin

14. June 2000 MSR Design 14 Washington WASHINGTON UNIVERSITY IN ST LOUIS Plugin Invocation based on Plugin IDs (con’t) Active Function Dispatcher (AFD) Plugin Control Unit IP Security Video Congestion Control IP User Space Kernel Plugin Management IP which plugin? plugin not present request plugin check policies request plugin from code server

15. June 2000 MSR Design 15 Washington WASHINGTON UNIVERSITY IN ST LOUIS Downloading a Plugin plugin request Our Router MPEG Video CC Code Server Plugin Database Internet check policies locate plugin send plugin Video CC

16. June 2000 MSR Design 16 Washington WASHINGTON UNIVERSITY IN ST LOUIS Plugin Invocation based on Plugin IDs Active Function Dispatcher (AFD) Plugin Control Unit IP Security Video Congestion Control User Space Kernel Plugin Management IP check digital signatures load plugin plugin present return plugin call plugin plugin from code server Video CC

17. June 2000 MSR Design 17 Washington WASHINGTON UNIVERSITY IN ST LOUIS ipintrq: ipintr ip_forwardip_output ip6_forwardip6_output atmc_input apic_israpic_intr ip6_input atmc_output input packetsinterruptsoutput packets T R A N S tcp/udp_inputtcp/udp_output socket layer P O R T apic_devoutput IP Packet Handling in ANN Kernel (APIC) Where in this loop is ANN processing done? Any guesses? Where is the Packet Scheduler? Any guesses? LINKLINK DATADATA NETWORKNETWORK UPPERUPPER LOWERLOWER apic_start packet_scheduler PS_Q Where is the fast forward path? Any guesses? Where are the gates? sgate dgate

18. June 2000 MSR Design 18 Washington WASHINGTON UNIVERSITY IN ST LOUIS IP Packet Handling in ANN Kernel (APIC) Device Driver -- Device Dependent –apic_intr() read INTR_ACK: 0 => not for us, return apic_isr() –apic_isr() read NOTIFY_LIST, next rcv channel needing attention –Process descriptors for RCV channel »swap words in each received mbuf (APIC BUG) »link mbufs until we find end of frame »verify CRC (just check flag set by APIC) »atmc_input(packet) »repeat –repeat ipintrq: ipintr ip_forwardip_output atmc_input apic_israpic_intr atmc_output apic_devoutput

19. June 2000 MSR Design 19 Washington WASHINGTON UNIVERSITY IN ST LOUIS IP Packet Handling in ANN Kernel (APIC) Device Driver - Dev. Independent atmc_input(): –LLC/SNAP processing (if needed) extract type (IP/RATM/…) –AAL5 processing extract length strip trailer –if gateway and !crossbow ipflow_fastforward() –aiu_getafix(packet) aiu_getafix stores a ptr to FTE in packet –aiu_fastforward(packet) currently commented out. WHY? –Enqueue in IP Queue ipintrq: ipintr ip_forwardip_output atmc_input apic_israpic_intr atmc_output apic_devoutput

20. June 2000 MSR Design 20 Washington WASHINGTON UNIVERSITY IN ST LOUIS IP Packet Handling in ANN Kernel (APIC) IP Input -- ipintr(): –Get next packet from IP Queue –Do some basic checks, header, length, checksum… –process IP options –aiu_dgate(&m,2) Router Plugins Dynamic Gate #2 –if packet is not for us and we can forward it: forward packet ip_forward() -- upcoming slide... return –aiu_dgate(&m,4) Router Plugins Dynamic Gate #4 –protocol specific input routine e.g. tcp_input() ipintrq: ipintr ip_forwardip_output atmc_input apic_israpic_intr atmc_output apic_devoutput

21. June 2000 MSR Design 21 Washington WASHINGTON UNIVERSITY IN ST LOUIS IP Packet Handling in ANN Kernel (APIC) IP Forwarding -- ip_forward(): –decide if we need to send any redirects to sender –ip_output() IP Output -- ip_output() –aiu_dgate(&m,3) Router Plugins Dynamic Gate #3 –get route –check for special processing ANEP Options –anep_output() »if DAN then afd_handle_dan_packet() :ACTIVE PROCESSING –send on interface ipintrq: ipintr ip_forwardip_output atmc_input apic_israpic_intr atmc_output apic_devoutput

22. June 2000 MSR Design 22 Washington WASHINGTON UNIVERSITY IN ST LOUIS IP Packet Handling in ANN Kernel (APIC) Device Driver Device Indep.-- atmc_output(): –LLC/SNAP processing –AAL5 processing (if needed) –if packet_scheduling: enqueue for PS –if !packet_scheduling: send to devoutput() Device Driver Device Dep. -- apic_devoutput() –configure an APIC descriptor for each mbuf in packet –Resume APIC TX channel ipintrq: ipintr ip_forwardip_output atmc_input apic_israpic_intr atmc_output apic_devoutput

23. June 2000 MSR Design 23 Washington WASHINGTON UNIVERSITY IN ST LOUIS Packet Classification - sys/crossbow/aiu_flow.c if (ipv6 flow_label) index = flow_label & (FLOWTABSIZE-1) else index = (src_addr[3] + dst_addr[3] +src_port + dst_port + protocol) & (FLOWTABSIZE-1) FTE = aiu_flowtable[index] while (FTE) { match FTE against packet for: dst_port, src_port, src_addr, dst_addr, protocol, interface, version if match return FTE else if ((!oldflow) && (FTE is OLD)) // OLD is not touched in last 10 seconds oldflow = fte FTE = FTE->next } if (oldflow) FTE = recycle oldflow // Garbage collection done twice per day else FTE = allocate a new entry and insert into table packet->m_cbh.fix = FTE // Store FTE in mbuf structure for later re-use return FTE

24. June 2000 MSR Design 24 Washington WASHINGTON UNIVERSITY IN ST LOUIS Active Packet Handling - sys/ann/afd.c fix = aui_getfix(packet) pgicode = extract Plugin Identifier from packet while (pgicode) acte = get class from active class table if (!acte) if packet wants to be queued enqueue packet request plugin from APL return NULL // signals caller to not send packet on else request plugin from APL return packet // signals caller to continue sending of packet fi else aite = get instance from instance table if (!aite) send message to plugin to create a new instance store new instance in instance table fi call instance entry() function fi done

25. June 2000 MSR Design 25 Washington WASHINGTON UNIVERSITY IN ST LOUIS Flow Related Tables Flow Table (global)...... fix = hash(packet 5-tuple) Plugin Class Table (global)...... Plugin Instance Table (1/class)............ Instance hash(pgicode) hash(fix)

26. June 2000 MSR Design 26 Washington WASHINGTON UNIVERSITY IN ST LOUIS Flow Table Entry - sys/crossbow/aiu_flow.h /* flow table entry structure */ struct aiu_flowte { u_int32_t interface; // interface the packet was received on u_int32_t protocol; // protocol u_int32_t src_port; // source port u_int32_t dst_port; // destination port u_int32_t flow_label; // IPv6 flow label u_int16_t flags; // various flags long touched; // last time touched long filtered[AIU_MAXDAGS]; // last time filtered struct aiu_filter *filter[AIU_MAXDAGS]; // ptrs to matching filters for each type of module supported caddr_t data[AIU_MAXDAGS]; // opaque module specific data u_int32_t version; // protocol version struct aiu_flowte *next; void (*cbpgi_flowcleanup[AIU_MAXDAGS])(short, void *); // called in case the flow gets recycled struct ann_flow ann;// info includes ptr to plugin instance (SAPF only?) u_int32_t hits; // hit count };

27. June 2000 MSR Design 27 Washington WASHINGTON UNIVERSITY IN ST LOUIS Class/Instance Table Entries - sys/ann/afd.h /* Class table entry structure */ struct afd_class_table_entry { u_int32_t pgicode; /* plugin code */ struct afd_inst_table_entry **instances; /* pointer to instance table */ struct afd_class_table_entry *next; /* next class in table */ }; /* Instance table entry structure */ struct afd_inst_table_entry { struct aiu_flowte *fix; /* flow index */ struct ann_pgiinst *inst; /* instance that handles that flow */ struct afd_inst_table_entry *next; /* next instance at this spot */ };

28. June 2000 MSR Design 28 Washington WASHINGTON UNIVERSITY IN ST LOUIS User Space Plugin Management

29. June 2000 MSR Design 29 Washington WASHINGTON UNIVERSITY IN ST LOUIS Active Plugin Loader Plugin Requestor Plugin DB Controller Policy Controller Security Gateway ANN Manager Key DB Plugin DB Policy Rules Code Server Active Code DB PE User Space, Switch Controller, & Remote Servers Control Processor... anetd Java VM ANTS NOSI APL AN EE’s Services SC... Routing RSVP Multi-Port Router Architecture 8-Port ATM Switch (20Gb/s)

30. June 2000 MSR Design 30 Washington WASHINGTON UNIVERSITY IN ST LOUIS ANPE Software Architecture Network Device Driver (DD) Class X Class Y Instances Plugin Control Unit (PCU) Plugin Management Proxy Active Function Dispatcher (AFD) Selector Dispatcher (SD) Instances Association Identification Unit (AIU) Packet Scheduler (PS) IPv4/v6 Forwarding RSVPd/routed Proxy

31. June 2000 MSR Design 31 Washington WASHINGTON UNIVERSITY IN ST LOUIS Summary ANN software architecture –built on Router Plugins –offers secure, automated kernel plugin download –provides reasonably good performance optimizations possible, e.g., polling vs. interrupts, SAPF shortcut processing Runs on the Smart Port Card and on standard PCs Integral Part of MSR and NMVC

32. June 2000 MSR Design 32 Washington WASHINGTON UNIVERSITY IN ST LOUIS Discussion Crossbow vs. ANN Performance Measurements –What do we want to learn? NetBSD redesign for SPC/MSR –User space daemons vs. kernel plugins for updates –Hardware/Software Interrupt usage –Redesign IP path for better performance –Redesign Interrupt structure for better performance