1. Washington WASHINGTON UNIVERSITY IN ST LOUIS fredk@arl.wustl.edu Design of the MultiService Router (MSR): A Platform for Networking Research Fred Kuhns Washington University Applied Research Laboratory http://www.arl.wustl.edu/arl/project/msr

2. 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Presentation Overview Overview: –Motivation, Context and Requirements System Architecture IP Forwarding in the MSR Introduction to the Control Protocol

3. 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Motivating Example open_session (type,args) code servers Session Establishment get app. spec. & plugin code determine best config. add destination host Gigabit links Traffic isolation Security and DOS Rapid prototyping Experimental protocols Resource reservations and guarantees Embedded applications and active processing

4. 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Identifying Design Requirements MSR must support Ethernet or ATM using PVCs Multiple hosts/routers per physical interface One or more routers a network/LAN Hosts LAN A CR AR Port 1 Net A Port 2 AR Hosts LAN B Port 1 Port 2 Net B Net C Net D Net E Net F Net G

5. 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 MSR Project Goals Develop an open, high performance and dynamically configurable Multiservice routing platform for use in Networking Research: –Support gigabit link speeds independent of specific link technologies –Port Processor independence: SPC, FPX or Both Configuration discovery at initialization time –Optimized packet processing in hardware (FPX) IP forwarding and advanced packet scheduling Active processing in hardware (FPX) Support prototyping of new functions in software (SPC) before migrating to the FPX

6. 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 –Create framework for experimenting with new protocols for traditional routing or resource management (QoS). Extensible and robust software frameworks –Router control and resource management –Support conventional routing protocols such as OSPF Avoid needless abstractions –Leverage existing and legacy efforts Leverage existing code or libraries Gigabit kits program, Extreme Networking project and Programmable Networks vision MSR Project Goals

7. 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Three Core Areas Hardware architecture and performance –High-performance forwarding path and core interconnect –Hardware Components: WUGS, APIC, SPC and FPX provide core components Top-Level Functional Requirements (Framework) –Captures the management and control operations. In the MSR, most top-level functions are implemented on a centralized control processor. Port-level Functional Requirements (Framework) –The IP forwarding path, active processing and port level control functions. Also statistics and monitoring.

8. 8 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Top-Level Framework System Initialization: default routes and resources Routing and Signaling –Extensible framework for routing protocol support (zebra). Active routing: extended version of OSPF (Ralph Keller) –Associating received LSA with correct MSR interface –Forwarding table management/distribution (SPC and FPX) Local Management and Control –Configuration discovery and initialization –Monitor and control local resources –Resource reservations: admission control, allocation and policing –API for plugin loading/unloading, instance creation/instance deletion, binding instances to filters, filter creation/deletion

9. 9 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Port-Level Functional Requirements Control and management interface Per flow or flow aggregate Resource allocation –Packet classification and general filters –Distributed Queuing (DQ) –Fair queuing at output ports (DRR) –Flow based resource allocations IP Forwarding –Standard IP best-effort forwarding –Reserved flow forwarding - exact match Active Processing Support –Plugin environment –Monitor and control resource usage –Dynamic plugin download and execution

10. 10 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Status: Complete Initial architectural design and implementation complete: –software modules and hardware components –Core software module design and implementation complete: general filter, exact match classifier, fipl, distributed queuing, active processing, virtual interfaces, port command protocol. Testing of IP forwarding function in FPX complete. Initial DQ testing complete Queue State DRR simulation and initial integration complete.

11. 11 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Status: Current Effort Analysis and Enhancement of the –Distributed Queuing algorithm and its implementation. –exact match classifier and route pinning with reservations Validate DRR Test and verify Wave Video plugin and demo Simple flow entry and route cache timeouts Validate plugin bindings to exact match classifier Routing protocol support (OSPF and Zebra)

12. 12 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Services Planned for Development Extreme Networking - http://www.arl.wustl.edu/arl/projects/extreme : –Lightweight Flow Setup Service one-way unicast flow with reserved bandwidth, soft-state stable rate and transient rate reservations –Network Access Service (NAS) provides controlled access to LFS registration/authentication of hosts, users resource usage data collection for monitoring, accounting –Reserved Tree Service (RTS) configured, semi-private network infrastructure reserved bandwidth, separate queues paced upstream forwarding with source-based queues

13. 13 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Future Extensions to Basic MSR Per source aggregate queues based on source prefix –DRR service; Discard policy is longest queue with hysteresis, discard front Super scalable packet scheduling –approximate radix sort w/compensation (timing wheels) Lightweight flow setup protocol implementation –flow identification in SPC, returns virtual output queue –framework for managing BW allocations and unfulfilled requests –interface to NAS Reserved Tree Service: Hardware only. Enhanced (per flow?) Distributed Queuing NAS implementations: –User authentication, policy enforcement, monitoring & feedback

14. 14 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Presentation Overview Overview of Project and status System Architecture IP Forwarding in the MSR Control Path SPC kernel processing Control Protocol

15. 15 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 MSR Hardware Components Control Processor Switch Fabric ATM Switch Core Port Processors PP LC PP LC PP LC PP LC PP LC PP LC Line Cards

16. 16 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Port Processors: SPC and/or FPX Control Processor Switch Fabric ATM Switch Core Port Processors FPX SPC LC IPPOPP FPX SPC LC IPPOPP FPX SPC LC IPPOPP FPX SPC LC IPPOPP FPX SPC LC IPPOPP FPX SPC LC IPPOPP Line Cards

17. 17 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Example SPC and FPX Design APIC IP Classifier DQ Module NID X.1 Z.2 shim Active Processing SPC FPX Flow Control Shim contains results of classification step

18. 18 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 MultiService Router - Overview PP DQ plugin PP PE FP PP Configure GBNSC (switch & ports) MM CP flexroutd Routing RA OSPF Local Interface framework Signaling Agents OSPFflexsig NCMO/Jammer RSVP NOC Net Manager App and GUI classify/lookup DRR classify/lookup DRR Signaling Resource WUGS MSR MSR control PE classify DQ classify CP - Control Processor RA - Route Agents MM – MSR Manager PP - Port Processor (SPC/FPX) PE – Processing Environment DQ – Distributed Queuing DRR – Deficit Round Robin FP – Forwarding Path

19. 19 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Top-Level Components MultiService Router (MSR) –Control Processor (CP): System monitoring and control: MSR Manager (MM): router configuration; signaling protocol; Forwarding db and Classifier rule set management; system monitoring; port level resource management and control; local admission control; discovers hardware configuration at startup. Routing Agent (RA): local instance of routing protocols, communicates with remote entities. Sends route updates to RM. Currently Zebra based. WUGS switch controller (GBNSC), used for monitoring functions: sends control cells to WUGS to read statistics and configuration information. –Port Processor (PP): Port level resource management. Forwarding Path (FP): modules/components performing the core IP forwarding and port level control operations. Route lookup, flow classification, filtering, distributed queuing and fair output queuing. Processing Environment (PE): infrastructure supporting active processing of IP datagrams. Application flows are bound to a software (or hardware module in the FPX) processing module.

20. 20 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Top-Level Components Network Operations Center (NOC) - GUI interface –Network Management Application Active metric collection Passive monitoring of DQ. Display formats include format, temporal resolution, processing overhead. Metric and display evaluation Active management not implemented. –Supports MSR Testing test/demo configuration and setup identify meaningful metrics and display architecture –Display and "manage" MSR configuration interface to init MSR, change per port attributes reset MSR set runtime parameters

21. 21 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 The Control Processor Native ATM Library SPC Libs Cmd/Msg FPX Control Logic/Cell Libs Switch & APIC Control/Cell Libs GBNSC Config INET API MSR Wrappers Policy Manager Resource Manager IP Routing Manager MSR Abstraction Layer Configuration MSR Manager Operational Control flexsig TCP UDP IP Device (APIC) Driver (Common Code) Native ATM "raw"

22. 22 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Control Processor Tasks The Top-Level framework, Currently supports: MSR Initialization Data collection for the Management tool Test and Demo Environment Support Routing support - in development. –OSPF - Standard Routing Protocols Local resource management and global signaling –Monitor resource usage –Support Active Processing - plugins

23. 23 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Requirements - RM Resource identification and initialization: Create and distribute routing tables to port processors –constructed using OSPF and a QoS routing protocol Distributed queuing (DQ) management –reserves output link BW and sets policies. Allocation of resources within individual ports –Static allocation - configuration script or admin commands –Dynamic allocation or re-allocation out-of-band allocation by manager out-of-band allocation by signaling agents in-band as needed by particular services

24. 24 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Distributed Routing Table Admin OSPF route tables RSVPOSPF# flow table EN VC space and BW - Admission Cntrl Merge Tables Port 1Port 2Port 3... Port N IP Resource Management...

25. 25 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Routing Support Context: Programmable Networks Focus: Software Component Architecture Issues: –Building, maintaining and distributing route tables –Delivery of updates (LSAs from neighbors) to CP –Format of route table entries –Support for logical interfaces (sub-interfaces) –CP component interactions (APIs): Routing - Zebra, OSPFd and msr_route_manager signaling and resource manager Assumptions: –All routing neighbors are directly attached

26. 26 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Presentation Overview Overview of Project and status System Architecture IP Forwarding in the MSR Control Path SPC kernel processing Control Protocol

27. 27 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 MSR 0.5 – Basic IP Forwarding Core functions implemented in basic MSR system (aka Version 0.5) –Control Processor System monitoring System configuration –Port level Software (SPC): Fast IP Lookup (FIPL) and Table management APIC Driver (the engine) MSR Memory management (buffers) High priority periodic callback mechanism Distributed Queuing General Classifier and an active processing environment

28. 28 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 SPC/FPX Phase 0.5 - Basic IP Forwarding IP WUGS SPC/FPX CP Does not show distributed queuing router One connected IP entity per port Control Traffic loopback

29. 29 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Distributed Queuing Goals –Maintain High Output Link Utilization –Avoid Switch Congestion –Avoid Output Queue Underflow Mechanisms –Virtual Output Queuing (VOQ) at the inputs –Broadcast VOQ and output backlogs every D  sec –Each PP i recalculates rate i,j every D  sec

30. 30 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 DQ - Cell Format Broadcast DQ summary cells every D  sec: Src port - sending port number (0-7) Overall Rate - total aggregate rate (BW) allocated to this port for the port-to-switch connection –currently not used Output queue length - bytes queued in output port’s output queue. VOQ X Queue Length - number of bytes queued in src port’s VOQ for output port X. Cell Header Src port 320 VOQ 0 Queue Length VOQ 1 Queue Length VOQ 2 Queue Length VOQ 3 Queue Length VOQ 4 Queue Length VOQ 5 Queue Length VOQ 6 Queue Length VOQ 7 Queue Length Padding Output Queue Length Overall Rate VCI = DQVC

31. 31 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 P4 P5 P6 P7 p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue wugs P0 P1 P2 P3 SPC/FPX DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr out Qs Determine per output port queue depth CP Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop 192.168.200.X 192.168.201.X 192.168.202.X 192.168.203.X 192.168.204.X 192.168.205.X 192.168.206.X 192.168.207.X MSR Router: Distributed Queuing 192.168.203.2 192.168.202.2 At each port, DQ runs every D  sec out Qs DQ updates packet scheduler to pace each VOQ according to backlog share DQ summary cells wait in queue for start of next cycle p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue Read all summary cells (including own) and calculate output rate for each VOQ. p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p0p0 p8p8 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue p8p8 p0p0 DQ data cell hdr DQ data cell hdr... queue DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr DQ data cell hdr Create DQ summary cell for this port and Broadcast cell to all input ports (including self)

32. 32 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Distributed Queuing Algorithm Goal: avoid switch congestion and output queue underflow. Let hi(i,j) be input i’s share of input-side backlog to output j. –can avoid switch congestion by sending from input i to output j at rate  L  S  hi(i,j) –where L is external link rate and S is switch speedup Let lo(i,j) be input i’s share of total backlog for output j. –can avoid underflow of queue at output j by sending from input i to output j at rate  L  lo(i,j) –this works if L  (lo(i,1)+···+lo(i,n))  L  S for all i Let wt(i,j) be the ratio of lo(i,j) to lo(i,1)+···+lo(i,n). Let rate(i,j)=L  S  lo(wt(i,j),hi(i,j)). Note: algorithm avoids congestion and avoids underflow for large enough S. –what is the smallest value of S for which underflow cannot occur?

33. 33 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 CP Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop WUGS P0 P1 P2 P3 P4 P5 P6 P7 192.168.200.X 192.168.201.X 192.168.202.X 192.168.203.X 192.168.204.X 192.168.205.X 192.168.206.X 192.168.207.X SPC/FPX DQ 192.168.203.2 192.168.202.2 IP fwd MSR IP Data Path - An Example 192.168.200.2 192.168.204.2

34. 34 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 MSR Version 1.0 - Enhanced Control Processor –Configuration discovery –enhanced download with broadcast –Command protocol implementation Port level Software (SPC): –Virtual Interface support and Shim processing –Dynamic update of FIPL table –MSR Memory Management enhancements –Distributed Queuing –Exact Match Classifier –Command protocol implementation –Embedded Debug facility

35. 35 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Virtual Interfaces on ATM SPC/FPX 2xx Port 1 lookup out 40 43 42 44 43 44 42 40 50 51 50 51 Port 2 Port 4 Port 3 CP ATM Switch Port 3 Port 0 Port 1 Port 2 R R Host VC=50 VC=51 X No PVC, No Traffic to/from MSR

36. 36 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 SPC WUGS SPC 63 Port 1 Port 2 lookup out 40 43 42 44 Port 3 41 43 44 40 41 43 44 43 44 42 40 63 40 41 42 44 43 41 42 40 43 41 42 41 42 44 40 50 51 53 52 50 51 53 52 lookup out 50 51 53 52 50 51 53 52 50 51 53 52 50 51 53 52 50 51 53 52 50 51 53 52 Port 4 control Port 0 CP SPC lookup/out processing 50 VP0 51 VP1 52 VP2 53 VP3 IP (udp/tcp) ospf RA config RM raw atm socket atm... Port loopback not shown IP Address bound to virtual interfaces only ip fwd path shown Virtual Interfaces, or Virtual Ports (VP) IP layer: routes pkts to/from sockets Sockets: Communication endpoints Driver: routes pkts between interface and net layer Internal MSR Connections (SPC only)

37. 37 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 SPC shim update shim demux Packet Routing, SPC and FPX WUGS... 64... 127 (out port + 64) 63... 64... 127 (in port + 64) IngressEgress VCs to next hop routers (p2p conn) From previous hop router add shim rem shim FIPL shim proc. FPX SPC shim demux shim update 63 Outbound VC = SPI + 128 0 <= SPI<= 15 Link Interface IP eval: IP processing for FPX. 1.Broadcast and Multicast destination address 2.IP options 3.Packet not recognized Inbound VC = SPI + 128 0 <= SPI <= 15 Currently Support at most 4 Inbound VCs: One for Ethernet or Four for ATM Current VCI Support: 1) 64 Ports (PN) 2) 16 sub-ports (SP) FIPL IP proc plugins FIPL IP proc plugins Ether only VC to endstations Ether only VC from endstations

38. 38 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 CP SPC/FPX WUGS SPC/FPX 50 Using Virtual Interfaces Port 1 Port 2Port 4 Port 0 lookup out shim 40 43 42 44 Port 3 41 43 44 40 41 43 44 43 44 42 40 41 42 44 43 41 42 40 43 41 42 41 42 44 40 50 51 50 51 50 51 52 50 51 52 50 51 50 51 50 VP0 SPC/FPX shim processing shim VINIP/MaskVCPort Sub Port 0192.168.200.1/245000 1192.168.201.1/245010 2192.168.202.1/245111 3192.168.203.1/245020 4192.168.204.1/245121 5192.168.205.1/245222 6192.168.206.1/245030 7192.168.207.1/245131 8192.168.208.1/245040 Input port lookup IP route Insert shim send to output Output port Reassemble frame get out VIN from shim remove shim send to next hop/host

39. 39 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 P0 WUGS PP Configure Resource Signaling Discover (switch & ports) RM CP flexroutd zebra Routing RA OSPF interfaces P1 P2 P3 P4 P6 P7 P5 src: 192.168.220.5 dst: 192.168.200.1 sport: 5050 dport: 89 data 50 51 52 53 IP packet Port: 5 SubPorts: 0-3 VIN 20: 192.168.220.1/24:Ext. Link VC 50 VIN 21: 192.168.221.1/24:Ext. Link VC 51 VIN 22: 192.168.222.1/24:Ext. Link VC 52 VIN 23: 192.168.223.1/24:Ext. Link VC 53 Packet Forwarding Port: 0 SubPorts: 0-3 VIN 0: 192.168.200.1/24:Ext. Link VC 50 VIN 1: 192.168.201.1/24:Ext. Link VC 51 VIN 2: 192.168.202.1/24:Ext. Link VC 52 VIN 3: 192.168.203.1/24:Ext. Link VC 53

40. 40 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Forwarding IP Traffic P0 WUGS PP Configure Resource Signaling Discover (switch & ports) RM CP flexroutd zebra Routing RA OSPF interfaces P1 P2 P3 P4 P6 P7 P5 ip lookup Lookup destination in table: 192.168.200.1/24 matches - Out VIN 0 Driver reads header and performs route lookup, returns fwd_key: –fwd_key = {Stream ID, Out VIN} –SID = reserved session ID, local only –VIN = {Port (10 bits), SubPort (6 bits)} Insert shim, update AAL5 trailer and IP header. calculate internal VC from output VIN’s port number (VC = 40) src: 192.168.220.5 dst: 192.168.200.1 sport: 5050 dport: 89 data shim 50 51 52 53 192.168.200.1 -> fwd_key lookup/out

41. 41 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Shim IP Header AAL5 Trailer IP Datagram Fragment offset VersionH-lengthTOSTotal length IdentificationFlags TTLProtocolHeader checksum Source Address Destination Address IP Options ?? IP data (transport header and transport data) AAL5 padding (0 - 40 bytes) CPCS-UU (0) Length (IP packet + LLC/SNAP) CRC (APIC calculates and sets) Internal IP Packet Format 8 Bytes

42. 42 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 IntraPort Shim: Field Definitions Flags - Used by SPC to demultiplex incoming packets.The FPX sets flags to indicate reason for sending packet to SPC. Note, may also use flags to implement flow control. –AF: Active Flow. –NR: No route in table. –OP: IP Options present (Correct version but incorrect header size). –UK: Unknown packet type (incorrect version for example). Stream Identifier (SID): Identifier for reserved traffic, locally unique label. – FPX fills in for reserved flows. Input VIN - The physical port and sub-port packet arrived on. PN is the physical port number and SPI is the sub-port identifier. There is a set map from SPI to VCI. –FPX sets these values. Not clear that we need this in the IntraPort shim. –VCI = Base VC + SPI Output VIN - output port and sub-port. –The FPX sets this if the route lookup succeeds. –If the SPC performs the lookup for the FPX then the SPC fills in. –The SPC may also modify this value in order to re-route a packet - modifying seems dangerous, but setting ok. Output VIN Input VIN Stream Identifier Not Used Flags Virtual Interface Number Format PN (10 bits) SPI (6 bits) 01531 AFNROPUK X Flags

43. 43 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 InterPort Shim: Field Definitions Used to convey forwarding information to output port. Currently only the Output SPI is necessary for forwarding. Flags: TBD. Input VIN – Same as IntraPort Shim. –Ingress FPX or SPC when FPX is not used. Output VIN – Same as IntraPort Shim. X Output VIN Not Used Input VIN Virtual Interface Number Format PN (10 bits) SPI (6 bits) 01531 Flags

44. 44 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 FIPL Table Entry Formats FPX version of FIPL table entry (36 bits): Output VIN Stream Identifier A 0163135 TBD 15 Output VIN Stream Identifier 0163115 SPC version of FIPL table entry (32 bits): Virtual Interface Number Format PN (10 bits) SPI (6 bits) 0155

45. 45 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Forwarding IP Traffic WUGS Configure Resource Signaling Discover (switch & ports) RM CP flexroutd zebra Routing RA OSPF interfaces P3 P4 P6 P7 At Port 0, driver extract shim and determines the destination VIN Output VIN converted to output VC = 50 Removes shim, updates AAL5 trailer and sends on VC 50 (in this case the packets goes to the CP) PP P5 50 51 52 53 shim processing src: 192.168.220.5 dst: 192.168.200.1 sport: 5050 dport: 89 data Route Advertisement P0 PP P1 P2 src: 192.168.100.5 dst: 192.168.214.1 sport: 5050 dport: 89 data shim (to VIN 0)

46. 46 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Example: Processing Route Updates P0 WUGS PP Configure Resource Signaling Discover (switch & ports) RM CP flexroutd zebra Routing RA OSPF Interfaces P1 P2 P3 P4 P6 P7 CP kernel delivers packet to socket layer Packet read by OSPFd, Senders IP address is mapped to interface ID OSPFd and Zebra associate received advertisement with an MSR input VIN (VIN20 in this case). PP P5 50 51 52 53 socket

47. 47 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 P0 WUGS PP Configure Resource Signaling Discover (switch & ports) RM CP flexroutd zebra Routing RA OSPF Interfaces P1 P2 P3 P4 P6 P7 Note on Sockets, IP and Interfaces src: 192.168.100.5 dst: 192.168.205.1 sport: 5050 dport: 89 data What if packet arrives on interface with different address? Do we care? Example: Packet sent to CP but arrives at port 7 on different VP. CP kernel will still send pkt to socket bound to 192.168.200.1 If all neighbor routers are directly attached, then it doesn't matter. We can distinguish by looking at sending IP address. socket PP P5 52 53 54 55

48. 48 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Example: Processing Route Updates P0 WUGS PP Configure Resource Signaling Discover (switch & ports) RM CP flexroutd zebra Routing RA OSPF Virtual Interfaces P1 P2 P3 P4 P6 P7 OSPFd notifies zebra of any route updates. Zebra provides a common framework for interacting with system Zebra passes route updates to he MSR Route Manager PP P5 52 53 54 55

49. 49 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 PP Route Distribution and Port Updates PP WUGS PP 192.168.205.1 Add destination vector and output port/VC "Broadcast" update P0 PP Configure Resource Signaling Discover (switch & ports) RM CP flexroutd zebra Routing RA OSPF Virtual Interfaces

50. 50 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Presentation Overview Overview of Project and status System Architecture IP Forwarding in the MSR Command Protocol and Debugging Messages –sendcmd() and the CP to PP Control Path

51. 51 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 MSR Command Overview The CP may send commands to individual ports (SPCs) using the command protocol. Use the sendcmd utility. A specific command type designates the SPC kernel modules to receive the command message. A sub- command may also be specified to further identify the particular operation to be performed. Command Protocol description –Control Processor sends command messages to a specific port and expects to receive a reply message indicating either Success or Failure. This is termed a Command Cycle. –There is the notion of a Command Transaction which may include one or more command cycles. A command transaction is terminated when the target (port) responds with a reply message containing an EOF

52. 52 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Predefined Top-Level Commands policy - Manage MSR policy object (see description) port_init - set local port number, enable DQ fipl –Send route updates to port (see the fipmgr utility) rp_pcu – Send commands to the plugin control unit. rp_inst - Send message to a plugin instance rp_class - send message to plugin base class apic – send commands to the APIC driver stats – Report port level statistics cfy – Classifier interface set_debug/get_debug – Set/Get debug flags and mask. port_up, port_down, dq and perf – Not Implemented

53. 53 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Policy Object Commands The policy object is referenced during runtime –modifications will generally have an immediate effect –get_XXX returns the current value, set_XXX will set it to a new value. get_gen/set_gen - enable/disable general classifier get_iprt/set_fipl/set_simple - specify which route lookup algorithm to use: fipl or simple linear lookup. get_dflags/set_dflags - specify where to send debug messages: disable, print to local console or send to cp. get_drr/set_drr - enable/disable drr (Not Implemented) get_dq/set_dq - enable/disable DQ (Not Implemented) get_flags/set_flags - return or set all policy control flags.

54. 54 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Plugin Framework Enhancements Integrated with Command framework –send command cells to PCU: create instance, free instance, bind instance to filter, unbind instance –Send command cells to particular plugin instances –Send command cells to plugin base class –instance access to: plugin class, instance id, filter id –pcu reports describing any loaded classes, instances and filters

55. 55 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 MSR RP_PCU Command rp_pcu Command sub-commands –addfltr/remfltr - add/remove pkt filter at gate x –flist - port prints current gate x filter list –bind/unbind - bind/unbind instance to fltr/gate combo –create - create plugin instance –free - remove plugin instance –clist - port prints current plugin class list –ilist - port prints current instance list –load/unload - load plugin - not implemented –null - no-op. Can be used as a ping operation

56. 56 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 APIC Command apic command - this command is useful for debugging and validating behavior. –info - a verbose printing of low level apic register, descriptor and buffer state. –resume - can specify a suspended channel to resume –desc - print information about a particular descriptor and any associated buffer. –pace - change the per VC pacing parameter –gpace - change the global pacing paramter

57. 57 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Classifier Command cfy command is currently needed to perform necessary maintenance –tbl_flush - flush (i.e. free) idle entries in the classifier table –rt_flush - flush (remove) cached routes in the classifier table entries. –info - print a list of the active classifier table entries.

58. 58 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Statisitics Command stats command - this is a potentially useful command set but currently is only nominally supported. –get_all - print all statistics using the DEBUG facility. –reset - reset all counters.

59. 59 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 MSR DEBUG Command Debug facility - mechanisms for sending text messages –defines a category and level –Integrated with Command framework Dynamic setting of debug mask - affects what messages are sent based on the category and level masks. set_debug or get_debug command –Valid debug categories/modules apic, ipfwd, ingress, egress, iprx, iptx, mem, dq, stats, ctl, conf, kern, natm, pcu, plugin, classify, perf, atmrx, atmtx, buff –Valid debug levels - 0 - 255 predefined: verbose, warning, error, critical Interface in MSR kernel –MSR_DEBUG((MSR_DEBUG_ |MSR_DEBUG_LEVEL_, “fmt”, args));

60. 60 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/7/02 Example Sending Cmd to Port CP Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop Next/Prev Hop wugs P0 P1 P2 P3 P4 P5 P6 P7 192.168.200.X 192.168.201.X 192.168.202.X 192.168.203.X 192.168.204.X 192.168.205.X 192.168.206.X 192.168.207.X SPC/FPX DQ 192.168.203.2 192.168.202.2 sendcmd(); create plugin instance: port id = 0, PluginID = 200 cmd data cell hdr msr_ctl reply(); plugin instance created: Status, Instance ID Report command completion status to application. Lookup sub-command perform function call then report results