1 - Charlie Wiseman - 05/11/07 Design Review: XScale Charlie Wiseman ONL NP Router

2 - Charlie Wiseman - 05/11/07 ONL NP Router SRAM Rx (2 ME) HdrFmt (1 ME) Parse, Lookup, Copy (3 MEs) TCAM SRAM Mux (1 ME) Tx (1 ME) QM (1 ME) xScale Assoc. Data ZBT-SRAM Plugin0Plugin1 Plugin2 Plugin3Plugin4 NN FreeList Mgr (1 ME) Tx, QM Parse Plugin XScale Stats (1 ME) Rx Mux HF Copy Plugins Tx SRAM NN Large SRAM Ring Scratch Ring NN Ring NN SRAM 64KW New Needs A Lot Of Mod. Needs Some Mod. 64KW 512W Small SRAM Ring xScale 512W Per-Plugin Ctrl Msgs LD Except Errors Mostly Unchanged 64KW

3 - Charlie Wiseman - 05/11/07 XScale SRAM Ring Scratch Ring PLC Mux RLI Plugin 0 Plugin 1 Plugin 2 Plugin 3 Plugin 4 Control, Configuration, Monitoring (TCP connection) Control Messages from RLI Control Responses to RLI and Configuration Updates to XScale Local Delivery Injected Packets (reclassify) Exceptions Errors QM Injected Packets (pass through)

4 - Charlie Wiseman - 05/11/07 PLC -> XScale XScale PLC Local Delivery Exceptions Errors Flags(8b): Why pkt is being sent to XScale TTL(1b): TTL expired Options(1b): IP Options present NoRoute(1b): No matching route or filter NonIP(1b): Non IP Packet received ARP_Needed(1b): NH_IP valid, but no MAC NH_Invalid(1b): NH_IP AND NH_MAC both invalid Reserved(2b): currently unused xScale SRAM L3 (IP, ARP, …) Pkt Length (16b) Buffer Handle(24b) Stats Index (16b) QID(16b) In Port (3b) Flags (8b) Plugin Tag (5b) NH MAC DA[47:16] (32b) NH MAC DA[15:0] (16b) Unicast/MCast bits (16b) Rsv (8b) Reserved (16b) EtherType (16b) Reserved (2b) NR (1b) TTL (1b) Opt (1b) NI (1b) ARP (1b) NH INV (1b) 45 6 Local Delivery: ARP Non-exception, non-error packets directed to XScale by route or filter Exceptions: TTL is zero No Route IP Options ARP Needed Errors: Next Hop invalid Non-IPv4, non-ARP

5 - Charlie Wiseman - 05/11/07 XScale -> Mux XScale Mux Injected Packets (reclassify) Flags: PT(1b): PassThrough(1)/Classify(0) Reserved (7b) L3 (IP, ARP, …) Pkt Length (16b) Buffer Handle(24b) Stats Index (16b) QID(16b) In Port (3b) Plugin Tag (5b) Flags (8b) Rsv (4b) Out Port (4b) Reserved (7b) PT (1b)

6 - Charlie Wiseman - 05/11/07 XScale -> QM XScale QM Injected Packets (pass through) Buffer Handle(24b) QID(16b) Rsv (4b) Out Port (4b) L3 (IP, ARP, …) Pkt Length (16b) Reserved(16b) Rsv (8b) Rsv (8b)

7 - Charlie Wiseman - 05/11/07 XScale -> Plugins XScale Plugin 0 Plugin 1 Plugin 2 Plugin 3 Plugin 4 Current Types: From RLI Control Msg (0) #words (8b)MsgID (16b) Value 2 (32b)... Control Messages from RLI Type (8b) Value 1 (32b)

8 - Charlie Wiseman - 05/11/07 Plugins -> XScale Plugin 0 Plugin 1 Plugin 2 Plugin 3 Plugin 4 XScale Control Responses to RLI and Configuration Updates to XScale Current Types: To RLI Control Message Resp (0) Debug Message* (1) To XScale Add Route ( 2) Remove Route ( 3) Add Primary Filter ( 4) Remove Primary Filter ( 5) Add Aux Filter ( 6) Remove Aux Filter ( 7) Update Queue Quantum ( 8) Update Queue Threshold ( 9) Update Port Rate (10) Update Mux Quanta (11) Update Aux Sampling Rates (12) Update Exception Destinations (13) #words (8b)MsgID (16b) Value 2 (32b)... Type (8b) Value 1 (32b)

9 - Charlie Wiseman - 05/11/07 Processing Paths ▪ Plugin Control Path: RLI XScale Plugins »Control messages between the RLI and plugins »XScale role: Pass messages ▪ Configuration Path: RLI,Plugins -> XScale (-> RLI) »System configuration requests »XScale role: Enact change ▪ Monitoring Path: RLI -> XScale -> RLI »Stats read requests »XScale role: Send values ▪ 'Data' Slow Path: PLC -> XScale -> Mux,QM »Packets that require additional processing not provided in the fast path »XScale role: Handle packets

10 - Charlie Wiseman - 05/11/07 Other Responsibilities ▪ Load 'core' MEs at start-up ▪ Initialize any values required by MEs and other system components ▪ Dynamically load and remove plugins

11 - Charlie Wiseman - 05/11/07 System Initialization ▪ Rx,Tx,Freelist Manager »Nothing ▪ Mux »Input ring quanta ▪ PLC »Default Routes and Filters (lookup) »Aux Filter sampling rates (copy) »Default destination for packets with no route, needing arp, or invalid next hop (copy) Green means the values are user configurable while the system is running

12 - Charlie Wiseman - 05/11/07 System Initialization ▪ QM »Port rates »Queue Descriptors »Queue Quanta »Queue Thresholds ▪ Header Format »Port MAC addresses ▪ Stats »Counters ▪ Other »SRAM occupancy counters »MSF? RTM? TCAM? SPI? AOA? Green means the values are user configurable while the system is running

13 - Charlie Wiseman - 05/11/07 Software Components ▪ ONL Daemon (C++) »User space application that talks with the RLI and Mes »Multiple processes? Threaded? ▪ System interface (memory, ME control) »Kernel module ▪ TCAM interface »User space library »Kernel module

14 - Charlie Wiseman - 05/11/07 Software Flow Read from PLC Read from RLI Read from Plugins Write to Mux Write to QM Write to RLI Write to Plugins ARPLog IPv4 Configu ration Monitor Plugin Control

15 - Charlie Wiseman - 05/11/07 Logging Packets ▪ Dump packet into log file on the XScale ▪ Can users log in to the XScale? »If so, then they can log in to read the log »If not, need to export the log ▪ Policy: Current types that get logged (and dropped) »Non-IPv4, non-ARP packets »Packets with invalid next hop »Non-exceptional IPv4 packets

16 - Charlie Wiseman - 05/11/07 IPv4 Exception Packets ▪ IP Options »Process options and send to QM (don't reclassify) ▪ No route, TTL expiration »Send appropriate ICMP message then drop »ICMP messages will be sent to PLC for classification

17 - Charlie Wiseman - 05/11/07 ARP ▪ Packets needing ARP »Keep track of outstanding ARP requests »For each outstanding request only keep the latest received (at the XScale) packet »Re-send request after 5 seconds if no response, up to three requests, then drop packet ▪ ARP replies »First, send the one saved packet »Then, update ARP table (and TCAM) »Drop later packets arriving at XScale ▪ ARP requests »If asking for the incoming port IP address, respond »Update ARP table ▪ Time out ARP table entries?

18 - Charlie Wiseman - 05/11/07 Non-classifiable packets ▪ For packets without valid IPv4 headers (e.g., ARP): »Policy: XScale sends directly to QM »Need to choose QID and stats index ▪ Add 64 'system' queues next to the datagram queues »Nothing special about them except that they are used by the system »User can still map other flows to them and monitor them ▪ Need at least one (slow) system stats index

19 - Charlie Wiseman - 05/11/07 Plugin Control ▪ Control message from RLI to plugin »Add message 'header' ▪ Control message from plugin to RLI »Strip message 'header' ▪ Debug message from plugin »Add to debug log file »Could send to RLI, but there is potential for a lot of traffic

20 - Charlie Wiseman - 05/11/07 Monitoring ▪ RLI sends requests to monitor a stats index and counter value »One time »Periodic ▪ Aggregate requests at XScale »No need to read the same counter multiple times if there are multiple requests for it »Currently, the RLI sends a request for each instance of a value being monitored

21 - Charlie Wiseman - 05/11/07 Configuration ▪ Read configuration »If plugins change the configuration, update should be sent to RLI »Push to RLI or wait for RLI to read value again? ▪ Configuration update from RLI or plugin »Write the appropriate memory or TCAM entry »What tables needed to be mirrored? ▪ Load plugin »Get compiled plugin from mounted file system »Load and start plugin »Could plugins instantiate other plugins?