An NP-Based Ethernet Switch for the Open Network Lab Design John DeHart and Charlie Wiseman

NP* Notes For NPS and OpenFlow all frames leave with Src MAC and Dst MAC unchanged from how they arrived For Overlay, there are two cases: Overlay acts as a Router and changes Src MAC and Dst MAC to be its MAC and its NextHop MAC respectively. Overlay uses underlying Ethernet switch and SMAC and DMAC are unchanged For Overlay doing multicast, the overlay plugins are responsible for making complete copies of the frames. CONCLUSIONS: Give Overlay plugins the ability to re-write the ethernet header so they can update the Src MAC and Dst MAC Just like plugins in the NPR do with IP Hdrs This eliminates the need for HF and allows Tx to NOT do anything special for Hdrs. No Chaining of buffers We will still use reference counts

NP* Notes: DBs and Lookups We think we can do what we need with 2 Databases: Overlay lookups DB Ethernet Routing DB Do 3 lookups, 2 of them may be Simultaneous: Overlay Ethernet Routing If we do simultaneous, we lose the top 3 bits of result!!! Saddr on Ethernet Routing DB This would use the Src MAC in the Dst MAC field and VLAN If no match or if input port does not agree with output port in result then send to Xscale for Ethernet Routing DB update This scheme simplifies Xscale Daemon A bit in the result would be defined for Overlays to indicate “Use Ethernet Routing” This allows us to support the “Use Ethernet Routing” feature without a second pass through PLC I believe TCAM performance for this meets spec.

NPR to NPS Plan is to use NPR as starting point and modify blocks to implement an NPS (NP based Ethernet Switch) Ethernet Issues to be addressed: VLAN Support Address Learning Spanning Tree Protocol Double VLAN Tagging ARP Design Issues to be addressed Lookup keys and results TCAM performance Two lookups needed for address learning?

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN Mostly Unchanged 64KW xScale 64KW 64KW 64KW 64KW 64KW Plugin to XScale Ctrl,Update & RLI Msgs 512W 512W 512W 512W 512W New NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM Needs A Lot Of Mod. 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Needs Some Mod. Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

VLANs Want users to be able to setup their own VLANs But VLANs already used in configuration switches to segregate traffic on every virtual link Have to use double VLAN tagging at the configuration switches Netgear switches support this Arista switches? NPS must have support for VLANs Be able to handle VLAN and non-VLAN frames Forwarding Table entries must include VLAN How do we manage the VLANs from the RLI? Per-VLAN spanning tree? Can NPS add VLAN tags to non-VLAN frames? Remove them from VLAN frames? What about other components? Does the NetFPGA ethernet switch support VLANs? What happens when VLAN frames show up at an NPR or NSP?

Broadcast/Multicast On multicast and broadcast packets, NPS must only send copies out ports participating in the VLAN All results contain the forwarding vector Have special TCAM entries at the ‘bottom’ of the database for every VLAN Set mask to only look at VLAN number Result contains all ports participating in the VLAN If no ‘top’ entry is matched (i.e., no matching destination address is found), then TCAM will return the ‘bottom’ entry for the VLAN This handles broadcasts and multicasts, as well as “misses” for unicast (all are treated as a broadcast)

Address Learning How do Ethernet switches learn what MAC address is connected to a port? Examine source address and VLAN of incoming packets and add to/update table How will the NPS learn the mapping? One possibility: two lookups per-packet, one TCAM database First lookup on source address/VLAN, if miss or if hit but input port != result port, then send mapping to XScale to update database Second lookup on destination address/VLAN (if unicast address) to find forwarding port(s), if miss then broadcast Another possibility: maintain two parallel databases and use clever masking tricks to only perform one lookup

Spanning Tree Protocol Ideally, NPS and all other Ethernet devices in ONL would support STP But, Virtual Ethernet Switches can not support it So, take the same approach as IP routing in the NPR Switches will not actively run STP RLI/User specifies the spanning trees (per-VLAN) and the daemons configure nodes accordingly RLI must enforce that each VLAN is tree structured RLI must also configure the tree for non-VLAN frames Each NPS is configured by setting the forwarding vector in the VLAN TCAM entry

ARP In the NPR ARP packets were delivered to the XScale In the NPS ARP packets should be handled based on Destination MAC Address Unicast Broadcast

TCAM Performance What is our NPU performance target? To hit 5 Gb rate per NPU: Minimum Ethernet frame: 84B 8B Preamble + 60B frame + 4B CRC + 12B InterFrame Spacing Note: CRC is counted in the minimum size of 64 Bytes 5 Gb/sec * 1B/8b * packet/84B = 7.44 Mpkt/sec per NPU Total TCAM Performace 2 Lookups per pkt * 2 NPUs * (7.44 Mpkt/sec)/NPU = 29.76 MLookups/sec Three aspects to determining TCAM Performance LA-1 Interface performance Based on Key size, how many requests can we push across the IXP/TCAM Interface TCAM Core Performance Based on Key size, how many lookup operations can the TCAM core support Attached SRAM Performance Based on Key size and Result size, how many operations can the TCAM core process and retrieve results from the Attached Associated SRAM. Our Lookup sizes Long: Key: 288 bit Result: 128 bit Short: Key: 72bit Result: 64 bits

Route DB Lookup Key and Result 264 Bit Key (24 additional bits available): Pl Tag (5b) P 3b Pri (4b) VLAN (12b) Ethernet DAddr (48b) Ethernet SAddr (48b) EtherType (16b) 136 bits (17 Bytes) IP DAddr (32b) IP SAddr (32b) Proto (8b) DPort (16b) SPort (16b) Reserved (24b) 128 bits (16 Bytes) OpenFlow Support Overlay Support: First 128 bits after Ethernet Header 64 Bit Result Preceded by Result Index: D (1b) H (1b) M H (1b) Rsvd (7b) Absolute Index (22b) QID (16b) Stats Index (16b) 64 bit result stored in TCAM Associated SRAM Ucast/MCast (12b) Address (20b) Pointer to QDR SRAM block for further result data (probably not used for basic ethernet switch. Used by OpenFlow. Multicast Copy Vector (11b) PPS (1b) Plugin/ Port Selection bit 5 Ports, 5 Plugins, 1 XScale

SAddr DB Lookup Key and Result 60 Bit Key: VLAN (12b) Ethernet SAddr (48b) 32 Bit Result Preceded by Result Index: D (1b) H (1b) M H (1b) Rsvd (7b) Saddr Absolute Index (22b) Rsvd (7b) Port (3b) Daddr Absolute Index (22b) 32 bit result stored in TCAM Associated SRAM The first 32 bits are TCAM control information that is always returned by the TCAM. The Absolute Index shown in the first word can be before the Associated SRAM result or after it. The D, H and MH bits are always in the first word. The Daddr Absolute Index in the second word is so the Xscale will know what entry to update if the Port field is not correct.

TCAM Performance On an LA-1 interface we want to do 7.44 Msps of size 288 bits and 7.44 Msps of size 72 bits Each of those consume less than 50% of the LA-1 bandwidth so we are fine here.

TCAM Performance Each NPU needs to do 7.44 Msps at size 288 for a total of 14.88 Msps Each NPU needs to do 7.44 Msps at size 72 for a total of 14.88 Msps Each of those totals are less than half of what the CAM Core can handle at the respective sizes, so we are ok.

TCAM Performance Each NPU needs to do 7.44 Msps at size 288 for a total of 14.88 Msps Each NPU needs to do 7.44 Msps at size 72 for a total of 14.88 Msps Each of those totals are less than half of what the TCAM/NSE SRAM can handle at the respective sizes, so we are ok.

TCAM Performance (Shared Eth/Saddr DB) On an LA-1 interface we want to do 7.44 Msps of size 288 bits and 2*7.44 Msps of size 128 bits Each of those consume less than 50% of the LA-1 bandwidth so we are fine here.

TCAM Performance (Shared Eth/Saddr DB) Each NPU needs to do 7.44 Msps at size 288 for a total of 14.88 Msps Each NPU needs to do 2*7.44 Msps at size 144 for a total of 29.76 Msps Each of those totals are less than half of what the CAM Core can handle at the respective sizes, so we are ok.

TCAM Performance (Shared Eth/Saddr DB) Each NPU needs to do 7.44 Msps at size 288/64 for a total of 14.88 Msps Each NPU needs to do 2*7.44 Msps at size 144/64 for a total of 29.76 Msps For a total of 44.64 Msps, which is less than the 50 Msps that TCAM/NSE SRAM can handle, so we are ok.

Indicates 8-Byte Boundaries Packet Format 1 VLAN DstAddr (6B) SrcAddr (6B) Ethernet Header Type=802.1Q (2B) VLAN (2B) Type=IP (2B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol = UDP (1B) Hdr Cksum (2B) Src Addr (4B) Header IP Dst Addr (4B) Src Port (2B) UDP Header Dst Port (2B) UDP length (2B) UDP checksum (2B) UDP Payload PAD (nB) Ethernet Trailer CRC (4B) Indicates 8-Byte Boundaries Assuming no IP Options

Indicates 8-Byte Boundaries Packet Format 2 VLANs DstAddr (6B) SrcAddr (6B) Ethernet Header Type=802.1Q (2B) VLAN (2B) Type=802.1Q (2B) VLAN (2B) Type=IP (2B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol = UDP (1B) Hdr Cksum (2B) Header IP Src Addr (4B) Dst Addr (4B) Src Port (2B) Dst Port (2B) UDP Header UDP length (2B) UDP checksum (2B) UDP Payload PAD (nB) Ethernet Trailer CRC (4B) Indicates 8-Byte Boundaries Assuming no IP Options

NP*: Random Notes What are the classes of things related to the NPS? NPS: NP Switch “Pure” ethernet switch with some plugin capabilties NPOV: NP OVerlay Platform for implementing Overlay Routers Support for L3 capabilties implemented in Lookups and User Plugins NPOF: NP OpenFlow Specific support for OpenFlow Core NP Ethernet Switch capabilities? Address learning Ethernet Routing Used by: NPS: Yes NPOV: ??? NPOF: Yes

NP*: Random Notes Characteristics of an NP* Device Is it acting as a router or a switch? Does a directly connected host use NP* Devices MAC Address as Dst MAC in frames it sends? Rewrites MAC Addresses on all Frames? Router: Yes Switch: No Terminates VLANs? Switch: Yes and No The ethernet payload of all copies of Multicast Frames are Identical? Dependent on user plugin functionality The Dst MAC Address of all copies of Multicast Frames are Identical???? How is L3 Address  L2 Address translation implemented for Multicast when L3 != IP? Do Overlay end hosts care? How is multicast group membership implemented on end hosts? In the NPR, Dst MACs are not necessarily the same. Auxiliary filters allow for them to differ. In the NPR, L3 == IP so we know how to translate to ethernet Multicast address when not Aux Filter. The VLAN on all copies of Multicast Frames are Identical????

Overlay Routers: Random Notes What are some examples of things we MIGHT want to consider as being able to be implemented via an Overlay Router on an NPS? IPv4 Forest/Ethernet (as opposed to Forest/UDP/IP/Ethernet) MPLS I3 OpenFlow IPv6 ???

Overlay Routers: Random Notes What are the capabilities that an Overlay Router should implement? Address resolution from Overlay Address (L3) to Ethernet Address (L2) MAC Address re-write in ethernet header Routing Flow specific routing What support from the NPOV does an Overlay Router need? What are the capabilities that an Overlay EndHost should implement? Adress Resolution Raw Socket interface What is the view of an Overlay EndHost? Does it think it is connected to a switch or a router? What is its next hop? Mapping Overlay multicast to Ethernet Multicast address? Does the Overlay Router terminate VLANs? Routers normally do. Do we need to support chained buffers NPR uses chained buffers to support copies that each have different Ethernet headers Can Tx handle this kind of transfer: Partial Ethernet header from DRAM Src MAC address from register Rest of pkt from DRAM ANSWER: NO

Overlay Routers: Functionality Customized Lookups Allow Overlay Routers to specify the content and layout of their lookup key. Options: N <Offset, Size> pairs Offset: # of bytes into L3 packet to start extraction Size: Number of bytes to include in key Positives: Very flexible, User controls ordering of key fields Negatives: more complicated to implement 32 bit mask Each bit represents 1 byte of packet. 1: Include byte in key 0: Do not include byte in key Positives: Some flexibility, simple specification of bytes to include Negatives: User has NO control of ordering of key fields, more complicated to implement Include first 16 – 19 Bytes of L3 packet in Key Mask also specified so user can mask off what they do not want matched against. Positives: Simpler to implement Negatives: Inflexible, User has no control of ordering of key fields, does not support IP, OF. Conclusions(JDD, CGW, MAH): We would specify a limit, X bytes, on how deep into the L3 pkt from which a key key field could be extracted. Given X, the bit mask above would be X bits long. Given this same limit the first two options above are basically equivalent. Interface to User and Interface to Parse could be different. Third option seems pointless

Overlay Routers: Functionality (continued) Multicast Address translation: L3 Multicast Addr  L2 Multicast Addr In the NPR, if we have an IP Multicast Filter, Copy translates the IP Destination address to an Ethernet Destination Address It is unclear to us at this point if L2 Multicast addresses are of any use if the Endhost Overlay program “owns” the interface and receives all the frames. Would it use the L2 Multicast address to identify its participation in the multicast of the L3 Address. Source Address resolution: Outgoing Port  L2 Address In the NPR, we implement this in HdrFormat via a preconfigured SRAM table Different L2 Header per copy? In the NPR we have cases (Auxiliary Filters, IP Multicast) that cause different copies of a packet to need different Ethernet headers (Source and Destination MAC Addresses) DMAC (and Ethertype) are stored in Header Buffer Descriptor SMAC is stored in SRAM table indexed by Port, read by Hdr Format Do we need similar capability in Overlay Routers? SMAC could/should be different in copies going to different ports. DMAC may not be different. We will not be supporting Auxiliary filters in NPOV. So, we should support different ethernet headers for copies to support different SMAC. To do that we should continue to pass ethernet header to Tx. Thus we need HF block to continue doing this. Anything, including VLAN, that needs to go in Ethernet header on output should be written into the Buffer Descriptor except the SMAC. HF can continue to get that from SRAM table. If the SMAC is the only thing that can be different than, we don’t need chained buffers and the rest of the ethernet header can go into the payload buffer descriptor. What about VLANs? Can they be different for different copies of a frame?

Overlay Routers: Functionality (continued) Destination Address resolution: L3  L2 address translation When a router determines the next hop for a routed packet it must set the L2 Destination address for that next hop. In IP Routers like the NPR this is done via the ARP protocol and an ARP table In the NPR we implement the ARP protocol via the Xscale. In the NPR we implement the ARP table in the TCAM There is an ARP Database against which the Packets IP Destination Address is matched. This does the L3  L2 resolution when the next hop is the final destination There is a Next Hop address field in the lookup result. This initially is the NH IP address and gets resolved via the ARP Protocol when the next hop is another router. Options for supporting this in an Overlay Router: Dynamic: Use OverlayARP protocol and OverlayARP Table OverlayARP protocol implemented in end hosts and in Plugin Can OverlayARP Table be implemented like it is in NPR? Static: Statically define OverlayARP Table RLI would generate a table of (L3_Address, L2_Address) pairs. Table would be written to each end host (something like /etc/ethers.overlay) Unclear at this point if the table should be identical on all hosts? Should each host have only its local “subnet”? How is the local “subnet” defined? Endhost Overlay Program would be responsible for reading table and extracting needed L2 Addresses Table would be written to SRAM of each Overlay Router Unclear at this point if the table should be identical on all routers? Should each router have only its local “subnet”? How is the local “subnet” defined? Overlay Plugin on Overlay Router would be responsible for reading table and extracting needed L2 Addresses.

Overlay Routers: Functionality (continued)

Copies How are copies for multicast and broadcast handled? Do we need anything different in the Ethernet Header? Can each copy be an exact copy of the original packet? Ethernet Switch All copies leaving switch should be identical OpenFlow Can OpenFlow make copies and re-write headers differently? Overlay Switch ???

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN 64KW xScale 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN 64KW xScale 64KW 64KW 64KW 64KW 64KW Blocks or text filled in pink indicate that they have changed from the NPR design. Blocks filled in green or text written in green indicate that they need to be changed from what they were in NPR. 512W 512W 512W 512W 512W Ethernet Frame Length (16b) Ethernet DA[15-0](16b) NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN 64KW xScale 64KW 64KW 64KW 64KW 64KW Buf Handle(32b) InPort (4b) Reserved (12b) Eth. Frame Len (16b) 512W 512W 512W 512W 512W NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch 1 2 3 7 xScale xScale (3 Rings?) SRAM TCAM SRAM Rx Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN Flags: Src: Source (2b): 00: Rx 01: XScale 10: Plugin 11: Undefined PT(1b): PassThrough(1)/Classify(0) Reserved (5b) Rsv (4b) Out Port (4b) Buffer Handle(24b) 64KW xScale 64KW 64KW 64KW 64KW 64KW Ethernet Frame Length (16b) QID(16b) 512W 512W 512W 512W 512W Plugin Tag (5b) In Port (3b) Flags (8b) Stats Index (16b) NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) Reserved (5b) Src (2b) PT (1b) 1 2 3 7 SRAM

ONL NP Switch QM will not do any Stats Operations xScale Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN QM will not do any Stats Operations so it does not need the Stats Index. But the QM code is nasty enough that it will not be easy to change the format for the input. We will attack that change when we do other optimizations for QM. 64KW xScale Rsv (8b) Buffer Handle(24b) 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W Rsv (4b) Out Port (4b) Rsv (8b) QID(16b) Ethernet Frame Length (16b) Reserved(16b) NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Buffer Handle(24b) Rsv (3b) Port (4b) V 1 Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN 64KW xScale 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN 64KW xScale 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W V 1 Rsv (3b) Port (4b) Buffer Handle(24b) NN Ethernet DA[47-16] (32b) NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM Ethernet DA[15-0](16b) Ethernet SA[47-32](16b) 512W Ethernet SA[31-0] (32b) 512W 512W 512W 512W Ethernet Type(16b) Reserved (16b) Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch 7 3 2 1 xScale xScale (3 Rings?) SRAM TCAM SRAM Rx Flags(8b): Why pkt is being sent to Plugin TTL(1b): TTL expired Options(1b): IP Options present NoRoute(1b): No matching route or filter NonIP(1b): Non IP Packet received ARP_Needed(1b): NH_IP valid, but no MAC NH_Invalid(1b): NH_IP AND NH_MAC both invalid Reserved(2b): currently unused xScale Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Rsv (8b) Buffer Handle(24b) Scratch Ring SRAM LD Ethernet Frame Length (16b) TCAM QID(16b) Except Errors SRAM NN NN Ring Plugin Tag (5b) In Port (3b) Flags (8b) Stats Index (16b) 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN NH MAC DA[47:16] (32b) NH MAC DA[15:0] (16b) EtherType (16b) 64KW Reserved (16b) xScale Unicast/MCast Bits (16b) 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) Reserved (2b) NH INV (1b) ARP (1b) NI (1b) NR (1b) Opt (1b) TTL (1b) SRAM 7 3 2 1

ONL NP Switch 1 2 3 7 xScale xScale (3 Rings?) SRAM TCAM SRAM Rx Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN Flags: PT(1b): PassThrough(1)/Classify(0) Reserved (7b) Rsv (4b) Out Port (4b) Buffer Handle(24b) 64KW xScale 64KW 64KW 64KW 64KW 64KW Ethernet Frame Length (16b) 512W 512W QID(16b) 512W 512W 512W Plugin Tag (5b) In Port (3b) Flags (8b) Stats Index (16b) NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Reserved (7b) PT (1b) 1 2 3 7 Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch 7 3 2 1 xScale xScale (3 Rings?) TCAM SRAM SRAM Rx Flags(8b): Why pkt is being sent to XScale TTL(1b): TTL expired Options(1b): IP Options present NoRoute(1b): No matching route or filter NonIP(1b): Non IP Packet received ARP_Needed(1b): NH_IP valid, but no MAC NH_Invalid(1b): NH_IP AND NH_MAC both invalid ARP_DB_Update(1b): Update ARP DB with result Reserved(1b): currently unused xScale Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Rsv (8b) Buffer Handle(24b) Scratch Ring Ethernet Frame Length (16b) TCAM QID(16b) SRAM LD Except Errors SRAM NN NN Ring Plugin Tag (5b) In Port (3b) Flags (8b) Stats Index (16b) 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NH MAC DA[47:16] (32b) NN NH MAC DA[15:0] (16b) EtherType (16b) Reserved (16b) 64KW xScale Unicast/MCast Bits (16b) 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Tx, QM Parse Plugin XScale Rsvd (1b) ARP DB (1b) NH INV (1b) ARP (1b) Stats (1 ME) NI (1b) NR (1b) Opt (1b) TTL (1b) FreeList Mgr (1 ME) SRAM 7 3 2 1

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Flags: PassThrough/Classify (1b): Reserved (7b) Ethernet Frame Length (16b) Buffer Handle(24b) Stats Index (16b) QID(16b) In Port (3b) Plugin Tag (5b) Flags (8b) Rsv (4b) Out Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN 64KW xScale 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN 64KW xScale 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 Stats Index (16b) Opcode (4b) Data (12b) SRAM 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM

ONL NP Switch xScale xScale (3 Rings?) SRAM TCAM SRAM Rx (2 ME) Mux Assoc. Data ZBT-SRAM xScale (3 Rings?) Small SRAM Ring Large SRAM Ring Scratch Ring SRAM TCAM LD Except Errors SRAM NN NN Ring 64KW Rx (2 ME) Mux (1 ME) Parse, Lookup, Copy (3 MEs) QM (1 ME) HdrFmt (1 ME) Tx (1 ME) NN 64KW xScale 64KW 64KW 64KW 64KW 64KW 512W 512W 512W 512W 512W NN NN NN NN Plugin0 Plugin1 Plugin2 Plugin3 Plugin4 SRAM Buffer Handle(24b) Reserved (8b) 512W 512W 512W 512W 512W Rx Mux HF Copy Plugins Tx Stats (1 ME) Tx, QM Parse Plugin XScale FreeList Mgr (1 ME) SRAM