1. Multicast Support in the NSP (the “low-level” details)
NSP Meeting
28 October 2002
Alex Chandra, Yuhua Chen, Sarang Dharmapurikar,
John Lockwood, Todd Sproull, Wen-Jing Tang,
David Taylor, Jon Turner

2. Multicast Tree Support
Support binary copy multicast without pruning
Extra complexity needed to avoid re-ordering
Example: If leaf E is removed from tree (a), then the tree will be maintained as shown in (b).
Note that recycling node B will only forward one copy to leaf C
Upstream copy suppression is possible

3. Multicast Use of InterPort Shim
InputVIN stamped at packet arrival
OutputVIN used to forward to next port
Previous Port Number (PPN) stamped by multicast recycling port
Used to suppress upstream copies
Added Multicast Tree Position (MTP) field
Denotes position of filter/packet in multicast tree
Allows multiple filters per port for a single multicast session

4. Multicast Tree Position (MTP)
Initialized by ISAR to 0x7F ( )
CARL performs exact match on 5-tuple and MTP
Returns a single bit (MB) as a branch indicator
Prior to send, MTP is modified by OSAR
Replace rightmost (least significant) zero with bit returned by CARL/QM
Change the bit to the right of this bit from 1 to 0

5. Classification & Route Lookup (CARL)

6. EM/FIPL Shared Memory Map
~38.5k FIPL entrieds
20,480 exact match filters
“Forced” pre-allocation of ingress/egress filter pairs
Head and list entries managed separately
All EM buffers must be aligned on 6 word boundaries relative to start of buffer
All EM pointers are entry pointers

7. Exact Match Search Engine

8. Exact Match Filter Entry Format

9. Exact Match Filter Entry Format
Multicast filters must set Filter Type field to 001
MB (Multicast Branch) bit identifies the branch of the multicast tree corresponding to the filter
Identifies which OutputVIN/QID pair to bind to packet
Forwarded to QM and used by OSAR to update MTP
Default = 0
NB (Number of Branches) bit indicates the number of branches exiting node
Needed for maintaining incomplete (non-pruned) trees
MTP field replaces Rate2 field for multicast filters

10. End of Presentation

11. MSR Configuration with FPX
SPC handles exceptional, optional, and “active” packet processing
FPX handles SAR, classification, and queueing
“Plain” ingress packets
Route lookup, exact match, general match classification  DQ
“Plain” egress packets
Exact match classification DRR
Supports 1 Gb/s links with 4 Sub-Port Identifiers (SPIs) per port
Dedicated ingress & egress comm. channels with SPC
Dedicated CP, SPC, and DQ control channels

12. RAD Logical Block Diagram

13. RAD Physical Block Diagram

14. NID Virtual Circuit Map
All traffic arriving from SW switched to RAD_SW port
Traffic arriving from SP0-SP3 switched to RAD_LC port
Due to physical partitioning of PSM and ISAR, require unique VCs for ingress and egress SPC traffic
Unique control VCIs for CP, SPC, and DQ rate distribution

15. InterPort Shim (FPX/SW)
5-bit Input and Output VINs (8 ports w/ 4 sub-ports)
Previous Port Number (PPN) stamped by multicast recycling port
Used to suppress upstream copies
Flags not defined for InterPort shim
Should we compress to 8 bytes? (Answer: No)

16. IntraPort Shim (FPX/SPC)
Same format for full-packet or header-only processing
Queue Identifier (QID) pins packet to a specific queue
Dedicated queue ranges for SPC traffic, LFS flows, etc.
Queue Length of the return queue for associated QID
Return SPC QIDs = transmit SPC queue + index
Only holds if the Re-Classify (RC) flag is not set

17. IntraPort Shim Flags
DP (Drop Packet): This flag is set when a packet should be dropped due to failed checksums, mismatched length fields, filters, or SPC directives.
RC (Re-Classify packet): This flag is set by the SPC when a packet requires re-classification in the FPX due to processing in the SPC.
NM (No Match): This flag is set when no filter or lookup entry exists for the packet. The packet is sent to the SPC for classification.
EX (Exception): This flag is set by ISAR when IP Options exist in the packet or the packet is not IPv4. The packet is sent to SPC for processing.
HO (Header-Only processing): This flag is set by CARL when only the packet header must be sent to the SPC for processing and the packet is more than 120B.
HR, FM, and TO flags are FPX-specific state
Must be retained with the packet (not modified by the SPC)

18. Classification & Route Lookup (CARL)
Route lookup, general filter match, exact filter match on ingress packets
64 priority levels
General filters have per-filter priority
Route lookup results have fixed priority (register config.)
Exact filter match results have fixed priority (register config.)
Exact filter match on egress packets
ISAR checks Protocol field, port numbers zeroed for non-TCP/UDP packets
Setup filter send ICMP packets directly to SPC… anything else?
General filter match on 5-tuple returns one exclusive and one non-exclusive filter
32-bit counter for matching packets
Exact filter match on 5-tuple returns OVINs and QIDs (2 for multicast packets), LFS rates, and flags
16-bit packet and 32-bit byte counters for matching packets
Route lookup on IPv4 destination address returns OVIN
16-bit packet counter for matching packets

19. Exact Filter Match

20. Exact Filter Match Entries (in SRAM)

21. Queue Manager (QMGR) DQ for switch-bound packets
DRR for link-bound packets
Dedicated SPC-bound packet queues
QID determines destination
0-127: to SPC
: from SPC
: reserved flow queues
: switch datagram queues
: line card datagram queues

22. Guidelines for SPC Processing
Must NOT modify the following fields
Lower 3-bits of flags and TOT Chunks
Packet and Second Chunk Pointers
May set the drop flag for any packet
May opt to drop a full packet locally
Must have consistent treatment of RC (Re-Classify) flag for all packets of a flow for uniform queuing treatment
If RC is not set, the QID currently in the shim will be used
RC bit must be set for LFS packets returned to FPX
Currently, ALL packets containing options bypass CARL and are sent to the SPC for processing

23. Guidelines for Header-Only Processing
FPX transmits first 120B of packet in three ATM cells
If filter specifies header-only processing, but the packet is less than 120B; the entire packet will be sent to the SPC
Must ALWAYS return one copy of header to FPX
Must NOT modify the length of header/data
May decide to set the drop flag
Must NOT modify the following fields
Lower 3-bits of flags and TTL Chunks
Packet and Second Chunk Pointers
Refer to previous slide regarding RC flag

24. Control Functions NID VCXT configuration
Register configuration (pass register number, operation, and value)
Read, write, reset all to default values
System counters and flags (pass counter number)
Read and reset all
Per-VCI input and output packet counters
For SPC traffic VCIs, full-packet and header-only counters
Per-VCI input and output control cell counters
Drop counters
Cell drops due to congestion at ISAR (no SDRAM, etc.)
Packet drops due to AAL5 checksum failures
Flags for IP Header checksum failure and IP and AAL5 length mismatch
Queue drop counters (SPC, LC, SW)

25. Control Functions (2) CARL SRAM updates
Maintain route database and exact match filter entries
Pass command, address, data
Read/write 32- or 36-bit words
Exact Filter Match hash table updates
Pass command, address (hash key), and data (pointer)
Maintain correct order of operations for adds/deletes
Adds: write entry in SRAM, update hash table entry
Deletes: update hash table entry, deallocate SRAM entry
General Filter Match
Pass filter table and priority resolution table entries
Consult document for details and data formats

26. Control Registers

27. References FPX Support for the MSR, Version 1.0 Team members
Detailed technical report documenting the hardware design
All interfaces, formats, control cells, OpCodes, etc.
Team members
Design still flexible in certain areas, however…
Coding has begun

28. “External” Shim Flag Definitions
AF (Active Flow): This flag is set when a lookup result returns a non-zero Stream Identifier (SID). The packet is sent to the SPC for active processing.
NR (No Route in table): This flag is set when no filter or lookup entry exists for the packet. The packet is sent to the SPC for classification.
OP (IP Options present): This flag is set when IP Options (other than Lightweight Flow Setup) exist in the packet header. The packet is sent to the SPC for options processing.
UK (Unknown packet type): This flag is set when a packet arrives that is not IPv4. The packet is sent to the SPC for processing.
RC (Re-Classify packet): This flag is set by the SPC when a packet requires re-classification in the FPX due to processing in the SPC.
HO (Header-Only processing): This flag is set when only the packet header must be sent to the SPC for processing.
HL (Header Lenth): This flag specifies how much packet data to send to the SPC for header only processing (0 = first 56 bytes of packet; 1 = first 116 bytes of packet). This flag is valid when only the packet header must be sent to the SPC for processing (HO = 1).
DP (Drop Packet): This flag is set when a packet should be dropped due to failed checksums, mismatched length fields, or filter directives.

29. “Internal” Shim Flag Definitions
IE (Ingress/Egress packet): This flag provides for implementation efficiency by allowing internal components to identify ingress and egress packets via a single bit compare. 0 = Ingress, 1 = Egress.
LP (LFS option present): Flag set to 1 when LFS is present.
LR (LFS rate request): Flag denotes requested rate type (0 = soft/transient, 1 = firm/constant). Flag is only valid when LP = 1.
SB (SPC-bound packet or header): Flag set to 1 when packet or header is to be sent to SPC.

30. Input Segmentation and Reassembly
Process IP traffic on 4 input sub-port VCs, 8 output port VCs, and 2 SPC VCs
Other traffic arriving from SW passes through control path
Reassemble IP packet
Check ATM HEC, IP Header checksum, and AAL5 checksum
Drop corrupt packets identified by checksum mismatches => set drop flag in shim
Compare IP Total Length and AAL5 length
Drop corrupt packets identified by length mismatches => set drop flag in shim
Check for IP Options
If not LFS Option, set option flag in shim for forwarding to SPC
Store payload in SDRAM
Maintain 14 contexts using local Packet Identifiers (PID) generated by Output Segmentation and Reassembly (OSAR)
Using BlockRAM as on-chip buffers, pass 60 byte “chunks” of packet to PSM
Packet Storage Manger (PSM) returns a packet pointer after first “chunk” of packet is transferred
Ingress traffic: insert 8-byte shim prior to IP header in first chunk
Write Input VIN based on packet arrival from SP0 – SP3
Forward header fields to CARL after entire packet is received
De-multiplex control cells to CCP

31. Output Segmentation and Reassembly
Dequeue packets based on packet pointers passed from Queue Manager
Select ingress/egress PSM based on shim flag
Create AAL5 frame for dequeued packets
Decrement IP TTL for egress packets
Update IP Header Checksum using incremental update algorithm
Generate AAL5 checksum
Ingress traffic: clear shim internal flags and SID to create InterPort Shim
Egress traffic: remove shim
Update rate field(s) for LFS option packets
Check for multicast count
Translate VCI of cells of outgoing AAL5 frames based on the shim fields and register set
Case 1: (Ingress traffic) Output VIN PN /= Local PN
Switch traffic VCI = Output VIN PN + Obase_VCI
SPC traffic (flags set) VCI = SPC VCI
Case 2: (Egress traffic) Output VIN PN = Local PN
Link traffic VCI = Output VIN SPI + Ibase_VCI
Multiplex control cell responses from CCP onto SW port

32. CARL Block Diagram

33. CARL Performance Line rate packet classification (3 Gb/s)
Assuming single-chunk (64B) packet I/O => 5.86M lookups/s
Route lookup = concatenate destination and source addresses (64-bits), perform longest-prefix match
Tree Bitmap w/ 4-bit stride => 16 nodes deep for exact match
Worst case memory accesses = 20
@ 75 MHz w/ 8 FIPL engines => 3.75M lookups/s
For typical 24-bit prefix lookups => 8.33M lookups/s
Returns Output VIN and SID
For multicast (binary copy), returns two Output VINs and one SID
SID2 = SID1 + 1
All results have fixed priority

34. CARL Performance (2)
General filter match = linear search over 64 filters using 5-tuple
@ 75 MHz => 1.17M table scans/s
Pipeline of 8 packet headers => 9.375M lookups/s
Provides for 37.5% overhead (62.5% efficiency)
Returns one exclusive filter match and up to three non-exclusive filter matches
Each filter has specific priority
May be relative to route lookup priority
Filter table under software control
Exact Filter Match = exact match over 5-tuple
Algorithm to be selected…
Must share SRAM capacity and bandwidth with Route Lookup
Dynamic filter updates based on implicit flow setup and tear-down for LFS support
Handled by on-chip hardware
Software filter insertion for reserved flows

35. SPC “Triggers” Packet exception cases Filter directives Others?
Not IPv4
Options other than LFS present
Filter directives
Exclusive/non-exclusive general filter or exact match filter specifies packet or packet header must be sent to SPC
Network monitoring, active processing, etc.
No matching filter or route
Will this ever happen?
Others?

36. Full-packet SPC Processing
SPC must have consistent treatment of RC flag for all packets of a given flow
Maintain uniform queuing treatment

37. Header Only SPC Processing
First cell contains reserved space for chunk pointers
12 bytes reserved regardless of HL
Single chunk = 56 bytes of packet
Requires 2 cells
Double chunk = 116 bytes of packet
Requires 3 cells
SPC may decide to drop packet by setting DP flag
SPC must NOT send copies of packet headers back to FPX
SPC must return header on the arrival VC
SPC_IN_VCI or SPC_EG_VCI

38. Task Assignments Overview (David) ISAR (Alex) OSAR (Yuhua)
PSM (Sarang)
CCP (David)
SRAM Interface (David)
SDRAM Interface (Sarang)
CARL (Alex & David)
General Filter Match (Alex)
Exact Filter Match (Alex/David)
Route Lookup (David)
I/O logic & priority resolution (Alex/David)
QM (Yuhua, Wen-Jing, Sarang)
DQ (Yuhua/Wen-Jing)
DRR (Yuhua/Wen-Jing/Sarang)
SPC/PFQ (Yuhua/Wen-Jing)
Document: /project/arl/fpx/papers/msr_design/msr_design_v1.pdf
Slides: /project/arl/fpx/papers/msr_design/talks/