1. Design of a Diversified Router: Line Card
John DeHart

2. Revision History 5/22/06 (JDD): 5/31/06 (JDD): 6/2/06 (JDD):
Updates to data passing from Block to Block.
Buffer descriptor stuff probably needs updating.
5/31/06 (JDD):
Clean up packet formats going from MR  LC_Egress
Add MN Shim to
6/2/06 (JDD):
Removed old slides that showed Table lookup stuff
Updated internal frame formats
Still need to revisit buffer descriptor stuff and block to block data formats.

3. Design Implementation
We will now look at how our model might be implemented.
Again, the primary focus will be on wired Ethernet as the access technology.
We will look at RX and TX separately.
First we will focus on the Substrate Router functionality in the LC.
As we do we will also show the packet formats as they traverse other parts of the Substrate Router.
Some of this will apply only to the Shared NP case.
For the sake of simpler diagrams we will reduce the IXP PE functional blocks as shown below:
Then we will look at the implementation of a common router framework in the IXP PE.
Parse
Lookup
Header Format
DeMux Rx Tx QM
Substrate RX
Substrate TX MR

4. Design Implementation
Packet arriving
On Port N LC … LC
Packet leaving
On Port M
Rxsubstrate MR
Txsubstrate
Switch
Switch …
IXP PE (Shared) RX TX

5. Substrate Link Configuration
RX Rules for determining type of Frame/Substrate Link:
P2P-DC is pre-configured on an interface by interface basis.
((EtherType = Substrate) AND (VLAN=VLAN0))  P2P-VLAN0 SL
VLAN0 is a predefined VLAN Id for use by the Substrate Network to connect peer substrate routers on a Multi-Access network
SL id = Senders Ethernet Address
Lookup MLI in SL Specific Table  MR:MI
((EtherType = Substrate) & (VLAN ≠ VLAN0)) OR
((EtherType = ARP) and (ProtoType = Substrate))  Multi-Access
MLI is unique across a Multi-Access Substrate Link
MLI Lookup in Multi-Access-SL Table  MR:MI
((EtherType = ARP) and (ProtoType ≠ Substrate)) OR
((EtherType ≠ Substrate) AND
(EtherType ≠ ARP))  Tunnel or Legacy
Substrate Link in a Tunnel (e.g. IPv4)
((EtherType = IP) AND (IP_Proto = Substrate))  Substrate Tunnel in IPv4
Legacy (e.g. IPv4, ARP):
((EtherType = IP) AND (IP_Proto = TCP))  Legacy IPv4
((EtherType = ARP) AND (ProtoType ≠ Substrate))  Legacy ARP
Etc.

6. Substrate Link Configuration
TX:
Substrate Link uniquely identified by MR:MI tuple
MR:MI  SL
MetaLink uniquely identified by MR:MI tuple
MR:MI  MLI
Hence:
MR:MI  SL:MLI
Different Header formats may be defined per SL
i.e. IPv4 Tunnel header vs. P2P-VLAN0 header
Located with lookup based on MLI
P2P-VLAN0
One or more MLI per MetaNet on the Multi-Access network
Multi-Access
One MLI per MetaNet on the Multi-Access network
Meta-Destination Address, to get Ethernet Address use ARP
Legacy:
Substrate Link in a Tunnel
IPv4
Gateway
Legacy to/from MetaNet

7. Ingress LC Input Frame: All SL Types
VLAN-tagged formats
Blue Shading: Determine SL Type
Black Outline: Key Fields from pkt
DstAddr (6B)
Type=IP (2B)
PAD (nB)
CRC (4B)
IP Payload
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B)
Legacy
SrcAddr (6B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=IP (2B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
P2P-VLAN0
Multi-Access
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol=Substrate (1B)
Hdr Cksum (2B)
Src Addr (4B)
Dst Addr (4B)
MLI (2B)
LEN (2B)
Meta Frame
PAD (nB)
CRC (4B)
P2P-DC
(Configured)
P2P-Tunnel

8. Ingress LC Input Frame: All SL Types
Non VLAN-tagged formats
Blue Shading: Determine SL Type
Black Outline: Key Fields from pkt
DstAddr (6B)
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type=IP (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
DstAddr (6B)
DstAddr (6B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
SrcAddr (6B)
TTL (1B)
Protocol=Substrate (1B)
SrcAddr (6B)
Protocol (1B)
Hdr Cksum (2B)
Type=Substrate (2B)
Hdr Cksum (2B)
Src Addr (4B)
Type=Substrate (2B)
MLI (2B)
Src Addr (4B)
MLI (2B)
LEN (2B)
Dst Addr (4B)
Dst Addr (4B)
LEN (2B)
Meta Frame
MLI (2B)
IP Payload
Meta Frame
LEN (2B)
Meta Frame
PAD (nB)
PAD (nB)
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
CRC (4B)
CRC (4B)
P2P-DC
(Configured)
P2P-Tunnel
Legacy
Multi-Access

9. Ingress LC Input Frame: All SL Types
Optional Extension: GRE formats
Blue Shading: Determine SL Type
Black Outline: Key Fields from pkt
DstAddr (6B)
SrcAddr (6B)
DstAddr (6B)
Type=802.1Q (2B)
SrcAddr (6B)
TCI ≠ VLAN0 (2B)
Type=IP (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
Protocol=GRE (1B)
Protocol=GRE (1B)
Hdr Cksum (2B)
Hdr Cksum (2B) `
Src Addr (4B)
Src Addr (4B)
Dst Addr (4B)
Dst Addr (4B)
Flags/recur/Ver (2B)
Flags/recur/Ver (2B) `
Type=Substrate (2B) `
Type=Substrate (2B)
Optional Fields (nB)
Optional Fields (nB)
MLI (2B) `
MLI (2B)
LEN (2B)
LEN (2B)
Meta Frame
MLI (2B)
Meta Frame
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
VLAN GRE
GRE

10. Egress LC Input Frame MR Switch … VLAN: Identifies MR
TxMI: With VLAN will identify SL and MLI
Src MPE:
LEN: Length in bytes of Meta Frame
Shim Flags: define what type of Shim Data is being given:
or perhaps what type is needed (maybe Substrate provides it for broadcast…)
Type (2b)
NhMnAddr: (01b)
We may need to use ARP to translate to MAC/Ethernet Address
MAC Address(10b)
Could be used for Broadcast/Multicast
MN Shim(11b) :
NhAddr field is used as MN Shim for PE to PE info
Things like:
why this frame is being sent on slow path
This type should NEVER show up at a LC but should only be for PE to PE. One of the PEs could be the control processor.
NULL (00b): No Next Hop Address given or needed.
Size (6b): Length of Shim Data field in Bytes
Shim Data: See flags above
Meta Frame:
Pad: Minimum Ethernet Frame size is 64 bytes
CRC: Ethernet Frame CRC LC MR
Substrate
Switch …
TxMI (2B)
TxMI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
Meta Frame
Shim Data (nB)
Shim Flags (2B)
Type=802.1Q (2B)
MR-Specific VLAN (2B)
Type=Substrate (2B)
LC Blade DstAddr (6B)
MR Blade SrcAddr (6B)
Src MPE (2B)
Src MPE (2B)
LEN (2B)
Shim Flags(2B)
Shim Data (nB)
Meta Frame

11. Substrate/MetaNet Model
To the Substrate, some Meta Links “Pass Through”
Pass through a Substrate Router without visiting a Meta Router
MetaLink 1
Substrate Link
MetaLink 2
MetaLink 3
MR_A
MR_A
MR_A MI MI MI MI MI MI
MR_B
MR_B
Pass Through
Meta Link MI MI MI MI
MR_C
MR_C
MR_C MI MI MI MI MI MI
Substrate
Router X
Substrate
Router Y
Substrate
Router Z

12. Pass-Through MetaLink
For Pass-Through ML, there is no support or need for NhAddr for the case where the SL leaving LC Y might be multi-access
Shim Flags = 0, Shim Data Length = 0
Allocate special set of MR/MI for use by Substrate when handling Pass-Through MetaLinks
TxMI == RxMI
Packet arriving
On Port N
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
TxMI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
Meta Frame
Shim Data (nB)
Shim Flags (2B)
Type=802.1Q (2B)
MR-Specific VLAN (2B)
Type=Substrate (2B)
LC Blade DstAddr (6B)
MR Blade SrcAddr (6B)
RxMI (2B)
TxMI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
Meta Frame
Shim Data (nB)
Shim Flags (2B)
Type=802.1Q (2B)
MR-Specific VLAN (2B)
Type=Substrate (2B)
LC Blade DstAddr (6B)
MR Blade SrcAddr (6B)
RxMI (2B)
Packet arriving
At LC Y
Packet arriving
On Port N
Of LC X
Ethernet
Switch LC X LC Y …

13. LC: Functional Blocks S W I T C H
Phy Int Rx
(2 ME)
Key
Extract
(2 ME)
Lookup
(2 ME)
Hdr
Format
(1 ME)
QM/Schd
(2 ME)
Switch Tx
(2 ME) S W I T C H
Phy Int Tx
(2 ME)
QM/Schd
(2 ME)
Hdr
Format
(1 ME)
Lookup
(2 ME)
Rate
Monitor
(1 ME)
Key
Extract
(1 ME)
Switch Rx
(2 ME)
ME counts are my first guesses for number needed.
For each block I’ll show:
Function:
What this block does.
Memory Accesses:
SRAM:
DRAM:
Buffer Descriptor Accesses:
Which fields in Buffer Descriptor the block needs to read and/or write.
Notes:
Additional thoughts about this block.
Next:
Analyze the SRAM Accesses and try to map them on to the available SRAM Channels.
Analyze the SRAM and DRAM accesses and calculate packet processing rates.

14. LC: Functional Blocks Ingress (Physical Interface  Switch):
PhyInt Rx
KeyExtractor
Lookup
Hdr Format
QM/Scheduler
Switch Tx
Egress (Switch  Physical Interface):
Switch Rx
This is only extracting the VLAN and the MI. Combine with previous or following block?
But it involves a DRAM Read so we probably want to leave it separate and use all 8 threads.
RateMonitor
Before or after the Lookup?
Is this different than what QM/Scheduler will/could do?
PhyInt Tx

15. LC: Buffer Descriptor
Hopefully we can use the same buffer descriptor for the LC and the CRF Processing Engine.
There might be some fields that are used on one and not on the other but that’s ok (MR_ID, TxMI, VLAN not needed on LC)
PE Buffer Descriptor:
LW0: buffer_next 32 bits Next Buffer Pointer (in a chain of buffers)
LW1: offset bits Offset to start of data in bytes
LW1: BufferSize 16 bits Length of data in the current buffer in bytes
LW2: reserved bits reserved/unused
LW2: reserved bits reserved/unused
LW2: free_list bits Freelist ID
LW2: packet_size 16 bits (Total packet size across multiple buffers)
LW3: MR_ID bits Meta Router ID
LW3: TxMI bits Transmit Meta Interface
LW4: VLAN bits VLAN
LW4: reserved 16 bits reserved/unused
LW5: reserved 32 bits reserved/unused
LW6: reserved 32 bits reserved/unused
LW7: packet_next 32 bits pointer to next packet (unused in cell mode)
Leave multi-buffer fields there as a template for the dedicated blade implementation of a jumbo-frame MR.
Also reduces changes to Rx, Tx, and QM and reduces potential problems.
So, far I haven’t found anything extra that we need on LC.
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size

16. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
RBUF
Buf Handle(32b)
CRC(32b)
Rx:
Function:
Coordinate transfer of packets from RBUF to DRAM
Memory Accesses:
SRAM:
Write Buffer Descriptor
Free List?
DRAM: Transfer from RBUF
Buffer Descriptor Accesses:
Write/Initialize: Buffer_Next, Buffer_Size, Offset, Free_List, Packet_Size
Monitoring:
Per Physical Interface
Pkt Counter
Byte Counter
Notes:
Buffer Handle:
contains the SRAM address of the buffer descriptor.
from the SRAM address of the descriptor we can calculate the DRAM address of the buffer data.
Offset of where packet starts should be a constant.
Port(8b)
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size

17. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Buf Handle(32b)
Buf Handle(32b)
CRC(32b)
Lookup Key(9B)
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size
Port(8b)
Key_Extract (2 Microengines):
Function:
Extracts lookup key based on type of frame that was received.
Peel ARP packets off and send to XScale
Compare CRC from Rx to CRC at end of Frame.
Drop if fail.
Memory Accesses:
DRAM:
Read as much of header as is necessary to extract key
May vary depending on type of Substrate Link
SRAM: None
Buffer Descriptor Accesses: None
Monitoring:
ARP Pkt Counter
Notes:
Calculates DRAM Address based on SRAM descriptor address in buffer handle and the Offset passed to it by RX.
Frame offset in buffer is a constant and does not need to be read from Buffer Descriptor

18. LC Ingress: Functional Blocks
Substrate Link Type:
Will be used as Database ID by Lookup Block
Lookup Keys:
SL Type (SL Type ID) (size)
P2P-DC(0x0) (24 bits)
SL(4b)
0100
Port
(4b)
MLI(16b)
P2P-Tunnel - IPv4(0x1) (72 bits)
SL(4b)
0100
Port
(4b)
EtherType (16b)
0x0800
IP SAddr (32b)
MLI (16b)
P2P-VLAN0(0x2) (72 bits)
SL(4b)
0100
Port
(4b)
Ethernet SAddr (48b)
MLI (16b)
MA(0x3) (24 bits)
SL(4b)
0100
Port
(4b)
MLI(16b)
Legacy IPv4(0x4) (24 bits)
SL(4b)
0100
Port
(4b)
EtherType (16b)
0x0800
Substrate Link Type
Physical Interface #

19. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Buf Handle(32b)
Buf Handle(32b)
Lookup Key(9B)
Lookup Result (10B)
Lookup:
Function:
Performs Lookup and passes result on to Hdr Format.
Memory Accesses:
DRAM: None
SRAM:
TCAM Lookup
Write Lookup command
Read Lookup Result (1-5 words from TCAM SRAM Controller or other SRAM Controller)
Buffer Descriptor Accesses: None
Monitoring:
Notes:
Lookup does no processing on the lookup result.
Need to decide how lookup result will be stored and retrieved.
See notes on TCAM for information about the issues involved.

20. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Lookup Result (8B)
Buf Handle(32b)
QID(24b)
Buf Handle(32b)
Size (16b)
Port(8b)
Hdr Format:
Function:
From lookup result:
re-writes headers in DRAM to make frame ready to transmit.
Extract QID to pass on to QM/Scheduler
Memory Accesses:
DRAM:
SRAM:
Read Descriptor and Re-Write Descriptor OR
Atomic Increment/Decrement some fields in Descriptor
Buffer Descriptor Accesses:
Update Size and Offset fields
Monitoring:
Notes:
Pass Size on to QM/Scheduler so it does not have to read buffer descriptor for Enqueue to update Q Length.
Buffer Offset should be a constant and should not need to be read from Buffer Descriptor
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size

21. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
QID(24b)
Buf Handle(32b)
Size (16b)
Port(8b)
Buf Handle(32b)
Port(8b)
QM/Scheduler (See Sailesh’s slides for more details)
Function:
Enqueue and Dequeue from queues
Scheduling algorithm
Drop Policy
Memory Accesses:
DRAM: None
SRAM:
Q-Array Reads and Writes
Scheduling Data Structure Reads and Writes
QLength Data Structure Reads and Writes
Dequeue: Read Buffer Descriptor to retrieve Packet Size
Buffer Descriptor Accesses: Read packet size
Monitoring:
Queue Lengths
Drops
Notes:

22. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Buf Handle(32b)
Port(8b)
TBUF
Switch TX:
Function:
Coordinate transfer of packets from DRAM to TBUF
Memory Accesses:
SRAM: Read Buffer Descriptor
DRAM: Transfer to TBUF
Buffer Descriptor Accesses:
Read Size and Offset
Monitoring:
Per Physical Interface
Pkt Counter
Byte Counter
Notes:
Calculate DRAM address based on SRAM Descriptor address in buffer handle

23. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Rate
Monitor
Key
Extract
Switch Rx S W I T C H
Buf Handle (32b)
RBUF
CRC (32b)
Rx:
Function:
Coordinate transfer of packets from RBUF to DRAM
Memory Accesses:
SRAM: Write Buffer Descriptor
DRAM: Transfer from RBUF
Buffer Descriptor Accesses:
Write/Initialize: Buffer_Next, Buffer_Size, Offset, Free_List, Packet_Size
Notes:
Buffer Handle:
contains the SRAM address of the buffer descriptor.
from the SRAM address of the descriptor we can calculate the DRAM address of the buffer data.
Passing the offset of where the packet starts in memory will save the next block from having to read the buffer descriptor.
Perhaps we should just pass the actual DRAM Buffer Pointer?
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size

24. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Rate
Monitor
Key
Extract
Switch Rx S W I T C H
Buf Handle(32b)
Buf Handle (32b)
VLAN (16b)
Lookup Key
TxMI (16b)
Key_Extract:
Function:
Extracts lookup key based on type of frame that was received.
Memory Accesses:
DRAM:
Read VLAN and TxMI from Frame
SRAM: None
Buffer Descriptor Accesses: None
Notes:
Calculates DRAM Address based on SRAM descriptor address in buffer handle and the Offset passed to it by RX.

25. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Rate
Monitor
Key
Extract
Switch Rx S W I T C H
Buf Handle(32b)
Buf Handle(32b)
VLAN (16b)
VLAN (16b)
TxMI (16b)
TxMI (16b)
Rate Monitor:
Function:
Ensures that MR/MI’s behave according to their Rate Specs.
Does this need to be a separate function from the QM/Scheduler?
Memory Accesses: Unknown at this point
DRAM:
SRAM:
Buffer Descriptor Accesses: Unknown at this point
Notes:

26. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Rate
Monitor
Key
Extract
Switch Rx S W I T C H
Buf Handle (32b)
Buf Handle(32b)
Lookup Result (20B)
VLAN (16b)
TxMI (16b)
Lookup:
Function:
Performs Lookup and passes result on to Hdr Format.
Memory Accesses:
DRAM: None
SRAM:
TCAM Lookup
Write Lookup command
Read Lookup Result (1-5 words from TCAM SRAM Controller or other SRAM Controller)
Buffer Descriptor Accesses: None
Notes:
Lookup does no processing on the lookup result.
Need to decide how lookup result will be stored and retrieved.
See notes on TCAM for information about the issues involved.

27. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Rate
Monitor
Key
Extract
Switch Rx S W I T C H
Buf Handle(32b)
Buf Handle (32b)
Size (16b)
Lookup Result (16B)
QID(24b)
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size
Port(8b)
Hdr Format:
Function:
From lookup result:
re-writes headers in DRAM to make frame ready to transmit.
Extract QID to pass on to QM/Scheduler
Memory Accesses:
DRAM:
SRAM:
Read Descriptor and Re-Write Descriptor OR
Atomic Increment/Decrement some fields in Descriptor
Buffer Descriptor Accesses:
Update Size and Offset fields
Notes:
Pass Size on to QM/Scheduler so it does not have to read buffer descriptor for Enqueue to update Q Length.

28. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Rate
Monitor
Key
Extract
Switch Rx S W I T C H
Buf Handle(32b)
Buf Handle(32b)
Port(8b)
Size (16b)
QID(24b)
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size
Port(8b)
QM/Scheduler (See Sailesh’s slides for more details)
Function:
Enqueue and Dequeue from queues
Scheduling algorithm
Drop Policy
Memory Accesses:
DRAM: None
SRAM:
Q-Array Reads and Writes
Scheduling Data Structure Reads and Writes
QLength Data Structure Reads and Writes
Dequeue: Read Buffer Descriptor to retrieve Packet Size
Buffer Descriptor Accesses: Read packet size
Notes:

29. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Rate
Monitor
Key
Extract
Switch Rx S W I T C H
TBUF
Buf Handle(32b)
Port(8b)
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size
Switch TX:
Function:
Coordinate transfer of packets from DRAM to TBUF
Memory Accesses:
SRAM: Read Buffer Descriptor
DRAM: Transfer to TBUF
Buffer Descriptor Accesses:
Read Size and Offset
Notes:
Calculate DRAM address based on SRAM Descriptor address in buffer handle

30. ML Loopback: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Loopback
Hdr
Re-Format
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Rate
Monitor
Key
Extract
Switch Rx
Loopback path should be able to re-use some of the blocks implemented for the LC
Loopback Hdr Re-Format:
Needs to be able to strip off the previous MR’s Header.
For Plain-IP  IPv4 MR  MR Y
When frame arrives at Loopback (between IPv4 MR and MR Y) it will still have IP Header which should be stripped off before frame is sent to MR Y.
Lookup Result should probably include a length field or a Buffer offset field that indicates where new Meta Frame should start.

31. LC: Notes on TCAM Lookups
See techX_Design_TCAM_Usage.ppt slides for notes on how the TCAM will be used by the Lookup Block

32. Extra
The next set of slides are for templates or extra information if needed

33. Text Slide Template

34. Image Slide Template

35. LC: SRAM Accesses S W I T C H SRAM Ch#A SRAM Ch#C SRAM Ch#D SRAM Ch#A
Ch#F
SRAM
Ch#H
SRAM
Ch#A
Phy Int Rx
(2 ME)
Key
Extract
(1 ME)
Lookup
(1 ME)
Hdr
Format
(1 ME)
QM/Schd
(1 ME)
Switch Tx
(2 ME) S W I T C H
SRAM Ch #A: SRAM Buffer Descriptors for Ingress (Phy Int  Switch)
SRAM Ch #B: SRAM Buffer Descriptors for Egress (Switch  Phy Int)
SRAM Ch #C: TCAM Access
SRAM Ch #D: Lookup Associated Data for Ingress (Phy Int  Switch)
SRAM Ch #E: Lookup Associated Data for Egress (Switch  Phy Int)
SRAM Ch #F: Q-Array for Ingress (Phy Int  Switch) QM
SRAM Ch #G: Q-Array for Egress (Switch  Phy Int) QM
SRAM Ch #H: QM Scheduling/Dequeue Data Structure
SRAM Ch #I: QM Scheduling/Dequeue Data Structure
Phy Int Tx
(1 ME)
QM/Schd
(1 ME)
Hdr
Format
(1 ME)
Lookup
(1 ME)
Rate
Monitor
(1 ME)
Key
Extract
(1 ME)
Switch Rx
(2 ME)
SRAM
Ch#B
SRAM
Ch#B
SRAM
Ch#G
SRAM
Ch#I
SRAM
Ch#B
SRAM
Ch#C
SRAM
Ch#E
SRAM
Ch#B

36. LC: Number of Per Pkt SRAM Accesses
Rx:
1 SRAM Buffer Descriptor WRITE (A,B)
Lookup
TCAM:
1 SRAM WRITE of TCAM Lookup Command (C, C)
1-8 SRAM READs of Results Mailbox (C, C)
1 READ if we are just retrieving an Index or Pointer
Up to 8 locations to read if we are retrieving the actual result data from Results Mailbox
Associated Data:
Read of up to 8 32-bit words in consectutive SRAM locations (D, E)
Hdr Format:
QM/Sched
Q-Array:
Enqueue: <= 1 SRAM Read Queue Descriptor (F,G)
Dequeue: <= 1 SRAM Read Queue Descriptor (F,G)
SRAM Buffer Descriptors:
Enqueue:
1 Buffer Descriptor Read (A,B)
1 Buffer Descriptor Write (A,B)
Dequeue
SRAM for Scheduling Data Structure
1 Write to add new queue which just became active (H,I)
1 Read (H,I)
1 Write (H,I)
Tx:
1 SRAM Buffer Descriptor READ (A,B)
Return Buffer Descriptor to Free List

37. LC: SRAM Accesses Buffer Descriptor Accesses (A,B): 3 Writes, 3 Reads
Rx: Write initial descriptor
Hdr Format: Write descriptor to update sizes, offsets, …
QM/Scheduler:
Enqueue Read Descriptor to get pkt size to do drop policy check
Enqueue Write Descriptor to update next pkt pointer (how does Q-Array do this for us?)
Dequeue Read Descriptor to get pkt size to do credit check?
Tx: Read Descriptor

38. LC: Number of Per Pkt SRAM Accesses
Channel A:
Ingress: W, R, W, R, R
Channel B:
Egress: W, R, W, R, R
Channel C:
Ingress: W, 1-8 R,
Egress: W, 1-8 R,
Channel D:
Ingress: <=8 R
Channel E:
Egress: <= 8 R
Channel F:
Ingress: R, W, R, W
Channel G
Egress: R, W, R, W
Channel H:
Ingress: W, R, W
Channel I:
Egress: W, R, W

39. LC: Functional Blocks . . . . . . S W I T C H QM/Schd (1 ME) Phy Int
Input
Hlpr
(1 ME)
QM/Schd
(1 ME)
Output
Hlpr
(1 ME)
Phy Int Rx
(2 ME)
Key
Extract
(1 ME)
Lookup
(1 ME)
Hdr
Format
(1 ME)
Switch Tx
(2 ME) S W I T C H
. . .
QM/Schd
(1 ME)
Scratch Rings
NN Ring
NN Ring
QM/
Sch
Hlpr
(1 ME)
QM/Schd
(1 ME)
QM/
Sch
Hlpr
(1 ME)
Phy Int Tx
(1 ME)
Hdr
Format
(1 ME)
Lookup
(1 ME)
Rate
Monitor
(1 ME)
Key
Extract
(1 ME)
Switch Rx
(2 ME)
. . .
QM/Schd
(1 ME)
Multiple QM/Scheds
MetaLinks assigned to QM/Sched to balance load based on their allocated BW
Input Helper ME demuxes from NN ring from Hdr Format to multiple Scratch Rings
Output Helper ME muxes from multiple Scratch Rings to NN to TX
Scratch Rings
NN Ring
NN Ring

40. OLD
The rest of these are old slides that should be deleted at some point.

41. LC: Notes on TCAM Lookups
The following slides show a way to use the TCAM for the lookups.
Slight adjustments might be desirable depending on:
Ease of doing operations on non-byte length bit fields
What we learn about methods for using the TCAM.
Field and Identifier sizes:
MN id: 32 bits
MI id: 16 bits (64K Meta Interfaces per Meta Router)
MLI: bits (64K Meta Links per Substrate Link)
Port: bits (256 Physical Interfaces per Line Card)
QID: bits (1M Queues per Queue Manager)
QM ID: 4 bits (16 Queue Managers per LC or PE.)
We probably can only support 4 QMs (2 bits)
(64 Q-Array Entries) / (16 CAM entries)  4 QMs per SRAM Controller.

42. LC: Notes on TCAM Lookups
Lookup Key size options:
32/36, 64/72, 128/144, 256/288, 512/576 (all in bits)
Lookup Result options:
Absolute Index: relative to beginning of TCAM array
Database Relative Index: relative to beginning of selected database
Memory Pointer Index: points to SRAM location of result data
Associated Data: 32, 64 or 128 bits of data associated with the lookup result.
Associated Data is stored in ZBT SRAM attached to TCAM.
TCAM Databases
How many to use?
1 for TX and 1 for RX?
1 for TX and 1 for each of the SL Types on Rx (5 types)?
Other…

43. LC: TCAM Lookup Keys on RX
P2P-DC(28b): SL_Type(4)/Port(8)/MLI(16)
P2P-Tunnel(IPv4)(76b): SL_Type(4)/Port(8)/EtherType(16)/IPSrc(32)/MLI(16)
P2P-VLAN0(76b): SL_Type(4)/Port(8)/EthSAddr(48)/MLI(16)
MA(28b): SL_Type(4)/Port(8)/MLI(16)
Legacy(28b): SL_Type(4)/Port(8)/EType(16)
Fields:
SL_Type (4b): Substrate Link Type
0000: DC
0001: IPv4 Tunnel
0010: VLAN0
0011: MA
0100: Legacy (non-substrate) without VLAN
0101: Legacy (non-substrate) with VLAN
Port(8b): Physical Interface number
MLI(16b): Meta Link Identifier
Ethertype(16b): Ethernet Header Type field
IPSrc(32b): IP Source Address
EthSAddr(48b): Ethernet Header Source Address

44. LC: TCAM Lookup Keys on RX
SL(4b)
0000
Port(8b)
MLI(16b)
PAD(100b)
SL (4b)
0001
Port (8b)
MLI (16b)
PAD (52b)
IP SAddr (32b)
EtherType (16b)
0x0800
SL (4b)
0010
Port (8b)
MLI (16b)
PAD (52b)
Ethernet SAddr (48b)
SL(4b)
0011
Port(8b)
MLI(16b)
PAD(100b)
SL (4b)
0100
Port (8b)
PAD (100b)
EtherType (16b)
0x0800

45. LC: TCAM Lookup Results on RX
Standard Fields (116b):
Type (4b)
0000: Default, not Pass Through, ignore MnFlags
1000: Pass Through with no extra lookup needed for NhAddr(nB), MnFlags(8b) should be 0x00
1001: Pass Through with extra lookup needed for NhAddr(nB)
VLAN (16b)
MI (16b)
Blade Eth Hdr (48b)
Only needs to have the DAddr.
SAddr can be configured and constant per LC
First EtherType can be constant: 802.1Q
Second EtherType can be constant: Substrate
QID (24b)
MnFlags(8b)
Can we say there will be no Pass Through Meta Links where one side will be on a Multi Access and hence might need a NhAddr field?
Pass Through Meta Link Fields:
NhAddr(nB)
Even a 32b IP NhAddr would not fit in a 128b TCAM Result.
The TCAM supports Multi-Hit Lookups: Need to better understand what this means.
Do a second lookup with same key except SL_Type=1111b, to retrieve the Next Hop bits?
Or do a second lookup with the following fields to retrieve the Next Hop bits?
SL_Type (4b) = 1111b
Port (8b)

46. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
The Lookup Result for TX will consist of several parts:
Lookup Result
Constant fields
Calculated fields
Fields that can be stored in Local Memory
Some of these are common across all SL Types
Other fields are specific to each SL Type
Common across all SL Types (100b):
From Result (44b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (24b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-DC Hdr:
P2P-MA Hdr:
P2P-VLAN0 Hdr
P2P-Tunnel Hdr for IPv4 Tunnel

47. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (100b):
From Result (52b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (24b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-DC Hdr (64b)
Constant (16b)
EtherType (16b) = Substrate
Calculated (0b)
From Result (48b)
Eth DA (48b)
Lookup Result Total (Common From Result + Specific From Result): 100 bits
Total (Common + Specific) : 164 bits

48. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (100b):
From Result (52b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (24b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-MA Hdr (64b) :
Constant (16b)
EtherType (16b) = Substrate
Calculated (0b)
From Result (48b)
Eth DA (48b)
Lookup Result Total (Common From Result + Specific From Result): 100 bits
Total (Common + Specific) : 164 bits

49. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (100b):
From Result (52b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (24b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-VLAN0 Hdr (80b):
Constant (16b)
EtherType1 (16b) = 802.1Q
EtherType2 (16b) = Substrate
Calculated (0b)
From Result (64b)
Eth DA (48b)
TCI (16b)
Lookup Result Total (Common From Result + Specific From Result): 116 bits
Total (Common + Specific) : 180 bits

50. LC: TCAM Lookups on TX Result (continued)
Common across all SL Types (100b):
From Result (52b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (24b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-Tunnel Hdr for IPv4 Tunnel (224b):
Constant (64b)
Eth Hdr EtherType (16b) = 0x0800
IPHdr Version(4b)/HLen(4b)/Tos(8b) (16b): All can be constant?
IP Hdr Flags(3b)/FragOff(13b) (16b) : what is our stance on Fragments? If never used, these are constants, if it is possible we will have to use them, then this has to be calculated. Either way, shouldn’t be in Result
IP Hdr TTL (8b): Constant
IP Hdr Proto (8b) = Substrate
Calculated (48b)
IP Hdr Len(16b) : needs to be calculated for each packet sent, so shouldn’t be in Result.
IP Hdr ID(16b): should be unique for each packet sent, so shouldn’t be in Result.
IP Hdr Checksum (16b): Needs to be calculated, so shouldn’t be in Result.
Local Memory (32b)
IP Hdr Src Addr (32b) : tied to port
From Result (80b)
Eth Hdr DA (48b)
IP Hdr Dst Addr (32b)
Lookup Result Total (Common From Result + Specific From Result): 132 bits
Total (Common + Specific) : 324 bits

51. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (100b):
From Result (52b)
SL Type(4b)
Port(8b)
MLI(16b)
Ignored for Legacy Traffic
QID (24b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
Legacy (IPv4) with VLAN Hdr (96b):
Constant (16b)
EtherType1 (16b) = 802.1Q
Calculated (0b)
ARP Lookup on NhAddr (48b)
Eth DA (48b)
From Result (32b)
EtherType2 (16b) = IPv4
TCI (16b)
Lookup Result Total (Common From Result + Specific From Result): 84 bits
Total (Common + Specific) : 196 bits

52. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (100b):
From Result (52b)
SL Type(4b)
Port(8b)
MLI(16b)
Ignored for Legacy Traffic
QID (24b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
Legacy (IPv4) without VLAN Hdr (64b):
Constant (0b)
Calculated (0b)
ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b)
Eth DA (48b)
From Result (16b)
EtherType (16b) = IPv4
Lookup Result Total (Common From Result + Specific From Result): 68 bits
Total (Common + Specific) : 164 bits

53. SUMMARY: LC: TCAM LookupsDC
Tunnel
VLAN0 MA
Legacyw/ vlan
Legacy w/o vlan RX
Key 28 76
Result
116 TX 32
100
132 84 68
Result in Loc. Mem 48 80
Combined
148
212
164
Rx Key Size: 128 bits
Rx Result Size: 128 bits
Tx Key Size: 32 bits
Tx Result Size: 256 bits (we should try to squeeze this in to 128 bits!)
What if QID went from 24 to 22 bits.
2 bits for QM id
20 bits for QID: 1M queues per QM instance, 4M Queues total.
And Port went from 8 bits to 6 bits (64 physical interfaces per Line Card)
If we can’t use the 128 bit Result size for Tx, we might as well include the Local Memory result in the TCAM lookup.
The Local Memory result was going to be the data that was only dependent on the Physical Interface id and was going to go into Local memory to try to keep the TCAM result below 128 bits.

54. LC Ingress: Functional Blocks
Substrate Link Type:
Will be used as Database ID by Lookup Block
Lookup Keys:
SL Type (SL Type ID) (size)
P2P-DC(0x0) (24 bits)
Port(8b)
MLI(16b)
P2P-Tunnel - IPv4(0x1) (72 bits)
Port (8b)
EtherType (16b)
0x0800
IP SAddr (32b)
MLI (16b)
P2P-VLAN0(0x2) (72 bits)
Port (8b)
Ethernet SAddr (48b)
MLI (16b)
MA(0x3) (24 bits)
Port(8b)
MLI(16b)
Legacy IPv4(0x4) (24 bits)
Port (8b)
EtherType (16b)
0x0800
Physical Interface #

55. LC: TCAM Lookup Keys on Ingress
P2P-DC
MLI(16b)
SL(4b)
0000
Port
(4b)
24 bits
IPv4 Tunnel
MLI (16b)
IP SAddr (32b)
EtherType (16b)
0x0800
SL(4b)
0001
Port
(4b)
72 bits
Legacy
Port
(4b)
EtherType (16b)
0x0800
SL(4b)
0010
24 bits
P2P-VLAN0
MLI(16b)
SL(4b)
0011
Port
(4b)
24 bits MA
SL(4b)
0100
Port
(4b)
Ethernet SAddr (48b)
MLI (16b)
72 bits
DstAddr (6B)
Type=IP (2B)
PAD (nB)
CRC (4B)
IP Payload
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B)
Legacy
Blue Shading: Determine SL Type
Black Outline: Key Fields from pkt
SrcAddr (6B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=IP (2B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
P2P-VLAN0
Multi-Access
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol=Substrate (1B)
Hdr Cksum (2B)
Src Addr (4B)
Dst Addr (4B)
MLI (2B)
LEN (2B)
Meta Frame
PAD (nB)
CRC (4B)
P2P-DC
(Configured)
P2P-Tunnel

56. LC Packet RX: All SL Types
VLAN-tagged formats
Type=IP (2B)
MLI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
Meta Frame
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol=Substrate (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
P2P-DC
P2P-Tunnel
P2P-VLAN0
Multi-Access

57. LC Packet RX: All SL Types
Non VLAN-tagged formats
Type=IP (2B)
MLI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
Meta Frame
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol=Substrate (1B)
Hdr Cksum (2B)
DstAddr (6B)
SrcAddr (6B)
DstAddr (6B)
Type=Substrate (2B)
Meta Frame
MLI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type=Substrate (2B)
MLI (2B)
LEN (2B)
Meta Frame
PAD (nB)
CRC (4B)
P2P-DC
P2P-Tunnel
Multi-Access

58. Tables and Lookups The following slides have some tables and lookups.
They are not meant as the only way to do things, just one way.
Some if not all of the tables and lookups will almost certainly be implemented using the TCAM.
For different types of frames we might need different types of headers.
For this a lookup result will probably indicate what format/template to use and some/all of the info to use.
The Point:
These slides are meant to show that we have the right information at the right place to do the needed lookup.
There is still some design work for the actual implementor(s) to do to get it all right and make it efficient.

59. LC Packet TX: All SL TypesMR
Substrate
Switch
VLANMNid
. . .
MnId
MnID …
MI (2B)
Blade-to-Blade
Ethernet
Header
(MR specific VLAN)
MnFlags(1B)
NhAddr (nB)
Arp Cache
. . .
MnID:MnAddr
DAddr
Key
Result
LEN (2B)
Meta Frame
TxMI (2B)
MnFlags (1B)
NhAddr (nB)
LEN (2B)
If SL_TYPE=Multi-Access Lookup(MnID, MnAddr)
Meta Frame
PAD (nB)
CRC (4B)
SL/ML Table
SL_Type
. . .
Port
MLI
Hdr Info/Templates
VLAN:TxMI  SL:MLI

60. LC Packet TX – P2P-DC . . . . . . SL/ML Table Recalculate
Arp Cache
. . .
MnID:MnAddr
DAddr
Key
Result
P2P-DC does not use ARP Cache.
DAddr is configured in the Hdr field of Per SL ML Tables
SL/ML Table
SL_Type
. . .
Port
MLI
Hdr Info/Templates
TxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
NhMnAddr (nB)
MnFlags (1B)
P2P-DC should not need to give NhMnAddr
MnFlags would indicate NULL
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
TCI (2B)
Type=Substrate (2B)
MLI (2B)
LEN (2B)
Meta Frame
Recalculate
PAD (nB)
CRC (4B)

61. LC Packet TX – P2P-Tunnel (IPv4)
Arp Cache
. . .
MnID:MnAddr
DAddr
Key
Result
P2P-Tunnel does not use ARP Cache.
DAddr is configured in the Hdr field of Per SL ML Tables
This may be an over-simplification.
The packet needs to be routed toward the other end of the Tunnel
SL/ML Table
SL_Type
. . .
Port
MLI
Hdr Info/Templates
Type=IP (2B)
MLI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
Meta Frame
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol=Substrate (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B)
This seems to assume that we have a static first hop for the IP Tunnel.
Is that reasonable?
Probably, Yes.
TxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
NhMnAddr (nB)
MnFlags (1B)
Recalculate

62. LC Packet TX – P2P-VLAN0 . . . . . . SL/ML Table Recalculate
Arp Cache
. . .
MnID:MnAddr
DAddr
Key
Result
Do we want the P2P-VLAN0 to use the ARP Cache?
Or is the DAddr configured in the Hdr fields of Per SL ML Tables?
SL/ML Table
SL_Type
. . .
Port
MLI
Hdr Info/Templates
TxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
NhMnAddr (nB)
MnFlags (1B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Recalculate

63. LC Packet TX – Multi-Access
Arp Cache
. . .
MnID:MnAddr
DAddr
Key
Result
Multi-Access does use ARP Cache.
If DAddr entry for MnAddr is missing, send packet up to XScale for it to perform ARP Request.
SL/ML Table
SL_Type
. . .
Port
MLI
Hdr Info/Templates
TxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
NhMnAddr (nB)
MnFlags (1B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Recalculate

64. LC Packet RX: P2P-DC Per SL ML Tables Packet arriving On Port N
There is one P2P-DC SL Table per Port (Physical Interface).
Blade info in tables provides info so we can fill in templates for appropriate header(s) to get frame to necessary blade.
QID is used on LC
VLANMR
RxMI
Blade
QID
Packet arriving
On Port N
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
(Port N P2P-DC Table)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
VLANMR
RxMI
Blade
QID
RxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR Specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
Do we want to include Sender’s MnAddr in Frame going to MR?
No.
But we may need to give it the MAC address.
Recalculate
MI (2B)
LEN (2B)
Meta Frame
Packet arriving
On Port N LC
Substrate MR
Switch …

65. LC Packet RX: P2P-Tunnel
Packet arriving
On Port N
Per SL ML Tables
There is one P2P-Tunnel SL Table per tunnel SL.
VLANMR
RxMI
Blade
QID
Type=IP (2B)
MLI (2B)
LEN (2B)
PAD (nB)
CRC (4B)
Meta Frame
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol=Substrate (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B)
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
Fct(PortN, IPv4,
IP Src_Addr)
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
(Port N IPv4 Tunnel(i) Table)
MI (2B)
LEN (2B)
Meta Frame
VLANMR
RxMI
Blade
QID
RxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR Specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
Do we want to include Sender’s MnAddr in Frame going to MR?
Probably not
IP Routers don’t
Recalculate
Packet arriving
On Port N LC
Substrate MR
Switch …

66. LC Packet RX: P2P-VLAN0 Per SL ML Tables Per SL ML Tables
There is one P2P-VLAN0 SL Table per SL.
Located based on SrcAddr of sender
VLANMR
RxMI
Blade
QID
Packet arriving
On Port N
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
(Port N P2P-VLAN0 Table)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
MI (2B)
LEN (2B)
Meta Frame
VLANMR
RxMI
Blade
QID
RxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR Specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
Do we want to include Sender’s MnAddr in Frame going to MR?
Recalculate
Packet arriving
On Port N LC
Substrate MR
Switch …

67. LC Packet RX: Multi-Access
Per SL ML Tables
There is one Multi-Access SL Table per port.
Contains the MLI entries for the Multi-Access SL
VLANMR
RxMI
Blade
QID
Packet arriving
On Port N
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
VLANMR
RxMI
Blade
QID
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
(Port N Multi-Access Table)
MI (2B)
LEN (2B)
Meta Frame
VLANMR
RxMI
Blade
QID
RxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR Specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
Do we want to include Sender’s MnAddr in Frame going to MR?
Recalculate
Packet arriving
On Port N LC
Substrate MR
Switch …

68. Protocol ≠ Substrate (1B)
Pure Legacy Traffic LC GE
IXP PE
Blade
IXP PE
Blade GP
Blade GP
Blade
Type=IP (2B)
PAD (nB)
CRC (4B)
IP Packet Body
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol ≠ Substrate (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B) LC GE
. . .
IPv4 MR
Switch
Blade
LC:
If ((EtherType=IPv4) AND (IP Proto ≠ Substrate))
then non-Tunnel Legacy
Send frame to pre-configured default IPv4 MR

69. LC Packet RX: Legacy IPv4
Packet arriving
On Port N
Per SL ML Tables
Fct(PortN)
There is one Legacy MR Table per Port.
Entries point to Legacy MR for the specified Protocol.
There is at most one Legacy MR for each legacy protocol
VLANMR
RxMI
Blade
QID
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
Type=IP (2B)
PAD (nB)
CRC (4B)
IP Packet Body
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol ≠ Substrate (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B)
VLANMR
RxMI
VLANMR
Blade
QID
(Port N P2P-DC Table)
RxMI
Blade
QID
Fct(IPv4)
VLANMR
RxMI
Blade
QID
(Port N Legacy MR Table)
VLANMR
RxMI
Blade
QID
RxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
MR Specific VLAN
PAD (nB)
CRC (4B)
Meta Frame
MI (2B)
LEN (2B)
Meta Frame
Recalculate
Recalculate
Packet arriving
On Port N LC
Substrate MR
Switch …

70. LC Packet TX: All SL TypesMR
Substrate
Switch
VLANMNid
. . .
MnId
MnID …
MI (2B)
TxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
(MR specific VLAN)
PAD (nB)
CRC (4B)
Meta Frame
NhAddr (nB)
MnFlags (1B)
MnFlags(1B)
NhAddr (nB)
Arp Cache
. . .
MnID:MnAddr
DAddr
Key
Result
LEN (2B)
Meta Frame
If SL_TYPE=Multi-Access Lookup(MnID, MnAddr)
SL/ML Table
SL_Type
. . .
Port
MLI
Hdr Info/Templates
VLAN:TxMI  SL:MLI

71. LC Packet TX – Legacy IPv4
SL/ML Table
SL_Type
. . .
Port
MLI
Hdr Info/Templates
Arp Cache
. . .
MnID:MnAddr
DAddr
Key
Result
Legacy IPv4 does use ARP Cache.
If DAddr entry for MnAddr is missing, send packet up to XScale for it to perform ARP Request.
Type=IP (2B)
PAD (nB)
CRC (4B)
IP Packet Body
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol ≠ Substrate (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B)
TxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
(MR specific VLAN)
PAD (nB)
CRC (4B)
Meta Frame
NhAddr (nB)
MnFlags (1B)
Recalculate

72. Protocol ≠ Substrate (1B)
Pure Legacy Traffic
Type=IP (2B)
PAD (nB)
CRC (4B)
IP Packet Body
Dst Addr (4B)
Src Addr (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol ≠ Substrate (1B)
Hdr Cksum (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
DstAddr (6B)
SrcAddr (6B) LC GE
IXP PE
Blade
IXP PE
Blade GP
Blade GP
Blade LC GE
. . .
IPv4 MR
Switch
Blade

73. Rx: Pass-Through MetaLink
Per SL ML Tables MR
. . .
RxMI
Blade
QID
(Port N P2P-DC Table)
Packet arriving
On Port N
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
RxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
(MR specific VLAN)
PAD (nB)
CRC (4B)
Meta Frame
NhAddr (nB)
MnFlags (1B)
Recalculate
Packet arriving
On Port N
Of LC X
Ethernet
Switch LC X LC Y …

74. Tx: Pass-Through MetaLink
Ethernet
Switch LC X LC Y
VLANMNid
. . .
MnId
MnID …
TxMI (2B)
LEN (2B)
Blade-to-Blade
Ethernet
Header
(MR specific VLAN)
PAD (nB)
CRC (4B)
Meta Frame
NhAddr (nB)
MnFlags (1B)
Arp Cache
. . .
MnID:MnAddr
DAddr
Key
Result
If SL_TYPE=Multi-Access Lookup(MnID, MnAddr)
SL/ML Table
SL_Type
. . .
Port
MLI
Hdr Info/Templates
VLAN:TxMI  SL:MLI
This is exactly like the common case shown for all SL Types.
After this things proceed as previously shown for the type of SL that is being used for the Transmit.

75. Design Implementation
We will now look at how our model might be implemented.
Again, the primary focus will be on wired Ethernet as the access technology.
We will look at RX and TX separately.
First we will focus on the Substrate Router functionality in the LC.
As we do we will also show the packet formats as they traverse other parts of the Substrate Router.
Some of this will apply only to the Shared NP case.
For the sake of simpler diagrams we will reduce the IXP PE functional blocks as shown below:
Then we will look at the implementation of a common router framework in the IXP PE.
Parse
Lookup
Header Format
DeMux Rx Tx QM
Substrate RX
Substrate TX MR