1. Design of a Diversified Router: Lookup Block
John DeHart

2. Lookup Block
Since the Lookup Block is focused on using the TCAM, first we need to understand what its capabilities and limitations are.

3. TCAM Software
Several software components exist, enough to be really confusing.
IDT Libraries:
Microengine Lookup Library (MLL)
IipcMll.uc
Lookup Management Library (LML)
NSE with Dual QDR Interfaces IDT75K234SLAM
NSE-QDR Data Plane Macro (DPM) API
Initialization Management and Search (IMS) Library
Intel Libraries:
TCAM Classifier Library
Microengine and XScale support for using TCAM.
Requires installation of MLL and LML.

4. TCAM Issues CAM Size: Segments: Data: 256K 72-bit entries
Organized into Segments.
Mask: 256K 72-bit entries
Segments:
Each Segment is 8k 72-bit entries
32 Segments
Segments are not shared between Databases.
Minimum database size is therefore 8K 72-bit entries.
Do databases wider than 72-bits use multiple parallel segments or do they use sequential entries in a segment to make up the long entries?
Sequential entries are used to comprise one longer entry.
36b DB has 16K entries per segment
72b DB has 8K entries per segment
144b DB has 4K entries per segment
288b DB has 2K entries per segment
576b DB has 1K entries per segment
Can a segment be dynamically added to a Database as it grows?
Yes, more on this feature in a future issue of the IDT User Manual…

5. TCAM Issues Number of Databases available: 16
Database Core Sizes: 36b, 72b, 144b, 288b, 576b
Coresize indicates how many CAM core entries are used per DB entry
Key/Entry size
Can be different for each Database.
<= Database Core Size
Key/Entry size is a multiple of 32 or one of the DB Core sizes
Key/Entry size indicates how many memory access cycles it will take to get the Key into the TCAM across the 16-bit wide QDR II SRAM interface.
Result Type
Absolute Index: relative to beginning of CAM
Database Relative Index: relative to beginning of Database
Memory Pointer: Translation based on configuration register values
TCAM Associated Data of width 32, 64 or 128 bits
Memory Constraints:
Actual Results can be stored in TCAM Associated DATA SRAM or IXP SRAM.
TCAM Associated Data
512K x 36 bit ZBT SRAM (4 bits of parity)
Supports 256K 64-bit Results
If used for Rx and Tx then 128K in each direction
Supports 128K 128-bit Results
If used for Rx and Tx then 64K in each direction
IXP QDR II SRAM Channel
2 x 2Mx18 (effective 4M x 18b)
4 times as much as the TCAM ZBT SRAM.
Supports 1024K 64-bit Results
If used for Rx and Tx then 512K in each direction
Supports 512K 128-bit Results
If used for Rx and Tx then 256K in each direction

6. TCAM Issues Lookup commands supported: Lookup (Direct)
Direct: Command is encoded in 2b Instruction field on Address bus
Indirect: Instruction field = 11b, Command encoded on Data bus.
Lookup (Direct)
1 DB, 1 Result
Multi-Hit Lookup (Direct)
1 DB, <= 8 Results
Simultaneous Multi-Database Lookup (Direct)
2 DB, 1 Result Each
Multi-Database Lookup (Indirect)
<= 8 DB, 1 Result Each
Simultaneous Multi-Database Lookup (Indirect)
Functionally same as Direct version but key presentation and DB selection are different.
Re-Issue Multi-Database Lookup (Indirect)
Search Key can be modified for each DB being searched.
First 32 bits of search key can be specified for each
Rest of key is same for each.

7. TCAM Issues Types of Databases: Priority: LC: NP/MR:
Longest Prefix Match (LPM):
Uses Ternary capability of TCAM
Exact Match (EM)
Does not use Ternary capability of TCAM
Best/Range Match: What we typically call General Match.
Priority:
Priority within a database is done by order of the entries.
Exact Match does not need priority within the database
LPM and Best/Range Match do use priority with the databases.
So, the order in which the entries are stored in these databases is important.
Changing priorities on existing entries may causes us some problems.
I don’t know yet how the software helps with this.
For LPM DBs we would want to group prefixes by length in the TCAM.
LC:
All Databases for RX and TX will be Exact Match
NP/MR:
Probably will support at least 3 databases:
Route Lookup (LPM)
Exact Match
General Match (Best/Range Match)

8. TCAM Issues Mask Registers Notes (mostly for reference)
When are these used?
I think we will need one of these for each database that is to be used in a Multi Database Lookup (MDL), where the database entries do not actually use all the bits in the corresponding core size.
For example: a 32-bit lookup would have a core size of 36 bits and so would need a GMR configured as 0xFFFFFFFF00 to mask off the low order 4 bits when it is used in a MDL where there are larger databases also being searched.
64 72-bit Global Mask Registers (GMR)
Can be combined for different database sizes
36-bit databases have access to 31 out of a total of 64 GMRs
A bit in the configuration for a database selects which half of the GMRs can be used
A field in each lookup command selects which specific GMR is to be used with the lookup key.
Value of 0x1F (31) is used in command to indicate no GMR is to be used. Hence, 36-bit lookups cannot use all 32 GMRs in its half.
72-bit databases have access to 31 out of a total of 64 GMRs
Value of 0x1F (31) is used in command to indicate no GMR is to be used. Hence, 72-bit lookups cannot use all 32 GMRs in its half.
144-bit lookups have 32 GMRs available to it.
288-bit lookups have 16 GMRs available to it.
576-bit lookups have 8 GMRs available to it.
Each lookup command can have one GMR associated with it.

9. TCAM Issues Performance Impacts IXP/TCAM Interface
16 bits wide
200 MHz QDR
Effective 32bits per clock tick
So getting Key in and Result out is 32bits/tick
Example: 128b Key would take 4 ticks to get clocked into TCAM.
Example: 128b results from ZBT SRAM via TCAM
Key Size (200MHz, 16 bit Interface for commands)
Rates are max PER LA-1 Interface, up to CAM Core max rate
32b (1W): 50M Lookups/sec (CAM Core constraint)
36b (2W): 50M Lookups/sec (CAM Core constraint)
64b (2W):100M Lookups/sec (Interface constraint)
72b (3W): 67M Lookups/sec (Interface constraint)
128b (4W): 50M Lookups/sec (Interface constraint)

10. TCAM Issues Performance Impacts (continued) Result Type
Index or Pointer Types:
Key Size: Max CAM Core lookup rate
36b: 50M/sec
72b: 100M/sec
144b: 100M/sec
288b: 50M/sec
576b: 25M/sec
Associated Data: CAM Core rates:
These rates are Total across both LA-1 Interfaces
Key Size: (32b Result Rate, 64b Result Rate, 128b Result Rate)
36b: , 50, 25
72b: 100, 50, 25
144b: 100, 50, 25

11. TCAM Performace Tables

12. TCAM Performace Graphs

13. TCAM Performace Graphs

14. LC: Notes on TCAM Lookups
The following slides show a way to use the TCAM for the lookups.
Slight adjustments might be desirable depending on:
Ease of doing operations on non-byte length bit fields
What we learn about methods for using the TCAM.
Field and Identifier sizes:
MR id: 16 bits
Is this local or global?
Seems like it just needs to be local and can be == VLAN
The VLAN is defined as 16 bits, but our switch blade supports 4K (12 bits)
MI id: 16 bits (64K Meta Interfaces per Meta Router)
Do we need to support 64K MIs?
MLI: bits (64K Meta Links per Substrate Link)
Port: bits (256 Physical Interfaces per Line Card)
QID: bits (128K Queues per Queue Manager)
QM ID: 3 bits (8 Queue Managers per LC or PE.)
We probably can only support 4 QMs (2 bits)
(64 Q-Array Entries) / (16 CAM entries)  4 QMs per SRAM Controller.

15. LC: Notes on TCAM Lookups
Lookup Key size options:
Key Sz 32 36 64 72 96
128
144
160
192
224
256
288
320
352
384
416
448
480
512
544
576
Core Sz
Ticks 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Lookup Result options:
Absolute Index: relative to beginning of TCAM array
Database Relative Index: relative to beginning of selected database
Memory Pointer Index: points to SRAM location of result data
Associated Data: 32, 64 or 128 bits of data associated with the lookup result.
Associated Data is stored in ZBT SRAM attached to TCAM.

16. Lookup Block Requirements
General:
Number of Packets per second required to handle?
Line Rate: 10Gb/s
Assume an average IP Packet Size of 200 Bytes (1600 bits)
(10Gb/s)/(1600 bits/pkt) = 6.25 Mpkt/s
Ethernet Header of 14 Bytes
Average Frame Size of 214 Bytes (1712 bits)
(10Gb/s)/(1712 bits/pkt) = Mpkt/s
Ethernet Inter-Frame Spacing: 96 bits
Average Frame Size with Inter-Frame Spacing: 1808 bits
(10Gb/s)/(1808 bits/pkt) = 5.53 Mpkt/s
Number of Lookups per second required?
Total Number of Keys to be supported?
Size of Keys?
Size of Results?

17. Lookup Block Requirements
LC Specific:
Number of Lookups per second required:
1 Rx and 1 Tx lookup required per packet
If we assume 6.25 MPkts/sec then we need 12.5 M Lookups/sec.
All lookups will be Exact Match.
Rx:
# Databases: 5
0000: DC
0001: IPv4 Tunnel
0010: VLAN0
0011: MA (with or without VLAN)
0100: Legacy (non-substrate) with or without VLAN
Key: 72b
Result: 64b
Fields necessary to get frame to the blade that will handle it next:
MR Blade: For Meta Router processing.
LC Blade: For Tx of Pass-Through MetaLink
Tx:
# Databases: 1
1000: all TX lookups use one database
Key: 32b
Result: 128b

18. SUMMARY: LC: TCAM LookupsDC
Tunnel
VLAN0 MA w/
vlan
w/o
Legacy w/ vlan
Legacy w/o vlan RX
Key 24 72
Result 60 TX 32 96
128
112 64 48 80
Rx Key: 72 bits
Rx Result Size: 60 bits
Tx Key Size: 32 bits
Tx Result Size: 128 bits
We need to watch out for the Tx Result for Tunnels. If we introduce anything else we want in there then we go beyond the 128 bits supportable through the TCAM’s Associated memory.

19. LC: TCAM Lookup Keys on RX
P2P-DC
Port(8b)
MLI(16b)
24 bits
IPv4 Tunnel
Port (8b)
EtherType (16b)
0x0800
IP SAddr (32b)
MLI (16b)
72 bits MA
Port (8b)
Ethernet SAddr (48b)
MLI (16b)
72 bits
P2P-VLAN0
Port(8b)
MLI(16b)
24 bits
Legacy
Port (8b)
EtherType (16b)
0x0800
24 bits
DstAddr (6B)
Blue Shading: Determine SL Type
Black Outline: Key Fields from pkt
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
TCI ≠ VLAN0 (2B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=IP (2B)
Multi-Access
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
P2P-VLAN0
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
Protocol=Substrate (1B)
Protocol (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)
Dst Addr (4B)
Dst Addr (4B)
MLI (2B)
IP Payload
LEN (2B)
Meta Frame
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
P2P-DC
Configured
P2P-Tunnel
Legacy

20. LC: TCAM Lookup Results on RX
We need the Ethernet Header fields to get the frame to the blade that is to process it next.
Ethernet header fields that are constants can be configured and do not need to be in the Lookup Result.
Ethernet Header fields:
DAddr: Depends on MetaLink
SAddr: Can be constant and configured per LC
EtherType1: Can be a constant: 802.1Q
VLAN(TCI): Different for each MR
EtherType2: Can be a constant: Substrate
Result (60b)
VLAN (16b)
MI (16b)
DAddr (8b)
We can control the MAC Addresses of the Blades, so lets say that 40 of the 48 bits of DAddr are constant across all blades and 8 bits are assigned and stored in the Lookup Result.
Will 8 bits be enough to support multiple chasses?
We could go up to 12 bits and still use 64bit Associated Data
QID (20b)

21. LC: TCAM Lookup Results on RX
Notes on Pass Through MetaLinks and Multi-Access SLs
When going MR  LC-TX the MR may provide a Next Hop MN Address for the LC to use to map to a MAC address.
This is particularly used when the destination Substrate Link is Multi-Access and there may be multiple MAC addresses used on the same Multi-Access MetaLink.
When going LC-RX  LC-TX for a pass through MetaLink, do we need to do something similar?
We could think that this would arise when a MetaNet has hosts on a multi-access network but the first Substrate Router that these hosts have access to does not have a MR for that MN.
However, I contend that if there is no MR on that access SR, then there is nothing there to discriminate between the multiple MN address on the single MetaLink.
So it will have to be configured as multiple P2P-VLAN0 Substrate Links, one per host in that MetaNet.

22. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
The Lookup Result for TX will consist of several parts:
Lookup Result
Constant fields
Calculated fields
Fields that can be stored in Local Memory
Some of these are common across all SL Types
Other fields are specific to each SL Type
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-DC Hdr:
P2P-MA Hdr:
P2P-VLAN0 Hdr
P2P-Tunnel Hdr for IPv4 Tunnel

23. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-DC Hdr (64b)
Constant (16b)
EtherType (16b) = Substrate
Calculated (0b)
Eth DA (48b)
Lookup Result Total (Common From Result + Specific From Result): 96 bits
Total (Common + Specific) : 160 bits

24. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
MA Hdr (64b) :
Constant (16b)
EtherType (16b) = Substrate
Calculated (0b)
ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b)
Eth DA (48b)
From Result (0b)
Lookup Result Total (Common From Result + Specific From Result): 48 bits
Total (Common + Specific) : 160 bits

25. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
MA with VLAN Hdr (96b) :
Constant (32b)
EtherType1 (16b) = 802.1Q
EtherType2 (16b) = Substrate
Calculated (0b)
ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b)
Eth DA (48b)
From Result (16b)
TCI (16b)
Lookup Result Total (Common From Result + Specific From Result): 64 bits
Total (Common + Specific) : 192 bits

26. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-VLAN0 Hdr (80b):
Constant (16b)
EtherType1 (16b) = 802.1Q
EtherType2 (16b) = Substrate
Calculated (0b)
From Result (64b)
Eth DA (48b)
TCI (16b)
Lookup Result Total (Common From Result + Specific From Result): 112 bits
Total (Common + Specific) : 176 bits

27. LC: TCAM Lookups on TX Result (continued)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
P2P-Tunnel Hdr for IPv4 Tunnel (224b):
Constant (64b)
Eth Hdr EtherType (16b) = 0x0800
IPHdr Version(4b)/HLen(4b)/Tos(8b) (16b): All can be constant?
IP Hdr Flags(3b)/FragOff(13b) (16b) : what is our stance on Fragments? If never used, these are constants, if it is possible we will have to use them, then this has to be calculated. Either way, shouldn’t be in Result
IP Hdr TTL (8b): ???
IP Hdr Proto (8b) = Substrate
Calculated (48b)
IP Hdr Len(16b) : needs to be calculated for each packet sent, so shouldn’t be in Result.
IP Hdr ID(16b): should be unique for each packet sent, so shouldn’t be in Result.
IP Hdr Checksum (16b): Needs to be calculated, so shouldn’t be in Result.
Local Memory (32b)
IP Hdr Src Addr (32b) : tied to physical interface (10 entry table)
From Result (80b)
Eth Hdr DA (48b)
IP Hdr Dst Addr (32b)
Lookup Result Total (Common From Result + Specific From Result): 128 bits
Total (Common + Specific) : 320 bits

28. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
Ignored for Legacy Traffic
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
Legacy (IPv4) with VLAN Hdr (96b):
Constant (16b)
EtherType1 (16b) = 802.1Q
Calculated (0b)
ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b)
Eth DA (48b)
From Result (32b)
EtherType2 (16b) = IPv4
TCI (16b)
Lookup Result Total (Common From Result + Specific From Result): 80 bits
Total (Common + Specific) : 192 bits

29. LC: TCAM Lookups on TX Key: Result VLAN(16b) TxMI(16b)
Common across all SL Types (96b):
From Result (48b)
SL Type(4b)
Port(8b)
MLI(16b)
Ignored for Legacy Traffic
QID (20b)
Local Memory (48b)
Eth Hdr SA (48b) : tied to Port
SL Type Specific Headers
Legacy (IPv4) without VLAN Hdr (64b):
Constant (0b)
Calculated (0b)
ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b)
Eth DA (48b)
From Result (16b)
EtherType (16b) = IPv4
Lookup Result Total (Common From Result + Specific From Result): 64 bits
Total (Common + Specific) : 160 bits

30. Lookup Block Requirements
Shared NP Lookup Engine specific:
Number of Lookups per second required:
1 lookup required per packet
5Gb/s per NP on a blade
If we assume 6.25 MPkts/sec for 10Gb/s then for 5Gb/s would be MPkt/s
We would want M Lookups/sec per LA-1 Interface, total of 6.25 M Lookups/sec for the TCAM Core.
Number of MRs to be supported?
Will each get its own database? No. This would limit it to 16 which is not enough.
How many keys will each MR be limited to?
How much of Result can be MR-specific?
How much of Key can be MR-specific?
How are masks to be supported?
Mask core is same size as Data core. One mask per Entry
Global Mask Registers also available for masking key to match size of Entry during Multi Database Lookups where the multiple databases have different sizes.
How will multiple hits across databases be supported?
How will priorities be supported?
Priorities within a database are purely by the order of the keys.
For example, in a GM filter table if Keys 4 and 7 both match, Key 4 is selected.
Priorities across databases will have to be included in the Entries
Do we need support for non-exclusive (make a copy) filters?
Later?
How are GM with fields with ranges supported?
The IDT libraries support this by adding multiple entries, each with its own mask, to the DB to cover the range of the field.

31. IPv4 MR Lookup Entry Examples
Route Lookup: Longest Prefix Match
Entry (64b):
MR ID (16b)
MI/Port (16b)
DAddr (32b)
Mask: (32 + Prefix length) high order bits set to 1
GM Match Lookup
Entry (148b):
If we could shorten the MR/MI fields by a total of 4 bits this would fit in a 144 bit core size.
SAddr (32b)
Protocol (8b)
Sport(16b)
Dport(16b)
TCP_Flags (12b)
Mask: Completely general, user defined.
EM Match Lookup
Entry (136b):
Mask: 136 high order bits set to 1

32. IPv4 MR Lookup Databases
How many databases to use?
Three Options:
3: a separate DB for each
2: one DB for GM and one for RL and EM
1: RL, GM and EM all in one DB
3 Databases:
Means we would use Multi Database Lookup (MDL) command
More efficient use of CAM core entries as each DB could be sized closer to its Entry size
Guaranteed at least one Result from each Database if an existing match existed in each database.
2 Databases:
We could use MDL command
Guaranteed one Result from GM and one from either EM or RL but not both!
Order is important:
EM filters would all go first in EM/RL DB, with full masks.
At most one entry would match
EM filters would always be higher priority than Routes.
This seems ok. That is what is typically implied by an Exact match filter.
If no EM filter match, we would get the best RL match.
RL entries would be sorted by prefix length so first match was the longest.
We could use two separate commands: Lookup or MHL for GM and MHL for EM/RL
Guaranteed at least one Result from each {GM,EM,RL} if an existing match existed in each.
Price: Two lookups per packet.
1 Database:
Use Multi Hit Lookup (MHL) command
Efficient use of the CAM core entries is a potential problem.
Would not be as bad, if we could get the GM filters down to 144 bits by making the MR/MI fields a combined 4 bits shorter.
Order is important
EM Filters first
GM Filters second
Ordering within GM filters defines priority to a certain extent
With Result of 64 bit AD, we can get back at most 4 Results
So, we could end up with 3 GM filter results and then arbitrate priority between them based on a priority field in Result.
RL Entries last
EM and GM always take priority over RL.
Seems ok, that is typically what is implied by the filters.
Priority field in Results could be used to arbitrate between matched EM and GM filters

33. IPv4 MR Lookup Key Examples
Order matters:
Same Key will be applied to all Databases(MDL)
Multi-Database Lookup (MDL)
Each Database will use the number of bits it was configured for, starting at the MSB.
DAddr field needs to be first
TCP_Flags field needs to be last
Route Lookup: Longest Prefix Match
Key (64b):
MR ID (16b)
MI/Port (16b)
DAddr (32b)
GM Match Lookup: Best/Range Match
Key (148b):
SAddr (32b)
Protocol (8b)
Sport(16b)
Dport(16b)
TCP_Flags (12b)
MASK/Ranges
How will we handle
Masks for Addr fields
Ranges for Port fields
Wildcard for Protocol field
EM Match Lookup: Exact Match
Key (136b):
DAddr (32b)
SAddr (32b)
Sport (16b)
TCP_Flags (12b)
Protocol (8b)
DPort (16b)
MI/Port (16b)
MR ID (16b)

34. IPv4 MR Lookup Key Examples
Lookup Key: 148 bits out of 5 32-bit words transmitted with Lookup command.
MR ID (16b)
MI/Port (16b)
DAddr(32b)
SAddr(32b)
DPort(16b)
SPort(16b)
Proto
(8b)
TCP_Flags
(12b)
Pad
(12b) W1 W2 W3 W4 W5
MDL
Mask
Mask
Data
Core Entries
for EM DB
136 bits
Core Size: 144 bits
GMR=0xFFFFFFFFF
0xFFFFFFFFF
0xFFFFFFF00
Mask
Data
Core Entries
For RL DB
64 bits
Core Size: 72 bits
GMR=0xFFFFFFFFF
0xFFFFFFF00
GMR
Data
GMR
GMR
Core Entries
for GM DB
148 bits
Core Size: 288 bits
GMR=0xFFFFFFFFF
0xFFFFFFFFF
0xF 0x

35. IPv4 MR Lookup Result Examples
QID(20b)
Output MI (16b)
Port(8b)
Priority(8b): range 0-255
Drop(1b)
Total of 53 bits
Each Database will have 64 bits of associated data, of which we will use the low order 53 bits.
And for MDL lookups only 61 of 64 bits of Associated Data is returned, so we are in luck. Whew.
RTN=1b
ADSP=1b
AD WIDTH=01b
Results Mailbox:
D: Done (1b): set to 1 when ALL searches are completed.
H: Hit (1b): set to 1 if the search was successful and result is valid, 0 otherwise
MH: MHit (1b): set to 1 if search was successful and there were additional hits in database.
R: Reserved bits.
AD (Associated Data): 61 of the 64 bits of Associated Data from the Associated Data ZBT SRAM attached to TCAM. D
AD[60:32] (1st Search) H MH
Results Mailbox
AD[31: 0] (1st Search)
AD[60:32] (2nd Search) R H MH
AD[31: 0] (2nd Search)
AD[60:32] (3rd Search) R H MH
AD[31: 0] 3rd Search)
Not Used
Not Used

36. IPv4 MR Lookup Result Examples
PROBLEM: one problem with using the MDL is that we do not get an index back with our results.
Hence we will not be able to easily increment a counter based on the lookup result.
Multi Database Lookup (MDL) cmd returns one and only one of the following per database searched:
Absolute Index
Translated Index
Associated Data
Lookup cmd returns
Absolute Index followed by Associated Data
Associated Data followed by Absolute Index
Options 1:
three back-to-back Lookup cmds, one for each database.
Each result would provide us with the
result data
Index
This requires 3 times the number of lookups in the TCAM.
Option 2:
Use MDL but have result include the index and not the data.
We would then have to have the result data in a separate memory that we would then read.
Option 3:
Add a Stats index to Result. Keep table of counters and increment based on index.
Result (for MDL, really only 61 out of the 64 bits available):
QID(20b)
Output MI (16b)
Port(8b)
Priority(6b): range 0-63
Drop(1b)
Stats Index (10b): 1024 indices (could even be 1024 indices per database)
Total of 61 bits
We can trade off bits between stats index and other fields (Priority, Port, …) to get more or less.
Note: If we increase the result size to 128 bits, then we CANNOT get 3 results back in the results mailbox.
I like Option 3 and will continue along those lines.

37. IPv4 MR Database Core Sizes
Route Database
Core Size: 72b
Entries per Segment: 8K
Number of Entries needed per route: 1
Number of Routes per Segment: 8K
GM Database
Core Size: 288b
Entries per Segment: 2K
Number of Entries needed per filter: dependent on filter
Number of Filters per Segment: <= 2K
EM Database
Core Size: 144b
Entries per Segment: 4K
Number of Entries needed per filter: 1
Number of Filters per Segment: 4K
Configuration used in our FPX-based Router:
32 GM filters
10K Ingress EM Filters
10K Egress EM Filters
~ 40K Route Entries
Configuration needed to achieve approximately the same numbers:
5 Segments for Route Database
1 Segment for GM Database
5 Segments for EM Database
Total of 11 Segments (out of a total of 32 in TCAM)

38. IPv4 MR Database AD Usage
Each Segment can be configured with a Base Address and a result size for calculating an address into the Associated Data.
The Associated Data is stored in a 512K x 36 bit ZBT SRAM
Using 64bit Results will give us 256K slots in the AD SRAM.
48K Route DB Entries
<= 2K GM DB Entries
20K EM DB Entries
Max Total of 70K Results needed.
Plenty of room in the AD for the IPv4 MR Results

39. MPLS Lookup Key Example
MPLS uses a 20 bit Label
Key (52 bits):
MR ID (16b)
MI/Port (16b)
MPLS_Label (20b)
Use an Exact Match Database
MPLS Label Database
Core Size: 72b
Entries per Segment: 8K
Number of Entries needed per label: 1
Number of Labels per Segment: 8K
We will not need the MDL command for MPLS MR, just the Lookup command which allows us to get the whole 64bit (or 32bit or 128bit)
Drop Bit:
Does MPLS need a Drop Bit? Perhaps it would use a Miss as the same thing as Drop. That is, the fact that a label is not entered in the Database is an indication that frames using that label should be dropped.
Or perhaps we can use a Port=0xFF as the Drop flag and only have a port range of
Either way, lets assume we do not need a drop bit and can use a 64 bit result.
What will MPLS Label Lookup Result look like?
New Label (20 bits)
QID(20b)
Output MI (16b)
Port(8b)
Total of 64 bits
But this gives us no Stats Index.

40. MPLS Lookup Result Examples
Note: No Stats Index included in Results
Option 1:
Use the Absolute Index returned with the Associated Data, to locate a counter to increment
Option 2:
Increase the result size to 128 bits.
This would also allow us to put the Drop bit in.
Result:
New Label (20b)
QID(20b)
Output MI (16b)
Port(8b)
Stats Index(10b)
Drop bit (1b)
Total of 75 bits
Lets go with Option 2.

41. MPLS Lookup Result Examples
New Label (20b)
QID(20b)
Output MI (16b)
Port(8b)
Stats Index(10b)
Drop bit (1b)
Total of 75 bits
DB will use 128 bits of associated data and will return the Associated Data followed by the Absolute Index.
We don’t need the Absolute Index and we don’t need the top 3 bits of the AD. With this ordering we just have to read the first 4 words on the results Mailbox instead of 5.
RTN=1b
ADSP=1b
AD WIDTH=10b
Results Mailbox:
D: Done (1b): set to 1 when search is completed.
H: Hit (1b): set to 1 if the search was successful and result is valid, 0 otherwise
MH: MHit (1b): set to 1 if search was successful and there were additional hits in database.
Absolute Index: Index offset from beginning of TCAM array.
Associated Data: 128 bits of Associated Data from the Associated Data ZBT SRAM attached to TCAM. D H MH
Reserved(7b)
Associated Data [124:96]
Associated Data [95:64]
Associated Data [63:32]
Associated Data [31:0]
Results Mailbox
Absolute Index(21:0)
Not Used
Not Used
Not Used

42. TCAM Latency Data
IDT App Note AN-459: “IDT75K72234 Instruction Latency”
Provides data and examples for latency calculations
Assumptions:
The NSE has no instructions in the pipeline
The measured instruction is the only instruction issued
The other NSE interfaces are idle.

43. TCAM Latency Data Example from IDT App Note AN-459
288-bit Lookup (288/32 = 9 QDR cycles to transfer 288 bit key)
32-bits of Associated ZBT SRAM data is returned
QDR clock frequency is 200 MHz
System clock frequency is 200 MHz
Description
Clock Domain
Freq (MHz)
# of clocks
Time (ns)
QDR xfer time
QDR
200 9 45
Instruction FIFO 2 10
Synchronizer
System 3 15
Execution Latency 32
160
Re-Synchronizer 1 5
Total Time 47
235
Execution Latency numbers are from Table 1 of AN-459

44. TCAM Latency Data Parameters for our LC RX Lookup
128-bit Lookup (128/32 = 4 QDR cycles to transfer 128 bit key)
128-bits of Associated ZBT SRAM data is returned
QDR clock frequency is 200 MHz
System clock frequency is 200 MHz
Core Blocking (CB) Delay: 8 cycles
Backend Latency: 14 cycles
Description
Clock Domain
Freq (MHz)
# of clocks
Time (ns)
QDR xfer time
QDR
200 4 20
Instruction FIFO 2 10
Synchronizer
System 3 15
Execution Latency 36
180
Re-Synchronizer 1 5
Total Time 46
230
Core Blocking 8 40
Total Time + CB 54
270

45. TCAM Latency Data Parameters for possible LC RX Lookup
72-bit Lookup (72/32 = 3 QDR cycles to transfer 72 bit key)
128-bits of Associated ZBT SRAM data is returned
QDR clock frequency is 200 MHz
System clock frequency is 200 MHz
Core Blocking (CB) Delay: 8 cycles
Backend Latency: 14 cycles
Description
Clock Domain
Freq (MHz)
# of clocks
Time (ns)
QDR xfer time
QDR
200 3 15
Instruction FIFO 2 10
Synchronizer
System
Execution Latency 36
180
Re-Synchronizer 1 5
Total Time 45
225
Core Blocking 8 40
Total Time + CB 53
265

46. TCAM Latency Data Parameters for possible LC RX Lookup
72-bit Lookup (72/32 = 3 QDR cycles to transfer 72 bit key)
64-bits of Associated ZBT SRAM data is returned
QDR clock frequency is 200 MHz
System clock frequency is 200 MHz
Core Blocking (CB) Delay: 4 cycles
Backend Latency: 10 cycles
Description
Clock Domain
Freq (MHz)
# of clocks
Time (ns)
QDR xfer time
QDR
200 3 15
Instruction FIFO 2 10
Synchronizer
System
Execution Latency 32
160
Re-Synchronizer 1 5
Total Time 41
205
Core Blocking 4 20
Total Time + CB 45
225

47. TCAM Latency Data Parameters for possible LC RX Multi Hit Lookup
72-bit Lookup (72/32 = 3 QDR cycles to transfer 72 bit key)
64-bits of Associated ZBT SRAM data is returned
QDR clock frequency is 200 MHz
System clock frequency is 200 MHz
Core Blocking (CB) Delay: 6 cycles
Backend Latency: 10 cycles
Description
Clock Domain
Freq (MHz)
# of clocks
Time (ns)
QDR xfer time
QDR
200 3 15
Instruction FIFO 2 10
Synchronizer
System
Execution Latency 32
160
Re-Synchronizer 1 5
Total Time 41
205
Core Blocking 6 30
Total Time + CB 47
235

48. IDT Data Plane Macro API
IDT provides a set of macros for creating commands to the TCAM
Here are some that will be particularly useful to us:
IipcMakeBase()
IipcMakeDirectInstruction()
IipcMakeIndirectInstruction()
IipcMakeSubInstruction()
IipcQDRDelay()
IipcNPUDelay()
IipcSignalXXXDone()
IipcSramRead()
IipcFormContextFromCsrMeCtx()
IipcMake36BitLookupInstruction()
IipcSramReadResultStatus()

49. IDT Data Plane Macro API
IipcSramRead() …
.sig sramread
.reg $t00, $t01, $t02, $t03, $t04, $t05, $t06, $t07
.xfer_order_rd $t00 $t01 $t02 $t03 $t04 $t05 $t06 $t07
.set $t00, $t01, $t02, $t03, $t04, $t05, $t06, $t07
; Read the first word, re-try the reading until Done bit is set
SRAMREAD#:
sram[ read, $t00, base, 0x0, 1 ], ctx_swap[ sramread ]
br_bclr[ $t00, 31, SRAMREAD# ]
; Now, read again to make sure the index is set correctly when the Done bit is cleared
sram[ read, $t00, base, 0x0, amount ], ctx_swap[ sramread ]
; Manually set the Done bit on the returned result
immed_w0[ result[0], 0x0000 ]
immed_w1[ result[0], 0x8000 ]
alu[ result[0], result[0], or, $t00 ]
; Transfer the rest of results depending on read amount
IipcSramReadResultStatus()
#macro IipcSramReadResultStatus( result, amount, base, regnum )
.begin
.reg lowword
.set lowword
immed_w0[ lowword, ( 0x1 << 5 | regnum << 2 ) ]
immed_w1[ lowword, 0 ]
;SRAMREAD#:
sram[ read, result, base, lowword, amount ], ctx_swap[ sramread ]
; sram[ read, result, base, 0x0, amount ], ctx_swap[ sramread ]
; br_bclr[ result, 31, SRAMREAD# ]
.end
#endm

50. IDT Data Plane Macro API: Example
Example of a Direct Lookup command
; channel = 0
; select = 0
; context = 4
IipcMakeBase[ iipc_base_word, 0x0, IIPC_DOUBLE_32MB_SELECT_0, 0x4 ]
; instruction = 0 (IIPC_LOOKUP)
; gmask = 31 (no GMR)
; database = 0 (database 0)
IipcMakeDirectInstruction[ iipc_command_word, IIPC_LOOKUP, IIPC_NO_GMASK, 0x0 ]
; Create the 72 bit search key to lookup in the write transfer registers.
; key = 0xBBBBBBBBBBBBBBBBBB
immed_w0[ data, 0xBBBB ]
immed_w1[ data, 0xBBBB ]
alu[ $w00, --, B, data ]
alu[ $w01, --, B, data ]
immed_w0[ data, 0x0000 ]
immed_w1[ data, 0xBB00 ]
alu[ $w02, --, B, data ]
; perform QDR write, sending command to NSE
sram[ write, $w00, iipc_base_word, iipc_command_word, 3 ], ctx_swap[sramwrite]
; compute approximate delay time and set signal
IipcSignalLookupDone[ 3, 72 ]
; perform a read that does not return until the NSE results mailbox Done bit is set
IipcSramRead[ $r00, 1, iipc_base_word ]

51. Lookup Block . CTX-0 QDR SRAM NSE Interface TCAM CTX-1 . . . CTX-2
SRAM Controller
CTX-1
In NN Ring
Out NN Ring
. . .
CTX-2
. . .
KEY
KEY
KEY
KEY
Result
Result
Result
Result .
CTX-7

52. Lookup Block CTX-x In NN !Empty Out NN !Full NSE Result Read Done
Input NN Ring is not empty, something for us to read.
Out NN !Full
Output NN Ring is not full, space for us to write to it.
NSE Result Read Done
Our Read of Results Mailbox has completed.
Next_Ctx Start
Our turn to read from the In NN Ring.
Next_Ctx Done
Our turn to write to the Out NN Ring.
Next_Ctx
Start
Next_Ctx
Done
CTX-x
NSE Result
Read Done
In NN
!Empty
Out NN
!Full
Next_Ctx
Start
Next_Ctx
Done

53. LC-RX Lookup Block Pseudocode
Initialization Phase
Start
Wait on ((Next_Ctx Start signal) and (In NN Ring !Empty signal))
Phase 1
Assert Next_Ctx Start signal
Read In NN Ring(buf_handle, Key, SL_Type)
Extract Key of correct size based on SL_Type
Build Lookup command (IDT Macro)
Send Lookup command to NSE (sram[] write instruction)
Calculate Delay Time and Wait (IDT Macro)
Phase 2
Issue Command to Read Result from Results Mailbox (IDT Macro)
Macro does Wait for Result and checks Done bit and continues to read until Done bit is set.
Wait for ((Next_Ctx Done signal) and (Out NN Ring !Full signal))
Phase 3
Assert Next_Ctx Done signal
Send (buf_handle, Result) to Out NN Ring
GoTo Phase 1

54. LC-TX Lookup Block Pseudocode
Initialization Phase
Start
Wait on ((Next_Ctx Start signal) and (In NN Ring !Empty signal))
Phase 1
Assert Next_Ctx Start signal
Read In NN Ring(buf_handle, Offset, Key)
Extract Key of VLAN and TxMI
Build Lookup command (IDT Macro)
Send Lookup command to NSE (sram[] write instruction)
Calculate Delay Time and Wait (IDT Macro)
Phase 2
Issue command to Read Result from Results Mailbox (IDT Macro)
Macro does Wait for Result and checks Done bit and continues to read until Done bit is set.
Wait for ((Next_Ctx Done signal) and (Out NN Ring !Full signal))
Phase 3
Assert Next_Ctx Done signal
Send (buf_handle, Offset, Result) to Out NN Ring
Wait on Next_Ctx Start signal
GoTo Phase 1

55. IPv4 MR Lookup Block Pseudocode
Initialization Phase
Initialize GMR_GM, GMR_EM, GMR_RL for each type/size of lookup database/key
Start
Wait on ((Next_Ctx Start signal) and (In NN Ring !Empty signal))
Phase 1
Assert Next_Ctx Start signal
Read In NN Ring(buf_handle, dram_ptr(?), Offset, MR_Id, Input_MI, MR_Mem_Ptr, Key)
Extract Key of 124 bits
Build Multi Database Lookup (MDL) command using Key and GMR_GM, GMR_EM, GMR_RL
IDT Macro
Send MDL command to NSE (sram[] write instruction)
Calculate Delay Time and Wait (IDT Macro)
Phase 2
Issue command to Read Result from Results Mailbox (IDT Macro)
Macro does Wait for Result and checks Done bit and continues to read until Done bit is set.
If no hits, zero Out_Result and then set Miss bit
Else compare priority of hits and select highest priority and write into Out_Result
Wait for ((Next_Ctx Done signal) and (Out NN Ring !Full signal))
Phase 3
Assert Next_Ctx Done signal
Send (buf_handle, dram_ptr(?), Offset, MR_Id, MR_Mem_ptr, Out_Result) to Out NN Ring
GoTo Phase 1

56. Intel/IDT NSE/TCAM Tools
Libraries
Macros
Etc.

57. Extra
The next set of slides are for templates or extra information if needed

58. Text Slide Template

59. Image Slide Template

60. OLD
The rest of these are old slides that should be deleted at some point.

61. START: LC-RX With MA SL on a PT ML
This set of slides is with the assumption that we DO need to support a MA SL on one of a Pass Through MetaLink

62. SUMMARY: LC: TCAM LookupsDC
Tunnel
VLAN0 MA w/
vlan
w/o
Legacy w/ vlan
Legacy w/o vlan RX
Key 24 72
Result
61* TX 32 96
128
112 64 48 80
Rx Key: 72 bits
Rx Result Size: 61* bits
61 bits if there is no need for NhAddr
Multiple results if there is a need for NhAddr, see earlier discussion.
Tx Key Size: 32 bits
Tx Result Size: 128 bits
We need to watch out for the Tx Result for Tunnels. If we introduce anything else we want in there then we go beyond the 128 bits supportable through the TCAM’s Associated memory.

63. LC: TCAM Lookup Keys on RX
P2P-DC
Port(8b)
MLI(16b)
24 bits
IPv4 Tunnel
Port (8b)
EtherType (16b)
0x0800
IP SAddr (32b)
MLI (16b)
72 bits MA
Port (8b)
Ethernet SAddr (48b)
MLI (16b)
72 bits
P2P-VLAN0
Port(8b)
MLI(16b)
24 bits
Legacy
Port (8b)
EtherType (16b)
0x0800
24 bits
DstAddr (6B)
Blue Shading: Determine SL Type
Black Outline: Key Fields from pkt
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
Type=802.1Q (2B)
TCI ≠ VLAN0 (2B)
TCI ≠ VLAN0 (2B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=IP (2B)
Multi-Access
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI≠VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
MLI (2B)
LEN (2B)
Meta Frame
TCI=VLAN0 (2B)
Type=Substrate (2B)
PAD (nB)
CRC (4B)
DstAddr (6B)
SrcAddr (6B)
P2P-VLAN0
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
Protocol=Substrate (1B)
Protocol (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)
Dst Addr (4B)
Dst Addr (4B)
MLI (2B)
IP Payload
LEN (2B)
Meta Frame
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
P2P-DC
Configured
P2P-Tunnel
Legacy

64. LC: TCAM Lookup Results on RX
Fields we definitely need:
VLAN (16b)
We could probably drop this to 12b since our switch is supposed to only support 4K VLANs but we might want to leave this open to switches supporting larger numbers of VLANs.
MI (16b)
Blade Eth Hdr (8b)
Only needs to have the DAddr.
The rest can be constant and configured:
SAddr can be configured and constant per LC
First EtherType can be constant: 802.1Q
Second EtherType can be constant: Substrate
We can control the MAC Addresses of the Blades, so lets say that 40 of the 48 bits of DAddr are fixed and 8 bits are assigned and stored in the Lookup Result.
Will 8 bits be enough to support multiple chasses?
QID (20b)
Possible Fields to handle pass-through Meta Links:
MnFlags(8b): (see next slide)
NhAddr(60b): (If needed, see next slide)

65. LC: TCAM Lookup Results on RX
Can we say there will be no Pass Through Meta Links where one side will be on a Multi Access and hence might need a NhAddr field?
If so then we can drop the MnFlags and NhAddr fields from result
Result size then becomes: 60b
Pass Through Meta Link Fields:
MnFlags(8b)
NhAddr(nB)
We have 60 bits left over that could be used for NhAddr
If we need more, options:
Do a second lookup with the following fields to retrieve the Next Hop bits from another Database for NH Bits?
Port (8b)
VLAN (16b)
MI (16b)
If the size indicated in the MnFlags is greater than 60 bits, use put an Memory Pointer in the bits after the MnFlags and lookup the NhAddr in memory.
We could even include 32 bits of the NhAddr in the original TCAM result and still have a memory pointer to get the rest. This might cut down on memory access time needed to retrieve the NhAddr.

66. Rx Handling NhAddr for Pass-Through MLs
For Tx to handle Multi-Access Substrate Links we need to provide an ARP capability on behalf of the MetaNets.
For MR  LC-TX: to do this we allow the MRs to give the LCs a MN Next Hop Address that the LC-TX will do a lookup on to see if we have a MAC address for and if not, issue an ARP to request it.
But, the LC does not know anything in general about MN Addresses or their size.
Included in the MnFlags fields is the size so Tx can handle variable sizes.
The MnFlags field has a 6 bit length in bytes, so it can be up to 63 bytes
But what is a good limit?
IPv4 is 32 bits
IPv6 is 128 bits
But IPv6 uses the Neighbor Discovery Protocol to do what ARP does, and NDP does a lot more.
We have a lot to learn about IPv6…
For Pass Through MetaLinks where one side is MA we need the LC-RX to have in its lookup result the Next Hop Address so the Tx can do the translation to MAC address in the same way.
Can we assume that this won’t happen?
Actually it is probably fairly likely to happen on access links. Perhaps the MetaNet does not have a MR located at the first Substrate Router but its access MetaLinks pass-through
If we find that we have to store the MnNhAddr in the Rx Result, I think we can make it variable sized by using Multi-Hit Lookups (MHL) and storing the same Key multiple times with different results for each one.
Each subsequent result would be the additional bits to concatenate on to the MnNhAddr.
In the Results Mailbox for a MHL, we can have at most
250 bits in bit AD Results or 244 bits in 4 64-bit AD Results or 232 bits in 8 32-bit AD Results
So, lets assume that in general we don’t need the NhAddr in the Rx entry. (next slide…)

67. Rx Handling NhAddr for Pass-Through MLs
Rx Result (61 bits)
VLAN (16b)
MI (16b)
Blade Eth Hdr (8b)
QID (20b)
Continuation bit (1b):
0: no need for MnFlags and NhAddr, MHL should report MHit of 0
1: MnFlags and NhAddr contained in subsequent results, MHL should report MHit of 0.
This then leaves us 3 more possible results to the MHL, giving us (3*61)-8 = 155 bits of Next Hop Address.
We have to be careful when writing the entries for Rx:
We write the main and subsequent entries in the right order.
I assume that the order of the results is based on the priority of the entries in the TCAM which is determined by order.
The continuation bit is set correctly.
Actually this is just a safety check. If we always do a MHL then the MH bit in the result should tell us if there are subsequent results.

68. END: LC-RX With MA SL on a PT ML
This set of slides is with the assumption that we DO need to support a MA SL on one of a Pass Through MetaLink