1. An NP-Based Router for the Open Network Lab Meeting Notes
John DeHart

2. Notes from 4/10/07 (HF Review)
Add Per block generic Error counters
HF should test for invalid port# and drop if found
Developers WorkBench has a code coverage utility
Can we read Local Memory from the XScale?
And if so, can we do it only while the ME is stopped or can we do it while it is running?
Counters for Rx and Tx per port:
Per port Rx counters in Mux
Per port Tx counters in HF
Rx and Tx will add sending Drop counters to Stats block
PLC drop if Plugin Ring is full
Plugins should wait if write ring is full
Rx will drop if SRAM ring to Mux is full
Are there MSF counters for dropped packets?
If so, can we access them?

3. Notes from 3/27/07
Schedule?
Bank1: check SRAM access bw with QM and Rings
QID: Use 1-5 for port numbers in top 3 bits so that a DG qid of 0 will not result in a full QID=0
NSP: Should we allow users to assign traffic to a DG queue
Do we need the Drop bit or is it sufficient to have a copy_vector of 0 to indicate a drop?
Define how/where sampling filters are implemented and how the sampling happens.
QIDs and Plugins are still fuzzy.
Should there be a restricted set of QIDs that the Plugins can allocate that the RLI and XScale are not allowed to allocate?
Can we put Output Port in data going to the XScale?
Buffer Offset: Should it point to the start of the ethernet header or start of the IP pkt?
Length fields in the buffer descriptor are they ethernet frame length or IP packet length?
Add Buffer Chaining slides.

4. Notes from 3/23/07 ONL Control Mtg
Using the same QID for all copies of a multicast does not work
The QM does not partition QIDs across ports
Do we need to support Datagram queues?
Yes, we will support 64 datagram queues per port
We will use the same Hash Function as in the NSP router
For testing purposes, can users assign the datagram queues to filters/routes?
Proposed partitioning of QIDs:
QID[15:13]: Port Number 0-4
QID[12]: Reserved by RLI vs XScale
0: RLI Reserved
1: XScale Reserved
QID[11: 0] : per port queues
4096 RLI reserved queues per port
4032 XScale reserved queues per port
64 datagram queues per port
yyy xx xxxx: Datagram queues for port <yyy>
IDT XScale software kernel memory issues still need to be resolved.

5. Notes from 3/13/07
Ethertype needs to be written to buffer descriptor so HF can get it.
Who tags non-IP pkts for being sent to XScale: Parse?
We will not be supporting ethernet headers with:
VLANs
LLC/SNAP encapsulation
Add In Plugin in data going to a Plugin:
In Plugin: tells the last plugin that had the packet
Plugins can write to other Plugins sram rings
Support for XScale participation in an IP multicast
For use with Control protocols?
Add In Port values for Plugin and XScale generated packets
Include both In Port and In Plugin to lookup key?
Should flag bits also go to Plugins
For users to use our IP MCast support they must abide by the IP multicast addressing rules.
i.e. Copy will do the translation of IP MCast DAddr to Ethernet MCast DAddr so if the IP DA does not conform it can’t do it.

6. Issues and Questions Upgrade to IXA SDK 4.3.1 Which Rx to use?
Techx/Development/IXP_SDK_4.3/{cd1,cd2,4-3-1_update}
Which Rx to use?
Intel Rx from IXA SDK is our base for further work
Which Tx to use?
Three options:
Our current Tx (Intel IXA SDK 4.0, Radisys modifications, WU Modifications)
Among other changes, we removed some code that supported buffer chaining.
Radisys Tx based on SDK 4.0 – we would need to re-do our modifications
This would get the buffer chaining code back if we need/want it
Intel IXA SDK Tx – no Radisys modifications, we would need to re-do our modifications
How will we write L2 Headers?
When there are >1 copies:
For a copy going to the QM, Copy allocates a buffer and buffer descriptor for the L2 Header
Copy writes the DAddr into the buffer descriptor
Options:
HF writes full L2 header to DRAM buffer and Tx initiates the transfer from DRAM to TBUF
Unicast: to packet DRAM buffer
Multicast: to prepended header DRAM buffer
HF/Tx writes/reads L2 header to/from Scratch ring and Tx writes it directly to TBUF
When there is only one copy of the packet:
No extra buffer and buffer descriptor are allocated
L2 header is given to Tx in same way as it is for the >1 copy case
How should Exceptions be handled?
TTL Expired
IP Options present
No Route
C vs. uc
Probably any new blocks should be written in C
Existing code (Rx, Tx, QM, Stats) can remain as uc. Freelist Mgr?
Continued on next slide…

7. Issues and Questions Need to add Global counters
See ONLStats.ppt
Global counters:
Per port Rx and Tx: Pkt and Byte counters
Drop counters:
Rx (out of buffers)
Parse (malformed IP header/pkt)
QM (queue overflow)
Plugin
XScale
Copy (lookup result has Drop bit set, lookup MISS?)
Tx (internal buffer overflow)
What is our performance target?
5-port Router, full link rates.
How should SRAM banks be allocated?
How many packets should be able to be resident in system at any given time?
How many queues do we need to support?
Etc.
How will lookups be structured?
One operation across multiple DBs vs. multiple operations each on one DB
Will results be stored in Associated Data SRAM or in one of our SRAM banks?
Can we use SRAM Bank0 and still get the throughput we want?
Multicast:
Are we defining how an ONL user should implement multicast?
Or are we just trying to provide some mechanisms to allow ONL users to experiment with multicast?
Do we need to allow a Unicast lookup with one copy going out and one copy going to a plugin?
If so, this would use the NH_MAC field and the copy vector field
Continued on next slide…

8. Issues and Questions Plugins: XScale:
Can they send pkts directly to the QM instead of always going back through Parse/Lookup/Copy?
Use of NN rings between Plugins to do plugin chaining
Plugins should be able to write to Stats module ring also to utilize stats counters as they want.
XScale:
Can it send pkts directly to the QM instead of always going through Parse/Lookup/Copy path?
ARP request and reply?
What else will it do besides handling ARP?
Do we need to guarantee in-order delivery of packets for a flow that triggers an ARP operation?
Re-injected packet may be behind a recently arrived packet for same flow.
What is the format of our Buffer Descriptor:
Add Reference Count (4 bits)
Add MAC DAddr (48 bits)
Does the Packet Size or Offset ever change once written?
Yes, Plugins can change the packet size and offset.
Other?
Continued on next slide…

9. Issues and Questions How will we manage the Free list?
Support for Multicast (ref count in buf desc) makes reclaiming buffers a little trickier.
Scratch ring to Separate ME
Do we want it to batch requests?
Read 5 or 10 from the scratch ring at once, compare the buffer handles and accumulate
Depending on queue, copies of packets will go out close in time to one another…
But vast majority of packets will be unicast so no accumulation will be possible.
Or, use the CAM to accumulate 16 buffer handles
Evict unicast or done multicast from CAM and actually free descriptor
Do we want to put Freelist Mgr ME just ahead of Rx and use NN ring into Rx to feed buffer descriptors when we can?
We might be able to have Mux and Freelist Mgr share an ME (4 threads per or something)
Modify dl_buf_drop()
Performance assumptions of blocks that do drops may have to be changed if we add an SRAM operation to a drop
It will also add a context swap. The drop code will need to do a test_and_decr, wait for the result (i.e. context swap) and then depending on the result perhaps do the drop.
Note: test_and_decr SRAM atomic operation returns pre-modified value
Usage Scenarios:
It would be good to document some typical ONL usage examples.
This might just be extracting some stuff from existing ONL documentation and class projects.
Ken?
It might also be good to document a JST dream sequence for an ONL experiment
Oh my, what I have done now…
Do we need to worry about balancing MEs across the two clusters?
QM and Lookup are probably heaviest SRAM users
Rx and Tx are probably heaviest DRAM users.
Plugins need to be in neighboring MEs
QM and HF need to be in neighboring MEs

10. Notes Need a reference count for multicast. (in buffer descriptor)
How to handle freeing buffer for multicast packet?
Drops can take place in the following blocks:
Parse QM
Plugin Tx
Mux  Parse
Reclassify bit
For traffic that does not get reclassified after coming from a Plugin or the XScale we need all the data that the QM will need:
QID
Stats Index
Output Port
If a packet matches an Aux filter AND it needs ARP processing, the ARP processing takes precedence and we do not process the Aux filter result.
Does anything other than ARP related traffic go to the XScale?
IP exceptions like expired TTL?
Can users direct traffic for delivery to the XScale and add processing there?
Probably not if we are viewing the XScale as being like our CPs in the NSP implementation.

11. Notes Combining Parse/Lookup/Copy
Dispatch loop
Build settings
TCAM mailboxes (there are 128 contexts)
So with 24 threads we can have up to 5 TCAM contexts per thread.
Rewrite Lookup in C
Input and Output on Scratch rings
Configurable priorities on Mux inputs
Xscale, Plugins, Rx
Should we allow plugins to write directly to QM input scratch ring for packets that do not need reclassification?
If we allow this is there any reason for a plugin to send a packet back through Parse/Lookup/Copy if it wants it to NOT be reclassified?
We can give Plugins the capability to use NN rings between themselves to chain plugins.