1. Flow Stats Module
James Moscola
September 12, 2007

2. SPP V1 LC Egress with 1x10Gb/s Tx
XScale
NAT Miss
Scratch Ring
SCR S W I T C H R B U F M S F
Rx1
Rx2
Key
Extract
Lookup
Hdr
Format NN NN NN NN
TCAM NN T B U F
QM0
SCR
Port
Splitter R T M M S F
Flow
Stats1
SCR
1x10G
Tx2
1x10G
Tx1
QM1
SCR NN NN
QM2
SCR
QM3
SCR
SCR
NAT Pkt
return
Stats
(1 ME)
SRAM3
SRAM1
SCR
Flow
Stats2
SRAM
Freelist
SRAM
XScale
XScale
SRAM2
Archive Records

3. SPP V1 LC Egress with 10x1Gb/s Tx
XScale
NAT Miss
Scratch Ring
SCR S W I T C H R B U F M S F
Rx1
Rx2
Key
Extract
Lookup
Hdr
Format NN NN NN NN
TCAM NN T B U F
5x1G
Tx1
(P0-P4)
QM0
SCR
Port
Splitter
SCR R T M M S F
Flow
Stats1
SCR
QM1
SCR
5x1G
Tx2
(P5-P9)
QM2
SCR
SCR
QM3
SCR
SCR
NAT Pkt
return
Stats
(1 ME)
SCR
SRAM3
SRAM1
Flow
Stats2
SRAM
Freelist
SRAM
XScale
XScale
SRAM2
Archive Records

4. Overview of Flow Stats Main functions Secondary functions
Uniquely identify flows based on 6-tuple
Hash header values to get an index into a table of records
Maintain packet and byte counts for each flow
Compare packet header with header values in record, and increment if same
Otherwise, follow hash chain until correct record is found
Send flow information to XScale for archiving every five minutes
Secondary functions
Maintain hash table
Identify and remove flows that are no longer active
Invalid flows are removed so memory can be resused

5. Design Considerations
Efficiently maintaining a hash table with chained collisions
Efficiently inserting and deleting records
Efficiently reading hash table records
Synchronization issues
Multiple threads modifying hash table and chains

6. Destination Address (32b)
Flow Record
Total Record Size = 8 32-bit words
V is valid bit
Only needed at head of chain
‘1’ for valid record
‘0’ for invalid record
Start timestamp (16-bits) is set when record starts counting flow
Reset to zero when record is archived
End timestamp (16-bits) is set each time a packet is seen for the given flow
Packet and Byte counters are incremented for each packet on the given flow
Next Record Number is next record in hash chain
0x1FFFF if record is tail
Address of next record = (next_record_num * record_size) + collision_table_base_addr
LW0
Source Address (32b)
LW1
Destination Address (32b)
LW2
SrcPort (16b)
DestPort (16b)
LW3
Reserved (12b)
Slice ID (12b)
Protocol (8b)
LW4 V
(1b)
Reserved (14b)
Next Record Number (17b)
LW5
Packet Counter (32b)
Can END timestamp be eliminated? Use archive time as the end timestamp.
Actual end of flow will have occurred within 5 minutes of archival time if records are archived every 5 minutes.
LW6
Byte Counter (32b)
LW7
Start Timestamp (16b)
End Timestamp (16b)
= Member of 6-tuple

7. Timestamp Details Timestamp on XScale is 64-bits
Storing 64-bit start and end timestamps would cause each flow record to be too large for a single SRAM read
Instead, only store the 16-bits of each timestamp required to represent a five minute time interval
Clock frequency = 1.4 GHz
Timestamp increments every 16 clock cycles
Use bits 41:26 for 16 bit timestamps
(226 * 16 cycles)/1.4GHz = .767 seconds
(241 * 16 cycles)/1.4GHz = seconds (418 minutes)
Time interval that can be represented using these bits
.767 seconds through 418 minutes

8. Hash Table Memory Allocating 4 MBytes in SRAM Channel 3 for hash table
Supports ~130K records
Divided memory 75% for the main table and 25% for the collision table
Memory required = Main_table_size + Collision_table_size .75*(#records * #bytes/record) + .25*(#records * #bytes/record) ~98K records + ~32K records ~3Mbytes + ~1Mbytes
Space for main table and collision table can be adjusted to tune performance
Larger main table means fewer collisions, but still need adequate space for collision table
Main
Table
~75%
Collision
Table
~25%

9. Inserting Into Hash Table
IXP has 3 different hash functions (48-bit, 64-bit, 128-bit)
Using 64-bit hash function is sufficient and takes less time than 128-bit hash function
Not including Source Addr or Protocol into address
HASH(D.Addr, S.Port, D.Port);
Result of hash is used to address the main hash table
Since we want ~100K records in main table, result of hash is used to get as close to 100K entries as possible by adding a 16bit and 15bit chunk from the hash result
hash_result(15:0) + hash_result(30:16) = record_number
Records in the main table represent the head of a chain
If slot at head of chain is empty (valid_bit=0), store record there
If slot at head of chain is occupied, compare 6-tuple
If 6-tuple matches
If packet_count == 0 then (existing flows will have 0 packet_counts when previous packets on flow have just been archived)
Increment packet_counter for record
Add size of current packet to byte_counter
Set start and end time stamps
If packet_count > 0 then
Set end time stamp
If 6-tuples doesn’t match then a collision has occurred and the record needs to be stored in collision table
Main
Table
Collision
Table

10. Hash Collisions Hash collisions are chained in linked list
Head of list is in the main table
Remainder of list is in collision table
SRAM ring maintains list of free slots in collision table
Slots are numbered from 0 to #_Collision_Table_Slots
Same as next_record_number
To convert to memory address (slot_num * record_size) + collision_table_base_addr
When a collision occurs, a pointer to an open slot in the collision table can be retrieved from the SRAM ring
When a record is removed from the collision table, a pointer is returned to the SRAM ring for the invalidated slot
Main
Table
Collision
Table
SRAM
Ring
Free list

11. Archiving Hash Table Records
Send all valid records in hash table to XScale for archiving every 5 minutes
For each record in the main table (i.e. start of chain) ...
For each record in hash chain ...
If record is valid ...
If packet count > 0 then
Send record to XScale via SRAM ring
Set packet count to 0
Set byte count to 0
Leave record in table
If packet count == 0 then
Flow has already been archived
No packet has arrived on flow in 5 minutes
Record is no longer valid
Delete record from hash table to free memory
Info Sent to XScale for each
flow every 5 minutes
LW0
Source Address (32b)
LW1
Destination Address (32b)
LW2
SrcPort (16b)
DestPort (16b)
LW3
Reserved (12b)
Slice ID (12b)
Protocol (8b)
LW4
Packet Counter (32b)
LW5
Byte Counter (32b)
LW6
Start Timestamp_high (32b)
LW7
Start Timestamp_low (32b)
LW8
End Timestamp_high (32b)
LW9
End Timestamp_low (32b)

12. Deleting Records from Hash Table
While archiving records
If packet count is zero then remove record from hash table
Record has already been archived, and no packets have arrived in the last five minutes
To remove a record
If ((record == head) && (record == tail))
Valid_bit = 0
Else If ((record == head) && (record != tail))
Replace record with record.next
Free the slot for the moved record
Else if record != head
Set previous records next pointer to record.next
Free slot for the deleted record
Main
Table
Collision
Table
SRAM
Ring
Free list

13. Memory Synchronization Issues
Multiple threads reading/writing same block of memory
Only allow 1 ME to modify structure of hash table
Inserting and deleting nodes
Use global registers to indicate that the structure of the hash table is being modified
Eight global lock registers (1 per thread) to indicate what chain in the hash table is being modified
When a thread wants to insert/delete a record from hash table
Store pointer to the head of the hash chain in the threads dedicated global lock register
If another thread is processing a packet that hashed to the same hash chain, wait for lock register to clear and restart processing packet
Otherwise, continue processing the packet normally
Clear global lock register when done with insert/deletes
Value of 0xFFFFFFFF indicates that lock is clear

14. Flow Stats Execution ME 1 ME 2 (thread numbers may need adjusting)
Init - Configure hash function
8 threads
Read packet header
Hash packet header
Send header and hash result to ME2 for processing
ME 2 (thread numbers may need adjusting)
Init - Load SRAM ring with addresses for each slot in the collision table Init - Set TIMESTAMP to 0
7 threads (ctx 1-7)
Insert records into hash table
Increment counter for records
1 thread (ctx 0)
Archive and delete hash table records

15. Diagram of Flow Stats Execution (ME1)
get buffer
handle from QM
60 cycles
read packet
header (DRAM)
300 cycles
300 cycles
read buffer
descriptor (SRAM)
150 cycles
300 cycles
send buffer
handle to TX
60 cycles
build hash key
~50 cycles
compute hash
100 cycles
send packet
info to ME2
60 cycles
~570 cycles

16. Diagram of Flow Stats Execution (ME2)
Incrementing Counters
Adds records to hash chain, but doesn’t remove them
Iterating through hash chain
Locking head of chain
Best: ~360 cycles
Worst: ~ x
60 cycles
get packet
info from ME1
150 cycles
read hash table
record (SRAM) x
valid?
Yes
match? No
tail? No
compare
record to header
read next
record in chain
~10 cycles
Yes
Yes
150 cycles
Yes
count==0?
set register
to lock chain No No
150 cycles
set register
to lock chain
Now locking all insert/increment operations. Since 2 counters + timestamps need to be written, it makes more sense to write them in a single burst write as opposed to 1 atomic increment (packet count), 1 atomic add (byte count) and 1 standard write to update the timestamps
set register
to lock chain
set register
to lock chain
get record slot
from freelist
150 cycles
150 cycles
150 cycles
150 cycles
insert
new record
Write START/END
time & new counts
Write END
time & new counts
insert
new record
clear lock
register
clear lock
register
clear lock
register
clear lock
register

17. Diagram of Flow Stats Execution (ME2)
Archiving Records
Removes records from hash chain, but doesn’t add them
Processing of archiving records occurs every five minutes
Waiting to archive
Locking head of chain
count == 0?
Yes
head of list?
Yes
tail of list?
Yes
read current
time No No No No
5 minutes?
send record
to XScale
set register
to lock chain
set register
to lock chain
set register
to lock chain
Yes
set register
to lock chain
read next record
from main table
write next_ptr to
previous list item
read
record.next
set valid bit
to zero
Yes
reset counters
and timestamps
valid?
clear lock
register
replace record
with record.next
clear lock
register No
clear lock
register
return record
slot to freelist
clear lock
register
read next record
in chain
return record.next
slot to freelist
Yes
more records
in chain? No No
done with
all records?
Yes

18. Return from Swap
When returning from each CTX switch, always check global lock registers
If any of the global locks contain the address of the hash chain that the current thread is trying to modify, then the hash chain is locked and the current thread must restart processing on the current packet
If none of the global locks contain the address of the hash chain that the current thread is trying to modify, then the current thread can just continue processing that packet as usual
check global
lock values
match current
chain?
Yes No
continue
processing packet
restart procssing
packet

19. SPP V1 LC Egress with 1x10Gb/s Tx
V: Valid Bit V 1
Rsv
(3b)
Port
(4b)
Buffer Handle(24b)
QM0
Flow
Stats1
SCR
1x10G
Tx1
QM1 NN
QM2
Source Address (32b)
QM3
SCR
Destination Address (32b)
SRAM3
SrcPort (16b)
DestPort (16b)
Flow
Stats2
SRAM
Freelist
SRAM
Reserved (12b)
Slice ID (12b)
Protocol (8b)
XScale
Archive Records
Packet Counter (32b)
Source Address (32b)
Byte Counter (32b)
Destination Address (32b)
Start Timestamp_high (32b)
SrcPort (16b)
DestPort (16b)
Start Timestamp_low (32b)
Reserved
(8b)
Packet Length (16b)
Protocol (8b)
End Timestamp_high (32b)
Rsvd
(3b)
Hash Result (17b)
Slice ID (12b)
End Timestamp_low (32b)

20. Flow Statistics Module
Scratch rings
QM_TO_FS_RING_1: 0x2400 – 0x27FF // for receiving from QM
QM_TO_FS_RING_2: 0x2800 – 0x2BFF // for receiving from QM
FS1_TO_FS2_RING: 0x2C00 - 0x2FFF // for sending data from FS1 to FS2
FS_TO_TX_RING_1: 0x x33FF // for sending data to TX1
FS_TO_TX_RING_2: 0x3400 – 0x37FF // for sending data to TX2
SRAM rings
FS2_FREELIST: 0x???? - 0x???? // stores list of open slots in collision table
FS2_TO_XSCALE: 0x???? – 0x???? // for sending record information to the XScale for archiving
LC Egress SRAM Channel 3 info for Flow Stats
HASH_CHAIN_TAIL 0x1FFFF // indicates the end of a hash chain
ARCHIVE_DELAY 0x0188 // 5 minutes
RECORD_SIZE 8 * 4 = 32 // 8 32-bit words/record * 4 bytes/word
TOTAL_NUM_RECORDS // MAX with 4 MB table is ~130K records
NUM_HASH_TABLE_RECORDS // NUM_HASH_TABLE_RECORDS<=TOTAL_NUM_RECORDS (mod 32 = 0)
NUM_COLLISION_TABLE_RECORDS TOTAL_NUM_RECORDS - NUM_HASH_TABLE_RECORDS = 32384
LCE_FS_HASH_TABLE_BASE SRAM_CHANNEL_3_BASE_ADDR + 0x = 0xC
LCE_FS_HASH_TABLE_SIZE 0x400000
LCE_FS_COLLISION_TABLE_BASE (HASH_TABLE_BASE + (RECORD_SIZE * NUM_HASH_TABLE_RECORDS)) = 0xC

21. End