1. Fred Kuhns and Alex Chandra Applied Research Laboratory
FPX/SPC Integration
Fred Kuhns and Alex Chandra
Applied Research Laboratory

2. Rule if GM filter: Exclusive, Ingress and Non-multicast
if OVIN is invalid use result of RL
if QID valid use it, else use dgram QID
else
both OVIN and QID must be valid
In all cases insert resulting OVIN in inter-port shim
if packet from SPC and RC == 0, then OVIN and QID must be valid. The SPC may change these fields.
if GM filter: Exclusive, Egress and Non-multicast
Always use OVIN from Shim
if QID is valid use it, else use dgram QID
If packet from SPC and RC == 0, then OVIN and QID in shim must be valid – SPC may alter these values.

3. Notes: mtg [09/16/02] and subsequent discussions
TTL Processing Note:
Ingress ISAR: If TTL = {0,1} and packet from line-card then send to SPC, bypassing CARL. Otherwise process normally (TTL is not changed or inspected further).
Egress OSAR: If TTL > 0 and packet to line-card, then decrement. Always send regardless of TTL value (TTL is not changed or inspected further).
ICMP messages: We will support sending and receiving the following
Time exceeded for datagram: FPX ingress port sends packet to SPC if TTL = 1/0. The SPC will decide whether to deliver locally or generate ICMP message.
Destination unreachable: if there is no route entry then send to SPC – we already do this. SPC responds with ICMP msg if it is unable to resolve route.
ICMP echo and reply: This will have the routers local IP address so we “can” implement at CP – need to look into how we do this at the CP. Alternatively, we can define a route table entry for the local interfaces IP address to send those packets to the SPC.
Filter on IP header Source Address:
Invalid Source addresses:
The high order 3 bits equal 111. Note class D (Multicast) has 1110, class E (Reserved) has 1111 and broadcast is of course all 1’s.
Address is all 0’s
Address high order byte equals 127d (0x7f) – i.e. the loopback interface
In an invalid Source Address then silently drop.
Note: while we “should” drop directed broadcasts and filter for IP spoofing we will not do this due to the difficulty of implementation. Discussion: ??

4. Notes (2) Filter on IP header Destination Address:
Invalid Destination addresses for forwarding (IP lookup):
{-1, -1}, {0, 0}, {prefix, 0}, Class E ( – )
If valid unicast then forward: high order 3 bits do not equal 111 and address is not all 0’s.
If valid broadcast address then send to CP
there are two broadcast address of concern: 0xffffffff and local “subnet” broadcast.
Can we use the General Match filter for this?
If multicast address (first 4 bits = 1110) then ISSUE??
If class E, all zeros or any other case then silently drop.
IP Fragmentation:
Assume MTU is same on all ports, so will not need to fragment
if receive an IP fragment, then silently drop. Rationale: describe??
Path MTU discovery:
support the IP option form: describe approach??
do not support the ICMP based approach: describe approach??
Other Packet Processing Errors:
IP header parameter errors other than version: Silently drop.
Errors: Checksum error, length error, invalid source address, Others?.
Rationale: Since the IP header may be corrupted or counterfeit we can not and should not rely on any of the field values.
AAL5 checksum or length errors – silently drop packet. Rationale same as for IP parameter errors.
Filter packets looking for source addresses – is this “doable” in current design. Protects against certain types of DOS attacks and required by rfc.

5. Notes(3)
Will support GM filtering on output port selectable via a configuration flag. Default case does not enable this feature.
How do we deal with Multicast and Broadcast?
with IGMP we could use a GM filter to send these multicast packet to the CP – can/should we do this??
General Match Filters and default routes:
Alex will add an additional bit to the GM filter indicating OVIN validity. If OVIN is not valid then the results of the FIPL lookup are used. If the fipl lookup fails then treat as you would any datagram without a valid route.
ICMP Redirect
Should we generate an ICMP redirect if the resulting OVIN == IVIN??

6. System Architecture Switch Fabric IPP OPP IPP OPP IPP OPP IPP OPP IPP
Control Processor
Switch Fabric
ATM Switch Core
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
FPX
FPX
FPX
FPX
FPX
FPX
Processors
Port
SPC
SPC
SPC
SPC
SPC
SPC LC LC LC LC LC LC
Line
Cards

7. Logical view of SPC/FPXLC
SPC CP
SPC/FPX
VCI = OBase + In_PN
VCI = IBase + SP
VCI = OBase + Out_PN
Ingress Traffic
(Input Port)
Egress Traffic
(Output Port)
IN_VCI
EG_VCI
Fast Path: Core IP processing in FPX (classification, packet scheduling, distributed queuing etc.)
Slow Path: Exceptional and “active” processing in SPC
Control Path: messages from the CP

8. ISAR – receive packet from Line card or SPC
Ingress
I only care about the resulting behavior. So some details are missing if they do not directly support the “story”
SPC
VC = 56
TO = 0
ingress
VC = 57
TO = 1
egress
FPX
ISAR – receive packet from Line card or SPC
Reassemble packet, verify AAL5 checksum.
If invalid checksum, silently drop packet.
From LC:
If bad IP parameters or invalid checksum then silently discard packet {ip hlen < 5, ip cksum invalid, AAL5 cksum invalid, ip tlen + shim_len != AAL5 len}
If IP options or not IPv4, Set EX flag and send to SPC
If TTL = 0/1, send to SPC
otherwise “send to” CARL
From SPC:
Drop packet if:
Not HO: if hlen < 5, IP/AAL5 cksum invalid, AAL5 len != hlen+16
HO: if hlen < 5, IP/AAL5 cksum invalid, AAL5 len != , DP=1
Note: for HO=0, ISAR expect SPC to drop packet if necessary.
if RC = 1, “send to” CARL for classification
Fill in Shim values:
InVIN using PN register, what about sub-port?
Flags:
RC = 1 unless SPC returns RC = 0, DP = 1 or EX = 1.
NM = 0 in all cases (set by CARL)
EX = 1 when IP options, 0 otherwise
HO = 0, CARL may reset
HR = 1 when HO = 1 and returned from SPC, 0 otherwise
FM = 1 from LC otherwise 0 from SW
TO = 1 => LC and 0 => SW. redefined by CARL
Switch 40
Line
Card 41 50 42 51 43 52 44 53 45 46 47

9. in: <destination>
Ingress
Notes:
MTP = Multicast Tree Position, 8 bit #
SPC
VC = 56
TO = 0
ingress
VC = 57
TO = 1
egress
FPX
Classify Packet
CARL GM
in: <5-tuple>
out: <E: priority, HO, DP, QID, OVIN>
<N:HO, QID, OVIN>
Switch
in: <FM, 5-tuple,
MTP>
out: <Flags (FT, HO, LR, TO), OVIN0, QID0, Rate0, OVIN1, OVIN1 Rate1, MTP?> EM
(priority)
fipl
(priority)
out: <OVIN>
QID = SW_DGRAM_BASE (440) + OVIN(4:2)
in: <destination> 40
Line
Card
in parallel, GM, EM and dgram lookup
update OVIN & QID with highest priority matching filter/route
if packet from SPC and RC = 0 and DP = 0 then assume TO and OVIN are correct (TO assume = 0/SW, erroneous for it to = 1/LC since route lookup performed at ingress port)
if (QID = 0) then QID = SW_DGRAM_BASE (440) + OVIN(4:2)
else QID assumed valid and not modified by CARL
QMGR remaps QID = QID + 128
Update TO and DP (GM Exclusive may require we drop packet)
Send to OSAR 41
ISAR 50 42 51 43 52 44 53 45 46 47

10. Ingress FPX SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress
QM/OSAR/PSM
Switch QM
Place packet in correct queue, schedule for transmission
When scheduled, send to OSAR
OSAR
encapsulate in AAL5 frame
To SW
update shim: flags (all zero), IVIN, OVIN, PPN, MTP
encapsulate in AAL5 frame and determine VC from OutVIN
To SPC (ingress and egress same)
Encapsulate on AAL5 frame, set VC
Insert Intra-port shim 40
Line
Card
ISAR/
CARL 41 50 42 51 43 52 44 53 45 46 47

11. FPX: Queue assignment after classification
*Note on QIDs to the SPC
are reserved for SPC bound traffic
QID Port Description Flag Supported
1 Ingress No match in CARL NM=1 Yes
2 Ingress Not IPv4 EX=1 Yes
3 Ingress TTL = 0/1 EX=1 Yes
4 Ingress IP Options EX=1 Yes
5 Egress Output link down EX=1 No
6 Not defined
7 Not defined
SPC
VC = 56
TO = 0
ingress
VC = 57
TO = 1
egress
FPX
After CARL classifies packet it is placed into a queue and sent on a VC based on TO, SR, QID and OutVIN. Does not show reclassification. Software assigns all QIDs except 1-7. The range 128 – 255 is calculated internally.
...
...
Switch
*QID 1-127
SR = 0, TO = 1/0
QID
(qid = qid + 128)
SR = 1, TO = 1/0
OutVIN (port) used to calculate VC
RC = 1
TO = 1
TO = 0 40
Line
Card
CARL 41
QID 440 – 447
datagram queues
to switch (TO = 0) 50
...
QID 448 – 511
Datagram Queues (64) 42 51 43 52 44 53
...
QID 256 – 439
Reserved Flows (184) 45 46 47
OutVIN (sub-port)
used to calculate VC
TO = 1
TO = 0

12. ISAR – receive packet from switch
Egress
SPC
VC = 56
TO = 0
ingress
VC = 57
TO = 1
egress
FPX
Switch
ISAR – receive packet from switch 40 41
Line
Card 42
Receive packet from switch; Set FM = 0 (SW)
IP len + 8 != AAL5 length, then silently drop
otherwise same as ingress ISAR 50 43 51 44 52 45 53 46 47

13. Egress FPX Classify Packet Exact Match only
SPC
VC = 56
TO = 0
ingress
VC = 57
TO = 1
egress
FPX
Classify Packet
Switch
CARL
in: <FM, 5-tuple,
“depth”> EM
out: <Flags (FT, HO, LR, TO), OVIN0, QID0, Rate0, OVIN1, OVIN1 Rate1, MTP?>
ISAR
Exact Match only
out 1 or 2 tupples {TO, OVIN, QID, Rate, MB}
From SPC and RC = 0 and DP = 0 (bypass pkt), then TO flag and OVIN are assumed correct
if TO = 1/LC
if QID = 0
QID = LC_DGRAM_BASE(448) + OVIN(1:0):SA(1:0):DA(1:0)
else
QID not modified by CARL
QMGR set QID = QID + 128
else same as ingress case
Send to QM include pkt pointer, rates, shim fields. 40 41
Line
Card 42 43 50 44 51 45
pkt ref 52 46 53 47

14. Egress FPX SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress Switch
QM/OSAR QM
send pkt, rates and shim fields to OSAR
OSAR
Sending to LC:
remove shim,
update LFS option (rate2)
decrement TTL if > 0 (always send even if TTL = 0)
update IP hdr checksum
encapsulate in AAL5 frame and calculate VC from OutVIN
Sending to SPC:
Encapsulate on AAL5 frame, set VC
Insert Intra-port shim
ISAR 40 41 42
Line
Card 43 50 44 51 45 52 46 53 47

15. FPX to SPC – Intra-Port Shim
Input VIN
Output VIN
PPN
Flags
Queue Identifier (QID)
Port #
Sub #
Port #
Sub #
Port # DP RC NM EX HO HR FM TO
TTL
Chunks
Queue Length
Packet Pointer
Second Chunk Pointer
Flag Values
DP (Drop Packet): SPC directive to FPX. Will always be zero coming into SPC.
RC (Reclassify Packet): SPC directive to FPX. May be 0 or 1 going to SPC. SPC must redefine.
NM (No Match): FPX can not classify (no route), FPX to SPC. Set to 0 before returning to FPX.
EX (Exception Packet): Requires non-fast path; IP options, Non IPv4. FPX to SPC. Set to 0 before returning to FPX.
HO (Header Only): FPX to SPC; Set if packet size > 120B and only first chunk sent to SPC. SPC must not modify. SPC can not drop internally if this bit is set.
HR (Header Only Return): Internal FPX flag. SPC must not modify.
FM (From LC/SW): Set by FPX. SPC can use but not modify.
TO (To LC/SW): Set by FPX. SPC can use but not modify.

16. FPX to SPC, Inter-port Shim
Flags
Input VIN
Output VIN
PPN
VIN PN SP
Inter-port Shim – expect the Input and Output VINs to be correct. No other information needed – in particular the PPN is not used by the SPC.

17. Looking at the Shim Fields (Intra-Port)
Flags: Input to SPC → indicates required processing {NM, EX, HO}
NM → FPX lookup failed, perform lookup
If SPC can resolve route then
set {Output VIN, QID}
set NM = 0,
return to FPX.
else
send ICMP to source and drop packet (internal to SPC).
EX → Implies IP Options, IP “parameter error”, or Non IPv4.
If IP option, process options and return to FPX with EX = 0. Further iff an EM filter has been added/removed then set RC = 1.
If Non IPv4 then SPC either “consumes” or process as if NM is set.
HO → Only the header (first 120 bytes) of packet.
if need to drop packet, then set DP = 1 and return to FPX

18. Scenario Overview
Default system configuration (routes, FPX register file)
FPX, SPC and CP
Basic datagram forwarding, no GM or EM filters
valid packets
various error conditions
get traffic statistics (counts)
Forwarding with GM and EM filters
CP adds default GM and EM filters
traffic: no matching filters
traffic: match GM and EM. Try mix of traffic (dgram, reserved) and sending to SPC
Control FPX from SPC
add/remove EM filter
Add/remove filters and route entries under load

19. Default Configuration for Scenarios
Initial, Basic Configuration
SPC configuration – none for datagram forwarding test
Will use NCHARGE where possible.
NID image stored in PAL on FPX. Must be configured (VC translation table) then correct RAD image selected and downloaded.
RAD Configuration
FPX register values (separate slide)
Download default FIPL table (separate slide)
Default general match and exact match filters - none for initial datagram forwarding test
Other FPX specific configuration issues?
None that I know of

20. RAD Configuration
CP configuration module will handle basic configuration issues
RAD configuration
initialize and configure SPCs (indicate if FPX present)
use NCHARGE to download and start RAD.
configure NID VC translation table
RAD Initialization – see separate slide with register file defaults
Set/verify default VCI values: CP Control VCI
Registers SPC Control VCI, DQ Control VCI, Ibase VCI, Obase VCI, SPC IN VCI and SPC EG VCI
Set port number:
register PN
Set/verify default route and exact match priorities
registers RL Priority and EM Priority
Set/verify default link rate, SPC rate and speed advantage
registers QM Speed, QM Link and QM SPC
Does software set or just read the registers RL RootNodePtr and EM Offset?
Configure CARL for default tests
add our usual default routes – see separate slide with default routes
For classifier tests, download default GM and EM filters – see separate slide

21. FPX Register File Configuration registers
We need to have test cases where we read, modify and verify.
Default values are correct for testing.
Register
Number
Name
Width
(bits)
Default
Value
Description
0x00
PN (Port Number) 3
Physical port number of the FPX
0x01
CP Control VCI 16
0/35
Control cell VCI: CP to/from FPX
0x02
SPC Control VCI
1/29
Control cell VCI: SPC to/from FPX
0x03
DQ Control VCI
1/61
Control cell VCI for DQ
0x04
Ibase VCI
0/128
Base VCI for link traffic
0x05
Obase VCI
0/64
Base VCI for switch traffic
0x06
SPC IN VCI
1/62
VCI for ingress traffic: SPC to/from FPX
0x07
SPC EG VCI
1/63
VCI for egress traffic: SPC to/from FPX
0x08
RL RootNodePtr 18
0 (0x0)
Root node pointer for Tree Bitmap
0x09
RL Priority 6
60 (0x3C)
Priority of route lookup results
0x0A
EM Offset
7 (0x7)
Address offset of exact match entries
0x0B
EM Priority
56 (0x38)
Priority of exact match results
0x0C
QM Speed 8
2 (0x02)
Speed advantage (ratio)
0x0D
QM Link
x=.875, y=13
(0xED)
(983 Mb/s)
Link rate in multiples of 64kb/s
0x0E
QM SPC
x=.5, y=11
(0x8B)
(197 Mb/s)
SPC rate in multiples of 64kb/s

22. Default Routes and Interface Addresses
Download default routing table at initialization.
SPC represents the VIN differently: This needs to be fixed.
At some point we need to have a better (i.e. more convenient) way of specifying routes and dynamically changing them using a standard routing protocol such as OSPF. PN SP
Prefix/Width
IP Address
SPCVIN
(10/6)
FPXVIN
(3/2)
/24
0x0000 (0)
0x00 1
/24
0x0001 (1)
0x01 2
/24
0x0002 (2)
0x02 3
/24
0x0003 (3)
0x03
/24
0x0040 (64)
0x04
/24
0x0041 (65)
0x05
/24
0x0042 (66)
0x06
/24
0x0043 (67)
0x07
/24
0x0080 (128)
0x08
/24
0x0081 (129)
0x09
/24
0x0082 (130)
0x0a
/24
0x0083 (131)
0x0b
/24
0x00c0 (192)
0x0c
/24
0x00c1 (193)
0x0d
/24
0x00c2 (194)
0x0e
/24
0x00c3 (195)
0x0f 4
/24
0x0100 (256)
0x10
/24
0x0101 (257)
0x11
/24
0x0102 (258)
0x12
/24
0x0103 (259)
0x13 5
/24
0x0140 (320)
0x14
/24
0x0141 (321)
0x15
/24
0x0142 (322)
0x16
/24
0x0143 (323)
0x17 6
/24
0x0180 (384)
0x18
/24
0x0181 (385)
0x19
/24
0x0182 (386)
0x1a
/24
0x0183 (387)
0x1b 7
/24
0x01c0 (448)
0x1c
/24
0x01c1 (449)
0x1d
/24
0x01c2 (450)
0x1e
/24
0x01c3 (451)
0x1f

23. Scenario 1: Datagram Forwarding, No G/E filters
Datagram Forwarding, Valid packet, Ingress and Egress Ports
Valid route, valid IP header. Forward packet
Datagram Forwarding, Error/Exceptional processing, Ingress Port
IP option other than LFS, send to SPC with EX flag = 1 (see LFS processing below)
Valid packet, no route, send to SPC with NM flag = 1 and QID = 1
SPC has route, update OutVIN and QID. Send back to FPX, NM = 0
SPC does not have route, send ICMP destination unreachable to source if HO = 0 then drop packet else set DP flag = 1, return packet to FPX.
Valid route, TTL = 0 or 1. Send to SPC with EX flag = 1 and QID = 3?.
If packet destination address is local then deliver to CP.
If not local then send ICMP Time Exceeded message to source, if HO = 0 then drop packet else set DP flag = 1 and return to FPX.
Checksum error, invalid length or other parameter error, FPX silently drops packet.
If the IP version is not valid then set EX flag = 1 and send to SPC
if IP version not supported then if HO = 0 drop pkt else set DP flag = 1 and return packet to FPX for dropping. Otherwise SPC processes packet – though it is not clear what this means.
Buffer overrun – silently drop packet.
Datagram Forwarding, Error/Exceptional processing, Egress Port
IP options are not checked, Route and GM filters are not checked, Assume SHIM correct
Checksum or other parameter error, drop silently
If indicated interface or VC not available?, set EX flag = 1 and send to SPC with QID = 4?. SPC will send ICMP destination unreachable message to source, set DP flag = 1 and return to FPX.

24. Scenario 2: Matching GM or EM Filters
Install GM and EM filters from CP
See word document for filter specification
Dynamically add and remove EM filters
Dynamically add and remove GM filters
Dynamically add and remove routes
Read packet counters for each filter type
Packet Matches GM – repeat for both Ingress and Egress ports
Non-Exclusive filter
Exclusive filter
Drop Packet, Send to SPC, Header Only
Both Exclusive and Non-Exclusive filter
Exclusive sent to SPC, Non-Exclusive send header only to SPC
Packet Matches EM filter – repeat for both Ingress and Egress ports
Reserved flow – do not send to SPC but set OVIN and QID
Send to SPC, complete packet
Send to SPC, Header Only
Packet Matches EM and GM Filters at Ingress port
Alter priorities to verify CARL operation (which lookup engine “wins”)
EM and GM exclusive, EM has higher priority, reserved
EM and GM exclusive, GM has higher priority, drop packet
Should we try with RL having the highest priority?

25. Scenario X: LFS Option Processing
Case 1: new LFS flow
FPX send packet with IP option to SPC before it is classified.
SPC processes IP option and send control cell to FPX to install new EM filter for flow.
SPC assigns an unused QID
Does the SPC have to also assign the OutVIN? Is so then the SPC has a complete routing table available.
SPC waits for FPX to reply to control message before sending packet back with the RC flag = 1.
The FPX classifies the packet and places in correct queue.
Case 2: Existing LFS reservation, no change required
RC flag = 1, packet returned to SPC.
Case 3: Existing LFS reservation, decrease allocation
Increase available count, send control message to FPX (updated reservation), wait for reply, RC flag = 1, send packet to FPX, look for any partial allocations, allocate freed bandwidth, send any necessary control cells to FPX with updates
Case 4: Existing LFS reservation, increase allocation
if request can be satisfied then send control cell to FPX updating entry, wait for reply, set RC flag = 1 and return to FPX.
if request can be partially satisfied then add entry to Partial Allocation Table, send control cell to FPX updating entry, wait for reply, set RC flag = 1 and return to FPX.
if request can not be satisfied then add entry to Partial Allocation Table, set RC flag = 1 and return to FPX.
Case 5: Existing LFS reservation, Release request
Increase available count, send control message to FPX removing EM filter, set the QID and OutVIN in shim to be same as the released EM filter, (RC = 0), send packet to FPX.
Case 6: No existing LFS reservation, Release request
do nothing, send packet to FPX with RC flag = 1.