1. Design of a High Performance PlanetLab Node
John DeHart

2. Revision History (10/06 - ??/??)
10/3/06 (JDD)
Added info on buffer descriptor
10/11/06 (JDD)
Corrections to order of Src and Dst IP Addr in IP packet headers
10/12/06 (JDD)
Add proposed change to Lookup input and output data formats
10/24/06 (JDD)
Updates to Egress Lookup Key
10/26/06 (JDD)
Added HW definitions
11/02/06 (JDD)
Added diagram of 2 code options, 4 slices, 4 interfaces per slice.
11/15/06 (JDD)
Adding November Intel Demo config info

3. Revision History (6/06 – 9/06)
6/27/06 (JDD):
Created from Diversified Router version
7/5/06 (JDD):
Still in progress…
7/6/06 (JDD):
7/10/06 (JDD)
Minor modifications based on comments from Jing and Brandon.
7/17/06 (JDD)
Updated Data between Key_Extract, Lookup and Hdr_format on LC projects to consistently use the same format for the second word of data: IP Pkt Length (16b), Reserved(8b), Eth Hdr Len(8b)
8/30/06 (JDD)
Changed fields in hdr_format to QM for IPv4 MR to make the agree with LC projects.
9/8/06 (JDD)
Modified IPv4 Memory Map
Clean separation between Init memory blocks and Dynamic memory blocks
Modified LC Egress Lookup Key (add Sport) and Result (add counter index)
9/12/06 (JDD and BDH)
Add Slice Memory Ptr to NN ring structures.
Add Code option to be passed through to Header Format
Update Egress Data formats for Key Extract, Lookup and Hdr Format

4. Overview
These slides are still a bit rough, but it should get us started
Three Project Goals
PlanetLab Node
ONL Router
Diversified Router
First Priority: PlanetLab Node for November Demo
Phase 0.0: external GE Switch, 1 GPE, 1 LC, >=1 NPE
Phase 0.5: same but with Switch Blade
What is impact on Rx, Tx, QM?
Phase 1: same but with Multiple GPEs (each as its own PlanetLab Node)
Phase 2: same but with Multiple GPEs, unified as one PlanetLab Node.
Heavy emphasis in these slides is on Phase 0.0
Probable hardware configuration:
1 NP Blade for LC
1 NP Blade for NPE
1 GP Blade
External GE Switch
Possible additions:
1 Switch Blade in place of external switch
1 NP Blade for a second NPE

5. Overview
Assume we will use VLANs internally to isolate and identify MRs.
Assume we can set Ethernet MAC addresses on blades
40 bits fixed,
8 bits variable
View PlanetLab UDP/IP as a new Substrate Link Type
No Substrate Headers used internally or externally
All packets/frames are IP packets in Ethernet Frames
External
LC  GPE, GPE  LC
LC  NPE, NPE  LC
NPE  GPE, GPE  NPE
In effect, UDP/IP Headers are the Substrate Header.
MN Internal Header still used between MPEs (NPE  GPE)
Static Shared NP implementation of MRs.
Limited, predefined MR code options
Statically loaded
Different MR slices run the same code, just use different Filters in Lookup table.
Queuing is slightly different on each system (LC_Ingress, LC_Egress, NPE)
What about ARP?
Do we need/want to support it for November?
We’ll need it eventually.

6. System View … exception packets use internal port numbers slice pkt
GPE
exception packets use internal port numbers VS VS
NPE …
slice pkt
IPH
MN-H
Kernel/VNET
Switch LC
Default filter directs packet to GPE
Filter directs packet to NPE
IPH
IPH
daddr= nextNode
daddr= thisNode
IPH
IPH
daddr= nextNode
slice pkt
slice pkt
slice pkt
slice pkt
daddr= thisNode

7. System View: External Switch
PLC
GPE
Switch / 5
1Gb / 1
1Gb
Net R T M LC / 1
1Gb R T M
NPUA / 5
1Gb 4
1Gb /
Local
Host x4
NPUB / 5
1Gb
Phase 0.0: External GE Switch (16 Ports connected)
LC RTM:
1 GE Interface connected to PLC/myPLC via Network
4 GE Interfaces connected directly to Local Hosts
Extra data sources and sinks
NP RTM
5 GE Interfaces used by NPUA
5 GE Interfaces used by NPUB
GPE
1 Front panel GE Interface

8. System View: External Switch
PLC
GPE
Switch / 5
1Gb / 1
1Gb
Net R T M LC / 1
1Gb R T M
NPUA / 5
1Gb 4
1Gb /
Local
Host x4
NPUB / 5
1Gb
TP GigE
Fiber GigE (max of 4 slots on switch)

9. System View: Switch Blade
PLC
GPE
Switch
Blade / 1
1Gb
Net R T M LC / 1
1Gb / 1
10Gb
NPE-1 / 1
10Gb
Local
Host x9 / 9
1Gb
Phase 05: Switch Blade
LC RTM
1GE Interface connected to PLC/myPLC via Network
9 GE Interfaces connected directly to Local Hosts
1 10Gb interface via backplane
NPE
Will not need an RTM
GPE
1 GE Interface on the Fabric connector

10. PlanetLab Ingress LC Input Frame
New PlanetLab Substrate Link Type:
Configured SL Type
LC is told at boot/init time that this is its one and only SL Type.
Similar to the way P2P-DC is handled.
SL Type: 0101b
Port: May be a don’t care
IP DAddr: Verifies that packet is for our node
IP Proto = UDP
Could be a UDP tunnel to a slice
UDP DPort: Indicates which slice
Default route is to the GPE
Key =
SL=0101b
Port: May be a don’t care.
IP DAddr = our node address
DstAddr (6B)
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
SrcAddr (6B)
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
Protocol = UDP (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)
Header IP
Dst Addr (4B)
Dst Addr (4B)
IP Options (0-40B)
IP Options (0-40B)
Src Port (2B)
Src Port (2B)
UDP
Header
Dst Port (2B)
Dst Port (2B)
UDP length (2B)
UDP length (2B)
UDP checksum (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
UDP Payload
(MN Packet)
PAD (nB)
PAD (nB)
Ethernet
Trailer
CRC (4B)
CRC (4B)
PlanetLab IPv4 Key(0x5) (64 bits)
SL(4b)
0101
Port
(4b)
IP DAddr (32b)
IP Proto
(8b)
UDP DPort (16b)

11. PlanetLab Ingress LC Processing
Ingress Processing:
Portions are similar to IPv4 MR Parse.
IP Header checks/validation:
Check that version is IPv4
Check IP Header checksum
Ignore options (Leave as is and forward on)
Drop if fragmented (What if GPE bound?)
Extract
IP Protocol
IP Dst Addr
UDP Dst Port
Or whatever is in the 2B that would be the UDP port if the IP Protocol were UDP
We shouldn’t have to worry about what might be in this field if the IP Protocol is not UDP
Perform Lookup
Result contains
Per MI Stats/Counter Index
Ethernet DAddr for destination blade
VLAN for destination slice (if needed)
No changes made to IP Header
Re-write Ethernet Header
Configured with Ethernet SAddr for LC
Total number of 8-Byte reads:
No VLAN, No IP Options: 4
No VLAN, Max IP Options: 9
VLAN, No IP Options: 5
VLAN, Max IP Options: 14
DstAddr (6B)
Ethernet
Header
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
Protocol = UDP (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)
Header IP
Dst Addr (4B)
Dst Addr (4B)
IP Options (0-40B)
IP Options (0-40B)
Src Port (2B)
Src Port (2B)
UDP
Header
Dst Port (2B)
Dst Port (2B)
UDP length (2B)
UDP length (2B)
UDP checksum (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
UDP Payload
(MN Packet)
PAD (nB)
PAD (nB)
Ethernet
Trailer
CRC (4B)
CRC (4B)
Indicates fields that
need to be read
Indicates 8-Byte Boundaries
Assuming no IP Options

12. PlanetLab Ingress LC Output Frame
Ethernet Header is only thing that changes:
DAddr: MAC Address of GPE/NPE (Result)
40 bits are static
8 bits variable and stored in Result
SAddr: MAC Address of LC (static)
Type = 802.1Q (static)
VLAN = Slice VLAN (Result)
Type = IP (static)
Total number of 8-Byte Reads: 1
Need to read first part of IP header so when we do the write of last part of ethernet header we can fill out the 8-Byte Write.
Total number of 8-Byte Writes: 3
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
PAD (nB)
Ethernet
Trailer
CRC (4B)
Indicates fields that
need to be written
Indicates 8-Byte Boundaries
Assuming no IP Options

13. PlanetLab NPE Input Frame from LC
Ethernet Header:
DstAddr: MAC address of NPE
SrcAddr: MAC address of LC
VLAN: One VLAN per MR (MR == Slice)
IP Header:
Dst Addr: IP address of this node (phase 0)
Src Addr: IP address of previous hop
Protocol: UDP
UDP Header:
Dst Port: Identifies input tunnel
Src Port: with IP Src Addr identifies sending entity
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
PAD (nB)
Ethernet
Trailer
CRC (4B)
Indicates 8-Byte Boundaries
Assuming no IP Options

14. PlanetLab NPE Output Frame to LC
Re-writes IP Header
Dst Addr: Next Hop
Src Addr: NPE’s IP Address
Re-writes UDP Header
Src Port: NPE’s end of the tunnel to next hop
Dst Port: Other end of tunnel to next hop.
UDP checksum?
Re-writes Ethernet Header
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
PAD (nB)
Ethernet
Trailer
CRC (4B)
Indicates 8-Byte Boundaries
Assuming no IP Options

15. PlanetLab NPE Local Delivery Frame FROM GPE
IP Header
Dst Addr: NPE’s IP Address
Src Addr: GPE’s IP Address
No IP Options
UDP Header
Src Port: GPE’s end of the tunnel
Dst Port: NPE’s end of the tunnel
UDP checksum?
MN Internal Header
As defined for diversified router
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
MN Internal Hdr (4-10B)
PAD (nB)
CRC (4B)
Ethernet
Trailer
Indicates 8-Byte Boundaries
Assuming no IP Options

16. PlanetLab NPE Local Delivery Frame TO GPE
Re-writes IP Header
Dst Addr: GPE’s IP Address
Src Addr: NPE’s IP Address
No IP Options
Re-writes UDP Header
Src Port: NPE’s end of the tunnel to GPE
Dst Port: GPE’s end of the tunnel from NPE
UDP checksum?
MN Internal Header
Need to look at the details of what needs to go in here now that we have no explicit RxMI and TxMI fields/values.
Ethernet Header
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
MN Internal Hdr (4-10B)
PAD (nB)
CRC (4B)
Ethernet
Trailer
Indicates 8-Byte Boundaries
Assuming no IP Options

17. PlanetLab NPE Exception Path Frame FROM GPE
IP Header
Dst Addr: NPE’s IP Address
Src Addr: GPE’s IP Address
No IP Options
UDP Header
Src Port: GPE’s end of the tunnel
Dst Port: NPE’s end of the tunnel
UDP checksum?
MN Internal Header
As defined for diversified router
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
MN Internal Hdr (4-10B)
PAD (nB)
Ethernet
Trailer
CRC (4B)
Indicates 8-Byte Boundaries
Assuming no IP Options

18. PlanetLab NPE Exception Path Frame TO GPE
Re-writes IP Header
Dst Addr: GPE’s IP Address
Src Addr: NPE’s IP Address
No IP Options
Re-writes UDP Header
Src Port: NPE’s end of the tunnel to GPE
Dst Port: GPE’s end of the tunnel from NPE
UDP checksum?
MN Internal Header
As defined for diversified router
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
MN Internal Hdr (4-10B)
PAD (nB)
CRC (4B)
Ethernet
Trailer
Indicates 8-Byte Boundaries
Assuming no IP Options

19. PlanetLab Egress LC Input Frame
Ethernet Header addressed to LC
IP Packet should be complete and LC does not need to touch it. (Phase 0)
Should not even need to do hdr checksum
Lookup Key:
IP Protocol (8b)
UDP Sport (16b)
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
PAD (nB)
Ethernet
Trailer
CRC (4B)
Indicates fields that
need to be read
Indicates 8-Byte Boundaries
Assuming no IP Options

20. PlanetLab Egress LC Processing
Egress Processing:
Extract
IP Dst Addr and UDP Sport
Perform Lookup
Need to generate
MAC DAddr for next hop
VLAN, maybe
Assume there are a limited number of next hops:
Local Hosts
Routers connected to the local subnet.
Result contains
Per MI Stats/Counter Index
Ethernet DstAddr
VLAN if needed OR
L2 Lookup Table Index
L2 Lookup Table Entry:
L2 Header Size (14B or 18B)
18 Bytes of Data
No changes made to IP Header
Re-write Ethernet Header
Configured with Ethernet SAddr for LC
Total number of 8-Byte reads: 2
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
PAD (nB)
Ethernet
Trailer
CRC (4B)
Indicates fields that
need to be read
Indicates 8-Byte Boundaries
Assuming no IP Options

21. PlanetLab Egress LC Output Frame
Ethernet Header is only thing that changes.
DAddr: MAC Address of next hop (Result)
SAddr: MAC Address of LC (static)
Optional
Type = 802.1Q (Result)
VLAN = Slice VLAN (Result)
Type = IP (static)
Total number of 8-Byte Reads: 1
Need to read first 2 or 6 Bytes of the IP header so when we do the write of last part of ethernet header we can fill out the 8-Byte Write.
Total number of 8-Byte Writes: 2 or 3
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
PAD (nB)
Ethernet
Trailer
CRC (4B)
Indicates fields that
need to be written
Indicates 8-Byte Boundaries
Assuming no IP Options

22. Queuing Assume we are using an external GE switch NPE Queueing:
LC RTM:
5 internal ports go to switch for traffic to/from NPE(s) and GPE(s)
5 external ports
1 to Internet
4 to local hosts
NPE RTM
5 ports used by NPUA to switch
Each port is associated with 1 of the LC’s external ports
5 ports used by NPUB to switch
GPE
1 GE port to switch
NPE Queueing:
Needs Rate control per port

23. NPE Queuing Queueing on a per port basis
Ports split across NPUA and NPUB (Phase 0.0)
Rate control per port
Rate control will need to be dynamically adjustable so we can balance bandwidth usage across NPs and GPE
Each MR gets N queues per port
MRs choose how to use the N queues
Quantum assigned per MR for each port
MRs can choose how to split among their N queues
May or may not make sense to assign queues on a per MI basis
Quantum being split across the queues means that an MR with only one active MI may not get its “fair” share.

24. NPE Queuing N P U A QM/Schd QM/Schd N P U B 5-Port Tx
...
Port 1(to LC Port 1)
5-Port Tx R T M
SPI
Switch
Port 2(to LC Port 2) N P U A
Port 3(to LC Port 3)
Port 4(to LC Port 4)
Port 5(to LC Port 5)
QM/Schd
...
Port 6 (to LC Port 1) N P U B
Port 7(to LC Port 2)
Port 8(to LC Port 3)
Port 9(to LC Port 4)
Port 10(to LC Port 5)

25. LC Ingress Queuing 1 5-port QM support 5 GPEs
Each “port” supports one GPE
Only one queue needed per port.
Rate control, 1 Gb/s per port
In Phase 0.0, HW flow control will provide/enforce this rate.
1 5-port QM support all NPEs
Each “port” supports one NPE
Rate control should not be needed

26. LC Egress Queuing Queueing on a per MR per port basis
Similar to NPE Queuing, but each MR gets 1 queue per port
Default is equal quantum assigned to each MR on each port
All get equal “fair share”
Allow for changing quanta.

27. LC: Functional Blocks S W I T C H Phy Int Rx (2 ME) Key Extract (2 ME)
Lookup
(2 ME)
Hdr
Format
(1 ME)
QM/Schd
(2 ME)
Switch Tx
(2 ME) S W I T C H
Phy Int Tx
(2 ME)
QM/Schd
(2 ME)
Hdr
Format
(1 ME)
Lookup
(2 ME)
Key
Extract
(1 ME)
Switch Rx
(2 ME)

28. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
RBUF
Buf Handle(32b)
Eth. Frame
Len (16b)
Reserved
(12b)
Port
(4b)
Rx (2 Microengines):
Function:
Coordinate transfer of packets from RBUF to DRAM

29. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Buf Handle(32b)
Buf Handle(32b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
Eth. Frame
Len (16b)
Reserved
(12b)
Port
(4b)
Lookup Key[63-32] (32b)
Lookup Key[ 31-0] (32b)
Key_Extract (1 Microengine):
Function:
Extracts lookup key.
Peel ARP packets off and send to XScale???
Lookup Key (64b):
SL Type (4b): 0101b
Port (4b): May not be needed
IP DAddr (32b)
IP Proto (8b)
UDP DPort (16b)
Notes:
Frame offset in buffer is a constant and does not need to be read from Buffer Descriptor
Ethernet Hdr Length should be passed along chain so Hdr Format can figure out where to start writing its stuff.
Ethernet Header could have different lengths depending on whether VLANs are present or not.
IP Hdr 1st Word (32b)

30. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Buf Handle(32b)
Buf Handle(32b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
Lookup Key[63-32] (32b)
VLAN (16b)
Stats Index (16b)
Lookup Key[ 31-0] (32b)
DAddr
(8b)
Port
(4b)
QID (20b)
IP Hdr 1st Word (32b)
IP Hdr 1st Word (32b)
Lookup: Notes on next page

31. LC Ingress: Functional Blocks
Lookup:
Function:
Performs Lookup and passes result on to Hdr Format.
Lookup Key (64b):
SL Type (4b): 0101b
Port (4b): May not be needed
IP DAddr (32b)
IP Proto (8b)
UDP DPort (16b)
Lookup Result (56b):
DAddr (8b): only 8 bits of Ethernet DAddr are variable, other 40 are static per node.
VLAN (12b)
QID (20b)
Stats Index (16b)
Port (4b):
For case with external switch it is the actual physical interface to use
Also one port per GPE and one port per NPE
For case with switch blade, it is just used to spread traffic across QM/Scheduler?
Notes:
Does Lookup Key need to include Port? Seems like it should not.
Does Lookup still need Frame Length? Will it be maintaining any Byte Counters?
Result should not have to include RxMI, it is not used for anything.
Stats Index may be a Per MI stats index if desired.

32. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Buf Handle(32b)
Buffer Handle(32b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
Rsv
(4b)
Port
(4b)
Rsv
(4b)
QID(20b)
VLAN (16b)
Stats Index (16b)
DAddr
(8b)
Port
(4b)
QID (20b)
Frame Length (16b)
Stats Index (16b)
IP Hdr 1st Word (32b)
Hdr Format:
Function:
From lookup result:
re-writes just the ethernet header in DRAM to make frame ready to transmit.
Extract QID, Port, Stats Index and Frame Length to pass on to QM/Scheduler
May need to increment a counter based on Stats Index.
Notes:
Pass Size on to QM/Scheduler so it does not have to read buffer descriptor for Enqueue to update Q Length.
Offset to beginning of old Ethernet header should be constant but we don’t necessarily know how long it was so we don’t know where to put our new one.
Ethernet Hdr Len is used to determine where new header should go

33. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Buffer Handle(32b)
Buffer Handle(24b)
Rsv
(3b)
Port
(4b) V 1
Rsv
(4b)
Port
(4b)
Rsv
(4b)
QID(20b)
V: Valid Bit
Frame Length (16b)
Stats Index (16b)
QM/Scheduler (See Sailesh’s slides for more details)
Function:
Enqueue and Dequeue from queues
Scheduling algorithm
Drop Policy
Notes:

34. LC Ingress: Functional Blocks
Phy Int Rx
Key
Extract
Lookup
Hdr
Format
QM/Schd
Switch Tx S W I T C H
Buffer Handle(24b)
Rsv
(3b)
Port
(4b) V 1
TBUF
V: Valid Bit
Switch TX:
Function:
Coordinate transfer of packets from DRAM to TBUF
Notes:

35. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Key
Extract
Switch Rx S W I T C H
Buf Handle(32b)
RBUF
Eth. Frame
Len (16b)
Reserved
(12b)
Port
(4b)
Rx:
Function:
Coordinate transfer of packets from RBUF to DRAM
Notes:
Do we need port?
May not make sense to remove it since, it is there for other versions of Rx.

36. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Key
Extract
Switch Rx S W I T C H
Buf Handle(32b)
Buf Handle(32b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
Eth. Frame
Len (16b)
Reserved
(12b)
Port
(4b)
IP DAddr (32b)
Lookup Key
IP Proto
(8b)
Lookup Key –
UDP SPort (16b)
Reserved
(8b)
IP Hdr 1st Word (32b)
Key_Extract:
Function:
Extracts lookup key
Notes:

37. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Key
Extract
Switch Rx S W I T C H
Buf Handle(32b)
Buf Handle(32b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
IP DAddr (32b)
IP DAddr (32b)
Lookup Result [63-32] (32b)
Lookup Key
IP Proto
(8b)
Lookup Key –
UDP SPort (16b)
Reserved
(8b)
Lookup:
Function:
Performs Lookup and passes result on to Hdr Format.
Lookup Key:
IP Protocol (8b)
UDP Sport (16b)
Lookup Result (52b):
VLAN (12b): Value of 0x000 or 0xFFF, indicates invalid?
QID (20b)
Port (4b)
Stats/Counter Index (16b)
Static values for Egress Ethernet address:
Ethernet SAddr
Types: IP and/or 802.1Q
Notes:
Lookup does no processing on the lookup result.
Lookup Result [31-0] (32b)
IP Hdr 1st Word (32b)
IP Hdr 1st Word (32b)

38. Stats/Counter Index (16b)
Lookup Result
Buf Handle(32b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
IP DAddr (32b)
Rsvd
(4b)
VLAN(12b)
Stats/Counter Index (16b)
Rsvd
(4b)
Port
(4b)
Rsvd
(4b)
QID (20b)
IP Hdr 1st Word (32b)

39. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Key
Extract
Switch Rx S W I T C H
Buffer Handle(32b)
Buf Handle(32b)
IP Pkt
Length (16b)
Reserved
(8b)
Eth Hdr
Len (8b)
Rsv
(4b)
Port
(4b)
Rsv
(4b)
QID(20b)
IP DAddr (32b)
Ethernet
Frame Length (16b)
Cntr Index (16b)
Lookup Result [63-32] (32b)
Lookup Result [31-0] (32b)
IP Hdr 1st Word (32b)
Hdr Format:
Function:
From lookup result:
re-writes ethernet header in DRAM to make frame ready to transmit.
Extract QID and frame length to pass on to QM/Scheduler
Notes:
Pass Size on to QM/Scheduler so it does not have to read buffer descriptor for Enqueue to update Q Length.

40. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Key
Extract
Switch Rx S W I T C H
Buffer Handle(32b)
Buffer Handle(24b)
Rsv
(3b)
Port
(4b) V 1
Rsv
(4b)
Port
(4b)
Rsv
(4b)
QID(20b)
V: Valid Bit
Ethernet
Frame Length (16b)
Cntr Index (16b)
QM/Scheduler (See Sailesh’s slides for more details)
Function:
Enqueue and Dequeue from queues
Scheduling algorithm
Drop Policy
Memory Accesses:
DRAM: None
SRAM:
Q-Array Reads and Writes
Scheduling Data Structure Reads and Writes
QLength Data Structure Reads and Writes
Dequeue: Read Buffer Descriptor to retrieve Packet Size
Buffer Descriptor Accesses: Read packet size
Notes:

41. LC Egress: Functional Blocks
Phy Int Tx
QM/Schd
Hdr
Format
Lookup
Key
Extract
Switch Rx S W I T C H
TBUF
Buffer Handle(24b)
Rsv
(3b)
Port
(4b) V 1
V: Valid Bit
Switch TX:
Function:
Coordinate transfer of packets from DRAM to TBUF
Memory Accesses:
SRAM: Read Buffer Descriptor
DRAM: Transfer to TBUF
Buffer Descriptor Accesses:
Read Size and Offset
Notes:
Calculate DRAM address based on SRAM Descriptor address in buffer handle

42. NPE Functional Blocks Phase 0.0
Each NPU will only support 5 LC Interfaces.
Only need 1 Tx ME
Only need 1 QM ME
Lookup
(1-2 ME) Rx
(2 ME)
5-Port Tx
(1 ME)
QM/Schd
Demux
Hdr
Format
(1-3 ME)
Parse

43. NPE Functional Blocks Later Phases
We may need some input queuing/buffering to absorb bursts at high input rates (10Gb/s, 20MPkts/s) to keep Parse from being overloaded.
Add an SRAM Ring between Demux and Parse instead of NN Ring
If needed, add an extra ME to read from SRAM ring and put into NN ring
Lookup
(1-2 ME) Rx
(2 ME)
5-Port Tx
(1 ME)
QM/Schd
Demux
Hdr
Format
Parse

44. IPv4 MR Functional Blocks
Lookup Rx Tx QM
Parse
Header
Format
DeMux
RBUF
Buf Handle(32b)
Port
(8b)
Reserved
Eth. Frame
Len (16b)
Rx:
Function
Coordinate transfer of packets from RBUF to DRAM
Notes:
We’ll pass the Buffer Handle which contains the SRAM address of the buffer descriptor.
From the SRAM address of the descriptor we can calculate the DRAM address of the buffer data.

45. IPv4 MR Functional Blocks
Lookup Rx Tx QM
Parse
Header
Format
DeMux
Buf Handle(32b)
Port
(8b)
Reserved
Eth. Frame
Len (16b)
Rx UDP DPort (16b)
Buf Handle(32b)
Slice ID (VLAN) (16b)
MN Frm Offset (16b)
MN Frm Length(16b)
Rx IP SAddr (32b)
Reserved
(12b)
Rx UDP SPort (16b)
Code
(4b)
DstAddr (6B)
SrcAddr (6B)
Ethernet
Header
Type=802.1Q (2B)
VLAN (2B)
Type=IP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol = UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
Header IP
Slice Data Ptr (32b)
Dst Addr (4B)
SRAM Pointer/Range to static data area need to be added.
IP Options (0-40B)
Src Port (2B)
UDP
Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload
(MN Packet)
Demux notes on next slide
PAD (nB)
Ethernet
Trailer
CRC (4B)

46. Demux Function Notes: Read Pkt Header from DRAM
Can we assume IP Header checksum is ok?
This frame comes from LC or from GPE. Are they each trusted?
Except for the possibility of options, we have to read all parts of the IP header anyway, so maybe go ahead and calculate the checksum if we have the cycles.
Extract the following fields from Pkt Header and pass to Parse
VLAN: equivalent to Slice ID (Slice == MR)
UDP DPort: equivalent to Substrate MI
IP SAddr, UDP Sport: extra interface information for planetlab slices.
UDP length: equivalent to MN frame length
Calculate offset into buffer of start of MN Pkt Header and pass to Parse
Based on Slice ID, determine code option.
How do we intended to do this?
Load a table in Local memory from SRAM at init time?
Notes:
Code: Identifies “Code Option”
Which set of static Parse code to execute.
We will allow a very small number of code options.
How does Demux identify code option?
Port should not be important in this implementation, right?
8-Byte Dram Reads: 5-10
depending on amount of IP Options and whether we need to do IP Hdr Checksum

47. IPv4 MR Functional Blocks
Lookup Rx Tx QM
Parse
Header
Format
DeMux
Buf Handle(32b)
Lookup Key[111-80] DA (32b)
Buf Handle(32b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
Lookup Key[ 79-48] SA (32b)
Lookup Key[ 47-16] Ports (32b)
Lookup Key
Proto/TCP_Flags
[15- 0] (16b)
Exception
Bits (12b)
Lookup Key[ ]
Slice ID/Rx UDP DPort (32b) L
Flags
(4b)
MN Frm Length(16b)
MN Frm Offset (16b)
Slice ID (VLAN) (16b)
Rx UDP DPort (16b)
Rx IP SAddr (32b)
Rx UDP SPort (16b)
Reserved
(12b)
Code
(4b)
Slice Data Ptr (32b)
Slice Data Ptr (32b)
Reserved
(28b)
Code
(4b)
Parse notes on following page

48. Parse Notes Parse Notes: Function
MR-specific header processing
Handles IPv4 header validation
Decrements TTL and recalculates Hdr Checksum.
Generate MR-specific lookup key (144 bits) from packet
Generate Exception bits to be passed on to Hdr Format (via Lookup) so Hdr Format can create shim fields for slow path packets going to Control Processor.
Input data to Parse is MN Frame Offset and Length
Output data from Parse is IP Pkt Offset and Length
If there is a MN Internal Header, Parse basically consumes it.
Hdr Format needs to start at the beginning of the IP Header and re-write headers upward in the Buffer.
If Parse receives a !Reclassify frame/pkt from the CP,
it should not decrement the TTL.
Assume this was done on the first trip through the MR.
it should not generate exceptions again.
Code: Identifies “Code Option”
Which set of static Parse code to execute.
We will allow a very small number of code options.
L Flags:
bit 0: 0: Normal, 1: Substrate Lookup
Need to revisit the details here of what is in the MN Internal Header
bit 1: 0: Normal, 1: NH MN Address present in Key Word[1]
Key Word[0] = MR/MI
Bit 1 should never be set without bit 0 also being set.
Notes:
UDP DPort (Rx MI) needs to be passed to Header Format (through Lookup) so that Header Format can include it in the MN Internal Header of packets that end up on the exception or Local delivery path. This will let the Control Processor know what interface the exception packets arrived on.

49. IPv4 MR Functional BlocksRx
DeMux
Parse
Lookup
Header
Format QM Tx
Lookup Key[111-80] DA (32b)
Buf Handle(32b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
Lookup Key[ 79-48] SA (32b)
Lookup Key[ 47-16] Ports (32b)
Lookup Key
Proto/TCP_Flags
[15- 0] (16b)
Exception
Bits (12b)
Lookup Key[ ]
Slice ID/Rx UDP DPort (32b) L
Flags
(4b)
Buf Handle(32b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
Slice ID (VLAN) (16b)
Rx UDP DPort(16b) R S V d
(1b) H
(1b) L D
(1b) D
(1b)
Exception
Bits (12b)
Cntr Index (16b)
Tx IP DAddr (32b)
Tx UDP DPort (16b)
Tx UDP SPort(16b)
DA(8b)
Port
(4b)
QID(20b)
Slice Data Ptr (32b)
Slice Data Ptr (32b)
Reserved
(28b)
Code
(4b)
Reserved
(28b)
Code
(4b)
Lookup notes on next page

50. IPv4 MR Functional BlocksRx
DeMux
Parse
Lookup
Header
Format QM Tx
Buf Handle(32b)
Buf Handle(32b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
Lookup Key[ ]
Slice ID/Rx UDP DPort (32b)
Slice ID (VLAN) (16b)
Rx UDP DPort(16b)
Lookup Key[111-80] DA (32b) R S V d
(1b) H
(1b) S R V d
(1b) D
(1b) L D
(1b)
Reserved (11b)
Cntr Index (16b)
Lookup Key[ 79-48] SA (32b)
Lookup Key[ 47-16] Ports (32b)
Tx IP DAddr (32b)
Lookup Key
Proto/TCP_Flags
[15- 0] (16b)
Reserved (16 b)
Tx UDP DPort (16b)
Tx UDP SPort(16b)
DA(8b)
Port
(4b)
QID(20b)
Slice Data Ptr (32b)
Slice Data Ptr (32b) L
Flags
(4b)
Exception Bits
(12b)
Reserved
(12 b)
Code
(4b)
Reserved
(4b)
Exception Bits
(12b)
Reserved
(12 b)
Code
(4b)
Lookup notes on next page
PROPOSED CHANGE TO LOOKUP INPUT AND OUTPUT DATA FORMATS (10/12/06)

51. Lookup Function Buf Handle(32b) Result: Output
Perform lookup in TCAM based on MR Id and lookup key
Result:
IP DAddr (32b)
UDP SPort (16b)
UDP DPort (16b)
Eth Daddr (8b) : low order 8 bits. Top 40 are pre-defined
Port (4b): Phase 0.0
QID (20b)
Cntr Index (16b): Used for incrementing Counters
Output
Buf Handle
Exception Bits: For Parse to communicate to Header format info about exception packets
Slice ID (VLAN)
Rx UDP DPort
IP Pkt Length: Length of just the IP Pkt
IP Pkt Offset: Offset from start of buffer to the start of IP Pkt header
IP/UDP Header fields: IP DAddr, UDP DPort, UDP SPort
QID
Port: Phase 0.0
Bits:
H: Hit
D:Drop,
NH: NH MN Address present,
MAC: MAC Address needed,
LD: Local Delivery
Rsv (3b): Reserved,
MR Bits: For MR-specific usage
Port
(4b)
QID(20b)
DA(8b)
Tx IP DAddr (32b)
Buf Handle(32b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
Cntr Index (16b) R V S d
(1b) D H
Exception
Bits (12b) L
Rx UDP DPort(16b)
Slice ID (VLAN) (16b)
Tx UDP SPort(16b)
Tx UDP DPort (16b)
Slice Data Ptr (32b)
Reserved
(28b)
Code
(4b)

52. IPv4 MR Functional BlocksRx
DeMux
Parse
Lookup
Header
Format QM Tx
Port
(4b)
QID(20b)
DA(8b)
Tx IP DAddr (32b)
Buf Handle(32b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
Cntr Index (16b) S R V d
(1b) D H
Exception
Bits (12b) L
Rx UDP DPort(16b)
Slice ID (VLAN) (16b)
Tx UDP SPort(16b)
Tx UDP DPort (16b)
Ethernet
Frame Length (16b)
Buffer Handle(32b)
Cntr Index (16b)
QID(20b)
Rsv
(4b)
Port
Slice Data Ptr (32b)
Reserved
(28b)
Code
(4b)
Header Format notes on next page.

53. IPv4 MR Functional Blocks
Header Format
Function
MR specific packet header formatting
MR specific Lookup Result processing
Drop bit, Hit/Miss bits, NH, MAC, LD, …
LD and Exceptions may go to different UDP Ports.
Result:
Ethernet DAddr of LC (Result)
Constant/Static fields needed:
IP DAddr of GPE for exception and LD (configured)
IP SAddr (configured)
UDP Sport and UDP DPort for exception and LD (configured)
QID and Port# to use for exception path and LD (configured)
IP Proto = UDP (constant)
Other IP Header fields: calculated or constants
Ethernet SAddr of NPE (configured)
VLAN (passed as input)
First Ethernet Type = 802.1Q (constant)
Second Ethernet Type = IP (constant)

54. IPv4 MR Functional BlocksRx
DeMux
Parse
Lookup
Header
Format QM Tx
Ethernet
Frame Length (16b)
Buffer Handle(32b)
Cntr Index (16b)
QID(20b)
Rsv
(4b)
Port
Buffer Handle(24b)
Rsv
(3b)
Port
(4b) V 1 QM
Function
CRF queue management for Meta Interface queues
For performance reasons, QM may actually be implemented as multiple instances
Each instance on a separate ME would support a separate set of Meta Interfaces.
See next slide for more details…

55. QM/Scheduler on Multiple MEs
Header Format
Input
Hlpr
(1 ME)
QM/Schd
(1 ME) Tx
MR-1
MR-n
. . .
QM/Schd
(1 ME) Tx
QID(20b)
Ethernet
Frame Length (16b)
Buffer Handle(32b)
Rsv
(4b)
Cntr Index (16b)
Port
NN/Scratch Rings
NN Ring
Buffer Handle(24b)
Rsv
(3b)
Port
(4b) V 1
QID(32b):
Reserved (8b)
QM ID (3b)
QID(17b): 1M queues per QM
Input Hlpr would use QM ID to select Scratch ring on which to put request.
QM/Sched then sends on its output NN/scratch ring to its associated Tx
With 64 entries in Q-Array and 16 entries in CAM, max number of QM/Schds is probably 4 (2 bits).
We’ll set aside 3 bits to give us flexibility in the future.

56. IPv4 MR Functional BlocksRx
DeMux
Parse
Lookup
Header
Format QM Tx
Buffer Handle(24b)
Rsv
(3b)
Port
(4b) V 1
TBUF Tx
Function
Coordinate transfer of packets from DRAM to TBUF

57. LC: Buffer Descriptor
Hopefully we can use the same buffer descriptor for the LC and the CRF Processing Engine.
There might be some fields that are used on one and not on the other but that’s ok (MR_ID, TxMI, VLAN not needed on LC)
PE Buffer Descriptor:
LW0: buffer_next 32 bits Next Buffer Pointer (in a chain of buffers)
LW1: offset bits Offset to start of data in bytes
LW1: BufferSize 16 bits Length of data in the current buffer in bytes
LW2: reserved bits reserved/unused
LW2: reserved bits reserved/unused
LW2: free_list bits Freelist ID
LW2: packet_size 16 bits (Total packet size across multiple buffers)
LW3: MR_ID bits Meta Router ID
LW3: TxMI bits Transmit Meta Interface
LW4: VLAN bits VLAN
LW4: reserved 16 bits reserved/unused
LW5: reserved 32 bits reserved/unused
LW6: reserved 32 bits reserved/unused
LW7: packet_next 32 bits pointer to next packet (unused in cell mode)
Leave multi-buffer fields there as a template for the dedicated blade implementation of a jumbo-frame MR.
Also reduces changes to Rx, Tx, and QM and reduces potential problems.
So, far I haven’t found anything extra that we need on LC.
VLAN
Packet_Next
MR_ID
TxMI
Free_List
Packet_Size
Buffer_Next
Offset
Buffer
Descriptor
Buffer_Size

58. Intel Buffer Descriptor
Buffer_Next (32b)
LW0
Buffer_Size (16b)
Offset (16b)
LW1
Packet_Size (16b)
Free_list
(4b)
Rx_stat
(4b)
Hdr_Type
(8b)
LW2
Input_Port (16b)
Output_Port (16b)
LW3
Next_Hop_ID (16b)
Fabric_Port
(8b)
Reserved
(4b)
NHID
type
(4b)
LW4
ColorID
(4b)
Reserved
(4b)
FlowID (32b)
LW5
Class_ID (16b)
Reserved (16b)
LW6
Packet_Next (32b)
LW7

59. Our Buffer Descriptor Buffer_Next (32b) Buffer_Size (16b) Offset (16b)
LW0
Buffer_Size (16b)
Offset (16b)
LW1
Packet_Size (16b)
Free_list
0000
(4b)
Reserved
(4b)
Reserved
(8b)
LW2
Reserved (16b)
Stats Index (16b)
LW3
Reserved (16b)
Reserved
(8b)
Reserved
(4b)
Reserved
(4b)
LW4
Reserved
(4b)
Reserved
(4b)
Reserved (32b)
LW5
Reserved (16b)
Reserved (16b)
LW6
Packet_Next (32b)
LW7

60. Extra
The next set of slides are for templates or extra information if needed

61. Text Slide Template

62. Image Slide Template

63. Hardware Definitions Blade Slot 1: Line Card Blade Slot 2: MetaRouter
NPUA: Ingress
NPUB: Egress
Linux Controller: foghorn.arl.wustl.edu
Windows Controller: techX01.arl.wustl.edu
Blade Slot 2: MetaRouter
NPUA: IPv4 MR #1
NPUB: IPv4 MR #2 (optional)
Linux Controller: coffee.arl.wustl.edu
Windows Controller: techX03.arl.wustl.edu
SPI Connections:
LC Blade:
RTM[0:4]  NPUA[0:4] External links to LC Ingress
NPUA[5:9]  RTM[5:9] LC Ingress to Internal Links
RTM[5:9]  NPUB[5:9] Internal links to LC Egress
NPUB[0:4]  RTM[0:4] LC Egress to External Links
MR Blade:
NPUA[0:4]  RTM[0:4]
NPUB[0:4]  RTM[5:9]
Hardwire Connections:
LC RTM[0]  coffee
LC RTM[4]  foghorn
LC RTM[5]  MR RTM[0]
LC RTM[6]  MR RTM[1]

64. HW Demo Quick diagram 6 5 7 4 10 1 8 3 9 2 NPUA 10 9 8 7 6 5 4 3 2 1MR
coffee 7 4 10 1 10 9 8 7 6 5 4 3 2 1 LC 8 3 9 2
NPUA – LC Ingress
NPUB – LC Egress

65. Multiple Slices, Code Options and MetaInterfaces
UDP Dport
VLAN, UDP Dport
VLAN, UDP Dport
UDP Dport
IPv4 MR
0xC100
0xC100
coffee
0x001,0xC100
0x001,0xC100
0xC200
Slice 1
( )
Code Option 1
coffee
0x001,0xC101
0x001,0xC101
0xC200
0xC300
0x001,0xC102
0x001,0xC102
0xC300
0xC400
0x001,0xC103
0x001,0xC103
0xC400
0xC101
foghorn
0x002,0xC200
0x002,0xC200
0xC101
0xC201
Slice 2
( )
Code Option 1
foghorn
0x002,0xC201
0x002,0xC201
0xC201
0xC301
0x002,0xC202
0x002,0xC202
0xC301
0xC401
0x002,0xC203
0x002,0xC203
0xC401
NPUA – LC Ingress
NPUB – LC Egress
0xC102
xxx
0x003,0xC300
0x003,0xC300
0xC102
Slice 3
( )
Code Option 2
xxx
0xC202
0x003,0xC301
0x003,0xC301
0xC202
0xC302
0x003,0xC302
0x003,0xC302
0xC302
0xC402
0x003,0xC303
0x003,0xC303
0xC402
0xC103
yyy
0x004,0xC400
0x004,0xC400
0xC103
Slice 4
( )
Code Option 2
yyy
0xC203
0x004,0xC401
0x004,0xC401
0xC203
0xC303
0x004,0xC402
0x004,0xC402
0xC303
0xC403
0x004,0xC403
0x004,0xC403
0xC403

66. Multi Slice IP Addresses for Test Packets
VLAN
UDP Rx DPort
Tunnel SA
Tunnel DA
IP SA
IP DA
Proto
SPort
DPort 1
0xC100
0x11 2
0xC101
0xC102
0xC103
0xC200
0xC201
0xC202
0xC203 3
0xC300
0xC301
0xC302
0xC303 4
0xC400
0xC401
0xC402
0xC403

67. November Intel Demo 4 MRs 4 MIs per MR 4 QIDs per MI 4 Hosts Monitor:
Numbered 1, 2, 3, 4
4 MIs per MR
4 QIDs per MI
To the User:
Numbered 1, 2, 3, 4 on each MI
Internally to the NP filters:
np_qid = 16 * (MR# - 1) + 4 * (MI# - 1 ) + USER_QID
4 Hosts
coffee, momcat, tabby, foghorn
Each with a presence on each of the MRs
Monitor:
Input BW and Pkts per MI at the LCI HF Pre-Q counters
Output BW and Pkts per MI at the LCE HF Pre-Q counters
Q-Length at the IPv4 MR QM
LC Stats Index:
Np_stats_index = 4 * (MR# - 1) + MI#

68. RTM Connections LC SWITCH? NPUB NPUA R T M R T M H4 H3 H2 H1 P10 P10

69. Demo Config MR-1 1 2 4 3 H1 H2 H4 H3 MR1_Flow12 MR1_Flow13 MR1_Flow14
H1 (coffee)
H2 (momcat)
H3 (tabby)
H4 (foghorn)
MR4_Flow21
MR4_Flow23
MR4_Flow24 1 2
MR-4 4 3
MR4_Flow41
MR4_Flow42
MR4_Flow43
MR4_Flow31
MR4_Flow32
MR4_Flow34 H4 H3

70. Tunnels, LCI Filters, QIDs and Stats IndicesMR M1
VLAN
UDP Rx DPort
Tunnel SA
Tunnel DA
Stats Index
QID 1
0xC100 2
0xC101 3
0xC102 4
0xC103
0xC200 5
0xC201 6
0xC202 7
0xC203 8
0xC300 9
0xC301 10
0xC302 11
0xC303 12
0xC400 13
0xC401 14
0xC402 15
0xC403 16

71. Tunnels, LCE Filters, QIDs and Stats IndicesMR M1
VLAN
UDP Rx DPort
Tunnel SA
Tunnel DA
Stats Index
QID 1
0xC100 2
0xC101 3
0xC102 4
0xC103
0xC200 5
0xC201 6
0xC202 7
0xC203 8
0xC300 9
0xC301 10
0xC302 11
0xC303 12
0xC400 13
0xC401 14
0xC402 15
0xC403 16

72. MR-1 Filters and QIDs DA SA MI USER_QID NP_QID 192.168.81.1 1 2 3 4 5 7 8 9 10 12 13 14 15

73. MR-2 Filters and QIDs DA SA MI USER_QID NP_QID 192.168.82.1 1 2 18 3 19 4 20 21 23 24 25 26 28 29 30 31

74. MR-3 Filters and QIDs DA SA MI USER_QID NP_QID 192.168.83.1 1 2 34 3 35 4 36 37 38 40 41 42 44 45 46 47

75. MR-4 Filters and QIDs DA SA MI USER_QID NP_QID 192.168.84.1 1 2 50 3 51 4 52 53 54 56 57 58 60 61 62 63

76. MR-1 Packet Flows VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA
Proto
SPort
DPort 1
0xC100
0x11 2 3 4
0xC101
0xC102
0xC103

77. MR-2 Packet Flows VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA
Proto
SPort
DPort 2
0xC200
0x11 1 3 4
0xC201
0xC202
0xC203

78. MR-3 Packet Flows VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA
Proto
SPort
DPort 3
0xC300
0x11 2 1 4
0xC301
0xC302
0xC303

79. MR-4 Packet Flows VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA
Proto
SPort
DPort 4
0xC400
0x11 2 1 3
0xC401
0xC402
0xC403

80. End of November Intel Demo Slides