Presentation is loading. Please wait.

Presentation is loading. Please wait.

SPP Version 1 Router Plans and Design

Similar presentations


Presentation on theme: "SPP Version 1 Router Plans and Design"— Presentation transcript:

1 SPP Version 1 Router Plans and Design
John DeHart

2 SPP Versions SPP Version 0: SPP Version 1: SPP Version 2:
What we used for SIGCOMM Paper SPP Version 1: Bare minimum we would need to release something to PlanetLab Users SPP Version 2: What we would REALLY like to release to PlanetLab users.

3 Upcoming Meetings and Dates
techX Meetings: 2/05/08: Status update SPP V2 SPP V1 SPP V1 Control NAT SPP V1 next demo ONL NPR 2/12/08: ONL NP Router Distribution issues 2/19/08: NAT Review 2/26/08: SPP V1 Demo 3/04/08: SPP V2 Review Other dates: 2/22/08: Department Visitors day for doctoral candidates Clean up labs and machine rooms Etc.

4 Work Items (Updated 4/9/08)
SPP V2 Design SPP V1 DONE: Get Init Scripts working again without Control Remove what is not needed in init scripts DONE: UDP Tunnel Traffic Generation

5 3/4/08: Fred Questions Add info on Substrate only NPE Lookups
NPE Lookup Key: What is the 1 bit type field? This differentiates between: 0: Substrate only lookup: when GPE returns a packet with no reclassify 1: Normal lookup Slice ID: 15 bits, but we only use 12 bits, right? Yes, it should be 12 bits and there should be 3 reserved bits. Actually we are limiting it to 11 bits now. Slides have been updated Move Exception bits in Lookup  HF data Implement LD bit in Lookup Result What is the order of the lookup key ports ? Lookup key fields, which come from Tunnel header and which come from encapsulated header. First word is all Substrate Info Rest is MN info Why don’t we have tunnel rx IP Daddr and port number in key to fully map meta-interface. Yes, good point. This has been raised before and we keep side stepping it. We will use an Index in the key that selects 1 of 16 possible Rx IP DAddrs. Slides updated but not implemented yet. NPE Lookup Result: What is H Flag? HIT What is D Flag? DROP 12 exception bits? Yes, there are 12 exception bits but they are not part of the lookup result. They get carried over from the input ring from Parse to the output ring to HF. MAC Address still there, but we don’t use it anymore right? Done (removed from slides) DONE: QParams: Have we added qlength in packets yet? GPE VLAN issues seem to require that a different set of IP Addresses be used for each different VLAN. This might require us to put the NPE Src IP Address for packets going to the GPE in the GPE Info table and ignore what is in the Per Scheduler IP Src address table . Can the data path do this? Looks like it should be easy since substrate encap uses the gpe info struct to figure out which scheduler is to be used so it can read the right src_ip struct. So the data is all there at the right place already.

6 QM: QLength in Packets QParams: Have we added qlength in packets yet?
No, I haven’t had time for that yet. Current: Flags: Length Valid Head Valid Tail Valid QLength (Bytes) (32b) Unused (29b) L V 1b H V 1b T V 1b Threshold (Bytes) (32b) QLength (Bytes) (32b) Quantum (32b) Threshold (Bytes) (32b) Unused (32b) Quantum (32b) QParams in SRAM QParams in Local Memory Flags(4b): Reserved Length Valid Head Valid Tail Valid Proposed: Unused (4b) QLength (Pkts) (28b) QLength (Pkts) (28b) T V 1b H L R S v QLength (Bytes) (32b) QLength (Bytes) (32b) Threshold (Bytes) (32b) Threshold (Bytes) (32b) Quantum (32b) Quantum (32b) QParams in SRAM QParams in Local Memory

7 Rx IP DAddr in Key Why don’t we have tunnel rx IP Daddr and port number in key to fully map meta-interface. Yes, good point. This has been raised before and we keep side stepping it. To fix it for real we need to make the key larger and take the performance hit. Actually, what we decided to do was to implement a small table (16 entries) of IP Addresses. The address that is matched by the incoming Rx IP DAddr will have its index used in the key as a 4-bit field. This is now reflected in the slides but has not yet been implemented in the code.

8 LD Bit in Lookup Result Implement LD bit in Lookup Result

9 November HW Test First test in HW should happen in Nov. 2007 Plan:
Retry SPP V0 demo on Dev. Chassis with new boards Finish all three projects in Simulation: LCI: Currently missing ICMP and NAT LCE: Currently missing NAT KE and Lookup are nearing completion HF MAC Address lookup fine tuning Flow stats: FS2 nearing completion, needs archive thread and testing testing testing Initialization scripts need work NPE: One more memory update needed for Substrate Encap (DRAM write) Working on initialization scripts. Testing Test all three in simulation including initialization scripts Convert *.ind initialization scripts to ‘cmd’ utility HW initialization scripts Review *.ind and ‘cmd’ scripts with Fred and control group TCAM utility for SPP V1? Plan A: Use Jonathon’s test utility Plan B: Packet generation for HW test? Use Traffic Generators? Test all in HW

10 November HW Test Status (11/29/07): Next Steps
DONE: Packets going through all three projects: DONE: TCAM Utilities for all three projects seem to be working Next Steps DONE: NPE config/init (JDD and MLW) Orchid (MLW) DONE: V1 Testing (JDD) This results in November HW Test milestone HF MAC Address usage and Initialization cleanup (JDD) LCI: DONE LCE: DONE NPE: Next Move NPE to NPUA (JDD): DONE We are still using the config we had to use because NPUA on the pre-production board was flakey. Orchid Integration (MLW and JDD) On hold. Performance testing (JDD and MLW) Expand lookup filters to exercise all Schedulers Need access to traffic generators or hosts with Charlie’s UDP Tunnel driver Flow Stats testing (JDD, JM) NAT (DMZ) ICMP pkt handling (JDD) LCI KeyExtract and LCE KeyExtract need to extract ICMP ID when Protocol==ICMP Control Integration (JDD, FK, etc) In Progress RLI Monitoring V2: To begin January 2008 (JDD, MLW)

11 Control Demo Notes Fast Path changes needed: QID changes : DONE
All three projects. Change to: QM_ID (2b) Sched_ID (3b) QID (15b) 32K Queues available with now implicit association with a QM or scheduler. Blocks Affected: QM: Format of Input, position of SchedID and QID LCI: Lookup: Format of output, change the format of TCAM result to report index in first word and results starting in second word so we can get a full 64 bit result HF: Format of Input, Format of Output, position of VLAN in input data PortSplitter: Format of Input, Format of Output, position of QM_ID in input data NPE: Lookup: Format of output HF : Format of Input, Format of Output SubstrateEncap : Format of Input, Format of Output, position of QM_ID in input data LCE: Lookup: Format of output , change the format of TCAM result to report index in first word and results starting in second word so we can get a full 64 bit result Initialization? How to test?

12 Control Demo Notes Max Buffer Limit (todo notes)
Fast Path changes needed (continued): LC HF MAC tables need to be dynamically read: DONE LCI Needs: Initialization of 1 SMAC Address (Fabric) Table of per scheduler DMAC addresses (1 per scheduler) LCE Needs: Table of per scheduler DMAC/SMAC addresses (1 pair per scheduler) To simplify things, lets have LCI and LCE use the same format table: Dst MAC (8 bytes) Src MAC (8 bytes) NPE Substrate Encap GPE info table entries need to be moved out of Slice data space and into a separate table of their own indexed by VLAN. DONE MAC Address coming from NPE to Egress is wrong FIXED NPE Substrate Encap has hard coded DMac address. Needs to read the MAC address from memory when it reads the IP Src address. Remove the 8 bits of DMAC from Lookup result DONE (no longer used) Max Buffer Limit (todo notes) Change JDD TCAM utilities to use DB IDs that Control uses.

13 Notes For NPE look at putting a limit on the number of outstanding buffers a Slice has at a time. Add a counter to the Substrate Decap VLAN/Slice table. When SD gets a packet, increment the counter for that Slice When a buffer is freed have the generic buf_free code decrement the counter for that slice. This will probably require recording the Slice ID in the buffer descriptor and having the buf_free code read the descriptor. Look at using all 10 external interfaces on LC Each interface that is used will be connected to different ports on the same router. Thus the SPP node does not have to worry about participating in routing protocols in V1. Use both fabric interfaces on GPEs V1 will use just one of them The one that is used will be associated with 1 external interface. The interfaces from different GPEs may or may not be associated with different external interfaces. There may be cases where GPEs share an IP Address There may be cases where GPEs have different IP Addresses We need to support both cases Check on how we handle fragmentation Add Ring specs to block diagram Schedule for upcoming meetings: 8/14: Charlie’s SIGCOMM talk and NAT 8/21: Plugin Framework (Shakir) 8/28: Flow Stats (JMM)

14 SPP V1 Plans Main focus today will be on the LC: SPP Version 1:
1 5-Port NPE (still don’t use NPUB) Support Multiple External IP Addresses Switch Blade integration 10GE Tx module integration ARP: Probably not needed in V1 NAT: Flow Stats: Egress Traffic monitoring MR Code Options Anything new? Control Local Control Booting NPU Add/Remove Slices MR Control Add/Remove Routes Node Manager GPE Multiple GPEs NAT SSH Forwarding PLC integration Main focus today will be on the LC: Block/ME design Lookups Flow stats ARP

15 Cycle Budget (min eth packets)
To hit 5 Gb rate: 76B per min IPv4 packet (64 min Eth + 12B IFS) 1.4Ghz clock rate 5 Gb/sec * 1B/8b * packet/76B = 8.22 Mp/sec 1.4Gcycle/sec * 1 sec/ 8.22 Mp = cycles per packet compute budget: 170 cycles latency budget: (threads*170) 8 threads: 1360 cycles To hit 10 Gb rate: 10 Gb/sec * 1B/8b * packet/76B = Mp/sec 1.4Gcycle/sec * 1 sec/ Mp = cycles per packet compute budget: 85 cycles latency budget: (threads*85) 8 threads: 680 cycles

16 Cycle Budget (IPv4 MN packets)
To hit 5 Gb rate: 90B per min IPv4 packet (78 min IPv4MN + 12B IFS) 1.4Ghz clock rate 5 Gb/sec * 1B/8b * packet/90B = 6.94 Mp/sec 1.4Gcycle/sec * 1 sec/ 6.94 Mp = cycles per packet compute budget: 201 cycles latency budget: (threads*201) 8 threads: 1608 cycles To support 6.94 M pkts/sec we can Read 28 Words and Write 28 Words per pkt per SRAM Bank (200M/6.94M) = To hit 10 Gb rate: 10 Gb/sec * 1B/8b * packet/90B = Mp/sec 1.4Gcycle/sec * 1 sec/ Mp = cycles per packet compute budget: 100 cycles latency budget: (threads*100) 8 threads: 800 cycles To support M pkts/sec we can Read 14 Words and Write 14 Words per pkt per SRAM Bank (200M/13.88M) =

17 Cycle Budget (Average Pkts)
To hit 5 Gb rate: 218B per min IPv4 packet (200 avg IPv4MN + 12B IFS) 1.4Ghz clock rate 5 Gb/sec * 1B/8b * packet/218B = 2.87 Mp/sec 1.4Gcycle/sec * 1 sec/ 2.87 Mp = cycles per packet compute budget: 487 cycles latency budget: (threads*487) 8 threads: 3896 cycles To support 2.87 M pkts/sec we can Read 69 Words and Write 69 Words per pkt per SRAM Bank (200M/2.87M) = To hit 10 Gb rate: 10 Gb/sec * 1B/8b * packet/218B = 5.74 Mp/sec 1.4Gcycle/sec * 1 sec/ 5.74 Mp = cycles per packet compute budget: 243 cycles latency budget: (threads*243) 8 threads: 1944 cycles To support 5.74 M pkts/sec we can Read 34 Words and Write 34 Words per pkt per SRAM Bank (200M/5.74M) =

18 SPP V1 ARP Notes Statically configure the Ethernet Addr of next hop(s). Don’t need ARP in V1. LC uses scheme similar to ONL LCE Lookup result contains Next Hop IP or NH Ethernet Addr. If NH Ethernet Addr is present than update packet and send If NH IP Addr present instead of NH Ethernet Addr then send to XScale Need to define shim/descriptor for LCE to XScale Physical Interface NH IP Address XScale will send ARP Broadcast on physical interface LCI receives Unicast ARP Response from RTM Port Sends to XScale indicating which physical interface recv’d on. XScale updates filter table If XScale has waiting packet, send to data path. LCI receives ARP Broadcast from RTM port XScale processes and sends ARP Response if needed. ARP Entry Aging.

19 SPP V1 ARP Interfaces LCI to XScale Interface LCE to XScale Interface
LCI just needs to detect EtherType Field of ARP Should be able to do this in Key Extract. Code already there to detect ARP and send to XScale. We may have to adjust for shim/descriptor to communicate additional info to XScale LCE to XScale Interface Needs to be Post Lookup and Pre QM. Needs to update the Shim/Descriptor to send info to the XScale. Hdr Format is probably the best place for this. XScale to LCE Interface Should be Queued to keep Port rate control sane. Does it need to be a separate scratch ring or can it go directly into the QM input ring(s)?

20 SPP V1 NAT Notes NAT Notes moved to separate file:
SPP_V1_NAT_design.ppt

21 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
SCR XScale NAT Scratch Rings R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

22 SPP V1 LC Egress with 1x10Gb/s Tx
SCR XScale NAT Scratch Rings S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN T B U F QM0 SCR Port Splitter Flow Stats1 SCR R T M M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR SCR Stats (1 ME) SCR SRAM3 SRAM1 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

23 SPP V1 LC Egress with 10x1Gb/s Tx
SCR XScale NAT Scratch Rings S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter R T M M S F Flow Stats1 SCR SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR Stats (1 ME) SCR SRAM3 SRAM1 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

24 INGRESS Block Interfaces

25 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
SCR XScale NAT Scratch Rings R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

26 Notes on Frame vs. Pkt Lengths
RX reports Ethernet Frame Length KE passes along IP Pkt length and Ethernet Hdr Length HF uses Ethernet Hdr Length and Buffer Offset to find start of IP Pkt so it can put on new ethernet header. HF passes along Ethernet Frame Length TX needs Ethernet Frame Length which it gets from buffer descriptor Buffer Size QM Dequeue gets length from buffer descriptor Thus it will get Ethernet Frame Length just like TX QM Enqueue gets a length from input ring which must agree with what QM Dequeue gets from buffer descriptor. Thus: HF must pass Ethernet Frame length in output ring AND it must write it to buffer descriptor. QM Link rates should include IFS, etc.

27 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
SCR XScale NAT Scratch Rings Buf Handle(24b) Intf (4b) Reserved (12b) Eth. Frame Len (16b) Rx Flags (8b) R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

28 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
Flags (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) Rsv (4b) Intf (4b) SCR XScale NAT Scratch Rings Lookup Key IP DAddr (32b) Protocol (8b) UDP DPort (16b) Type (8b) IP Hdr 1st Word (32b) R B U F IP Hdr 2nd Word (32b) R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM Buf Handle(24b) Intf (4b) Reserved (12b) Eth. Frame Len (16b) Rx Flags (8b) NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

29 SPP V1 LC Ingress Flags(8b) NAT Scratch Rings R B U F R T M M S F
XScale ICMP ERR Flags(8b) Rsvd 2b IE 1b T 1b U 1b I 1b N 1b H 1b NAT Scratch Rings SCR Rsv 1b IE 1b TCP Flags 6b Hit TCP UDP ICMP NAT ICMP ERR SCR R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Flags (8b) Buf Handle(24b) Flags (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) Rsv (4b) Intf (4b) TCAM IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) Lookup Key IP DAddr (32b) NN Lookup Result VLAN (12b) QM 2b Sch 3b PerSchedQID (15b) Protocol (8b) TCP/UDP DPort Or ICMP ID (16b) ICMP Type (8b) Translated DPort/ID (16b) Stats Index (16b) S W I T C H T B U F IP SAddr (32b) Scr2NN QM0 IP Hdr 1st Word (32b) SCR Port Splitter SCR M S F 1x10G Tx2 IP Hdr 1st Word (32b) 1x10G Tx1 IP Hdr Top 16 bits Of 2nd Word (16b) QM1 SCR Original DPort/ID (16b) IP Hdr Top 16 bits Of 2nd Word (16b) NN TCP/UDP SPort (16b) NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

30 SPP V1 LC Ingress NAT Scratch Rings R B U F R T M M S F S W I T C H T
Hit TCP Flags 6b H 1b Rsv S R P A F U FIN SYN RST PSH ACK URG XScale Flags (8b) Reserved (8b) Buf Handle(24b) NAT Scratch Rings SCR IP Pkt Length (16b) Eth Hdr Len (8b) Rsv (4b) Intf (4b) IP DAddr (32b) SCR R B U F Protocol (8b) TCP/UDP DPort Or ICMP ID (16b) ICMP Type (8b) R T M M S F Rx1 IP_SAddr (32b) Rx2 Key Extract Lookup Hdr Format NN NN NN NN IP Hdr 1st Word (32b) IP Hdr Top 16 bits Of 2nd Word (16b) TCP/UDP SPort (16b) Flags (8b) Buf Handle(24b) TCAM Hit Index (32b) TCAM IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) ICMP ERR ICMP NAT Hit UDP TCP NN VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b Rsvd 2b IE 1b T 1b U 1b I 1b N 1b H 1b Translated DPort/ID (16b) Stats Index (16b) S W I T C H T B U F Scr2NN QM0 IP Hdr 1st Word (32b) SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 IP Hdr Top 16 bits Of 2nd Word (16b) QM1 SCR Original DPort/ID (16b) NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

31 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
Flags (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) SCR XScale NAT Scratch Rings VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b Translated DPort/ID (16b) Stats Index (16b) IP Hdr 1st Word (32b) R B U F IP Hdr 2nd Word (32b) R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buffer Handle(24b) TCAM Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b NN Frame Length (16b) Stats Index (16b) S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

32 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
SCR XScale NAT Scratch Rings R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buffer Handle(24b) TCAM Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b NN Frame Length (16b) Stats Index (16b) S W I T C H T B U F QM0 Scr2NN SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) Reserved (8b) Buffer Handle(24b) SRAM1 SRAM3 SCR Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b SRAM2 Frame Length (16b) Stats Index (16b)

33 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
SCR XScale NAT Scratch Rings R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format Reserved (8b) Buffer Handle(24b) NN NN NN NN Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b Frame Length (16b) TCAM Stats Index (16b) NN Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

34 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
SCR XScale NAT Scratch Rings R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 TCAM NN Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

35 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
SCR XScale NAT Scratch Rings R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR Stats Index (16b) Opcode (4b) Data (12b) QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

36 EGRESS Block Interfaces

37 Egress Buffer Descriptor
Buffer_Next (32b) LW0 Buffer_Size (16b) Offset (16b) LW1 Packet_Size (16b) Free_list 0000 (4b) Reserved (4b) Reserved (8b) LW2 Reserved (4b) SliceID (VLAN) (12b) Stats Index (16b) LW3 Reserved (16b) Reserved (8b) Reserved (4b) Reserved (4b) LW4 Reserved (4b) Reserved (4b) Reserved (32b) LW5 Reserved (16b) Reserved (16b) LW6 Packet_Next (32b) LW7

38 SPP V1 LC Egress with 1x10Gb/s Tx
SCR XScale NAT Scratch Rings ToXScale:43 From:XS:44 S W I T C H R B U F M S F ToHF:33 Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN From:LK:34 DROP:13 DROP:14 IN:F2 DROP:12 TCAM NN T B U F QM0 SCR Port Splitter SCR R T M M S F Flow Stats1 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Rcv: 25 SCR Stats (1 ME) SRAM3 SRAM1 SCR Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

39 SPP V1 LC Egress with 10x1Gb/s Tx
SCR XScale NAT Scratch Rings S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter R T M M S F Flow Stats1 SCR SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR Stats (1 ME) SCR SRAM3 SRAM1 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

40 SPP V1 LC Egress with 10x1Gb/s Tx
SCR XScale NAT Scratch Rings Buf Handle(24b) Port (4b) Reserved (12b) Eth. Frame Len (16b) Rx Flags (8b) S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter SCR R T M M S F Flow Stats1 SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR Stats (1 ME) SCR SRAM3 SRAM1 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

41 SPP V1 LC Egress with 10x1Gb/s Tx
SCR XScale NAT Scratch Rings Buf Handle(24b) Port (4b) Reserved (12b) Eth. Frame Len (16b) Rx Flags (8b) S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Buf Handle(24b) IP_SAddr (32b) IP Pkt Length (16b) SrcMAC (8b) Eth Hdr Len (8b) UDP SPort (16b) IP Proto Type(8b) IP Hdr 1st Word (32b) Reserved IP Hdr 2nd Word (32b) IP DAddr (32b) TCAM Lookup Key NN T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter Flow Stats1 SCR R T M M S F SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR Stats (1 ME) SRAM1 SCR SRAM3 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

42 SPP V1 LC Egress with 10x1Gb/s Tx
Flags (8b) Buf Handle(24b) H IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) XScale Lookup Result Flags(8b) Reserved 5b N 1b H I VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b Hit Translated SPort(16b) Stats Index (16b) NAT Miss Scratch Ring ICMP NAT IP DAddr (32b) SCR S W I T C H R B U F IP Hdr 1st Word (32b) M S F IP Hdr Top 16 bits Of 2nd Word (16b) Rsv (4b) SliceID (12b) Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) SrcMAC (8b) TCAM IP_SAddr (32b) NN IP Proto (8b) UDP SPort (16b) Type(8b) T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter R T M M S F IP DAddr (32b) Flow Stats1 SCR SCR QM1 SCR IP Hdr 1st Word (32b) 5x1G Tx2 (P5-P9) QM2 IP Hdr Top 16 bits Of 2nd Word (16b) SCR Rsv (4b) SliceID (12b) SCR QM3 SCR SCR NAT Pkt return Stats (1 ME) SRAM3 SRAM1 SCR Flow Stats2 SRAM Freelist SRAM XScale XScale SRAM2 Archive Records

43 Proposed Change: SPP V1 LC Egress
ICMP ERR XScale Flags(8b) Rsvd 2b IE 1b T 1b U 1b I 1b N 1b H 1b SCR Rsv 1b IE 1b TCP Flags 6b ICMP NAT Hit UDP TCP NAT Scratch Rings ICMP ERR S W I T C H SCR R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buf Handle(24b) Flags (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) SrcMAC (8b) IP Pkt Length (16b) TCAM Eth Hdr Len (8b) Reserved (8b) Lookup Result IP_SAddr (32b) VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b NN IP Proto (8b) TCP/UDP SPort Or ICMP ID (16b) ICMP Type (8b) Translated SPort(16b) Stats Index (16b) T B U F 5x1G Tx1 (P0-P4) IP DAddr (32b) QM0 SCR Port Splitter R T M M S F IP DAddr (32b) Flow Stats1 SCR IP Hdr 1st Word (32b) SCR QM1 SCR IP Hdr 1st Word (32b) IP Hdr Top 16 bits Of 2nd Word (16b) Original SPort/ID (16b) 5x1G Tx2 (P5-P9) QM2 IP Hdr Top 16 bits Of 2nd Word (16b) TCP/UDP DPort (16b) Reserved (16b) SCR SCR QM3 SCR SCR NAT Pkt return Stats (1 ME) SRAM3 SRAM1 SCR Flow Stats2 SRAM Freelist SRAM XScale XScale SRAM2 Archive Records

44 Proposed Change: SPP V1 LC Egress
Hit TCP Flags 6b H 1b Rsv S R P A F U FIN SYN RST PSH ACK URG Flags (8b) Buf Handle(24b) XScale IP Pkt Length (16b) Eth Hdr Len (8b) SrcMAC (8b) NAT Scratch Rings SCR IP_SAddr (32b) S W I T C H IP Proto (8b) TCP/UDP SPort Or ICMP ID (16b) ICMP Type(8b) SCR R B U F M S F Rx1 IP_DAddr (32b) Rx2 Key Extract Lookup Hdr Format IP Hdr 1st Word (32b) NN NN NN NN IP Hdr Top 16 bits Of 2nd Word (16b) TCP/UDP DPort (16b) Flags (8b) Buf Handle(24b) TCAM Hit Index (32b) TCAM IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) ICMP ERR ICMP NAT Hit UDP TCP VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b NN T B U F 5x1G Tx1 (P0-P4) Rsvd 2b IE 1b T 1b U 1b I 1b N 1b H 1b Translated SPort(16b) Stats Index (16b) SCR QM0 SCR Port Splitter Flow Stats1 SCR IP DAddr (32b) R T M M S F SCR IP Hdr 1st Word (32b) QM1 SCR IP Hdr Top 16 bits Of 2nd Word (16b) Original SPort/ID (16b) 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR NAT Pkt return Stats (1 ME) SRAM3 SRAM1 SCR Flow Stats2 SRAM Freelist SRAM XScale XScale SRAM2 Archive Records

45 NAT Changes: SPP V1 LC Egress
XScale Flags(8b) Reserved 3b N 1b H I U T SCR ICMP NAT Hit UDP TCP NAT Scratch Rings S W I T C H SCR R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buf Handle(24b) Flags (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) SrcMAC (8b) IP Pkt Length (16b) TCAM Eth Hdr Len (8b) Reserved (8b) Lookup Result IP_SAddr (32b) VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b NN IP Proto (8b) TCP/UDP SPort Or ICMP ID (16b) ICMP Type (8b) Translated SPort(16b) Stats Index (16b) T B U F 5x1G Tx1 (P0-P4) IP DAddr (32b) QM0 SCR Port Splitter R T M M S F IP DAddr (32b) Flow Stats1 SCR IP Hdr 1st Word (32b) SCR QM1 SCR IP Hdr 1st Word (32b) IP Hdr Top 16 bits Of 2nd Word (16b) Reserved (16b) 5x1G Tx2 (P5-P9) QM2 IP Hdr Top 16 bits Of 2nd Word (16b) Reserved (16b) SCR SCR QM3 SCR SCR Stats (1 ME) SRAM1 SCR SRAM3 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

46 NAT Change: SPP V1 LC Egress
Hit TCP Flags 6b H 1b Rsv S R P A F U FIN SYN RST PSH ACK URG Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) Flags (8b) IP_SAddr (32b) SrcMAC (8b) TCP/UDP SPort Or ICMP ID (16b) IP Proto ICMP Type(8b) IP_DAddr (32b) TCP/UDP DPort (16b) TCAM Hit Index (32b) IP Hdr 1st Word (32b) IP Hdr Top 16 bits Of 2nd Word (16b) XScale NAT Scratch Rings SCR S W I T C H SCR R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Flags (8b) Buf Handle(24b) TCAM IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) ICMP NAT Hit UDP TCP VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b NN T B U F 5x1G Tx1 (P0-P4) Reserved 3b N 1b H I U T Translated SPort(16b) Stats Index (16b) SCR QM0 SCR Port Splitter SCR IP DAddr (32b) R T M M S F Flow Stats1 SCR IP Hdr 1st Word (32b) QM1 SCR IP Hdr Top 16 bits Of 2nd Word (16b) Reserved (16b) 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR Stats (1 ME) SRAM1 SCR SRAM3 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

47 SPP V1 LC Egress with 10x1Gb/s Tx
NAT MISS! XScale NAT Miss Scratch Ring Buf Handle(24b) IP Pkt Length (16b) Reserved (8b) Eth Hdr Len (8b) Flags (8b) SCR S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Flags (8b) Buf Handle(24b) IP Pkt Length (16b) TCAM Eth Hdr Len (8b) Reserved (8b) VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b NN Translated SPort(16b) Stats Index (16b) T B U F 5x1G Tx1 (P0-P4) IP DAddr (32b) SCR QM0 SCR Port Splitter SCR R T M M S F Flow Stats1 IP Hdr 1st Word (32b) SCR QM1 SCR IP Hdr Top 16 bits Of 2nd Word (16b) Rsv (4b) SliceID (12b) 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR NAT Pkt return Stats (1 ME) SRAM3 SRAM1 SCR Flow Stats2 SRAM Freelist SRAM XScale XScale SRAM2 Archive Records

48 SPP V1 LC Egress with 10x1Gb/s Tx
Flags (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) SCR XScale NAT Scratch Rings VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b Translated SPort(16b) Stats Index (16b) IP DAddr (32b) S W I T C H R B U F IP Hdr 1st Word (32b) M S F IP Hdr Top 16 bits Of 2nd Word (16b) Rsv (4b) SliceID (12b) Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buffer Handle(24b) TCAM Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b NN Ethernet Frame Length (16b) Cntr Index (16b) T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter Flow Stats1 SCR R T M M S F SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR Stats (1 ME) SRAM1 SCR SRAM3 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

49 SPP V1 LC Egress with 10x1Gb/s Tx
SCR XScale NAT Scratch Rings S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN Reserved (8b) NN Buffer Handle(24b) NN Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b TCAM Ethernet Frame Length (16b) Cntr Index (16b) NN T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter R T M M S F Flow Stats1 SCR SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR Reserved (8b) Buffer Handle(24b) SCR Stats (1 ME) SRAM3 Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b SRAM1 SCR Flow Stats2 SRAM Freelist SRAM Ethernet Frame Length (16b) Cntr Index (16b) XScale SRAM2 Archive Records

50 SPP V1 LC Egress with 10x1Gb/s Tx
SCR XScale NAT Scratch Rings S W I T C H R B U F M S F Rx1 Rx2 Key Extract Reserved (8b) Buffer Handle(24b) Lookup Hdr Format NN NN NN NN Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b Ethernet Frame Length (16b) Cntr Index (16b) TCAM Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 NN T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter SCR R T M M S F Flow Stats1 SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR Stats (1 ME) SRAM1 SCR SRAM3 Flow Stats2 SRAM Freelist SRAM XScale SRAM2 Archive Records

51 SPP V1 LC Egress with 10x1Gb/s Tx
SCR XScale NAT Scratch Rings S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 TCAM Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 NN T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter SCR R T M M S F Flow Stats1 SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 SCR SCR Stats (1 ME) SCR SRAM3 SRAM1 Flow Stats2 SRAM SRAM XScale SRAM2 Archive Records Freelist

52 SPP V1 LC Egress with 1x10Gb/s Tx
SCR XScale NAT Scratch Rings S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 TCAM Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 NN T B U F QM0 SCR Port Splitter SCR R T M M S F Flow Stats1 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR SCR Stats (1 ME) SCR SRAM3 SRAM1 Flow Stats2 SRAM SRAM XScale SRAM2 Archive Records Freelist

53 SPP V1 LC Egress with 10x1Gb/s Tx
SCR XScale NAT Scratch Rings S W I T C H R B U F M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN TCAM NN T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter Flow Stats1 SCR R T M M S F SCR QM1 SCR 5x1G Tx2 (P5-P9) QM2 SCR SCR QM3 Stats Index (16b) Opcode (4b) Data (12b) SCR SCR Stats (1 ME) SRAM1 SCR SRAM3 Flow Stats2 SRAM SRAM XScale SRAM2 Archive Records Freelist

54 NPE Next we’ll look at the design for the NPE for SPP V1

55 NPE Buffer Descriptor Buffer_Next (32b) Buffer_Size (16b) Offset (16b)
LW0 Buffer_Size (16b) Offset (16b) LW1 Packet_Size (16b) Free_list 0000 (4b) Reserved (4b) Reserved (8b) LW2 Reserved (4b) VLAN (12b) Stats Index (16b) LW3 Reserved (16b) Reserved (16b) LW4 Reserved (4b) Reserved (4b) Reserved (32b) LW5 Reserved (16b) Reserved (16b) LW6 Packet_Next (32b) LW7

56 SPP V1 NPE (MetaRouters)
Substrate Decap S W I T C H R B U F M S F Rx1 Rx2 NN Lookup Hdr Format NN NN NN NN Parse TCAM NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

57 SPP V1 NPE (MetaRouters)
Substrate Decap S W I T C H R B U F M S F Rx1 Rx2 NN Lookup Hdr Format NN NN NN NN Parse Buf Handle(24b) Port (4b) Reserved (12b) Eth. Frame Len (16b) Rx Flags (8b) TCAM NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

58 SPP V1 NPE (MetaRouters)
Buf Handle(32b) Substrate Decap MN Frm Length(16b) MN Frm Offset (16b) S W I T C H V S R 1 RxId (4b) Slice ID (VLAN) (11b) Rx UDP DPort (16b) R B U F M S F Rx IP SAddr (32b) Rx1 Rx2 NN Lookup Hdr Format NN NN NN Rx IP DAddr (32b) NN Parse Rx UDP SPort (16b) Reserved (12b) Code (4b) Buf Handle(24b) Port (4b) Reserved (12b) Eth. Frame Len (16b) Rx Flags (8b) Slice Data Ptr (32b) TCAM NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

59 SPP V1 NPE (MetaRouters)
Substrate Decap Buf Handle(32b) S W I T C H MN Frm Length(16b) MN Frm Offset (16b) R B U F v S R 1 RxId (4b) Slice ID (VLAN) (11b) Rx UDP DPort (16b) M S F Rx1 Rx2 NN Lookup Hdr Format Rx IP SAddr (32b) NN NN NN NN Rx IP DAddr (32b) Parse Rx UDP SPort (16b) Reserved (12b) Code (4b) Buf Handle(32b) TCAM Slice Data Ptr (32b) IP Pkt Length (16b) IP Pkt Offset (16b) NN Lookup Key[ ] Type(1b)/RxID(4b)/ Slice ID(11b)/Rx UDP DPort (16b) S W I T C H T B U F Lookup Key[111-80] DA (32b) Scr2NN QM0 SCR Port Splitter SCR M S F Lookup Key[ 79-48] SA (32b) 1x10G Tx2 1x10G Tx1 Lookup Key[ 47-16] Ports (32b) QM1 SCR NN NN Lookup Key Proto/TCP_Flags [15- 0] (16b) Rsv (4b) Exception Bits (12b) QM2 SCR Slice Data Ptr (32b) QM3 SCR Stats (1 ME) Rx UDP SPort (16b) Reserved (12b) Code (4b) SRAM1 SRAM3 SCR Rx IP SAddr (32b) Rx IP DAddr (32b) SRAM2

60 SPP V1 NPE (MetaRouters)
Buf Handle(32b) IP Pkt Length (16b) IP Pkt Offset (16b) Substrate Decap Lookup Key[ ] Type(1b)/RxID(4b)/ Slice ID(11b)/Rx UDP DPort (16b) S W I T C H R B U F Lookup Key[111-80] DA (32b) Lookup Key[ 79-48] SA (32b) M S F Rx1 Lookup Key[ 47-16] Ports (32b) Rx2 NN Lookup Hdr Format NN Lookup Key Proto/TCP_Flags [15- 0] (16b) Rsv (4b) NN NN Exception Bits (12b) NN Parse Slice Data Ptr (32b) Buf Handle(32b) IP Pkt Length (16b) TCAM IP Pkt Offset (16b) Rx UDP SPort (16b) Reserved (12b) Code (4b) NN Rsvd (5b) Slice ID (VLAN) (11b) Rx UDP DPort(16b) Rx IP SAddr (32b) R V S d (1b) H (1b) R V S d (1b) L D (1b) D (1b) Reserved (11b) Cntr Index (16b) S W I T C H T B U F Rx IP DAddr (32b) Scr2NN QM0 SCR Port Splitter SCR Tx IP DAddr (32b) M S F 1x10G Tx2 1x10G Tx1 Tx UDP DPort (16b) QM1 SCR Tx UDP SPort(16b) NN NN Exception Bits (12b) QM2 PerSchedQID (15b) Sch 3b QM 2b SCR Slice Data Ptr (32b) QM3 SCR Rx UDP SPort (16b) Reserved (12b) Code (4b) Stats (1 ME) Rx IP SAddr (32b) SRAM1 SRAM3 SCR Rx IP DAddr (32b) SRAM2

61 NPE Lookup Key and Result
Generic MN Key: Lookup Result as written in TCAM: Rx UDP DPort (16b) T (1b) Slice ID (VLAN) (11b) Rx IP DA Index (4b) MN Key Bits (32b) MN IP Key Bits (32b) MN Key Bits (32b) MN Key Bits 15-0 (16b) Rsvd (3b) L D (1b) D R O (1b) p Reserved (11b) Cntr Index (16b) Tx IP DAddr (32b) Tx UDP DPort (16b) Tx UDP SPort(16b) Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b Lookup Result as passed to HF (Note the movement of Exception bits): IPv4 MN Key: Rx UDP DPort (16b) T (1b) Slice ID (VLAN) (11b) Rx IP DA Index (4b) IPv4_MN IP DAddr (32b) IPv4_MN IP SAddr (32b) IPv4_MN SPort (16b) IPv4_MN DPort (16b) IPv4_MN Proto/TCP Flags (16b) R S V (1b) d H (1b) R S V (1b) d L D (1b) D R p O (1b) Reserved (11b) Cntr Index (16b) Tx IP DAddr (32b) Tx UDP DPort (16b) Tx UDP SPort(16b) Exception Bits (12b) PerSchedQID (15b) Sch 3b QM 2b Move of exception bits not Implemented yet. TCP TCP Flags (12b) 01 (2b) r s v DONE !TCP 00 (2b) Proto (8b) Reserved (6b)

62 NPE Substrate ONLY Lookup Key and Result
Generic MN Key: Lookup Result as written in TCAM: Rx IP Da Index Is calculated based on RX IP Daddr In Int. Hdr T 1 (1b) Rx IP DA Index (4b) Slice ID (VLAN) (11b) Rx UDP DPort (16b) (from Int. Hdr) Rsvd (3b) L D (1b) D R O (1b) p Reserved (11b) Cntr Index (16b) TX IP DAddr (from Int. Hdr) (32b) Tx IP DAddr (32b) 0x (32b) Tx UDP DPort (16b) Tx UDP SPort(16b) 0x0000 (16b) TX UDP DPort (16b) (from Int. Hdr) Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b Do we want The Rx MI Specified in The substrate Only key? Should Tx UDP Sport Be included also? 0x0000 (16b) Lookup Result as passed to HF (Note the movement of Exception bits): MN Internal Header Type (16b) Int. Hdr Length (12b) Rx UDP DPort (2B) Tx UDP SPort (2B) Tx UDP DPort (2B) Tx IP DAddr (4B) 0000 Rx UDP SPort (2B) Rx IP Saddr (4B) Rx IP Daddr (4B) R S V (1b) d H (1b) R S V (1b) d L D (1b) D R p O (1b) Reserved (11b) Cntr Index (16b) Tx IP DAddr (32b) Tx UDP DPort (16b) Tx UDP SPort(16b) Exception Bits (12b) PerSchedQID (15b) Sch 3b QM 2b Move of exception bits not Implemented yet. STILL WORKING ON THIS DONE DONE

63 NPE Bypass Lookup Key and Result
Lookup Result as written in TCAM: GPE Rx UDP Dport (anything needed?) T 1 (1b) Rx IP DA Index (4b) Slice ID (VLAN) (11b) 0x0000 (16b) Rsvd (3b) L D (1b) D R O (1b) p Reserved (11b) Cntr Index (16b) USER Output MI ID (32b) 0x (32b) 0x (32b) 0x0000 (16b) 0x0000 (16b) 0x0000 (16b) 0x0000 (16b) 0x000 (12b) PerSchedQID (15b) Sch 3b QM 2b 0x0000 (16b) USER Output MI ID Has to come from MN Internal Header Lookup Result as passed to HF (Note the movement of Exception bits): MN Internal Header Type (16b) Int. Hdr Length (12b) Rx UDP DPort (2B) Tx UDP SPort (2B) Tx UDP DPort (2B) Tx IP DAddr (4B) 0000 Rx UDP SPort (2B) Rx IP Saddr (4B) Rx IP Daddr (4B) R S V (1b) d H (1b) R S V (1b) d L D (1b) D R p O (1b) Reserved (11b) Cntr Index (16b) Tx IP DAddr (32b) Tx UDP DPort (16b) Tx UDP SPort(16b) Exception Bits (12b) PerSchedQID (15b) Sch 3b QM 2b Exception bits come from Input ring. STILL WORKING ON THIS

64 SPP V1 NPE (MetaRouters)
Substrate Decap S W I T C H R B U F Buf Handle(32b) M S F Rx1 IP Pkt Length (16b) IP Pkt Offset (16b) Rx2 NN Lookup Hdr Format NN Rsv (5b) Slice ID (VLAN) (11b) Rx UDP DPort(16b) NN NN NN R S V d (1b) H (1b) R S V d (1b) L D (1b) D (1b) Reserved (11b) Cntr Index (16b) Parse Tx IP DAddr (32b) Buffer Handle(24b) Reserved (8b) TCAM TCAM TCAM Tx UDP DPort (16b) Tx UDP SPort(16b) NN Exception Bits (12b) PerSchedQID (15b) Sch 3b QM 2b MN Pkt Length (16b) MN Pkt Offset (16b) S W I T C H T B U F Slice Data Ptr (32b) Reserved (12b) QM 2b Sch 3b PerSchedQID (15b) Scr2NN QM0 SCR Substrate Encap SCR M S F Rx UDP SPort (16b) Reserved (12b) Code (4b) 1x10G Tx2 1x10G Tx1 Rsv (5b) Slice ID (VLAN) (11b) QM1 Cntr Index (16b) SCR Rx IP SAddr (32b) NN NN Tx IP DAddr (32b) QM2 SCR Rx IP DAddr (32b) Tx UDP SPort(16b) Tx UDP DPort (16b) QM3 SCR Reserved (8b) Rsvd G P E (1b) L D (1b) Code (4b) S R V d (2b) Stats (1 ME) SCR Slice Data Ptr (32b) SRAM1 SRAM3 SRAM2

65 IPv4 Internal Header Format
Path Category Type field Reason Outgoing MN Internal Hdr GPE->NPE [0] Reclassify Rx UDP DPort if set, otherwise Rx UDP Dport + FwdKey NPE-> Egress LC Fast path No MN Int Hdr NPE->GPE Exception [2] No route Rx UDP DPort [3] Expired TTL [4] IP w/ options Rx UDP DPort + FwdKey [5] Redirect due to Rx UDP DPort =Tx UDP SPort Control [6] Local delivery [7] Inspect Debug [8] Monitor [9] Log due to error in pkts FwdKey = [Tx UDP DPort + Tx UDP Sport + Tx IP DAddr]

66 IPv4 Internal Header Format
0000 Type (28b) Int. Hdr Length (2B) Reserved (2B) Rx IP Saddr (4B) Rx IP Daddr (4B) Rx UDP SPort (2B) Rx UDP DPort (2B) Tx IP DAddr (4B) MN Specific Data IPv4: Fwd Key Tx UDP SPort (2B) Tx UDP DPort (2B) Type (16b) Int. Hdr Length (12b) Rx UDP DPort (2B) Tx UDP SPort (2B) Tx UDP DPort (2B) Tx IP DAddr (4B) 0000 Rx UDP SPort (2B) Rx IP Saddr (4B) Rx IP Daddr (4B) MN Specific Data IPv4: Fwd Key

67 SPP V1 NPE (MetaRouters)
Buffer Handle(24b) Reserved (8b) Substrate Decap MN Pkt Length (16b) MN Pkt Offset (16b) S W I T C H R B U F Reserved (12b) QM 2b Sch 3b PerSchedQID (15b) M S F Rx1 Rx2 NN Rsv (5b) Slice ID (VLAN) (11b) Lookup Cntr Index (16b) Hdr Format NN NN NN NN Tx IP DAddr (32b) Parse Tx UDP SPort(16b) Tx UDP DPort (16b) DA(8b) Rsvd G P E (1b) L D (1b) Code (4b) R S V d (2b) NN Slice Data Ptr (32b) S W I T C H T B U F Scr2NN QM1 SCR Substrate Encap SCR M S F 1x10G Tx2 1x10G Tx1 QM2 SCR NN NN QM3 SCR QM4 Reerved (8b) Buffer Handle(24b) SCR Stats (1 ME) Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b SRAM1 SCR TCAM SRAM3 TCAM Ethernet Frame Length (16b) SRAM2 Cntr Index (16b)

68 SPP V1 NPE (MetaRouters)
Substrate Decap S W I T C H R B U F M S F Rx1 Rx2 NN Lookup Hdr Format NN NN NN NN Parse TCAM NN Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 S W I T C H T B U F Scr2NN QM0 SCR Substrate Encap SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 Reerved (8b) Buffer Handle(24b) SCR Stats (1 ME) Reserved (12b) PerSchedQID (15b) Sch 3b QM 2b SRAM1 TCAM SRAM3 SCR TCAM Ethernet Frame Length (16b) SRAM2 Cntr Index (16b)

69 SPP V1 NPE (MetaRouters)
Substrate Decap S W I T C H R B U F M S F Rx1 Rx2 NN Lookup Hdr Format NN NN NN NN Parse TCAM Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 NN S W I T C H Buffer Handle(24b) Rsv (3b) Intf (4b) V 1 T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

70 SPP V1 NPE (MetaRouters)
Substrate Decap S W I T C H R B U F M S F Rx1 Rx2 NN Lookup Hdr Format NN NN NN NN Parse TCAM NN S W I T C H T B U F Scr2NN QM0 SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 QM1 SCR NN NN QM2 SCR Stats Index (16b) Opcode (4b) Data (12b) QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

71 TCAM Performance (Rates in M/sec)
Lookup Size #LA-1 Words Core Size Assoc. Data Single LA-1 Max Rate Max Core Rate Avg Shared Rate (Each of 2 LA-1s) 32 1 36 50 25 64 128 12.5 2 72 100 3 67 4 144 5 40 160 288 LC_Ingress/LC_Egress IPv4 MR

72 Changes for Key Extract
Input: No changes Output Move Eth Hdr Len field Add Type field to Lookup Key If ICMP extract TYPE field from ICMP pkt Otherwise, set to 0. Add Src MAC to Lookup Key Extract low 8 bits of Src MAC from ethernet header Add VLAN/IP_SAddr to Lookup Key If low 6 bits of Src MAC are all 1’s then Src MAC is from NPE Use VLAN in the VLAN/IP_SAddr field VLAN goes in lower 12 bits, upper 20 bits are all 0’s If low 6 bits of Src MAC are NOT all 1’s then it is from GPE Use IP_SAddr in the VLAN/IP_SAddr field

73 SPP V1 LC Egress with 10x1Gb/s Tx
Flags (8b) Buf Handle(24b) H IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) XScale Lookup Result VLAN (12b) rsv 4b PerSchedQID (11b) Sch 3b QM 2b Translated SPort(16b) Stats Index (16b) NAT Miss Scratch Ring IP DAddr (32b) SCR S W I T C H IP Hdr 1st Word (32b) R B U F IP Hdr 2nd Word (32b) M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) SrcMAC (8b) TCAM VLAN/IP_SAddr (32b) NN IP Proto (8b) UDP SPort (16b) Type(8b) T B U F 5x1G Tx1 (P0-P4) QM0 SCR Port Splitter SCR R T M M S F IP DAddr (32b) Flow Stats1 SCR QM1 SCR IP Hdr 1st Word (32b) 5x1G Tx2 (P5-P9) QM2 IP Hdr 2nd Word (32b) SCR SCR QM3 SCR SCR NAT Pkt return Stats (1 ME) SRAM1 SCR SRAM3 Flow Stats2 SRAM Freelist SRAM XScale XScale SRAM2 Archive Records

74 Old Slides The following are the old “Changes to ..” slides from when we were going from V0 to V1 They have gotten confusing now that we are going from V1 to V2

75 Changes for Key Extract
Lookup Key Changes: Old Lookup Key (64b): SL Type (4b) Port (4b) IP DAddr (32b) IP Proto (8b) UDP DPort (16b) New Lookup Key (72b): Reserved (4b) Interface (4b) ICMP Type (8b) Move Ethernet Hdr Length field

76 Changes for Lookup Lookup Key Changes: See KE notes
Lookup Result Changes: Add Translated DPort/ID Move MAC DAddr Move Vlan Remove Port field QM ID and Scheduler ID QM_ID (2b) SchedID(3b) QID(15b) PerSchedQID(15b) QM_ID and SchedID are used to demux and get packet to correct QM and Scheduler The QM/Scheduler uses just the PerSchedQID bits as the QID given to the SRAM Controller. Change in size of lookup result Move Eth Hdr Len. Add Flags in 1st word Bit 0: HIT (Result is valid) Bit 1: NAT (NAT translation required, orig port != xlated Port) Bit 2: ICMP (Protocol == 1)

77 Changes for Header Format
Lookup Result Changes: See Lookup notes Process NAT Miss If (H==0) then NAT Miss Send to XScale Ingress/Egress XScales will generate ICMP Error msg. Move Eth Hdr Len in input ring No Interface field to pass along Perform DPort/ID translation recalculate IP Hdr checksum. Calculate incremental Transport (TCP and UDP) checksum Check for arriving UDP checksum of 0 which implies that packet is not using the optional UDP checksum

78 Changes for Port Splitter
No Interface field in input ring QM(2b) determines which QM Scr ring to use QM will use the Sch(3b) to determine which scheduler to use. Port Splitter does not have to give a separate field anymore. The QM(2b), Sch(3b), QID(15b) should be left unchanged in the low 20 bits The QM will use the 15-bit QID field as the qid value given to the SRAM controller.

79 Changes for QM QM will extract the Sch(3b) to identify the scheduler (called the port_id in the code) instead of getting a separate ‘port’ field. Association of a Scheduler with a physical interface: Dequeue currently reads the interface rate from SRAM periodically. We could extend this to have it also read the interface it is associated with at the same time it is reading the rate. This would also work for the LC_Ingress and NPE where we need to change the interface to 0 before sending it to TX. This could be accomplished by setting the interface read by Dequeue to 0 and then all the Dequeue engines (schedulers) would send to Interface 0.

80 Changes for Scr2NN New Block based on Dave’s “Port_Concentrator”
QM now takes care of giving appropriate value for the Interface field

81 Changes for Stats Do we want to incorporate the improvements we made for the ONL Stats block?

82 Changes for Key Extract
Input: No changes Output Move Eth Hdr Len field Add Type field to Lookup Key If ICMP extract TYPE field from ICMP pkt Otherwise, set to 0. Add Src MAC to Lookup Key Extract low 8 bits of Src MAC from ethernet header Low 8 bits of each blade MUST be unique across the chassis These 8 bits will be the differentiating factor on lookups where a GPE is trying to use the same IP SAddr and Sport pair as an NPE is. In this case the GPE use will get NAT translated at the LC Egress Add IP_SAddr to Lookup Key At one point in time we had this field being VLAN/IP_SAddr but this is not longer needed. The VLAN was going to be used when the traffic came from an NPE and the IP_SAddr when it came from a GPE.

83 Changes for Lookup Input: Output: Move Eth Hdr Len field
Add Type field to Lookup Key Add Src MAC to Lookup Key Add IP SAddr to Lookup Key Output: Re-organize output Move IP DAddr to 5th word (Do we still need this?) Move Eth Hdr Len Add Flags in 1st word Bit 0: HIT (Result is valid) Bit 1: NAT (NAT translation required, orig port != xlated Port) Bit 2: ICMP (Protocol == 1) Add Translated Sport to Lookup Result

84 Changes for Header Format
Input: Re-organize input 72 bit lookup result Move IP DAddr to 5th word Move Eth Hdr Len Add Flags in 1st word Add Translated Sport to Lookup Result Output to PortSplitter/QM: No changes Output to XScale: New Function: Write Buffer descriptor including: Packet Size Buffer Size Freelist Offset SliceID (VLAN) Stats Index Should we also write the ethernet header length? Test HIT Flag to determine if NAT Hit or Miss Send Miss to XScale Send Hit to PortSplitter/QM Perform SPort/ID translation recalculate IP Hdr checksum. Calculate incremental Transport (TCP and UDP) checksum Check for arriving UDP checksum of 0 which implies that packet is not using the optional UDP checksum

85 Changes for Port Splitter
No Interface field in input ring QM(2b) determines which QM Scr ring to use Sch(3b) needs to be copied up to the low 3 bits of the top byte to comply with QM’s current input format. We will look into removing this requirement and see if it is easy to have the QM extract the scheduler bits itself The QM(2b), Sch(3b), QID(15b) should also be left unchanged in the low 20 bits The QM will give the 15-bit QID field to the SRAM Controller as the qid being used.

86 Changes for QM QM will extract the Sch(3b) to identify the scheduler (called the port_id in the code) instead of getting a separate ‘port’ field. Association of a Scheduler with a physical interface: Dequeue currently reads the interface rate from SRAM periodically. We could extend this to have it also read the interface it is associated with at the same time it is reading the rate. This would also work for the LC_Ingress and NPE where we need to change the interface to 0 before sending it to TX. This could be accomplished by setting the interface read by Dequeue to 0 and then all the Dequeue engines (schedulers) would send to Interface 0.

87 Changes for FlowStats New Block Output for 10x1Gb/s Tx:
To 1 of two scratch rings dependent on outgoing interface Output for 1x10Gb/s Tx: To NN Ring QM will now take care of setting the appropriate interface, FlowStats doesn’t have to do anything special.

88 Changes for Stats Do we want to incorporate the improvements we made for the ONL Stats block?

89 Changes for Lookup No Port field in input data
But Lookup doesn’t look at the data anyway so no changes.

90 Changes for HF No Port field in input data
Use QM/Sched bits to determine Src IP Address to use on the outgoing Tunnel Header. Src MAC Addr should be a constant Dst MAC Addr should be configured: LCE GPE Separation of HF and Substrate Encap

91 Changes for Substrate Encap
New Block

92 Changes for QM Use Sched bits to determine which Scheduler to use.

93 NAT Changes: SPP V1 LC Ingress
SCR XScale NAT Scratch Rings Flags(8b) Reserved 3b N 1b H I U T ICMP NAT Hit UDP TCP R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buf Handle(24b) Flags (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) Rsv (4b) Intf (4b) TCAM IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) Lookup Key IP DAddr (32b) NN Lookup Result VLAN (12b) QM 2b Sch 3b PerSchedQID (15b) Protocol (8b) TCP/UDP DPort Or ICMP ID (16b) ICMP Type (8b) Translated DPort/ID (16b) Stats Index (16b) S W I T C H T B U F IP Hdr 1st Word (32b) Scr2NN QM0 IP Hdr 1st Word (32b) SCR Port Splitter SCR M S F 1x10G Tx2 IP Hdr Top 16 bits Of 2nd Word (16b) 1x10G Tx1 Reserved (16b) IP Hdr Top 16 bits Of 2nd Word (16b) QM1 SCR Reserved (16b) NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

94 NAT Changes: SPP V1 LC Ingress
Hit TCP Flags 6b H 1b Rsv S R P A F U FIN SYN RST PSH ACK URG XScale Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) Flags (8b) IP DAddr (32b) Intf (4b) TCP/UDP DPort Or ICMP ID (16b) Protocol ICMP Type (8b) Rsv IP_SAddr (32b) TCP/UDP SPort (16b) TCAM Hit Index (32b) IP Hdr 1st Word (32b) IP Hdr Top 16 bits Of 2nd Word (16b) NAT Scratch Rings SCR SCR R B U F R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Flags (8b) Buf Handle(24b) TCAM IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) ICMP NAT Hit UDP TCP NN VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b Reserved 3b N 1b H I U T Translated DPort/ID (16b) Stats Index (16b) S W I T C H T B U F Scr2NN QM0 IP Hdr 1st Word (32b) SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 IP Hdr Top 16 bits Of 2nd Word (16b) QM1 SCR Reserved (16b) NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

95 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
XScale NAT MISS! NAT Miss Scratch Ring Flags (8b) Reserved (8b) Buf Handle(24b) SCR R B U F IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Flags (8b) Buf Handle(24b) TCAM IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) NN VLAN (12b) PerSchedQID (15b) Sch 3b QM 2b Translated DPort/ID (16b) Stats Index (16b) S W I T C H T B U F Scr2NN QM0 IP Hdr 1st Word (32b) SCR Port Splitter SCR M S F 1x10G Tx2 1x10G Tx1 IP Hdr 2nd Word (32b) QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2

96 SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)
Flags (8b) Buf Handle(24b) IP Pkt Length (16b) Eth Hdr Len (8b) Reserved (8b) XScale Flags(8b) H Reserved 5b N 1b H I Lookup Result VLAN (12b) QM 2b Sch 3b PerSchedQID (15b) Hit ICMP NAT Translated DPort/ID (16b) Stats Index (16b) NAT Miss Scratch Ring IP Hdr 1st Word (32b) SCR R B U F IP Hdr 2nd Word (32b) R T M M S F Rx1 Rx2 Key Extract Lookup Hdr Format NN NN NN NN Reserved (8b) Buf Handle(24b) TCAM IP Pkt Length (16b) Eth Hdr Len (8b) Rsv (4b) Intf (4b) NN Lookup Key IP DAddr (32b) Protocol (8b) UDP DPort (16b) Type (8b) S W I T C H T B U F Scr2NN QM0 SCR Port Splitter IP Hdr 1st Word (32b) SCR M S F 1x10G Tx2 1x10G Tx1 IP Hdr 2nd Word (32b) QM1 SCR NN NN QM2 SCR QM3 SCR Stats (1 ME) SRAM1 SRAM3 SCR SRAM2


Download ppt "SPP Version 1 Router Plans and Design"

Similar presentations


Ads by Google