1. Washington WASHINGTON UNIVERSITY IN ST LOUIS fredk@arl.wustl.edu QManager Notes Fred

2. 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 3/17/2016 QM Dispatcher STATE_TRANS FULL_DP_DCHR_OUTC_F (in) FULL_DCHR_SWQM_F (in) FULL_DCHR_QLENC_F (in) EMPTY_QMGR_F (in) EMPTY_DCHR_FREE_PTR_F (in) COUNT_DCHR_PKTC_F (in) READY_IN (in) DATA_QMGR_F (in, 31..0) next_state CLK0 > CLK state LATCH_WORD > CLK FC_flag copycount (1..0) DEST_CTL > CLK DATA_DCHR_FREE_PTR_F (in, 17..0) DP_flag SR_flag TO_flag qid (9..0) get_comref_entry qid_new (9..0) WRITE_HEADER > CLK data_int (95..0) pkt_ptr (19..0) comm_ref_ptr (17..0) destination (1..0) RD_DCHR_FREE_PTR_F_DCHR (out) FM_flag SC_flag SB_flag DATA_DCHR_QM_F_DCHR (out, 95..0) DATA_DCHR_PKTC_F_DCHR (out, 35..0) DATA_DP_DCHR_OUTC_F_DCHR (out, 35..0) WEN_DP_DCHR_OUTC_F_DCHR (out) WEN_DCHR_PKTC_F_DCHR (out) WEN_DCHR_SWQM_F_DCHR (out) WEN_DCHR_QLENC_F_DCHR (out) QMGR_F_CTL > CLK RD_QMGR_F_QMGR (out) DEC_HD_CNT_DCHR (out) HEADER_ONLY_CTL > CLK CARL FIFO: EMPTY_QMGR_F RD_QMGR_F_QMGR DATA_QMGR_F Line Card (QLNC) FIFO: FULL_DCHR_QLENC_F WEN_DCHR_QLENC_F_DCHR DATA_DCHR_QM_F_DCHR SW (SWQM) FIFO: FULL_DCHR_SWQM_F WEN_DCHR_SWQM_F_DCHR DATA_DCHR_QM_F_DCHR Drop (OUTC) FIFO: FULL_DP_DCHR_OUTC_F WEN_DP_DCHR_OUTC_F_DCHR DATA_DP_DCHR_OUTC_F_DCHR Signals Not Used: prev_state DG_flag Constants Defined but not used: SW_DATAGRAM_BASE LC_DATAGRAM_BASE

3. 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 3/17/2016 Q Hdr table next_q deficit next_q deficit active (0) head_ptr tail_ptr deficit active (1) head_ptr tail_ptr next_q deficit active (1) head_ptr tail_ptr next_q deficit active (0) head_ptr tail_ptr next_q deficit active (1) head_ptr tail_ptr next_q active (0) head_ptr tail_ptr 1 2 3 4 5 511 qid (index) ext flags cpy next pkt ptrint flags total lengthppnspipkt ptr (Valid) ext flags cpy next pkt ptr int flags total lengthppnspipkt ptr (Valid) ext flags cpy next pkt ptr int flags total lengthppnspipkt ptr (InValid) ext flags cpy next pkt ptr int flags total lengthppnspipkt ptr (Valid) ext flags cpy next pkt ptrint flags total lengthppnspipkt ptr (InValid) ext flags cpy next pkt ptr int flags total lengthppnspipkt ptr (InValid) ext flags cpy next pkt ptr int flags total lengthppnspipkt ptr (Invalid) ext flags cpy next pkt ptrint flags total lengthppnspipkt ptr (InValid) ext flags cpy next pkt ptr int flags total lengthppnspipkt ptr (Valid) ext flags cpy next pkt ptr int flags total lengthppnspipkt ptr (Invalid) SRAM (2 words/entry, 36-bits/word) deficit active (1) head_ptr tail_ptr next_q Previous deficit active (1) head_ptr tail_ptr next_q Current cached copies in LNQM, may be out of sync with the table next_q (11-bit)head_ptr (18-bit)deficit (32-bit)Ahead_ptr (18-bit) 0294711 Qhdr Table Entry (word size 80-bit)) 79

4. 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 3/17/2016 CARL QMGR External Flags (8 bit) NM EX HO HR FMTO Internal Flags (8 bit) NN-15N-7 DGSBSR IC FCLPSC -- 5 4 3 2 10765 4 3210 DPRC 76 Total Length (11 bit)LFS Rate2LFS Rate1 310101624 PPNPacket Pointer (20 bit)OVIN (5 bit) 6332515661 Flags (8 bit)Queue Identifier (10 bit) DPRCNM EX HO HR FMTO Internal Flags (8 bit) 9564738088 Cpy Cnt MB DGSBSR IC FCLPSC-- 7779 9161315310192429 CARL QMGR QMGR OSAR PPNOVIN (5 bit)Total Length (11 bit) LFS Rate2LFS Rate1 010 08 Packet Pointer (20 bit) Flags (8 bit) Queue Identifier (10 bit) DPRCNM EX HO HR FMTO Internal Flags (8 bit) 311624 MB DGSBSR IC FCLPSC -- 29 Queue Length (24 bit) 0 3124 23 213101922 External Flag Values –DP (Drop Packet): SPC directive to FPX. Will always be zero coming into SPC. –RC (Reclassify Packet): SPC directive to FPX. May be 0 or 1 going to SPC. SPC must redefine. –NM (No Match): FPX can not classify (no route), FPX to SPC. Set to 0 before returning to FPX. –EX (Exception Packet): Requires non-fast path; IP options, Non IPv4. FPX to SPC. Set to 0 before returning to FPX. –HO (Header Only): FPX to SPC; Set if packet size > 120B and only first chunk sent to SPC. SPC must not modify. SPC can not drop internally if this bit is set. –HR (Header Only Return): Internal FPX flag. SPC must not modify. –FM (From LC/SW): Set by FPX. SPC can use but not modify. –TO (To LC/SW): Set by FPX. SPC can use but not modify. Internal Flag Values –DG (DataGram) –SB (SPC-Bound) –SR (SPC-Return) –IC (Initial Copy): CARL to QMGR, qmgr always sets to 0. –FC (Final Copy); CARL to QMGR, qmgr always sets to 0. –LP (LFS oPtion) –SC (Single-Chunk) –MC???

5. 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 3/17/2016 packet Header Flow through QMGR SRAM Single Copy Packet Header Format next packet header in SRAM Total LengthPacket Pointer External Flags DPRCNM EX HO HR FMTO Internal Flags DGSBSR LP SC 203528 OPN 17180 23 MBMC 20352519022 Total Length (11 bit)LFS Rate2LFS Rate1 310101924 PPNPacket Pointer (20 bit)OVIN (5 bit) 6332515661 Flags (8 bit) Queue Identifier (10 bit) DPRCNM EX HO HR FMTO Internal Flags (8 bit) 9564738088 cpy cnt MB DGSBSR IC FCLPSC-- 7779 9161315310192429 CARL QMGR PPNOVIN (5 bit)Total Length (11 bit) LFS Rate2LFS Rate1 010 08 Packet Pointer (20 bit) Flags (8 bit) Queue Identifier (10 bit) DPRCNM EX HO HR FMTO Internal Flags (8 bit) 311624 MB DGSBSRLPSC -- 29 Queue Length (24 bit) 0 3124 23 213101922 QMGR OSAR ‘0’ SPIOPN 310101624 SPI PPN MC ‘0’ current QID(8..0) SPIOPN Notes: MC = (cpycnt == 0) ? 0 : 1; In Out

6. 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 3/17/2016 SRAM Table Entries SRAM Single Copy Packet Header Format total lengthxx ppn common reference ptr 350192225 SRAM Multi-copy Packet Header Format XXX 35 X 0 01933 common reference ptrcpyX SRAM Multi-copy Common Reference Entry 35 0 1720 27 external flagsinternal flagscpynext header 35 0 1720 27 external flagsinternal flagscpynext header 350192225 total lengthxxppncommon reference ptr External Flags (8 bit) DPRCNM EX HO HR FMTO Internal Flags (8 bit) NN-15N-7 DGSBSR IC FCLPSC-- 353433 32 31 30 2928272625 24 23222120 FIX!

7. 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 3/17/2016 Token Bucket Constrained Output tokens BiBi B max Fill rate: add z tokens every tick (T seconds) remove L·x tokens to send a packet with L-bits Sending continuously: T·R av ·x tokens used each period; drain rate R av ·x; z tokens added every tick; replenishment rate = z/T z = ceil(R av ·T·x), for R av (z) < R max ; z max = floor(R max ·T·x) So the corresponding sending rates are R av (z) = z·(F/x), R av = {F/x, 2F/x, 3F/x, …, z max F/x} Ticks to send packet = L/TR max (ticks/pkt), Tokens to send packet Lx Given packet length in Bytes, L B, convert x B = 8x b ; R B = R b /8; R b = zF/x b = zF/(x B /8) VHDL uses x B = 8192 (tok/B) => x b = 1024 (tok/b) Burst size:(xL tokens to send L bytes, zL/TR tokens added during send) d = B – x B L B + zL B /TR max = B – L(x – z/TR max ), stop sending when = (B/L) / (x – z/TR max ) But, z = ceil(R av Tx) >= R av Tx so a lower bound is (B/xL) / (1 – R av /R max ) Let B = MTU · x and L = MTU then n = ceil(R max /(R max -R AV )), or R AV = R max (n-1)/n for n = 1, 2, …, N Bucket depth range: d = n(tick) * z(tok/tick) = B; ticks to fill = n = B/z; it takes MTU’ = MTU*x B tokens to send; –MTU’ < d <= B; so scale d as d’ = d + MTU’ then 0 < d’ <= B + MTU’; So max scaled bucket depth = B’ = B + MTU’; B = MTU’ so B’ = 2MTU’ = 2·MTU·x MTU = 2048 = 2 11, x = 8192 = 2 13, B’ = 2 2 11 2 13 = 2 25 Example: x = 64, F = 62.5MHz, R max = 1Gbps, F/x = 976.6 Kbps, z max = 1024, R (Mbps) = {.98, 1.9, 2.9, 3.9, …, 1,000} Packet length= L (b), Desired Rate= R av (b/s), Actual rate= R max (b/s) Interval (tick)= T (s), F -1, F is nominally 62.5MHz, Fill rate= z (tokens/tick), Drain rate = x (tokens/bit), Example: x = 1024 (x B = 8192), F = 62.5MHz, R max = 1Gbps, F/x = 61 Kbps (F/x B = 7.6KBps), z max = 16384 (2 14 ), R (Mbps) = {.061, …,.98, 1.04, 1.1, …, 1,000} B max = 2 25, B min = 2 24,

8. 8 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 3/17/2016 using 8-bits for Z, x = 64 (tokens/Byte) Definitions: Packet length = L (b), Desired Sending Rate = R (b/s), Actual link rate = R max (b/s), Clock period = T (s) (or 1/F) Tokens replenishment/tick = z (tok/tick), Tokens consumed/Byte = x (tok/B), Max packet size = MTU = 2KB, Max bucket depth = B max = 2·MTU ·x, Min bucket depth = MTU ·x Configuration: X = 2 6 (tok/B), R max = 125MB/s, B max =2 MTU 2 6 = 2 18 (tok), B min = 2 17 (tok), F = 55MHz R(z) = z·F/X = z·55·10 6 /64 (B/s) = z· 55·10 6 /8 (b/s) = z·55/8 (Mbps), alternatively, z(R) = ceil(R(b/s)·8/55·10 6 ) = ceil(R(Mb/s)·8/55) SPC traffic is limited to 200Mbps z = 30 (for 206.25Mb/s) Test scenarios: 0) Link at 1Gbps, z = 145/0x91 (996.88 Mb/s) 1) Link at 900Mbps,z = 131/0x83 (900.62 Mb/s) 2) Link at 500Mbps,z = 73/0x49 (501.88 Mb/s) 3) Link at 200Mbps,z = 30/0x1e (206.25 Mb/s) 4) Link at 100Mbps,z = 15/0x0f (103.12 Mb/s) 5) Link at 50Mbps,z = 8/0x08 (55.00 Mb/s) 6) Link at 10Mbps,z = 2/0x02 (13.75 Mb/s) 7) Link at 1 Mbps,z = 1/0x01 (6.88 Mb/s)