1. Michael Wilson mlw2@arl.wustl.edu http://www.arl.wustl.edu/projects/techX Block Design Review: Line Card Key Extract (Ingress and Egress)

2. 2 - Michael Wilson - 11/23/2015 Revision History 10/10/06 (MLW): »Released

3. 3 - Michael Wilson - 11/23/2015 Contents slide taken from PlanetLab_Design.ppt For both Ingress and Egress Key Extract: »overview »block diagram »code locations »test procedures »implementation status »performance analysis Lookup Phy Int Rx Switch Tx QM/Schd Key Extract Hdr Format Lookup Key Extract Switch Rx Phy Int Tx QM/Schd Hdr Format SWITCHSWITCH

4. 4 - Michael Wilson - 11/23/2015 Key Extract Overview (Both) For both Ingress and Egress Key Extract, we follow the same general method to lower the cycle budget. 1.Issue a DRAM read of the first portion of the packet 2.Issue a second DRAM read that encompasses all possible IP Options 3.Process any outputs that do not rely on the packet 4.Wait for the first DRAM read 5.Process everything before the IP Options 6.Set up the index register for anything after the IP Options 7.Wait for the second DRAM read 8.Process the remaining data Both are written in microcode, not C

5. Egress Key Extract

6. 6 - Michael Wilson - 11/23/2015 Egress Key Extract slide taken from PlanetLab_Design.ppt Main functions: »Extract lookup keys from packet payload and pass to Lookup block »Note: no validation! Outbound packets are assumed to be correct. Single code path NN communication »May need scratch ring communication for exception path Uses 8 threads Lookup Key Extract Switch Rx Phy Int Tx QM/Schd Hdr Format SWITCHSWITCH

7. 7 - Michael Wilson - 11/23/2015 LC Egress: Functional Blocks Lookup Key Extract Switch Rx Phy Int Tx QM/Schd Hdr Format SWITCHSWITCH Buf Handle(32b) Port (4b) Reserved (12b) Eth. Frame Len (16b) Buf Handle(32b) IP Pkt Length (16b) Reserved (8b) Eth Hdr Len (8b) IP DAddr (32b) Lookup Key – UDP SPort (16b) Lookup Key IP Proto (8b) Reserved (8b) Type=802.1Q (2B) PAD (nB) CRC (4B) UDP Payload (MN Packet) Src Addr (4B) Dst Addr (4B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol = UDP (1B) Hdr Cksum (2B) DstAddr (6B) SrcAddr (6B) IP Options (0-40B) Src Port (2B) Dst Port (2B) UDP length (2B) UDP checksum (2B) VLAN (2B) Type=IP (2B) Ethernet Header IP Header UDP Header Ethernet Trailer Indicates fields that need to be read Indicates 8-Byte Boundaries Assuming no IP Options slide taken from PlanetLab_Design.ppt

8. 8 - Michael Wilson - 11/23/2015 Egress KE Block Diagram dl_source_1ME_NN_2words() Start DRAM Reads egress_key_extract() dl_sink_1ME_NN_4words() Wait for first Read Load IP Len, IP Dest Addr, IP Proto, and IHL Set Eth Hdr Len Read UDP Sport Prepare Indexed Read No IP Options IP Options Wait for second Read Read UDP Sport 3 cycles (once NN ring ready) 10 cycles until ctx_arb 9 cycles (counting branch) 4 1 12 Set BlockID 1 19 cycles (in thread)

9. 9 - Michael Wilson - 11/23/2015 File locations (in …/LC_Egress/) Code »src/key_extractor/PL/key_extract.uc Includes »../dispatch_loop/dl_source_WU.uc dl_source() and dl_sink() functions »Stubs/PL/dispatch_loop/dl_system.h functions for ordered thread synchronization

10. 10 - Michael Wilson - 11/23/2015 Egress Key Extract Validation All validation tests done with 8 threads »Because the Egress can assume all outgoing packets are valid, no testing needs to be done for invalid packets »Unit Testing flow is …/src/key_extractor/PL/test/eth_top.flw Sends VLAN frames only, with UDP/IP payload, incrementing payload size No IP Options in this set Verified all fields of output ring data were as expected No full speed test in simulation yet

11. 11 - Michael Wilson - 11/23/2015 Egress Key Extractor Other Initialization »The PL Egress Key Extractor needs no initialization Bugs »None known! Untested »Hardware – this block has never been tested on hardware. Optimizations still available »Many minor re-ordering optimizations still available, but these are unnecessary (see performance!) To be done »The source still needs cleanup and commentary. »The T_INDEX code needs factoring for clarity. »Some local constants (E.g., the DRAM packet offset of 384) should be moved to a shared file.

12. 12 - Michael Wilson - 11/23/2015 Egress Key Extractor Other Performance »KE has CPU time of roughly 54 cycles. (Some time in other threads during unnecessary context yield not counted). No CPU optimization is necessary. »I/O latency for a single packet (no contention) was 562 cycles. »Total time for a single packet (from simulation) was 606 cycles. »Egress KE is under budget on paper; still need to do tests for contention and RX synchronization. »Expect Engress Key Extract to be extremely sensitive to DRAM contention!

13. Ingress Key Extractor

14. 14 - Michael Wilson - 11/23/2015 LC Ingress Key Extractor Lookup Phy Int Rx Switch Tx QM/Schd Key Extract Hdr Format Main function »Extracts lookup key. Lookup Key (64b): Ø SL Type (4b): 0101b Ø Port (4b): from RX Ø IP DAddr (32b) Ø IP Proto (8b) Ø UDP DPort (16b) Separate code path for each Substrate Link type (including VLAN) NN communication »Will use scratch ring communication for exception path Uses 8 threads SWITCHSWITCH Lookup Key[63-32] (32b) Buf Handle(32b) IP Pkt Length (16b) Reserved (8b) Lookup Key[ 31-0] (32b) Buf Handle(32b) Port (4b) Reserved (12b) Eth. Frame Len (16b) Eth Hdr Len (8b) slide taken from PlanetLab_Design.ppt

15. 15 - Michael Wilson - 11/23/2015 PlanetLab Ingress LC Input Frame New PlanetLab Substrate Link Type: »Configured SL Type LC is told at boot/init time that this is its one and only SL Type. Similar to the way P2P-DC is handled. »SL Type: 0101b »Port: May be a don’t care »IP DAddr: Verifies that packet is for our node »IP Proto = UDP Could be a UDP tunnel to a slice »UDP DPort: Indicates which slice »Default route is to the GPE Key = Ø SL=0101b Ø Port: May be a don’t care. Ø IP DAddr = our node address Type=IP (2B) PAD (nB) CRC (4B) UDP Payload (MN Packet) Src Addr (4B) Dst Addr (4B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol = UDP (1B) Hdr Cksum (2B) DstAddr (6B) SrcAddr (6B) IP Options (0-40B) Src Port (2B) Dst Port (2B) UDP length (2B) UDP checksum (2B) Type=802.1Q (2B) PAD (nB) CRC (4B) UDP Payload (MN Packet) Src Addr (4B) Dst Addr (4B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol = UDP (1B) Hdr Cksum (2B) DstAddr (6B) SrcAddr (6B) IP Options (0-40B) Src Port (2B) Dst Port (2B) UDP length (2B) UDP checksum (2B) VLAN (2B) Type=IP (2B) PlanetLab IPv4 Key(0x5) (64 bits) SL(4b) 0101 Port (4b) IP DAddr (32b) IP Proto (8b) UDP DPort (16b) Ethernet Header IP Header UDP Header Ethernet Trailer slide taken from PlanetLab_Design.ppt

16. 16 - Michael Wilson - 11/23/2015 Ingress KE Block Diagram dl_source_1ME_NN_2words() Start DRAM Reads ingress_key_extract() dl_sink_1ME_NN_4words() Wait for first Read Check Ethernet Type Set SL type & port Get IP Len, Eth Hdr Len, IP Daddr, IP Proto; setup for UDP dport read VLAN Wait for second Read Drop Read UDP Sport 13 cycles until ctx_arb 4 cycles (discounting branch) 1 Set Next BlockID IPv4 Other Get IP Len, Eth Hdr Len, IP Daddr, IP Proto; setup for UDP dport read Wait for second Read Read UDP Sport 20 cycles Wait for second Read 1

17. 17 - Michael Wilson - 11/23/2015 File locations (in …/LC_Ingress/) Code »src/key_extractor/PL/key_extract.uc Includes »../dispatch_loop/dl_source_WU.uc dl_source() and dl_sink() functions »Stubs/PL/dispatch_loop/dl_system.h functions for ordered thread synchronization

18. 18 - Michael Wilson - 11/23/2015 Ingress Key Extract Validation All validation tests done with 8 threads »Unit Testing flow is …/src/key_extractor/PL/test/eth_top.flw Sends VLAN frames alternating with non-VLAN frames, with UDP/IP payload, incrementing payload size Verified all fields of output ring data were as expected No drops in this set Ø ARP Ø Invalid packets No IP Options in this set No full speed test in simulation yet

19. 19 - Michael Wilson - 11/23/2015 Ingress Key Extractor Other Initialization »The PL Ingress Key Extractor needs no initialization Bugs »None known! Untested »Hardware – this block has never been tested on hardware. Optimizations still available »Many minor re-ordering optimizations still available, but these are unnecessary (see performance!) To be done »The source still needs cleanup and commentary. »The T_INDEX code needs factoring for clarity. »Some local constants (E.g., the DRAM packet offset of 384) should be moved to a shared file. »RFC 1812 packet validation »Arp packets being dropped instead of forwarded to XScale

20. 20 - Michael Wilson - 11/23/2015 Ingress Key Extractor Other Performance »KE has CPU time of roughly 60 cycles. (Some time in other threads during unnecessary context yield not counted). No CPU optimization is necessary. »I/O for a single packet (no contention) was 422 cycles. »Total time for a single packet (from simulation) was 599 cycles. »Ingress KE is under budget on paper; still need to do tests for contention and RX synchronization. »Expect Ingress Key Extract to be extremely sensitive to DRAM contention!

21. 21 - Michael Wilson - 11/23/2015 Image Slide Template

22. 22 - Michael Wilson - 11/23/2015 Text Slide Template

23. 23 - Michael Wilson - 11/23/2015 Extra Slides

24. 24 - Michael Wilson - 11/23/2015 Cycle Budget (min eth packets) To hit 5Gb rate: »76B per min IPv4 packet (64 min Eth + 12B IFS) »1.4Ghz clock rate »5 Gb/sec * 1B/8b * packet/76B = 8.22 Mp/sec »1.4Gcycle/sec * 1 sec/ 8.22 Mp = 170.3 cycles per packet »compute budget: 170 cycles »latency budget: (threads*170) 4 threads : 680 cycles 8 threads: 1360 cycles

25. 25 - Michael Wilson - 11/23/2015 Cycle Budget (IPv4 MN packets) To hit 5Gb rate: »90B per min IPv4 packet (78 min IPv4MN + 12B IFS) »1.4Ghz clock rate »5 Gb/sec * 1B/8b * packet/90B = 6.94 Mp/sec »1.4Gcycle/sec * 1 sec/ 6.94 Mp = 201.7 cycles per packet »compute budget: 201 cycles »latency budget: (threads*201) 4 threads : 804 cycles 8 threads: 1608 cycles

26. 26 - Michael Wilson - 11/23/2015 RFC 1812 5.2.2 IP Header Validation (1) The packet length reported by the Link Layer must be large enough to hold the minimum length legal IP datagram (20 bytes) (2) The IP checksum must be correct. (3) The IP version number must be 4. If the version number is not 4 then the packet may be another version of IP, such as IPng or ST-II. 4) The IP header length field must be large enough to hold the minimum length legal IP datagram (20 bytes = 5 words). (5) The IP total length field must be large enough to hold the IP datagram header, whose length is specified in the IP header length field. from http://www.faqs.org/rfcs/rfc1812.html, by way of Brandon Heller’s Block Review