2. FPX Network Platform 1 John Lockwood, Lockwood@arl.wustl.edu Assistant Professor Washington University Department of Computer Science Applied Research Lab 1 Brookings Drive Saint Louis, MO 63130 http://www.arl.wustl.edu/arl/projects/fpx/workshop_0801/agenda.html Supported by: NSF ANI-0096052 and Xilinx Corp. Field-programmable Port Extender (FPX) August 2001 Workshop

3. FPX Network Platform 2 Workshop Objectives Learn to accelerate network processing with reprogrammable hardware Understand System-On-Chip design Perform Hardware/Software Co-design Obtain hands-on experience with the Field Programmable Port Extender (FPX)

4. FPX Network Platform 3 Outline Motivation –Hardware enables packet processing at link speed –FPGAs provide flexibility for dynamically reconfiguration –Networks can be extended with FPGAs to provide enhanced functionality and performance. Research Results –Field programmable Port Extender (FPX) allows modules to be dynamically installed in a network. –FPX serves as an open platform for rapid prototype of firewall and router plug-in modules –Courses and Workshops held at Washington University to develop new System-On-Chip networking modules

5. FPX Network Platform 4 Workshop Activities Integrate FPX Infrastructure Components –SDRAM Memory Controller –Internet Protocol Wrappers Operate FPX Software Tools –NCHARGE Control Software –PARBIT FPGA Tools Implement Networking Modules on the FPX –Cell processing module in VHDL –Packet Processing Module –KCPSM Active Networking Module

6. FPX Network Platform 5 FPX Workshop Agenda: Times and Location –Wednesday, Aug 15, 2001 8am: Breakfast –5th floor Jolley Atrium 9am-Noon: Session I –Sever 201 Lab Lunch –WashU Campus 1pm-5pm: Session II –Sever 201 Lab –Thursday, Aug 16, 2001 8am: Breakfast –5 th floor Jolley Atrium 9am-Noon: Session III –Sever 201 Lab Lunch –5th floor Jolley Atrium 1pm-5pm: Session IV –Sever 201 Lab On-line Agenda: http://www.arl.wustl.edu/arl/projects/fpx/workshop_0801/agenda.html

7. FPX Network Platform 6 Building Networks with Reprogrammable hardware

8. FPX Network Platform 7 Technology Options for Evolvable Internet Hardware... Reprogrammable Hardware Network Processors Flexibility Performance Fully Reprogrammable High Performance Microprocessor ASIC

9. FPX Network Platform 8 Reprogrammable Device Configuration Routing Module : Interconnection of Blocks CLB : Primitive element of FPGA FPGA : Matrix of CLBs and Routing Modules

10. FPX Network Platform 9 The FPX Network Platform

11. FPX Network Platform 10 Configuration of Network Packet Processor Packet processing hardware performs: –Packet classification –Packet forwarding –Address Translation –Data modification –Packet buffering –Active Networking (Application-level data processing) FPX Extender Port programmable Field- Card OC3/ OC12/ OC48 Line Card OC3/ OC12/ OC48 Line Network Packets Network Packets

12. FPX Network Platform 11 Field Programmable Port Extender

13. FPX Network Platform 12 Configuration of Internet Router Additionally, Router interface performs: –Internet route lookup –Traffic policing and shaping IPP OPP Switch Fabric Gigabit IPP OPP FPX Extender Port programmable Field- Card OC3/ OC12/ OC48 Line Network Packets FPX Extender Port programmable Field- Card OC3/ OC12/ OC48 Line Network Packets

14. FPX Network Platform 13 Port Processing at edge of Gigabit Switch Original Network Switch –Line card connects to Gigabit switch backplane FPX-Enhanced Router –Line card connects to Gigabit switch backplane

15. FPX Network Platform 14 Complete Router Platform

16. FPX Network Platform 15 Combination Router Hardware and Software Implement link speed opertions on hardware Implement higher-level functions in software Migrate functionality on the critical path

17. FPX Network Platform 16 Port Configuration Data Module Intel Embedded PCI Bus PCI Interface

18. FPX Network Platform 17 The FPX Architecture

19. FPX Network Platform 18 Architecture of the FPX RAD –Large Xilinx FPGA –Attaches to SRAM and SDRAM –Reprogrammable over network –Provides two user-defined Module Interfaces NID –Provides Utopia Interfaces between switch & line card –Forwards cells to RAD –Programs RAD

20. FPX Network Platform 19 Field Programmable Port Extender –NID : Network Interface Device –RAD : Reprogrammable Application Device

21. FPX Network Platform 20 FPX SRAM –Provide low latency for fast table-lookups –Zero Bus Turnaround (ZBT) allows back-to-back read / write operations every 10ns –Dual, Independent Memories –36-bit wide bus

22. FPX Network Platform 21 FPX SDRAM –Dual, independent SDRAM memories –64-bit wide, 100 MHz –64MByte / Module : 128 Mbyte total [expandable] –Burst-based transactions [1-8 word transfers] –Latency of 14 cycles to Read/Write 8-word burst

23. FPX Network Platform 22 Hardware Device

24. FPX Network Platform 23 FPX Module Interface

25. FPX Network Platform 24 FPX Interfaces Provides Well defined Interface –Utopia-like 32-bit fast data interface –Flow control allows back-pressure Flow Routing –Arbitrary permutations of packet flows through ports Dynamically Reprogrammable –Other modules continue to operate even while new module is being reprogrammed Memory Access –Shared access to SRAM and SDRAM –Request/Grant protocol

26. FPX Network Platform 25 Reprogrammable Application Device (RAD) Spatial Re-use of FPGA Resources –Modules implemented using FPGA logic –Module logic can be individually reprogrammed Shared Access to off-chip resources –Memory Interfaces to SRAM and SDRAM –Common Datapath to send and receive data

27. FPX Network Platform 26 Network Module Hardware Interface SRAM_D_OUT[35:0] SRAM_ADDR[17:0] SRAM_WR_RD SRAM_REQ SRAM_GR SRAM_D_IN[35:0] SRAM Interface FPX Network Module READY_L CLK RESET_L ENABLE_L Module Interface SDRAM_RQ SDRAM_BL[4:0] SDRAM_ADDR[26:0] SDRAM_WR_RD SDRAM_GR SDRAM_DATA[63:0] SDRAM Interface SDRAM_EN SDRAM_OP_FIN TCA_MOD_OUT D_MOD_IN[31:0] SOC_MOD_IN D_MOD_OUT[31:0] SOC_MOD_OUT TCA_MOD_IN Data Interface fpx_module.vhd

28. FPX Network Platform 27 RAD Module Interface Cell I/O and Flow Control –32-bit wide UTOPIA-style interface w/ unique timing Off-chip Memory Access –Arbitrated access to SRAM and SDRAM via standard interface Control (clock, reset, and reconfiguration control)

29. FPX Network Platform 28 Infrastructure Services

30. FPX Network Platform 29 Routing Traffic Flows Between Modules Functions –Check packets for errors –Process commands Control, status, & reprogramming –Implement per-flow forwarding NID LineCardSwitch EC VC ccp Traffic flows routed among –Switch –Line Card –RAD.Switch –RAD.Linecard

31. FPX Network Platform 30 Example NID Routing Configurations Hardware Module Bypass (Default) 0 VC EC LineCard VC ccp EC VC Switch NID RAD No Modules Installed 1 23 Default Flow Action (Bypass) Egress Processing Ingress Processing Ingress+Egress Loopback Chained Egress Processing

32. FPX Network Platform 31 NID Flow Entry Example Problem: –Route flow on VCI 45 from Switch to Mod_sw –Route Flow on VCI 45 from Mod_sw to LC –Route Flow on VCI 45 from Linecard to Switch Solution –VCI [45] = 1,3,X,0 __ VCI 45 d1d2d3d4

33. FPX Network Platform 32 FPX Control : NCHARGE

34. FPX Network Platform 33 Fpx_control {0-7}.{0/1} Pictorial view of fpx_control interfaced with hardware Switch Controller

35. FPX Network Platform 34 Controlling the FPX Methods of Communication - NCHARGE - Telnet - Web Interface / CGI - Basic_send - User Applications Software Plug-ins - Concepts - Functionality Emulation –Nid_listener –Rad_listener Basic Send CGI Fip Memory Manager Access WEBBasic Telnet Send NID RAD 0.0 Gigabit Switch OC-3 Link NCHARGE 7.1 Software Controller Fip Remote Applications VCI 76 (NID), VCI 100 (RAD) VCI 115 (NID), VCI 123 (RAD) (up to 32 VCIs) Read Washington University RAD NCHARGE

36. FPX Network Platform 35 Web Access to NCHARGE - Radio Button Interface - Allows user to submit commands using CGI scripts - Provides for Switch Reset - http://fpx.arl.wustl.edu Web Access Provides:

37. FPX Network Platform 36 FPX Control and Reconfiguration

38. FPX Network Platform 37 Switch Element IPP OPP IPP LC WUGS Example: Write Configuration Memory FPX VPI PTI = 00 VCI = 0x34 HEC Reserved 70654213 CRC Sequence # CMData OPCODE Control Cell Switch Controller Switch Controller generates command to write 32-byte data element to RAD Configuration Memory ccp Control Cell sent containing Memory Address and Data Switch Element routes control cell to FPX NID Element on FPX writes Data from Payload into Rad Configuration Memory RAD Configuration Memory

39. FPX Network Platform 38 Reprogramming Logic Switch Controller writes RAD configuration memory to NID NID reads RAD config memory to program RAD NID programs at boot from EPROM –Bitfile for RAD arrives transmitted over network via control cells –Performs complete or partial reprogramming of RAD Switch Controller issues {Full/Partial} reconfigure command

40. FPX Network Platform 39 System-On-Chip Design using Dynamic Hardware Plugins (DHP)

41. FPX Network Platform 40 Module Loading / Unloading SRAM SDRAM SRAM FPX Module SDRAM SRAM FPGA’s Long Lines Intrachip Module Switching... Data Combining Modules within the Chip Modules fit together at static I/O interfaces Partial reprogramming of FPGA used to install/remove modules Modules added and removed while other modules process packts Statically-configured ‘Long Lines’ provide chip-wide routing

42. FPX Network Platform 41 Implementing modules in Virtex1000E Virtex 1000E logic resources –Globally accessible IOBs –64 x 96 CLB array 4 flops/LUTs per CLB –96 Block SelectRAMs 4096 bits per block 6 columns of 16 blocks 6 columns of dedicated interconnect Ingress Path Egress Path Double DHP Module DHP Module DHP Modules –64 x 12 CLB array (768 CLBs, 3072 flops) Double DHP Modules –64 x 24 CLB array (1536 CLBs, 6144 flops) 16 BRAMs (8KB) per Module 3 DHP Modules per path 1 SRAM interface per path 1 SDRAM interface per path IOB Ring VersaRing CLB columns BRAMs BRAM Interconnect

43. FPX Network Platform 42 Dynamic Hardware Plugins [On RAD FPGA] Module Synthesis Constraints –Infrastructure with target regions reserved for DHP modules insertions –PARBIT parameters Target Locations (Row, Col)

44. FPX Network Platform 43 Example Application : String Processing

45. FPX Network Platform 44 “Hello, World” Module Function

46. FPX Network Platform 45 Evolvable Internet Hardware Place and Route FPGA Gates (Xilinx) Module to RAD (Ncharge) Download Synthesize To Logic (Synplicity) Compile Design (Vcom) Observe Network Behaviour Through Module (Ncharge) Route Traffic Tweak Design

47. FPX Network Platform 46 Modular Interface to SDRAM

48. FPX Network Platform 47 SDRAM Controller Overview Module 0 Module 1 Module 2 SDRAM SDRAM Controller

49. FPX Network Platform 48 Example : Leaky Bucket - Buffers the incoming cells in a FIFO - Generates tokens at a regular interval - Gives out a cell from the FIFO when the number of tokens > 0 - Destroys a token when a cell is given out bursty data leaking data

50. FPX Network Platform 49 Internet Protocol Wrappers

51. FPX Network Platform 50 Payload Processing Environment –Higher-Level data processing on the FPX –Wrapper Framework Net App Wrapper

52. FPX Network Platform 51 Frame / IP Packet / UDP / Application Layers UDP Processor IP Processor Cell Processor Frame Processor Data Output Data Input Application-level Hardware Module Interfaces to Off-Chip Memories

53. FPX Network Platform 52 Soft-core Active Network Processor : The KCPSM Network Module

54. FPX Network Platform 53 The FPX KCPSM Module

55. FPX Network Platform 54 Simulating a KCPSM Program The input data cell The output data cell The 1 st – 2 nd word is the ATM Cell Header The 3 rd – 7 th word is the Internet Protocol Header The 8 th – 9 th word is the UDP Header The 10 th word specifies it is a data packet The 11 th – 13 th word is the data string ‘Hello World’ The 1 st – 2 nd word is the ATM Cell Header The 3 rd – 7 th word is the Internet Protocol Header The 8 th – 9 th word is the UDP Header The 10 th word specifies it is a data packet The 11 th – 13 th word is the data string ‘Uryyb Jbeyq’

56. FPX Network Platform 55 Applications for the FPX: Fast IP Lookup (FIPL)

57. FPX Network Platform 56 Fast IP Lookup Algorithm –Function: Search for best matching prefix using Trie algorithm –Contributors Will Eatherton, Zubin Dittia, Jon Turner, David Taylor, David Wilke, PrefixNext Hop * 01* 4 7 10*2 110*9 0001*1 1011*0 00110*5 01011*3 0 1 0 00 0 0 0 11 1 11 1 1 1 1

58. FPX Network Platform 57 Hardware Implementation in the FPX SRAM 1 SRAM 2 IP Lookup Engine counter On-Chip Cell Store SRAM 1 Interface Control Cell Processor Packet Reassembler RAD FPGA NID FPGA Extract IP Headers Remap VCIs for IP packets LC SW Request Grant 0 1 0 00 0 0 0 11 1 11 1 1 1 1

59. FPX Network Platform 58 Fast IP Lookup (FIPL) Application Route add 141.142.5.0/24 8 Route delete 141.142.0.0/16 Control cells FIPL Fast IP Lookup Lookup (X.Y.Z.W) Nexthop FPGA RAM External SRAM FIPL Memory Manager Software Commands Hardware Lookup

60. FPX Network Platform 59 Conclusion Field programmable Port Extender (FPX) Platform –Open Platform for hardware development Modular Interfaces –Allows integration components to build System-on-Chip (SoC) Library of Internet packet processing functions –Simplifies design of new functionality Interoperable w/existing software systems –Unifies Active Networking Hardware with Software –FPX Homepage http://www.arl.wustl.edu/arl/projects/fpx/

61. FPX Network Platform 60 Acknowledgements Several Individuals have contributed to this work: –Washington University Jon Turner Sarang Dharmapurikar Todd Sproull David Taylor Florian Braun Henry Fu Dave Lim Edson Horta –Xilinx Dave Parlour