1. Hands-on Training – June 18-19, 2012 1 NetFPGA Hands-on Training Day 1 Presented by: Adam Covington (Stanford University) Indiana University, Bloomington June 18 - 19, 2012 http://NetFPGA.org

2. Hands-on Training – June 18-19, 2012 2 Tutorial Outline Background –Introduction –The NetFPGA Platform The Stanford Base Reference Router –Motivation: Basic IP review –Example: Reference Router running on the NetFPGA Infrastructure –Tree –Build System –Scripts The Life of a Packet Through the NetFPGA –Hardware Datapath –Interface to software: Exceptions and Host I/O Implementation –Module Template –User Data Path –Write Crypto NIC using a static key Simulation and Debug –Write and Run Simulations for Crypto NIC Concluding Remarks

3. Hands-on Training – June 18-19, 2012 3 Section I: Motivation

4. Hands-on Training – June 18-19, 2012 4 NetFPGA = Networked FPGA A line-rate, flexible, open networking platform for teaching and research

5. Hands-on Training – June 18-19, 2012 5 NetFPGA 1G Board NetFPGA consists of… Four elements: NetFPGA board Tools + reference designs Contributed projects Community NetFPGA 10G Board

6. Hands-on Training – June 18-19, 2012 6 NetFPGA 1GNetFPGA 10G 4 x 1Gbps Ethernet Ports4 x 10Gbps Ethernet Ports 4.5 MB ZBT SRAM 64 MB DDR2 SDRAM 27 MB QDRII-SRAM 288 MB RLDRAM-II PCIPCI Express x8 Virtex II-Pro 50Virtex 5 TX240T NetFPGA Board Comparison

7. Hands-on Training – June 18-19, 2012 7 FPGA Memory 1GE NetFPGA board PCI CPU Memory NetFPGA Board PC with NetFPGA Networking Software running on a standard PC A hardware accelerator built with a Field Programmable Gate Array driving Gigabit network links

8. Hands-on Training – June 18-19, 2012 8 Tools + Reference Designs Tools: Compile designs Verify designs Interact with hardware Reference designs: Router (HW) Switch (HW) Network Interface Card (HW) Router Kit (SW) SCONE (SW)

9. Hands-on Training – June 18-19, 2012 9 Contributed Projects More projects: http://netfpga.org/foswiki/NetFPGA/OneGig/ProjectTable ProjectContributor OpenFlow switchStanford University Packet generatorStanford University NetFlow ProbeBrno University NetThreadsUniversity of Toronto zFilter (Sp)routerEricsson Traffic MonitorUniversity of Catania DFAUMass Lowell

10. Hands-on Training – June 18-19, 2012 10 Community Wiki Documentation –User’s Guide –Developer’s Guide Encourage users to contribute Forums Support by users for users Active community - 10s-100s of posts/week

11. Hands-on Training – June 18-19, 2012 11 International Community Over 1,000 users, using 2,000 cards at 150 universities in 40 countries

12. Hands-on Training – June 18-19, 2012 12 NetFPGA’s Defining Characteristics Line-Rate –Processes back-to-back packets Without dropping packets At full rate of Gigabit Ethernet Links –Operating on packet headers For switching, routing, and firewall rules –And packet payloads For content processing and intrusion prevention Open-source Hardware –Similar to open-source software Full source code available BSD-Style License –But harder, because Hardware modules must meeting timing Verilog & VHDL Components have more complex interfaces Hardware designers need high confidence in specification of modules

13. Hands-on Training – June 18-19, 2012 13 Test-Driven Design Regression tests –Have repeatable results –Define the supported features –Provide clear expectation on functionality Example: Internet Router –Drops packets with bad IP checksum –Performs Longest Prefix Matching on destination address –Forwards IPv4 packets of length 64-1500 bytes –Generates ICMP message for packets with TTL <= 1 –Defines how packets with IP options or non IPv4 … and dozens more … Every feature is defined by a regression test

14. Hands-on Training – June 18-19, 2012 14 Who, How, Why Who uses the NetFPGA? –Teachers –Students –Researchers How do they use the NetFPGA? –To run the Router Kit –To build modular reference designs IPv4 router 4-port NIC Ethernet switch, … Why do they use the NetFPGA? –To measure performance of Internet systems –To prototype new networking systems

15. Hands-on Training – June 18-19, 2012 15 Summer Camp Objectives Overall picture of NetFPGA How reference designs work How you can work on a project –NetFPGA Design Flow –Directory Structure, library modules and projects –How to utilize contributed projects Interface/Registers –How to verify a design (Simulation and Regression Tests) –Things to do when you get stuck AND… You can build your own projects!

16. 16 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 16 Section II: Network review

17. 17 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 17 Internet Protocol (IP) Data IP Hdr Eth Hdr Data IP Hdr Data to be transmitted: IP packets: Ethernet Frames: Data IP Hdr Data IP Hdr Eth Hdr Data IP Hdr Eth Hdr Data IP Hdr … …

18. 18 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 18 Internet Protocol (IP) Data IP Hdr … 16 32 41 Options (if any) Destination Address Source Address Header ChecksumProtocolTTL Fragment Offset Flags Fragment ID Total Packet LengthT.Service HLen Ver 20 bytes

19. 19 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 19 Basic operation of an IP router R3 A B C R1 R2 R4D E F R5 F R3E D Next HopDestination D

20. 20 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 20 Basic operation of an IP router A B C R1 R2 R3 R4D E F R5

21. 21 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 21 Forwarding tables EntryDestinationPort 1 2 ⋮ 2 32 0.0.0.0 0.0.0.1 ⋮ 255.255.255.255 1 2 ⋮ 12 ~ 4 billion entries Naïve approach: One entry per address Improved approach: Group entries to reduce table size EntryDestinationPort 1 2 ⋮ 50 0.0.0.0 – 127.255.255.255 128.0.0.1 – 128.255.255.255 ⋮ 248.0.0.0 – 255.255.255.255 1 2 ⋮ 12 IP address 32 bits wide → ~ 4 billion unique address

22. 22 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 22 IP addresses as a line 0 2 32 -1 EntryDestinationPort 1234512345 Stanford Berkeley North America Asia Everywhere (default) 1234512345 All IP addresses North America Asia BerkeleyStanford Your computerMy computer

23. 23 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 23 Longest Prefix Match (LPM) EntryDestinationPort 1234512345 Stanford Berkeley North America Asia Everywhere (default) 1234512345 Universities Continents Planet Data To: Stanford Matching entries: Stanford North America Everywhere Most specific

24. 24 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 24 Longest Prefix Match (LPM) EntryDestinationPort 1234512345 Stanford Berkeley North America Asia Everywhere (default) 1234512345 Universities Continents Planet Data To: Canada Matching entries: North America Everywhere Most specific

25. 25 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 25 Implementing Longest Prefix Match EntryDestinationPort 1234512345 Stanford Berkeley North America Asia Everywhere (default) 1234512345 Most specific Least specific Searching FOUND

26. 26 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 26 Basic components of an IP router Control Plane Data Plane per-packet processing Switching Forwarding Table Routing Table Routing Protocols Management & CLI Software Hardware Queuing

27. 27 S T A N F O R D U N I V E R S I T Y Hands-on Training – June 18-19, 2012 27 IP router components in NetFPGA SCONE Routing Table Routing Protocols Management & CLI Output Port Lookup Forwarding Table Input Arbiter Output Queues SwitchingQueuing Linux Routing Table Routing Protocols Management & CLI Router Kit OR Software Hardware

28. 28 Hands-on Training – June 18-19, 2012 28 Section III: Example

29. 29 Hands-on Training – June 18-19, 2012 29 Operational IPv4 router Control Plane Data Plane per-packet processing Software Hardware Routing Table Routing Protocols Management & CLI SCONE Switching Forwarding Table Queuing Reference router Java GUI

30. 30 Hands-on Training – June 18-19, 2012 30 Streaming video

31. 31 Hands-on Training – June 18-19, 2012 31 Streaming video PC & NetFPGA (NetFPGA in PC) NetFPGA running reference router

32. 32 Hands-on Training – June 18-19, 2012 32 Streaming video Video streaming over shortest path Video client Video server

33. 33 Hands-on Training – June 18-19, 2012 33 Streaming video Video client Video server Link breaks

34. 34 Hands-on Training – June 18-19, 2012 34 Streaming video.1.1.1.2.3.1.30.2.4.1.4.2.6.1.3.2.7.1.7.2.9.1.6.2.10.1.10.2.12.1.9.2.13.1.13.2.15.1.12.2.16.1.16.2.15.2.28.1.28.2.27.1.30.1.25.1.25.2.24.1.27.2.22.1.22.2.21.1.24.2.19.1.19.2.21.2.18.2.18.1.23.1

35. 35 Hands-on Training – June 18-19, 2012 35 Observing the routing tables Columns: Subnet address Subnet mask Next hop IP Output ports

36. 36 Hands-on Training – June 18-19, 2012 36 Demo

37. 37 Hands-on Training – June 18-19, 2012 37 Review NetFPGA as IPv4 router: Reference hardware + SCONE software Routing protocol discovers topology Demo: Ring topology Traffic flows over shortest path Broken link: automatically route around failure

38. 38 Hands-on Training – June 18-19, 2012 38 Section IV: Infrastructure

39. 39 Hands-on Training – June 18-19, 2012 39 Infrastructure Tree structure NetFPGA package contents –Reusable Verilog modules –Verification infrastructure –Build infrastructure –Utilities –Software libraries

40. 40 Hands-on Training – June 18-19, 2012 40 Tree Structure (1) netfpga bin lib projects bitfiles (scripts for running simulations and setting up the environment) (contains the bitfiles for all projects that have been synthesized) (shared Verilog modules, libraries needed for simulation/synthesis/design) (user projects, including reference designs)

41. 41 Hands-on Training – June 18-19, 2012 41 Tree Structure (2) lib C java Makefiles Perl5 python scripts verilog (common software and code for reference designs) (contains software for the graphical user interface) (makefiles for simulation and synthesis) (libraries to interact with reference designs, create test data, and manage simulations/regression tests) (common libraries to aid in regression tests) (utility scripts – less commonly used than those in the bin directory) (modules that can be reused in designs)

42. 42 Hands-on Training – June 18-19, 2012 42 Tree Structure (3) projects/crypto_nic doc include src sw synth test (project specific documentation) (XML files defining project and any local modules, auto-generated Verilog register defines) (non-library Verilog code used for synthesis and simulation) (software elements of the project) (project-specific.xco files to generate cores, Makefile to implement the design) (simulation and hardware tests) lib (C/Perl defines for registers)

43. 43 Hands-on Training – June 18-19, 2012 43 NetFPGA package contents Projects: –HW: router, switch, NIC, buffer sizing router –SW: router kit, SCONE Reusable Verilog modules Verification infrastructure: –simulate full board with PCI + physical interfaces –run tests against hardware –test data generation libraries (eg. packets) Build infrastructure Utilities: –register I/O, packaging, … Software libraries

44. 44 Hands-on Training – June 18-19, 2012 44 Reusable Verilog modules CategoryModules I/O interfacesEthernet MAC CPU DMA queues CPU register queues MDIO PCI Output queuesSRAM-based DRAM-based BRAM-based Output port lookupRouter (CAM-based) Learning switch (CAM-based) NIC Hardwired Memory interfacesSRAMDRAM MiscellaneousFIFOs Generic register module Rate limiter

45. 45 Hands-on Training – June 18-19, 2012 45 Verification Infrastructure Simulation: nf_test.py sim –allows testing before synthesis –catches many bugs Hardware tests: nf_test.py hw –test synthesized hardware Test data generation libraries: –easily create test data: –many standard packet formats supported out of the box –easily add support for custom formats

46. 46 Hands-on Training – June 18-19, 2012 46 Build infrastructure Register system: –allocates memory to modules –generates “include” files for various languages Build/synthesis: –required shared modules documented XML (shared with register system) –shared modules pulled in during synthesis –resultant bitfile checked for timing errors

47. 47 Hands-on Training – June 18-19, 2012 47 Utilities Bitfile download: nf_download Register I/O: regread, regwrite Device querying: nf_info SRAM dumping: lib/scripts/sram_dump

48. 48 Hands-on Training – June 18-19, 2012 48 Software libraries Libraries for interfacing with NetFPGA: –C, Perl, Java, Python support

49. Hands-on Training – June 18-19, 2012 49 Section V: Life of a Packet

50. Hands-on Training – June 18-19, 2012 50 Reference Router Pipeline Five stages –Input –Input arbitration –Routing decision and packet modification –Output queuing –Output Packet-based module interface Pluggable design MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ Input Arbiter Output Port Lookup MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ Output Queues

51. Hands-on Training – June 18-19, 2012 51 Full System Components Software PCI Bus NetFPGA CPU RxQ CPU RxQ CPU TxQ nf2_reg_grp user data path nf2c0nf2c1nf2c2nf2c3ioctl MAC TxQ MAC TxQ MAC RxQ MAC RxQ Ethernet CPU RxQ CPU RxQ CPU TxQ CPU RxQ CPU RxQ CPU TxQ CPU RxQ CPU RxQ CPU TxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ

52. Hands-on Training – June 18-19, 2012 52 port0port2 192.168.2.y 192.168.1.x Life of a Packet through the Hardware IP packet

53. Hands-on Training – June 18-19, 2012 53 Inter-Module Communication Using “Module Headers”: IP Hdr Eth Hdr … 0 0 0 Last word of packet0x10 Last Module Hdry …… Module Hdrx Contain information such as packet length, input port, output port, … Data Word (64 bits) Ctrl Word (8 bits)

54. Hands-on Training – June 18-19, 2012 54 Module i Module i+1 data Inter-Module Communication ctrl wr rdy

55. Hands-on Training – June 18-19, 2012 55 MAC Rx Queue IP Hdr: IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4 Eth Hdr: Dst MAC = port 0, Ethertype = IP Data

56. Hands-on Training – June 18-19, 2012 56 Rx Queue IP Hdr: IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4 Eth Hdr: Dst MAC = port 0, Ethertype = IP Data 0 0 0 Pkt length, input port = 0 0xff

57. Hands-on Training – June 18-19, 2012 57 Input Arbiter Rx Q 0 Rx Q 1 … Rx Q 7 Pkt

58. Hands-on Training – June 18-19, 2012 58 Output Port Lookup IP Hdr: IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4 EthHdr: Dst MAC = 0 Src MAC = x, Ethertype = IP Data 0 0 0 Pkt length, input port = 0 0xff

59. Hands-on Training – June 18-19, 2012 59 Output Port Lookup IP Hdr: IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4 IP Hdr: IP Dst: 192.168.2.3, TTL: 63, Csum:0x3ac2 Output Port Lookup EthHdr: Dst MAC = 0 Src MAC = x, Ethertype = IP Data 0 0 0 Pkt length, input port = 00xff 1- Check input port matches Dst MAC 2- Check TTL, checksum 3- Lookup next hop IP & output port (LPM) 4- Lookup next hop MAC address (ARP) 5- Add output port header 6- Modify MAC Dst and Src addresses 7-Decrement TTL and update checksum EthHdr: Dst MAC = nextHop Src MAC = port 4, Ethertype = IP Pkt length, input port = 0 output port = 4

60. Hands-on Training – June 18-19, 2012 60 Output Queues Pkt OQ0 OQ4 OQ7

61. Hands-on Training – June 18-19, 2012 61 MAC Tx Queue IP Hdr: IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4 IP Hdr: IP Dst: 192.168.2.3, TTL: 63, Csum:0x3ac2 EthHdr: Dst MAC = nextHop Src MAC = port 4, Ethertype = IP Data 0 0 0 Pkt length, input port = 0 output port = 4 0xff

62. Hands-on Training – June 18-19, 2012 62 MAC Tx Queue IP Hdr: IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4 IP Hdr: IP Dst: 192.168.2.3, TTL: 63, Csum:0x3ac2 EthHdr: Dst MAC = nextHop Src MAC = port 4, Ethertype = IP Data 0 0 0 Pkt length, input port = 0 output port = 4 0xff

63. Hands-on Training – June 18-19, 2012 63 Exception Packets Example: TTL = 0 or TTL = 1 Packet has to be sent to the CPU Host generates an ICMP packet response Difference starts at the Output Port Lookup

64. Hands-on Training – June 18-19, 2012 64 nf2c0nf2c1nf2c2nf2c3ioctl Ethernet Exception Packet Path CPU RxQ CPU RxQ CPU TxQ CPU TxQ CPU RxQ CPU RxQ CPU TxQ CPU TxQ CPU RxQ CPU RxQ CPU TxQ CPU TxQ CPU RxQ CPU RxQ CPU TxQ CPU TxQ nf2_reg_grp user data path MAC TxQ MAC TxQ MAC RxQ MAC RxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ Ethernet Software PCI Bus NetFPGA

65. Hands-on Training – June 18-19, 2012 65 Output Port Lookup IP Hdr: IP Dst: 192.168.2.3, TTL: 1, Csum:0x3ab4 Output Port Lookup EthHdr: Dst MAC = 0, Src MAC = x, Ethertype = IP Data 0 0 0 Pkt length, input port = 00xff 1- Check input port matches Dst MAC 2- Check TTL, checksum – EXCEPTION! 3- Add output port module Pkt length, input port = 0 output port = 1

66. Hands-on Training – June 18-19, 2012 66 Output Queues Pkt OQ0 OQ1 OQ2 OQ7

67. Hands-on Training – June 18-19, 2012 67 CPU Tx Queue IP Hdr: IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4 IP Hdr: IP Dst: 192.168.2.3, TTL: 1, Csum:0x3ab4 EthHdr: Dst MAC = 0, Src MAC = x, Ethertype = IP Data 0 0 0 Pkt length, input port = 0 output port = 1 0xff

68. Hands-on Training – June 18-19, 2012 68 CPU Tx Queue IP Hdr: IP Dst: 192.168.2.3, TTL: 1, Csum:0x3ab4 EthHdr: Dst MAC = 0, Src MAC = x, Ethertype = IP Data 0 0 0 Pkt length, input port = 0 output port = 1 0xff

69. Hands-on Training – June 18-19, 2012 69 ICMP Packet Packet arrives at the CPU Rx Queue from the PCI Bus Same path as a packet from the MAC until it reaches the Output Port Lookup (OPL) The OPL module sees the packet is from the CPU Rx Queue 1 and sets the output port directly to 0 The packet continues on the same path as the non-exception packet to the Output Queues and then MAC Tx queue 0

70. Hands-on Training – June 18-19, 2012 70 nf2c0nf2c1nf2c2nf2c3ioctl Ethernet ICMP Packet Path CPU RxQ CPU RxQ CPU TxQ CPU TxQ CPU RxQ CPU RxQ CPU TxQ CPU TxQ CPU RxQ CPU RxQ CPU TxQ CPU TxQ CPU RxQ CPU RxQ CPU TxQ CPU TxQ nf2_reg_grp user data path MAC TxQ MAC TxQ MAC RxQ MAC RxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ MAC TxQ MAC TxQ MAC RxQ MAC RxQ Ethernet Software PCI Bus NetFPGA

71. Hands-on Training – June 18-19, 2012 71 NetFPGA-Host Interaction Linux driver interfaces with hardware –Packet interface via standard Linux network stack –Register reads/writes via ioctl system call with wrapper functions: readReg(nf2device *dev, int address, unsigned *rd_data); writeReg(nf2device *dev, int address, unsigned *wr_data); eg: readReg(&nf2, OQ_NUM_PKTS_STORED_0, &val);

72. Hands-on Training – June 18-19, 2012 72 NetFPGA-Host Interaction NetFPGA to host packet transfer PCI Bus 2. Interrupt notifies driver of packet arrival 3. Driver sets up and initiates DMA transfer 1. Packet arrives – forwarding table sends to CPU queue

73. Hands-on Training – June 18-19, 2012 73 NetFPGA-Host Interaction NetFPGA to host packet transfer (cont.) PCI Bus 4. NetFPGA transfers packet via DMA 5. Interrupt signals completion of DMA 6. Driver passes packet to network stack

74. Hands-on Training – June 18-19, 2012 74 NetFPGA-Host Interaction Host to NetFPGA packet transfers PCI Bus 3. Interrupt signals completion of DMA 1. Software sends packet via network sockets Packet delivered to driver 2. Driver sets up and initiates DMA transfer

75. Hands-on Training – June 18-19, 2012 75 NetFPGA-Host Interaction Register access PCI Bus 1. Software makes ioctl call on network socket ioctl passed to driver 2. Driver performs PCI memory read/write

76. Hands-on Training – June 18-19, 2012 76 NetFPGA-Host Interaction Packet transfers shown using DMA interface Alternative: use programmed IO to transfer packets via register reads/writes –slower but eliminates the need to deal with network sockets

77. Hands-on Training – June 18-19, 2012 77 Section VI: Example Project

78. Hands-on Training – June 18-19, 2012 78 Project: Cryptographic NIC Implement a network interface card (NIC) that encrypts upon transmission and decrypts upon reception

79. Hands-on Training – June 18-19, 2012 79 Cryptography XOR function XOR written as: ^ ⊻ XOR is commutative: (A ^ B) ^ C = A ^ (B ^ C) ABA ^ B 000 011 101 110 XORing a value with itself always yields 0

80. Hands-on Training – June 18-19, 2012 80 Cryptography (cont.) Simple cryptography: –Generate a secret key –Encrypt the message by XORing the message and key –Decrypt the ciphertext by XORing with the key Explanation: (M ^ K) ^ K =M ^ (K ^ K) =M =M ^ 0 Commutativity A ^ A = 0

81. Hands-on Training – June 18-19, 2012 81 Cryptography (cont.) Example: Message:00111011 Key:10110001 Message ^ Key:10001010 Key:10110001 Message ^ Key ^ Key:00111011

82. Hands-on Training – June 18-19, 2012 82 Cryptography (cont.) Idea: Implement simple cryptography using XOR –32-bit key –Encrypt every word in payload with key Note: XORing with a one-time pad of the same length of the message is secure/uncrackable. See: http://en.wikipedia.org/wiki/One-time_pad PayloadHeader Key

83. Hands-on Training – June 18-19, 2012 83 Section VII: Implementation

84. Hands-on Training – June 18-19, 2012 84 Getting started with a new project (1) Projects: –Each design represented by a project –Location: netfpga/projects/ netfpga/projects/crypto_nic –Consists of: Verilog source Simulation tests Hardware tests Libraries Optional software

85. Hands-on Training – June 18-19, 2012 85 Getting started with a new project (2) –Normally: copy an existing project as the starting point –Today: pre-created project –Missing from pre-created project: Verilog files (with crypto implementation) Simulation tests Hardware tests Custom software

86. Hands-on Training – June 18-19, 2012 86 Getting started with a new project (3) Typically implement functionality in one or more modules inside the user data path MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ MAC RxQ MAC RxQ CPU RxQ CPU RxQ Input Arbiter Output Port Lookup MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ MAC TxQ MAC TxQ CPU TxQ CPU TxQ Output Queues Crypto Crypto module to encrypt and decrypt packets User data path

87. Hands-on Training – June 18-19, 2012 87 Getting started with a new project (4) –Shared modules included from netfpga/lib/verilog Generic modules that are re-used in multiple projects Specify shared modules in project’s include/project.xml –Local src modules override shared modules –crypto_nic: LocalShared user_data_path.v crypto.v Everything else

88. Hands-on Training – June 18-19, 2012 88 Exploring project.xml (1) Location: project/ /include Crypto NIC NIC with basic crypto support 0 1 0 Short name Description Version information indicate bitfile version Unique ID to identify project See: http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/DeviceIDList

89. Hands-on Training – June 18-19, 2012 89 Exploring project.xml (2) core/io_queues/cpu_dma_queue core/io_queues/ethernet_mac core/input_arbiter/rr_input_arbiter core/nf2/generic_top core/nf2/reference_core core/output_port_lookup/nic core/output_queues/sram_rr_output_queues core/sram_arbiter/sram_weighted_rr core/user_data_path/reference_user_data_path core/io/mdio core/cpci_bus core/dma core/user_data_path/udp_reg_master core/io_queues/add_rm_hdr core/strip_headers/keep_length core/utils/generic_regs core/utils Shared modules to load from lib/verilog

90. Hands-on Training – June 18-19, 2012 90 Exploring project.xml (3) Specify where to instantiate modules, the number of instances, and the memory addresses to use

91. Hands-on Training – June 18-19, 2012 91 Getting started with a new project (5) Tasks: Set the project that we’ll be working with: 1.Add the following lines to the end of ~/.bashrc: export NF_DESIGN_DIR=$NF_ROOT/projects/crypto_nic export PERL5LIB=$NF_ROOT/lib/Perl5:$NF_DESIGN_DIR/lib/Per l5 2.Type: source ~/.bashrc Copy reference files as starting points: 3.Copy the following files from netfpga/lib/verilog/core into netfpga/projects/crpyto_nic/src user_data_path/reference_user_data_path/src/user_data_path.v module_template/src/module_template.v

92. Hands-on Training – June 18-19, 2012 92 Getting started with a new project (6) Create crypto.v from module_template.v: 1.Rename the local module_template.v to crypto.v 2.Change the module name inside crypto.v (first non- comment line of the file) 3.Add the crypto module to the user data path

93. Hands-on Training – June 18-19, 2012 93 user_data_path.v (1) module user_data_path #( parameter DATA_WIDTH = 64,... ) (... ) //------------------ Internal parameters -----------------------... //----------------- Input arbiter wires/regs -------------------... Module port declaration

94. Hands-on Training – June 18-19, 2012 94 user_data_path.v (2) //-------------- output port lut wires/regs -------------------- wire [CTRL_WIDTH-1:0] op_lut_in_ctrl; wire [DATA_WIDTH-1:0] op_lut_in_data; wire op_lut_in_wr; wire op_lut_in_rdy;... //------- output queues wires/regs ------... Wire declarations for the output port lookup module. Duplicate this section, and replace op_lut with crypto

95. Hands-on Training – June 18-19, 2012 95 user_data_path.v (3) //--------- Connect the data path ----------- input_arbiter #(... ) input_arbiter (... ) output_port_lookup #(... ) output_port_lookup (... )... Module instantiations. 1.Duplicate the output_port_lookup instantiation 2.Rename to crypto 3.Remove all parameters (inside the first set or parentheses) 4.In the output_port_lookup instantiation, replace oq_ with crypto_ 5.In the crypto instantiation, replace op_lut_ with crypto_ We’ve inserted the new module into the pipeline

96. Hands-on Training – June 18-19, 2012 96 Getting started with a new project (7) Run a simulation to verify changes: 1.nf_test.py sim --major nic --minor short Now we can implement the crypto functionality

97. Hands-on Training – June 18-19, 2012 97 Implementing the Crypto Module (1) What do we want to encrypt? –IP payload only Plaintext IP header allows routing Content is hidden –Encrypt bytes 35 onward Bytes 1-14 – Ethernet header Bytes 15-34 – IPv4 header (assume no options) –Assume all packets are IPv4 for simplicity

98. Hands-on Training – June 18-19, 2012 98 Implementing the Crypto Module (2) State machine (draw on next page): –Module headers on each packet –Datapath 64-bits wide 34 / 8 is not an integer!  Inside the crypto module

99. Hands-on Training – June 18-19, 2012 99 Crypto Module State Diagram Hint: We suggest 4 states (or 3 if you’re feeling adventurous) Skip Module Headers

100. Hands-on Training – June 18-19, 2012 100 Implementing the Crypto Module (3) Implement your state machine inside crypto.v –Use a static key initially Suggested sequence of steps: 1.Create a static key value Constants can be declared in the module with localparam: localparam MY_EXAMPLE = 32’h01234567; 2.Implement your state machine without modifying the packet 3.Update your state machine to modify the packet by XORing the key and the payload Use two copies of the key to create a 64-bit value to XOR with data words

101. Hands-on Training – June 18-19, 2012 101 module_template.v (1) module module_template #( parameter DATA_WIDTH = 64, parameter CTRL_WIDTH = DATA_WIDTH/8, parameter UDP_REG_SRC_WIDTH = 2 ) (... ) //----------------------- Signals----------------------------... //------------------ Local assignments -----------------------... Module port declaration

102. Hands-on Training – June 18-19, 2012 102 module_template.v (2) //------------------------- Modules------------------------------- fallthrough_small_fifo #(.WIDTH(CTRL_WIDTH+DATA_WIDTH),.MAX_DEPTH_BITS(2) ) input_fifo (.din ({in_ctrl, in_data}), // Data in.wr_en (in_wr), // Write enable.rd_en (in_fifo_rd_en), // Read the next word.dout ({in_fifo_ctrl, in_fifo_data}),.full (),.nearly_full (in_fifo_nearly_full),.prog_full (),.empty (in_fifo_empty),.reset (reset),.clk (clk) ); Packet data dumped in a FIFO. Allows some “decoupling” between input and output.

103. Hands-on Training – June 18-19, 2012 103 module_template.v (3) generic_regs #(.UDP_REG_SRC_WIDTH (UDP_REG_SRC_WIDTH),.TAG (0),.REG_ADDR_WIDTH (1),.NUM_COUNTERS (0),.NUM_SOFTWARE_REGS (0),.NUM_HARDWARE_REGS (0) ) module_regs (... ); Generic registers. Ignore for now – we’ll explore this later

104. Hands-on Training – June 18-19, 2012 104 module_template.v (4) //------------------------- Logic------------------------------- always @(*) begin // Default values out_wr_int = 0; in_fifo_rd_en = 0; if (!in_fifo_empty && out_rdy) begin out_wr_int = 1; in_fifo_rd_en = 1; end Combinational logic to read data from the FIFO. (Data is output to output ports.) You’ll want to add your state in this section.

105. Hands-on Training – June 18-19, 2012 105 Inter-module Communication Module i+1 ` Module i data ctrl wr rdy

106. Hands-on Training – June 18-19, 2012 106 Implementing the Crypto Module (3) Implement your state machine inside crypto.v –Use a static key initially Suggested sequence of steps: 1.Create a static key value Constants can be declared in the module with localparam: localparam MY_EXAMPLE = 32’h01234567; 2.Implement your state machine without modifying the packet 3.Update your state machine to modify the packet by XORing the key and the payload Use two copies of the key to create a 64-bit value to XOR with data words

107. Hands-on Training – June 18-19, 2012 107 Section VIII: Simulation and Debug

108. Hands-on Training – June 18-19, 2012 108 Testing: Simulation (1) Simulation allows testing without requiring lengthy synthesis process NetFPGA simulation environment allows: –Send/receive packets Physical ports and CPU –Read/write registers –Verify results Simulations run in ModelSim/VCS/ISim

109. Hands-on Training – June 18-19, 2012 109 Testing: Simulation (2) Simulations located in project/test Multiple simulations per project –Test different features Example: –crypto_nic/test/both_nic_short Send one packet from CPU, expect packet out physical port Send one packet in physical port, expect packet to CPU Note: This test will not work once your crypto module is implemented!

110. Hands-on Training – June 18-19, 2012 110 Testing: Simulation (3) Useful functions: Register access: nftest_regwrite(addr, value) nftest_regread_expect(addr, expect) Packet generation: make_IP_pkt(…) – see documentation encrypt_pkt(key, pkt) decrypt_pkt(key, pkt) Packet transmission/reception: nftest_send_phy(interface, pkt) nftest_expect_phy(interface, pkt) nftest_send_dma(interface, pkt) nftest_expect_dma(interface, pkt)

111. Hands-on Training – June 18-19, 2012 111 Testing: Simulation (4) Task: Implement tests for encryption and decryption Modify the following tests: netfpga/projects/crypto_nic/test/both_crypto_encrypt/run.py netfpga/projects/crypto_nic/test/both_crypto_decrypt/run.py Look at both_nic_short as an example of creating IP packets and sending/receiving them

112. Hands-on Training – June 18-19, 2012 112 Running Simulations Use command nf_test.py –Required Parameter sim or hw (right now only use sim) –Optional parameters --major --minor --gui (starts the default viewing environment) both_crypto_encrypt majorminor

113. Hands-on Training – June 18-19, 2012 113 Running Simulations Non-GUI execution example: # 10756.00ns testbench.host32.service_interrupt: Info: Interrupt signaled # 10935 Host read 0x00000044 with cmd 0x6: Disconnect with Data, # 10995 CPCI Interrupt: DMA ingress xfer complete # 11175 Host read 0x00000148 with cmd 0x6: Disconnect with Data, # 11415 Host read 0x00000150 with cmd 0x6: Disconnect with Data, # 11475.00ns testbench.host32.service_interrupt: Info: DMA ingress transfer complete. # 11655 Host read 0x00000040 with cmd 0x6: Disconnect with Data, # Timecheck: 13645.00ns # 20100 Simulation has reached finish time - ending. # ** Note: $finish : /home/summercamp/netfpga/lib/verilog/core/testbench/target32.v # Time: 20100 ns Iteration: 0 Instance: /testbench/target32 --- Simulation is complete. Validating the output. Comparing simulation output for port 1... Port 1 matches [1 packets] Comparing simulation output for port 2... Port 2 matches [0 packets] --- Test PASSED (test_nic_short) Test test_nic_short passed! ------------SUMMARY--------------- PASSING TESTS: test_nic_short FAILING TESTS: TOTAL: 1 PASS: 1 FAIL: 0

114. Hands-on Training – June 18-19, 2012 114 Running Simulations GUI execution example:

115. Hands-on Training – June 18-19, 2012 115 Running Simulations GUI execution example (cont) Try the following: nf_test.py sim --major crypto --minor encrypt –gui In the transcript window of the GUI: do wave.do run 10us You should see waveforms of packets going in and coming out of the crypto module

116. Hands-on Training – June 18-19, 2012 116 Running Simulations When running ModelSim interactively: –Click "no" when simulator prompts to finish –Changes to code can be recompiled without quitting ModelSim: bash# cd /tmp/$(whoami)/test/ ; make model_sim VSIM 5> restart -f; run -a –Ensure $NF_DESIGN_DIR is correct

117. Hands-on Training – June 18-19, 2012 117 Section IX: Conclusion

118. Hands-on Training – June 18-19, 2012 118 Nick McKeown, Glen Gibb, Jad Naous, David Erickson, G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller, Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng, Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo, Sam D’Amico Acknowledgments NetFPGA Team at Stanford University (Past and Present): NetFPGA Team at University of Cambridge (Past and Present): Andrew Moore, David Miller, Martin Zadnik, Muhammad Shahbaz All Community members (including but not limited to): Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek, Yahsar Ganjali, Martin Labrecque, Jeff Shafer, Eric Keller, Tatsuya Yabe, Bilal Anwer, Yashar Ganjali, Martin Labrecque Ram Subramanian, Kees Vissers, Michaela Blott, Shep Siegel

119. Hands-on Training – June 18-19, 2012 119 Special thanks to our Partners: Other NetFPGA Tutorials Presented At: SIGMETRICS Ram Subramanian, Patrick Lysaght, Veena Kumar, Paul Hartke, Anna Acevedo Xilinx University Program (XUP) See: http://NetFPGA.org/tutorials/

120. Hands-on Training – June 18-19, 2012 120 Thanks to our Sponsors: Support for the NetFPGA project has been provided by the following companies and institutions Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these materials do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.