1. What’s needed to receive? A look at the minimum steps required for programming our anchor nic’s to receive packets

2. A disappointment Our former ‘nicwatch.cpp’ application does not seem to work reliably to show packets being received by the 82573L controller It was based on the ‘raw sockets’ protocol implemented within the Linux kernel’s vast networking subsystem, thus offering us the prospect of a ‘hardware-independent’ tool -- if only it would show us all the packets!

3. Two purposes… So let’s discard ‘nicwatch.cpp’ in favor of writing our own hardware-specific module that WILL be able to show us all the nic’s received packets, independently of Linux’s various layers of networking protocol code And let’s keep it as simple as possible, so we can see which programming steps are the truly essential ones for the 82573L nic

4. Accessing 82573L registers Device registers are hardware mapped to a range of addresses in physical memory We can get the location and extent of this memory-range from a BAR register in the 82573L device’s PCI Configuration Space We then request the Linux kernel to setup an I/O ‘remapping’ of this memory-range to ‘virtual’ addresses within kernel-space

5. i/o-memory remapping dynamic ram nic registers vram IO-APIC Local-APIC user space APIC registers kernel code/data nic registers vram ‘virtual’ address-spacephysical address-space 1-GB 3-GB

6. Kernel memory allocation The NIC requires that some host memory for packet-buffers and receive descriptors The kernel provides a ‘helper function’ for reserving a suitable region of memory in kernel-space which is both ‘non-pageable’ and ‘physically contiguous’ (i.e., kzalloc()) It’s our job is to decide how much memory our network controller hardware will need

7. the packet’s data ‘payload’ goes here (usually varies from 56 to 1500 bytes) Ethernet packet layout Total size normally can vary from 64 bytes up to 1522 bytes (unless ‘jumbo’ packets and/or ‘undersized’ packets are enabled) The NIC expects a 14-byte packet ‘header’ and it appends a 4-byte CRC check-sum destination MAC address (6-bytes) source MAC address (6-bytes) Type/length (2-bytes) Cyclic Redundancy Checksum (4-bytes) 0 6 12 14

8. Rx-Descriptor Ring-Buffer Circular buffer (128-bytes minimum – and must be a multiple of 128 bytes) RDBA base-address RDLEN (in bytes) RDH (head) RDT (tail) = owned by hardware (nic) = owned by software (cpu) 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80

9. Our ‘nicspy.c’ module It will be a ‘character-mode’ device-driver It will only implement ‘read()’ and ‘ioctl()’ The ‘read()’ function will cause a task to sleep until a network packet has arrived An interrupt-handler will wake up the task A ‘get_info’ function will be provided as a debugging aid, so the NIC’s Rx descriptor- queue can be conveniently inspected

10. Sixteen packet-buffers Our ‘nicspy.c’ driver allocates 16 buffers of size 1536 bytes (i.e., for normal ethernet) unused 32-KB allocated (16 packet-buffers, plus Rx-Descriptor Queue) #define KMEM_SIZE0x8000// 32KB = size of kernel memory allocation void*kmem = kzalloc( KMEM_SIZE, GFP_KERNEL ); if ( !kmem ) return –ENOMEM; for the Rx Descriptor Queue (256 bytes) for the sixteen packet-buffers

11. Format for an Rx Descriptor Base-address (64-bits) status Packet- length Packet- checksum VLAN tag errors 16 bytes The device-driver initializes this ‘base-address’ field with the physical address of a packet-buffer The network controller will ‘write-back’ the values for these fields when it has transferred a received packet’s data into this packet-buffer

12. Suggested C syntax typedef struct{ unsigned long longbase_address; unsigned shortpacket_length; unsigned shortpacket_cksum; unsigned chardesc_status; unsigned chardesc_errors; unsigned shortVLAN_tag; } RX_DESCRIPTOR; ‘Legacy Format’ for the Intel Pro1000 network controller’s Receive Descriptors

13. RxDesc Status-field PIFIPCSTCPCSVPIXSMEOPDD 7 6 5 4 3 2 1 0 DD = Descriptor Done (1=yes, 0=no) shows if nic is finished with descriptor EOP = End Of Packet (1=yes, 0=no) shows if this packet is logically last IXSM = Ignore Checksum Indications (1=yes, 0=no) VP = VLAN Packet match (1=yes, 0=no) USPCS = UDP Checksum calculated in packet (1=yes, 0=no) TCPCS = TCP Checksum calculated in packet (1=yes, 0=no) IPCS = IPv4 Checksum calculated on packet (1=yes, 0=no) PIF = Passed In-exact Filter (1=yes, 0=no) shows if software must check UDPCS

14. RxDesc Error-field RXEIPETCPE reserved =0 SECE 7 6 5 4 3 2 1 0 RXE = Received-data Error (1=yes, 0=no) IPE = IPv4-checksum error TCPE = TCP/UDP checksum error (1=yes, 0=no) SEQ = Sequence error (1=yes, 0=no) SE = Symbol Error (1=yes, 0=no) CE = CRC Error or alignment error (1=yes, 0=no) SEQ reserved =0

15. Essential ‘receive’ registers enum{ E1000_CTRL0x0000,// Device Control E1000_STATUS0x0008,// Device Status E1000_ICR0x00C0,// Interrupt Cause Read E1000_IMS0x00D0,// Interrupt Mask Set E1000_IMC0x00D8,// Interrupt Mask Clear E1000_RCRL0x0100,// Receive Control E1000_RDBAL0x2800,// Rx Descriptor Base Address Low E1000_RDBAH0x2804,// Rx Descriptor Base Address High E1000_RDLEN0x2808,// Rx Descriptor Length E1000_RDH0x2810,// Rx Descriptor Head E1000_RDT0X2818,// Rx Descriptor Tail E1000_RXDCTL0x2828,// Rx Descriptor Control E1000_RA0x5400,// Receive address-filter Array };

16. Receive Control (0x0100) R =0 00 FLXBUF SE CRC BSEX R =0 PMCF DPF R =0 CFI EN VFE BSIZE BAMBAM R =0 MODTYPRDMTS ILOSILOS SLUSLU LPEUPE 0 R =0 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 SBP ENEN 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 LBMMPE EN = Receive Enable DTYP = Descriptor TypeDPF = Discard Pause Frames SBP = Store Bad Packets MO = Multicast OffsetPMCF = Pass MAC Control Frames UPE = Unicast Promiscuous Enable BAM = Broadcast Accept ModeBSEX = Buffer Size Extension MPE = Multicast Promiscuous Enable BSIZE = Receive Buffer SizeSECRC = Strip Ethernet CRC LPE = Long Packet reception Enable VFE = VLAN Filter EnableFLXBUF = Flexible Buffer size LBM = Loopback ModeCFIEN = Canonical Form Indicator Enable RDMTS = Rx-Descriptor Minimum Threshold SizeCFI = Canonical Form Indicator bit-value We used 0x0000801C in RCTL to prepare the ‘receive engine’ prior to enabling it

17. Device Control (0x0000) PHY RST VME R =0 TFCERFCE RST R =0 R =0 R =0 R =0 R =0 ADV D3 WUC R =0 D/UD status R =0 R =0 R =0 R =0 R =0 FRC DPLX FRC SPD R =0 SPEED R =0 SLUSLU R =0 R =0 R =1 0 FDFD 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 GIO M D R =0 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 FD = Full-DuplexSPEED (00=10Mbps, 01=100Mbps, 10=1000Mbps, 11=reserved) GIOMD = GIO Master DisableADVD3WUP = Advertise Cold Wake Up Capability SLU = Set Link UpD/UD = Dock/Undock statusRFCE = Rx Flow-Control Enable FRCSPD = Force SpeedRST = Device ResetTFCE = Tx Flow-Control Enable FRCDPLX = Force DuplexPHYRST = Phy ResetVME = VLAN Mode Enable 82573LWe used 0x040C0241 to initiate a ‘device reset’ operation

18. 0 Device Status (0x0008) ?0 0 000 000000 GIO Master EN 000 0000 PHY RA ASDV ILOSILOS SLUSLU 0 TX OFF 0 FDFD 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 Function ID LULU 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 SPEED FD = Full-Duplex LU = Link Up TXOFF = Transmission Paused SPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved) ASDV = Auto-negotiation Speed Detection Value PHYRA = PHY Reset Asserted 82573L some undocumented functionality?

19. PCI Bus Master DMA 82573L i/o-memory RX and TX FIFOs (32-KB total) Host’s Dynamic Random Access Memory Rx Descriptor Queue packet-buffer DMA on-chip RX descriptors on-chip TX descriptors

20. Our ‘read()’ algorithm unsigned intrx_curr; ssize_t my_read( struct file *file, char *buf, size_t len, loff_t *pos ) { // our global variable ‘rx_curr’ is the descriptor-array index // for the next receive-buffer descriptor to be processed if ( this descriptor’s status is zero ) put calling task to sleep; // wakeup the task when a fresh packet has been received copy received data from the packet-buffer to user’s buffer clear this descriptor’s status advance our global variable ‘rx_curr’ to the next descriptor return the number of data-bytes transferred }

21. ‘nicspy.cpp’ This application calls our device-driver’s ‘read()’ function repeatedly, and displays the ‘raw’ ethernet packet-data each time It requires our ‘nicspy.c’ device-driver to be installed in the kernel, obviously There’s no ‘clash’ of filenames here – and their similarity helps keep them together: nicspy.c and nicspy.ko(the kernel-side) nicspy.cpp and nicspy( the user-side )

22. in-class demo We can install ‘nicspy.ko’ on one of our anchor machines – making sure ‘eth1’ is ‘down’ before we do our module-install – and then we run ‘nicspy’ on that machine Next we install our ‘nicping.ko’ module on some other anchor machine – be sure its ‘eth1’ interface is ‘down’ beforehand – and then use ‘cat /proc/nicping’ for a transmit