Our ‘xmit1000.c’ driver Implementing a ‘packet-transmit’ capability with the Intel 82573L network interface controller

Remenber ‘echo’ and ‘cat’? Your device-driver module (named ‘uart.c’) was supposed to allow two programs that are running on a pair of adjacent PCs to communicate via a “null-modem” cable $ echo Hello > /dev/uart $ _ $ cat /dev/uart Hello _ Receiving…Transmitting…

‘keep it simple’ Let’s try to implement a ‘write()’ routine for our Intel Pro/1000 ethernet controllers that will provide the same basic functionality as we achieved with our serial UART driver It should allow us to transmit a message by using the familiar UNIX ‘cat’ command to redirect output to a character device-file Our device-file will be named ‘/dev/nic’

This function will program the actual data-transfer Driver’s components write my_fops my_write() module_init() module_exit() This function will allow us to inspect the transmit-descriptors This function will detect and configure the hardware, define page-mappings, allocate and initialize the descriptors, start the ‘transmit’ engine, create the pseudo-file and register ‘my_fops’ This function will do needed ‘cleanup’ when it’s time to unload our driver – turn off the ‘transmit’ engine, free the memory, delete page-table entries, the pseudo-file, and the ‘my_fops’ ‘struct’ holds one function-pointer my_get_info()

Kzalloc() Linux kernels since offer this convenient function for allocating pre-zeroed kernel memory It has the same syntax as the ‘kmalloc()’ function (described in our texts), but adds the after-effect of zeroing out the newly-allocated memory-area Thus it does two logically distinct actions (often coupled anyway) within a single function-call void *kmem = kmalloc( region_size, GFP_KERNEL ); memset( kmem, 0x00, region_size ); /* can be replaced with */ void *kmem = kzalloc( region_size, GFP_KERNEL );

Single page-frame option Packet-Buffer (3-KB) (reused for successive transmissions) 4KB Page- Frame Descriptor-Buffer (1-KB) (room for up to 256 descriptors)

Our Tx-Descriptor ring descriptor 0 Our ‘reusable’ transmit-buffer (1536 bytes) descriptor 1 descriptor 2 descriptor 3 descriptor 4 descriptor 5 descriptor 6 descriptor 7 Array of 8 transmit-descriptors one packet-buffer TAILHEAD After writing the data into our packet-buffer, and writing its length to the the current TAIL descriptor, our driver will advance the TAIL index; the NIC responds by reading the current HEAD descriptor, fetching its data, then advancing the HEAD index as it sends our data out over the wire.

‘/proc/xmit1000’ This pseudo-file can be examined anytime to find out what values (if any) the NIC has ‘written back’ into the transmit-descriptors (i.e., the descriptor-status information) and current values in registers TDH and TDT: $ cat /proc/xmit1000

Direct Memory Access The NIC is able to ‘fetch’ descriptors from host-system’s memory (and also can read the data from our packet-buffer) as well as ‘store’ a status-report back into the host’s memory by temporarily becoming the Bus Master (taking control of the system-bus away from the CPU so that it can perform the ‘fetch’ and ‘store’ operations directly, without CPU involvement or interference)

Configuration registers TIPG TCTL TDBAL TDBAH TDLEN TDH TDT TXDCTL CTRL CTRL_EXT Device Control Extended Device Control Transmit Inter-Packet Gap Transmit Control Transmit Descriptor-queue Base-Address (LOW) Transmit Descriptor-queue Base-Address (HIGH) Transmit Descriptor-queue Length Transmit Descriptor-queue HEAD Transmit Descriptor-queue TAIL Transmit Descriptor-queue Control

The ‘initialization’ sequence Detect the network interface controller Obtain its i/o-memory address and size Remap the i/o-memory into kernel-space Allocate memory for buffer and descriptors Initialize the array of transmit-descriptors Reset the NIC and configure its operations Create the ‘/proc/xmit1000’ pseudo-file Register our ‘write()’ driver-method

The ‘cleanup’ sequence Usually the steps here follow those in the initialization sequence -- but in backwards order: Unregister the device-driver’s file-operations Delete the ‘/proc/xmit1000’ pseudo-file Disable the NIC’s ‘transmit’ engine Release the allocated kernel-memory Unmap the NIC’s i/o-memory region

Our ‘write()’ algorithm Get index of the current TAIL descriptor Confine the amount of user-data Copy user-data into the packet-buffer Setup the packet’s Ethernet Header Setup packet-length in the TAIL descriptor Now hand over this descriptor to the NIC (by advancing the value in register TDT) Tell the kernel how many bytes were sent

Recall Tx-Descriptor Layout special 0x0 0x4 0x8 0xC CMD Buffer-Address high (bits ) Buffer-Address low (bits 31..0) 31 0 Packet Length (in bytes)CSO statusCSS reserved =0 Buffer-Address = the packet-buffer’s 64-bit address in physical memory Packet-Length = number of bytes in the data-packet to be transmitted CMD = Command-field CSO/CSS = Checksum Offset/Start (in bytes) STA = Status-field

Suggested C syntax typedef struct { unsigned long long base_addr; unsigned shortpkt_length; unsigned charcksum_off; unsigned chardesc_cmd; unsigned chardesc_stat; unsigned charcksum_org; unsigned shortspecial; } TX_DESCRIPTOR;

Transmit IPG (0x0410) 82573L IPG R =0 IPG After Deferral (Recommended value = 7) IPG Part 1 (Recommended value = 8) IPG Back-To-Back (Recommended value = 8) IPG = Inter-Packet Gap This register controls the Inter-Packet Gap timer for the Ethernet controller. Note that the recommended TIPG register-value to achieve IEEE compliant minimum transfer IPG values in full- and half-duplex operations would be (hexadecimal), equal to (7<<20) | (8<<10) | (8<<0).

Transmit Control (0x0400) R =0 R =0 R =0 MULRTXCSCMT UNO RTX RTLC R =0 SW XOFF COLD (upper 6-bits) (COLLISION DISTANCE) COLD (lower 4-bits) (COLLISION DISTANCE) 0ASDV ILOSILOS SLUSLU TBI mode PSPPSP 0 R = R =0 ENEN SPEED CT (COLLISION THRESHOLD) EN = Transmit EnableSWXOFF = Software XOFF Transmission PSP = Pad Short PacketsRLTC = Retransmit on Late Collision CT = Collision Threshold (=0xF)UNORTX = Underrun No Re-Transmit COLD = Collision Distance (=0x3F)TXCSCMT = TxDescriptor Minimum Threshold MULR = Multiple Request Support 82573L

Our driver’s elections int tx_control = 0; tx_control |= (0<<1);// EN-bit (Enable Transmit Engine) tx_control |= (1<<3);// PSP-bit (Pad Short Packets) tx_control |= (15<<4);// CT=15 (Collision Threshold) tx_control |= (63<<12);// COLD=63 (Collision Distance) tx_control |= (0<<22);// SWXOFF-bit (Software XOFF Tx) tx_control |= (1<<24);// RTLC-bit (Re-Transmit on Late Collision) tx_control |= (0<<25);// UNORTX-bit (Underrun No Re-Transmit) tx_control |= (0<<26);// TXCSMT=0 (Tx-descriptor Min Threshold) tx_control |= (0<<28);// MULR-bit (Multiple Request Support) iowrite32( tx_control, io + E1000_TCTL );// Transmit Control register 82573L Here’s a C programming style that ‘documents’ the programmer’s choices.

An ‘e1000.c’ anomaly? The official Linux kernel is delivered with a device-driver supporting Intel’s ‘Pro/1000’ gigabit ethernet controllers (several) Often this driver will get loaded by default during the system’s startup procedures But it will interfere with your own driver if you try to write a substitute for ‘e1000.ko’ So you will want to remove it with ‘rmmod’

Side-effect of ‘rmmod’ We’ve observed an unexpected side-effect of ‘unloading’ the ‘e1000.ko’ device-driver The PCI Configuration Space’s command register gets modified in a way that keeps the NIC from working with your own driver Specifically, the Bus Mastering capability gets disabled (by clearing bit #2 in the PCI Configuration Space’s word at address 4)

What to do about it? This effect doesn’t arise on our ‘anchor’ cluster machines, but you may encounter it when you try using our demo elsewhere Here’s the simple “fix” to turn Bus Master capability back on (in your ‘module_init()’) u16pci_cmd;// declares a 16-bit variable pci_read_config_word( devp, 4, &pci_cmd );// read current word pci_cmd |= (1<<2);// turn on the Bus Master enabled-bit pci_write_config_word( devp, 4, pci_cmd );// write modification

In-class demo We demonstrate our ‘xmit1000.c’ driver on an ‘anchor’ machine, with some help from a companion-module (named ‘recv1000.c’) which is soon-to-be discussed in class $ echo Hello > /dev/nic $ _ $ cat /dev/nic Hello _ Receiving… Transmitting… anchor01anchor05 LAN

In-class exercise Open three or more terminal-windows on your PC’s graphical desktop, and login to a different ‘anchor’ machine in each one Install the ‘xmit1000.ko’ module on one of the anchor machines, and then install our ‘recv1000.ko’ module on the other stations Execute the ‘cat /dev/nic’ command on the receiver-stations, and then run an ‘echo’ command on the transmitter-station