More 82573L details Getting ready to write and test a character-mode device-driver for our anchor-LAN’s ethernet controllers

A ‘nic.c’ character driver? open read write my_fops ioctl my_ioctl() my_open() my_read() my_write() my_release() my_isr() module_init() module_exit() release

Statistics registers The 82573L has several dozen statistical counters which automatically operate to keep track of significant events affecting the ethernet controller’s performance Most are 32-bit ‘read-only’ registers, and they are automatically cleared when read Your module’s initialization routine could read them all (to start counting from zero)

Initializing the nic’s counters The statistical counters all have address- offsets in the range 0x04000 – 0x04FFF You can use a very simple program-loop to ‘clear’ each of these read-only registers // Here ‘io’ is the virtual base-address of the nic’s i/o-memory region { intr; // clear all of the Pro/1000 controller’s statistical counters for (r = 0x4000; r < 0x4FFF; r += 4) ioread32( io + r ); }

A few ‘counter’ examples 0x4000 CRCERRSCRC Errors Count 0x400C RXERRCReceive Error Count 0x4014SCCSingle Collision Count 0x4018ECOLExcessive Collision Count 0x4074GPRCGood Packets Received 0x4078BPRCBroadcast Packets Received 0x407CMPRCMulticast Packets Received 0x40D0TPRTotal Packets Received 0x40D4TPTTotal Packets Transmitted 0x40F0MPTCMulticast Packets Transmitted 0x40F4BPTCBroadcast Packets Transmitted

the packet’s data ‘payload’ goes here (usually varies from 56 to 1500 bytes) Ethernet packet layout Total size normally can vary from 64 bytes up to 1536 bytes (unless ‘jumbo’ packets and/or ‘undersized’ packets are enabled) The NIC expects a 14-byte packet ‘header’ and it appends a 4-byte CRC check-sum destination MAC address (6-bytes) source MAC address (6-bytes) Type/length (2-bytes) Cyclic Redundancy Checksum (4-bytes)

Filter registers All the modern ethernet controllers have a built-in ‘filtering’ capability which allows the NIC to automatically discard any packets having a destination-address different from the controller’s own unique MAC address But the 82573L offers a more elaborate filtering mechanism (and can also ‘reject’ packets based on the ‘source’ addresses)

How ‘receive’ works descriptor0 descriptor1 descriptor2 descriptor Buffer0 Buffer1 Buffer2 Buffer3 List of Buffer-Descriptors We setup memory-buffers where we want received packets to be placed by the NIC We also create a list of buffer-descriptors and inform the NIC of its location and size Then, when ready, we tell the NIC to ‘Go!’ (i.e., start receiving), but to let us know when these receptions have occurred Random Access Memory

Receive Control (0x0100) R =0 00 FLXBUF SE CRC BSEX R =0 PMCF DPF R =0 CFI EN VFE BSIZE BAMBAM R =0 MODTYPRDMTS ILOSILOS SLUSLU LPEUPE 0 R = SBP ENEN LBMMPE EN = Receive Enable DTYP = Descriptor TypeDPF = Discard Pause Frames SBP = Store Bad Packets MO = Multicast OffsetPMCF = Pass MAC Control Frames UPE = Unicast Promiscuous Enable BAM = Broadcast Accept ModeBSEX = Buffer Size Extension MPE = Multicast Promiscuous Enable BSIZE = Receive Buffer SizeSECRC = Strip Ethernet CRC LPE = Long Packet reception Enable VFE = VLAN Filter EnableFLXBUF = Flexible Buffer size LBM = Loopback ModeCFIEN = Canonical Form Indicator Enable RDMTS = Rx-Descriptor Minimum Threshold SizeCFI = Cannonical Form Indicator bit-value

Registers’ Names Memory-information registers RDBA(L/H) = Receive-Descriptor Base-Address Low/High (64-bits) RDLEN = Receive-Descriptor array Length RDH = Receive-Descriptor Head RDT = Receive-Descriptor Tail Receive-engine control registers RXDCTL = Receive-Descriptor Control Register RCTL = Receive Control Register Notification timing registers RDTR = Receive-interrupt packet Delay Timer RADV = Receive-interrupt Absolute Delay Value

Rx-Desc Ring-Buffer Circular buffer (128-bytes minimum) RDBA base-address RDLEN (in bytes) RDH (head) RDT (tail) = owned by hardware (nic) = owned by software (cpu) 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80

Rx-Descriptor Control (0x2828) R =0 R =0 R =0 R =0 R =0 R =0 R =0 GRANGRAN R =0 R = ADV D3 WUC SDP1 DATA SDP0 DATA D/UD status 0 WTHRESH (Writeback Threshold) R =0 R =0 0 FRC DPLX FRC SPD 0 HTHRESH (Host Threshold) R =0 R =0 ASDEASDE 0 LRSTLRST PTHRESH (Prefetch Threshold) GRAN (Granularity): 1=descriptor-size, 0=cacheline-size Prefetch Threshold – A prefetch operation is considered when the number of valid, but unprocessed, receive descriptors that the ethernet controller has in its on-chip buffer drops below this threshold. Host Threshold - A prefetch occurs if at least this many valid descriptors are available in host memory Writeback Threshold - This field controls the writing back to host memory of already processed receive descriptors in the ethernet controller’s on-chip buffer which are ready to be written back to host memory

Legacy Rx-Descriptor Layout VLAN tag 0x0 0x4 0x8 0xC Packet Checksum Buffer-Address high (bits ) Buffer-Address low (bits 31..0) 31 0 Packet Length (in bytes) StatusErrors Buffer-Address = the packet-buffer’s 64-bit address in physical memory Packet Length = number of bytes in the data-packet that has was received Packet Checksum = the16-bit one’s-complement of the entire logical packet Status = shows if descriptor has been used and if it’s last in a logical packet Errors = valid only when DD and EOP are set in the descriptor’s Status field

Suggested C syntax typedef struct { unsigned long long base_addr; unsigned shortpkt_length; unsigned shortchecksum; unsigned chardesc_stat; unsigned chardesc_errs; unsigned shortvlan_tag; } rx_descriptor;

RxDesc Status-field PIFIPCSTCPCSVPIXSMEOPDD DD = Descriptor Done (1=yes, 0=no) shows if nic is finished with descriptor EOP = End Of Packet (1=yes, 0=no) shows if this packet is logically last IXSM = Ignore Checksum Indications (1=yes, 0=no) VP = VLAN Packet match (1=yes, 0=no) USPCS = UDP Checksum calculated in packet (1=yes, 0=no) TCPCS = TCP Checksum calculated in packet (1=yes, 0=no) IPCS = IPv4 Checksum calculated on packet (1=yes, 0=no) PIF = Passed In-exact Filter (1=yes, 0=no) shows if software must check UDPCS

RxDesc Error-field RXEIPETCPE reserved =0 SECE RXE = Received-data Error (1=yes, 0=no) IPE = IPv4-checksum error TCPE = TCP/UDP checksum error (1=yes, 0=no) SEQ = Sequence error (1=yes, 0=no) SE = Symbol Error (1=yes, 0=no) CE = CRC Error or alignment error (1=yes, 0=no) SEQ reserved =0

Network Administration Some higher-level networking protocols require the Operating System to setup a translation between the ‘hostname’ for a workstation and the hardware-address of its Network Interface Controller One mechanism for doing this is creation of a specially-named textfile (‘/etc/ethers’) that provides database for translations

In-class exercise #1 We put a file named ‘ethers’ on our course website that offers a template for defining the translation database that software can consult on our ‘anchor’ cluster’s LAN One of the eight workstations’ entries has been filled in already: Can you complete this database by adding the MAC addresses for the other 7 machines? 00:30:48:8A:30:03 anchor00.cs.usfca.edu

Our ‘seereset.c’ demo We created this LKM to demonstrate the sequence of ‘state-changes’ that three of our network controller’s registers undergo in response to initiating a ‘reset’ operation The programming technique used here is one which we think could be useful in lots of other hardware programming situations where a vendor’s manual may not answer all our questions about how devices work

In-class exercise #2 Try redirecting the output from this ‘cat’ command to a file, like this: $ cat /proc/seereset > seereset.out Then edit this textfile, adding a comment to each line which indicates the bit(s) that experienced a ‘change-of-state’ from the line that came before it (thereby providing yourself with a running commentary as to how the NIC proceeds through a ‘reset’)