1. Dr A Sahu Dept of Comp Sc & Engg. IIT Guwahati

2. PCI Devices NIC Cards NIC card architecture Access to NIC register – PCI access

3. RealTek (Example RTK 8169S) Broadcom (Example BCM5751) Intel (Example 82573L NIC) Download manual from vender site – Or search “Realtek 8169 pdf” in google

4. Peripheral devices in the early PCs used fixed i/o- ports and fixed memory-addresses, e.g.: – Video memory address-range: 0xA0000-0xBFFFF – Programmable timer i/o-ports: 0x40-0x43 – Keyboard and mouse i/o-ports: 0x60-0x64 – Real-Time Clock’s i/o-ports:0x70-0x71 – Hard Disk controller’s i/o-ports: 0x01F0-01F7 – Graphics controller’s i/o-ports: 0x03C0-0x3CF – Serial-port controller’s i/o-ports: 0x03F8-0x03FF – Parallel-port controller’s i/o-ports: 0x0378-0x037A

5. It became clear in the 1990s that there would be contention among equipment vendors for ‘fixed’ resource-addresses, which of course were in limited supply Among the goals that motivated the PCI Specification was the creation of a more flexible scheme for allocating addresses that future peripheral devices could use

6. PCI Configuration Space Body (48 doublewords – variable format) PCI Configuration Space Body (48 doublewords – variable format) 64 doublewords PCI Configuration Space Header (16 doublewords – fixed format) PCI Configuration Space Header (16 doublewords – fixed format) A non-volatile parameter-storage area for each PCI device-function

7. Status Register Status Register Command Register Command Register Device ID Device ID Vendor ID Vendor ID BIST Cache Line Size Cache Line Size Class Code Class/SubClass/ProgIF Class Code Class/SubClass/ProgIF Revision ID Revision ID Base Address 0 Subsystem Device ID Subsystem Device ID Subsystem Vendor ID Subsystem Vendor ID CardBus CIS Pointer reserved capabilities pointer capabilities pointer Expansion ROM Base Address Minimum Grant Minimum Grant Interrupt Pin Interrupt Pin reserved Latency Timer Latency Timer Header Type Header Type Base Address 1 Base Address 2 Base Address 3 Base Address 4 Base Address 5 Interrupt Line Interrupt Line Maximum Latency Maximum Latency 31 0 16 doublewords Dwords 1 - 0 3 - 2 5 - 4 7 - 6 9 - 8 11 - 10 13 - 12 15 - 14

8. memory space (4GB) memory space (4GB) i/o space (64KB) i/o space (64KB) PCI configuration space (16MB) PCI configuration space (16MB) accessed using a large variety of processor instructions (mov, add, or, shr, push, etc.) and virtual-to-physical address-translation accessed only by using the processor’s special ‘in’ and ‘out’ instructions (without any translation of port-addresses) i/o-ports 0x0CF8-0x0CFF dedicated to accessing PCI Configuration Space

9. reserved CONFADD ( 0x0CF8) CONFDAT ( 0x0CFC) 31 23 16 15 11 10 8 7 2 0 ENEN ENEN bus (8-bits) bus (8-bits) device (5-bits) device (5-bits) doubleword (6-bits) doubleword (6-bits) function (3-bits) function (3-bits) 00 PCI Configuration Space Address Port (32-bits) PCI Configuration Space Data Port (32-bits) 31 0 Enable Configuration Space Mapping (1=yes, 0=no)

10. Step one: Output the desired longword’s address (bus, device, function, and dword) with bit 31 set to 1 (to enable access) to the Configuration-Space Address-Port Step two: Read the designated data from the Configuration-Space Data-Port Already discussed PCI-probes pciprobes.c – Lect 29..Showing vram, pciprobe.cpp

11. We can identify the network interface controller in PC’s by class-code 0x02 The subclass-code 0x00 is for ‘ethernet’ We can identify the NIC from its VENDOR and DEVICE identification-numbers: VENDOR_ID = 0x14E4 DEVICE_ID = 0x1677 You can use the ‘grep’ command to search for these numbers in this header-file:

12. The VENDOR-ID 0x14E4 belongs to the Broadcom Corporation Information about this firm may be learned from the corporation’s website The DEVICE-ID 0x1677 is used to signify Broadcom’s BCM5751 ethernet product

13. nic TX FIFO RX FIFO transceiver LAN cable BUSBUS main memory main memory packet buffer CPU

14. Network Interface’s hardware needs to implement ‘filtering’ of network packets Otherwise the PC’s memory-usage and processor-time will be wasted handling packets not meant for this PC to receive network packet’s layout Destination-address (6-bytes) Source-address (6-bytes) Each data-packet begins with the 6-byte device-address of the network interface which is intended to receive it

15. You can see the Hardware Address of the ethernet controller on your PC by typing: $ /sbin/ifconfig Look for it in the first line of screen-output that is labeled ‘eth0’, for example: eth0 Link encap: Ethernet HWaddr 00:11:43:C9:50:3A

16. Lets write a kernel module that lets users see certain register-values which pertain to the network interface in your system : – (1) the PCI Configuration Space registers – (2) the Media Access Controller’s address It also shows your machine’s node-name (in case you want to save the information)

17. We do not have NIC’s programming datasheet -- but we do have Linux source code for the ‘nic_pci_info.c’ device-driver, which includes a header-file ‘tg3.h’ found here: If you scroll through the #define directives you will see the offset where the hardware address is stored in the memory-mapped register-space of the ‘nic_pci_info.c’ interface

18. Status Register Status Register Command Register Command Register DeviceID 0x1677 DeviceID 0x1677 VendorID 0x14E4 VendorID 0x14E4 BIST Cache Line Size Cache Line Size Class Code Class/SubClass/ProgIF Class Code Class/SubClass/ProgIF Revision ID Revision ID Base Address 0 Subsystem Device ID Subsystem Device ID Subsystem Vendor ID Subsystem Vendor ID CardBus CIS Pointer reserved capabilities pointer capabilities pointer Expansion ROM Base Address Minimum Grant Minimum Grant Interrupt Pin Interrupt Pin reserved Latency Timer Latency Timer Header Type Header Type Base Address 1 Base Address 2 Base Address 3 Base Address 4 Base Address 5 Interrupt Line Interrupt Line Maximum Latency Maximum Latency 31 0 16 doublewords Dwords 1 - 0 3 - 2 5 - 4 7 - 6 9 - 8 11 - 10 13 - 12 15 - 14

19. #include struct pci_dev*devp; unsigned intiomem_base, iomem_size; void*io; devp = pci_get_device( 0x14E4, 0x1677, NULL ); if ( !devp ) return –ENODEV; iomem_base = pci_resource_start( devp, 0 ); iomem_size = pci_resource_len( devp, 0 ); io = ioremap( iomem_base, iomem_size ); if ( !io ) return -EBUSY; #include struct pci_dev*devp; unsigned intiomem_base, iomem_size; void*io; devp = pci_get_device( 0x14E4, 0x1677, NULL ); if ( !devp ) return –ENODEV; iomem_base = pci_resource_start( devp, 0 ); iomem_size = pci_resource_len( devp, 0 ); io = ioremap( iomem_base, iomem_size ); if ( !io ) return -EBUSY;

20. mac 1 mac 1 mac 0 mac 0 mac 5 mac 5 mac 4 mac 4 mac 3 mac 3 mac 2 mac 2 0x0410 0x0411 0x0412 0x0413 0x0414 0x0415 0x0416 0x0417 Broadcom network interface storage-addresses Intel IA-32 character-array storage mac 0 mac 0 mac 1 mac 1 mac 2 mac 2 mac 3 mac 3 mac 4 mac 4 mac 5 mac 5

21. mac 0 mac 0 mac 1 mac 1 mac 2 mac 2 mac 3 mac 3 mac 4 mac 4 mac 5 mac 5 0x5400 0x5401 0x5402 0x5403 0x5404 0x5405 0x5406 0x5407 Intel network interface storage-addresses Intel IA-32 character-array storage mac 0 mac 0 mac 1 mac 1 mac 2 mac 2 mac 3 mac 3 mac 4 mac 4 mac 5 mac 5

22. For Intel NICs : #define VENDOR_ID 0x8086// Intel Corp #define DEVICE_ID 0x109A// 82573L NIC Intel’s filter-register at offset 0x5400 uses the ‘little endian’ storage-convention

23. host-1 host-2 host-3 host-4 HUB “Collision Domain” CSMA/CD = “Carrier Sense Multiple Access/Collision Detection”

24. PCI = Peripheral Component Interconnect MAC = Media Access Controller Phy = Physical-layer functions AMT = Active Management Technology LOM = LAN On Motherboard

25. 32K configurable RX and TX packet FIFO IEEE 802.3x Flow Control support Host-Memory Receive Buffers 16K/256K IEEE 802.3ab Auto-Negotiation TCP/UDP checksum off-loading Jumbo-frame support (up to 16KB) Interrupt-moderation controls

26. PCI/PCI-e Bus 10/100/1000 PHY MAC/Controller MDI interface SM Bus interface EEPROM Flash interface LED indicators S/W Defined pins GMII/MII interface MDIO interface

27. Device registers are hardware mapped to a range of addresses in physical memory You obtain the location (and the length) of this memory-range from a Base Add register in the nic device’s PCI Configuration Space Then you request the Linux kernel to setup an I/O ‘remapping’ of this memory-range to ‘virtual’ addresses within kernel-space

28. dynamic ram dynamic ram nic registers nic registers vram IO-APIC Local-APIC user space user space APIC registers kernel code/data nic registers vram ‘virtual’ address-spacephysical address-space 1-GB 3-GB

29. Linux provides device-driver writers with some macros for accessing i/o-memory: #include unsigned int datum; iowrite32( datum, address ); datum = ioread32( address );

30. #include #define E1000_STATUS0x0008 unsigned int iomem_base, iomem_size; void *io; // remap the device’s i/o-memory into kernel space devp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL ); if ( !devp ) return –ENODEV; iomem_base = pci_resource_start( devp, 0 ); iomem_size = pci_resource_len( devp, 0 ); io = ioremap_nocache( iomem_base, iomem_size ); if ( !io ) return –ENOSPC; // read and display the nic’s STATUS register device_status = ioread32( io + E1000_STATUS ); printk( “ Device Status Register = 0x%08X \n”, status ); #include #define E1000_STATUS0x0008 unsigned int iomem_base, iomem_size; void *io; // remap the device’s i/o-memory into kernel space devp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL ); if ( !devp ) return –ENODEV; iomem_base = pci_resource_start( devp, 0 ); iomem_size = pci_resource_len( devp, 0 ); io = ioremap_nocache( iomem_base, iomem_size ); if ( !io ) return –ENOSPC; // read and display the nic’s STATUS register device_status = ioread32( io + E1000_STATUS ); printk( “ Device Status Register = 0x%08X \n”, status );

31. 0 ?0 0 000 000000 GIO Master EN GIO Master EN 000 0000 PHY reset PHY reset ASDV ILOSILOS SLUSLU 0 TX OFF TX OFF 0 FDFD FDFD 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 Function ID Function ID LULU LULU 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 SPEED FD = Full-Duplex LU = Link Up TXOFF = Transmission Paused SPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved) ASDV = Auto-negotiation Speed Detection Value 82573L some undocumented functionality?