January 10, Kits Workshop 1 Washington WASHINGTON UNIVERSITY IN ST LOUIS A Smart Port Card Tutorial --- Software John DeHart Washington University

January 10, Kits Workshop 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS Software Overview SPC Specific Kernel Modifications Building a Kernel –config –compile Loading Kernel onto SPC to execute it –download via APIC Initiating Kernel execution –boot loader Configuring Devices –APIC

January 10, Kits Workshop 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC Kernel Modifications Kernel Configuration File: –Memory Disk –Serial Console –Most device drivers unneeded (e.g no Ethernet…) Source Code Changes: –PCI Device Configuration:./dev/pci/if_apic_pci.c APIC Specific configuration handled by driver –Change some APIC defaults:./dev/ic/apic.c made switch port the default –No HLT instruction: locore.s –Kernel Memory Page Allocation:./arch/i386/include/pmap.new.h –RTC:./arch/i386/isa/clock.c,./ufs/ffs/ffs_vfsops.c We still had to remove some RTC code...

January 10, Kits Workshop 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS Steps to an Executing Kernel on an SPC Configure Kernel –Start with a typical NetBSD kernel config file –Add SPC specific option: options SPC –add memory disk (filesystem will reside in kernel!) options MEMORY_DISK* options MINIROOTSIZE –serial console port options CONSDEVNAME options CONS_OVERRIDE –remove “extra” devices (Ethernet, mouse, …)

January 10, Kits Workshop 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS Steps to an Executing Kernel on an SPC Filesystem –we use a vnode disk device: vnd(4) disk device in a regular file pseudo-device vnd –must be included in your kernel where you are building the FS –Does NOT need to be included in your kernel for the SPC –build filesystem dd(1): create the file vnconfig(8) - configure vnode pseudo disk as a regular file disklabel(8) - label it newfs(8) - construct a new file system mount(8)/umount(8): mount/unmount filesystem –mknod(8) - make device files –populate regular files from an existing system

January 10, Kits Workshop 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS … Steps to an Executing Kernel on an SPC Compile Kernel Make a symbol only version of kernel for /netbsd –ps, netstat, … use symbols from /netbsd –real kernel with file system CANNOT reside in itself! Add File System to Kernel –mdsetimage(8) Create an image version of Kernel –sets up bss section and symbol table Kernel bss file is ready to download!

January 10, Kits Workshop 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS … Steps to an Executing Kernel on an SPC Reset SPC Download Boot Loader –via APIC control cells –loaded at physical memory location: 0x0 Flip bit at location 0xFFE00 to start boot loader –configure NorthBridge (before kernel download!!) cache, memory size, memory type, timing, etc –wait for kernel download to complete looks for location 0x3FC to change from 0 to 1 –set up parameters on stack and jump to kernel kernel located at 0x Download kernel bss file Flip bit at location 0x3FC - kernel starts booting...

January 10, Kits Workshop 8 Washington WASHINGTON UNIVERSITY IN ST LOUIS The Download Process 0x0 Bootloader 0x Kernel x0FFE00 0x03FC Boot ROM 0xFFFFFFF RESET Download Program

January 10, Kits Workshop 9 Washington WASHINGTON UNIVERSITY IN ST LOUIS Host APIC to SPC Control Connections APIC PC WUGS Switch Port 0Port 1 Port 7 Port 6 Port 3 Port 5 SPC/ Port 4 Port 2 0/42 0/41 0/33 (0x21) 0/0x0321 APIC ID 410x0321 Example

January 10, Kits Workshop 10 Washington WASHINGTON UNIVERSITY IN ST LOUIS APIC Driver at SPC Boot Time PCI Configuration Space (sys/dev/pci/if_apic_pci.c): –Base memory address: APIC Physical memory in SPC starts at 0xF –Memory mapping options: Memory Enable Master Enable Write Invalidate Enable –IRQ Driver normally would read this from PCI config space APIC statically assigned IRQ 5 by System FPGA For consistency, driver writes 5 to Interrupt Line Register

January 10, Kits Workshop 11 Washington WASHINGTON UNIVERSITY IN ST LOUIS PCI Configuration Space Configuration Space Header Device IDVendor ID StatusCommand Class Code Revision ID BIST Header Type Cardbus CIS Pointer Subsystem ID Subsystem Vendor ID Expansion ROM Base Address Reserved Max_LatMin_Gnt Latency Timer Cache Line Size Base Address Registers Interrupt Pin Interrupt Line

January 10, Kits Workshop 12 Washington WASHINGTON UNIVERSITY IN ST LOUIS Defining a Route on APIC Network Interface Configure the APIC >ifconfig apic0 inet netmask 0xffffff00 -network interface: apic0 -our IP address: Configure a VCI >atm_ifconfig apic0 0xc5 open -aal5 -llc -besteffort -maxqueuebufs 100 -inport 0 -outports 1 -network interface: apic0 -vci: 0xc5 -besteffort (vs. paced or lowdelay) -Tx buffer queue limit for this VCI: ‘maxqueuebufs 100’ Configure the route >route add -iface link apic0:0.0.0.c5 -destination: network interface: apic0 -vci: 0xc5

January 10, Kits Workshop 13 Washington WASHINGTON UNIVERSITY IN ST LOUIS atm_ifconfig >atm_ifconfig Usage: atm_ifconfig [open|close|modify|getall [options...]] Options are: -aal0 | -aal5 -llc | -nollc : use LLC/SNAP? protocol is inet | ratm | value of ethernet type -loopback -inport -outports bitvector: 1 => port 0, 2 => port 1, 3 => both -lowdelay | -paced | -besteffort -maxqueuebufs is in Kbits/sec

January 10, Kits Workshop 14 Washington WASHINGTON UNIVERSITY IN ST LOUIS IPPOPP SPC TI Switch Fabric IPPOPP SPC TI IPPOPP TI Control Processor APIC Configuration IPPOPP SPC TI IPPOPP SPC TI... Sys. FPGA MB Pentium Cache North Bridge APIC SPC

January 10, Kits Workshop 15 Washington WASHINGTON UNIVERSITY IN ST LOUIS IPPOPP TI Sys. FPGA MB Pentium Cache North Bridge APIC Inport 1 Fiber Outport 2 Outport 1 Inport 0 Defining APIC Ports for Routing atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs Switch INPORT OUTPORTs

January 10, Kits Workshop 16 Washington WASHINGTON UNIVERSITY IN ST LOUIS Caveats You cannot change the IRQ assignments: –They are fixed in the System FPGA –And there should be no reason to change them HLT instruction in locore.s commented out PCI bus may be 25MHz or 33 MHz –Adjustment of NetBSD clock may be needed Memory/Filesystem tradeoffs –we have not pushed the limits yet Understand APIC port numbering SPC needs to be reset before EACH download