1. Washington University
Smart Port Card
Tutorial
---
Software
John DeHart
Washington University

2. Software Overview No MSR details included in this talk
That will come in the MSR Tutorial
SPC Specific Kernel Modifications
Building a Kernel
config
compile
Loading Kernel onto SPC to execute it
download via APIC
Initiating Kernel execution
boot loader
Configuring Devices
APIC

3. SPC Kernel Modifications
Kernel Configuration File:
Memory Disk
Serial Console
Most device drivers unneeded (e.g no Ethernet…)
Source Code Changes:
PCI Device Configuration: ./dev/pci/if_apic_pci.c
APIC Specific configuration handled by driver
Change some APIC defaults: ./dev/ic/apic.c
made switch port the default
No HLT instruction: locore.s
Kernel Memory Page Allocation:
./arch/i386/include/pmap.new.h
RTC: ./arch/i386/isa/clock.c, ./ufs/ffs/ffs_vfsops.c
We still had to remove some RTC code...

4. Steps to an Executing Kernel on an SPC
Configure Kernel
Start with a typical NetBSD kernel config file
Add SPC specific option:
options SPC
add memory disk (filesystem will reside in kernel!)
options MEMORY_DISK*
options MINIROOTSIZE
serial console port
options CONSDEVNAME
options CONS_OVERRIDE
remove “extra” devices (Ethernet, mouse, …)

5. Steps to an Executing Kernel on an SPC
Filesystem
we use a vnode disk device: vnd(4)
disk device in a regular file
pseudo-device vnd
must be included in your kernel where you are building the FS
Does NOT need to be included in your kernel for the SPC
build filesystem
dd(1): create the file
vnconfig(8) - configure vnode pseudo disk as a regular file
disklabel(8) - label it
newfs(8) - construct a new file system
mount(8)/umount(8): mount/unmount filesystem
mknod(8) - make device files
populate regular files from an existing system

6. … Steps to an Executing Kernel on an SPC
Compile Kernel
Make a symbol only version of kernel for /netbsd
ps, netstat, … use symbols from /netbsd
real kernel with file system CANNOT reside in itself!
Download kernel and File System separately
see following slides on AAL5download
Or, Download as one file:
Add File System to Kernel
mdsetimage(8)
Create an image version of Kernel
sets up bss section and symbol table
Kernel bss file is ready to download!

7. … Steps to an Executing Kernel on an SPC
Reset SPC
Download Boot Loader
via APIC control cells
loaded at physical memory location: 0x0
Flip bit at location 0xFFE00 to start boot loader
configure NorthBridge (before kernel download!!)
cache, memory size, memory type, timing, etc
wait for kernel download to complete
looks for location 0x3FC to change from 0 to 1
set up parameters on stack and jump to kernel
kernel located at 0x100000
Download kernel bss file
Flip bit at location 0x3FC - kernel starts booting...
This is the OLD WAY
Please stop using download_APIC
It is slow and painful

8. AAL5_Download Replaces old download_APIC utility
Can “shotgun” download to <= 8 SPCs at once
Puts exactly same kernel on all 8
This works fine for MSR where we want same kernel
May not be good for all uses
“Shotgun” number controlled from command line
Can download kernel and file system separately
no need to load file system into kernel before download
still need to load symbol version of kernel into file system before downloading
Takes about 1 minute for the whole download and boot process.

9. AAL5_Download: For SPC Implemented and runs in Stages:
Stage1 uses APIC Control cells to:
reset the SPC(s)
download Stage2 to run on the SPC(s)
Stage2 (boot.out) runs on the SPC
runs SPC boot code to configure CPU and Northbridge
configures APIC on SPC to use AAL5 descriptors
sends pings back to CP, waiting for Stage3
receives AAL5 frames from Stage3 and sends ACK/NACK
Stage3 runs on PC
syncs with Stage2
downloads the kernel or FS via 36KB AAL5 frames
retransmits if any errors

10. AAL5_Download: For SPC-II
We are still working out the final details
We are starting to test the first batch of SPC-IIs
Implemented and runs in Stages:
Stage2 (boot.out) runs on the SPC-II
Already loaded on the IDE Disk on Module.
no need to download it
runs SPC boot code to configure CPU and Northbridge
configures APIC on SPC to use AAL5 descriptors
sends pings back to CP, waiting for Stage3
receives AAL5 frames from Stage3 and sends ACK/NACK
Stage3 runs on PC
syncs with Stage2
downloads the kernel or FS via 36KB AAL5 frames
retransmits if any errors

11. Simple View Stage1 PC SPC-I Stage1 VCI=0x321 APIC VCI=21 VCI=21 VCI=25

12. Sequential View RESET Kernel and Filesystem Boot ROM SPC Memory PC SPC
Boot and
Stage2
0x0
Stage1
APIC
Reset
cell
Control
cells
RESET
0x0FFE00
Stage3
AAL5
Frames
Stage2
Kernel and
Filesystem 0x
0xFFFFFFF0
Boot ROM

13. Source and Executables
source location: wu_arl/utilities/SPC/AAL5Download
binary location:
wu_arl/utilities/bin/$OSTYPE/Stage1
wu_arl/utilities/bin/$OSTYPE/Stage3
wu_arl/utilities/bin/$OSTYPE/AAL5_download.sh
wu_arl/utilities/bin/$OSTYPE/boot.out
wu_arl/utilities/bin/$OSTYPE/FindNM

14. Usage Individual command line commands AAL5download.sh script
./Stage1 –shotgun 8 –svci 21 –rvci 21 –dvci 23 –uvci 24 –cvci 25
./Stage3 –shotgun 8 –native netbsd.MSR –both 24
./Stage3 –shotgun 8 –bss SPC24MB.fs –address 0x –both 24 -go
AAL5download.sh script
./AAL5download 8 netbsd.MSR SCP24MB.fs
Use from MSR Configuration Module
system(msrDownload.sh);

15. Special Usage Note on Linux
AAL5download.sh script:
needs to be able to determine the start location for the File System
NetBSD uses nm(1)
Linux nm(1) won’t work on a NetBSD binary
We have written a version of nm specifically for AAL5_download.sh: FindNM
FindNM is used automatically when AAL5_download.sh is run on Linux

16. AAL5_download.sh Usage > ./AAL5_download.sh
Usage: ./AAL5_download.sh <netbsd_file> <filesystem_file> <start_vci> [<second_vci>] [<third_vci>][<fourth_vci>] [<fifth_vci>]

17. Stage1 Usage > ./Stage1 ./Stage1: [vpi/vci data] [load info]
./Stage1: [-svpi #] (VPI for local sending of AAL5 control cells (0))
./Stage1: [-svci #] (VCI for local sending of AAL5 control cells)
./Stage1: [-rvpi #] (VPI for local reading of AAL5 control cells (0))
./Stage1: [-rvci #] (VCI for local reading of AAL5 control cells)
./Stage1: [-cvpi #] (VPI for remote CC writes of AAL5 control cells (0))
./Stage1: [-cvci #] (VCI for remote CC writes of AAL5 control cells)
./Stage1: [-dvpi #] (VPI for local write of AAL5 data frames (0))
./Stage1: [-dvci #] (VCI for local write of AAL5 data frames)
./Stage1: [-uvpi #] (VPI for local read of AAL5 data frames (0))
./Stage1: [-uvci #] (VCI for local read of AAL5 data frames)
./Stage1: [-resp #] (Response port of AAL5 control cells)
./Stage1: [-start #] (Initial VCI sequence (41, 42, 43, 44))
./Stage1: [-watch 0x#] (Monitor location for changes)
./Stage1: [-apic #] (Linux: Apic Id (0 <= X <= 3))
./Stage1: [-abit #] (Alternating bit setting)
./Stage1: [-file <path>] (Name of boot loader (boot.out))
./Stage1: [-swap] (use htonl() on output)
./Stage1: [-shotgun #] (multicast loading 8X)
./Stage1: [-msr #] (VCI for MSR ONLY AAL5 data frames)
./Stage1: [-address #] (Physical data write address (0))
./Stage1: [-noreset] (do not write reset register)

18. Stage3 Usage > ./Stage3 -help [-switches] (-help) args
-bss <file> Load .bss format file
-native <file> Load native uncompressed file
-strip Remove Symbols from kernel
-debug print lots of stuff
-verbose Dump Elf headers
-elf Elf format execution
-rvpi <VPI> Load VPI number
-rvci <VCI> Load VCI number
-svpi <VPI> Load VCI number
-svci <VCI> Load VCI number
-both <VCI> Set both VCI numbers
-address <PhysAddr> Load file at <addr>
-start <PhysAddr> Start file at <addr>
-go Start kernel
-shotgun <count> Multicast loading 8X

19. Host APIC to SPC Control Connections
Example
WUGS Switch
Port 0
Port 1
Port 7
Port 6
Port 3
Port 5
SPC/
Port 4
Port 2
APIC PC 41
0x0321
0/41
0/42
0/33 (0x21)
0/0x0321
APIC ID

20. APIC Driver at SPC Boot Time
PCI Configuration Space (sys/dev/pci/if_apic_pci.c):
Base memory address:
APIC Physical memory in SPC starts at 0xF
Memory mapping options:
Memory Enable
Master Enable
Write Invalidate Enable
IRQ
Driver normally would read this from PCI config space
APIC statically assigned IRQ 5 by System FPGA
For consistency, driver writes 5 to Interrupt Line Register

21. PCI Configuration Space
Device ID
Vendor ID
Status
Command
Class Code
Revision ID
BIST
Header
Type
Cardbus CIS Pointer
Subsystem ID
Subsystem Vendor ID
Expansion ROM Base Address
Reserved
Max_Lat
Min_Gnt
Latency
Timer
Cache Line
Size
Base Address Registers
Interrupt
Pin
Line
Configuration Space Header

22. Defining a Route on APIC Network Interface
NOTE: With the MSR you do not use ifconfig and atm_ifconfig
the MSR uses a different APIC driver then “plain” NetBSD for the SPC
Configure the APIC
ifconfig apic0 inet netmask 0xffffff00
network interface: apic0
our IP address:
Configure a VCI
atm_ifconfig apic0 0xc5 open -aal5 -llc -besteffort -maxqueuebufs 100 -inport FIBER -outports FIBER
vci: 0xc5
besteffort (vs. paced or lowdelay)
Tx buffer queue limit for this VCI: ‘maxqueuebufs 100’
Configure the route
route add -iface link apic0:0.0.0.c5
destination:

23. atm_ifconfig Changes HERE -inport and -outport values have changed.
Usage: atm_ifconfig <interface> <vpivci> [open|close|modify|getall [options...]]
Options are:
-aal0 | -aal5
-llc | -nollc <protocol>: use LLC/SNAP?
protocol is inet | ratm | value of ethernet type
-loopback
-inport <input_port_number>
-outports <bitvector_of_output_ports>
bitvector: 1 => port 0, 2 => port 1, 3 => both
-lowdelay | -paced <rate> | -besteffort
-maxqueuebufs <max_descriptors_in_tx_queue>
<rate> is in Kbits/sec
Changes
HERE
-inport and
-outport
values
have
changed.

24. APIC Configuration Switch Fabric . . . Control Processor SPC TI IPP6 5
IPP
OPP
SPC TI
IPP
OPP
SPC TI
IPP
OPP
SPC TI
IPP
OPP
SPC TI
IPP
OPP
Sys. FPGA
32-64 MB
APIC
North Bridge
. . .
SPC
Pentium
Cache TI 3 6

25. Defining APIC Ports for Routing
Switch
IPP
OPP
Sys. FPGA
32-64 MB
INPORT OUTPORTs
LINK LINK
APIC
North Bridge
Pentium
Cache TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs

26. Defining APIC Ports for Routing
Switch
IPP
OPP
Sys. FPGA
32-64 MB
INPORT OUTPORTs
LINK SWITCH
APIC
North Bridge
Pentium
Cache TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs

27. Defining APIC Ports for Routing
Switch
IPP
OPP
Sys. FPGA
32-64 MB
INPORT OUTPORTs
SWITCH LINK
APIC
North Bridge
Pentium
Cache TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs

28. Defining APIC Ports for Routing
Switch
IPP
OPP
INPORT OUTPORTs
SWITCH SWITCH
Sys. FPGA
32-64 MB
APIC
North Bridge
Pentium
Cache TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs

29. Defining APIC Ports for Routing
Switch
IPP
OPP
INPORT OUTPORTs
LINK BOTH
Sys. FPGA
32-64 MB
APIC
North Bridge
Pentium
Cache TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs

30. Defining APIC Ports for Routing
Switch
IPP
OPP
Sys. FPGA
32-64 MB
INPORT OUTPORTs
SWITCH BOTH
APIC
North Bridge
Pentium
Cache TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs

31. Caveats You cannot change the SPC-I IRQ assignments:
They are fixed in the System FPGA
And there should be no reason to change them
HLT instruction in locore.s commented out
remnant of when we were having INTEL problems
we can leave it this way
gives a slight performance improvement
SPC needs to be reset before EACH download