July 7-8, 2000 Kits Workshop 1 Washington WASHINGTON UNIVERSITY IN ST LOUIS A Smart Port Card Tutorial - The Exercises John DeHart Washington University

July 7-8, 2000 Kits Workshop 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS Question? How many of you have never built a NetBSD kernel?

July 7-8, 2000 Kits Workshop 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS Outline Exercises –Download a kernel –Build a kernel for the SPC –Modify a file system –Build a file system from scratch –Configure an SPC for IP connectivity –Install the SPC hardware in a Switch Use completely separate set of switches from other exercises Do this as you have free time throughout SATURDAY

July 7-8, 2000 Kits Workshop 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS If time permits... Open Exercises: –things we have not done yet –things we can discuss and/or start working on. Building a kernel - you add the SPC mods Download program using AAL5 frames Investigate Linux kernel for SPC

July 7-8, 2000 Kits Workshop 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS Setup We need to divide into 7 Groups: Each group gets its own: –login (spc1, spc2, spc3, … spc7 passwords: Spc1Tut, Spc2Tut, …) –switch with SPC already installed on Port 4 –PC with APIC connected to Port 3 /usr/SPC directory tree APIC PC Fiber Pair WUGS Switch Port 0Port 1 Port 7 Port 6Port 3 Port 5 SPC/ Port 4 Port 2 2 Serial Cables

July 7-8, 2000 Kits Workshop 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS Directory Tree for each Group: /usr/SPC/NetBSD NetBSD source directories /usr/SPC /bin WUGS and SPC utilities binaries /usr/SPC /boot_loader Source for SPC boot loader /usr/SPC /configuration GBNSC config files and Jammer scripts /usr/SPC /download Directory from which to download kernels /usr/SPC /root_wdir Directory for portion of kernel build done as root /usr/SPC /spc_utilities Source code for SPC utilities

July 7-8, 2000 Kits Workshop 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS Directory Tree for each Group: WUGS Utilities: –GBNSC –Jammer –GBNSC config file –Jammer script for configuring switch SPC Utilities: –resetSPC –download_APIC –download_ENI –mknl: make symbol-only netbsd –mkimg: make BSS image download directory –pre-built bootloader boot.out –bit file –Pre-built kernel spcKernel/netbsd.bss.ORIG SPC kernel source tree non-SPC kernel source tree Scripts for building kernel SPC Filesystem

Exercise 1: Kernel Downloading

July 7-8, 2000 Kits Workshop 9 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 1 - Download and boot a Kernel Reset switch Run GBNSC and Jammer to configure switch: –as root: >cd /usr/SPC/configuration >sh config.apic.connections –as regular user: >cd /usr/SPC/configuration >../bin/GBNSC -r -a config.switch.APIC & >../bin/Jammer 0.1 wooster 3551 SPC_Control.js

July 7-8, 2000 Kits Workshop 10 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 1 - continued Configure the PC (/etc/remote) so you can use tip: spc1:dv=/dev/tty01:dc:br#9600:pa=none spc0:dv=/dev/tty00:dc:br#9600:pa=none Start the tip processes in two separate windows as root: >tip spc0 >tip spc1

July 7-8, 2000 Kits Workshop 11 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 1 - continued Download bootloader and kernel >cd /usr/SPC/download >../bin/download_APIC -a –download will take 4-5 minutes… make sure you see this fairly quickly: boot loader started, sleep(2) Sending entire kernel...sendfile(filename=netbsd.bss, load_addr = 0x100000, start = 0x0, sendlen = 0x ) –while you’re waiting look at the scripts and source code: /usr/SPC/configuratoin/SPC_Control.js –Look at the WUGS connections to get to the SPC /usr/SPC/spc_utilities/download_APIC/main.cc

July 7-8, 2000 Kits Workshop 12 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 1 - continued Boot to multi-user mode –When you see this: “Enter pathname of shell or RETURN for sh:” hit –Then at ‘#’ prompt, type ‘exit’ Log in as root (no password) on the serial console –Use Terminal Type: vt100 –Notice which port is the console port. tty00 or tty01? –Check the size of the netbsd kernel in the file system: >ls -l /netbsd >du -s /netbsd –Check the size of the file system: >df -k / –While looking at your watch, try this: > sleep 20 what is the effect of these SPCs running at 25 MHz? Why?

July 7-8, 2000 Kits Workshop 13 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 1 - continued Send a reset control cell >/usr/SPC/bin/resetSPC -a –See that the serial console is now dead –Why does resetSPC hang? Look at code... –Kill it with a Download bootloader and kernel again See boot messages in tip window…

Exercise 2: Kernel and Filesystem Building

July 7-8, 2000 Kits Workshop 15 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2A - Building a kernel Do these steps as regular user (i.e. not as root) Make sure the kern and pci source code is configured: >cd /usr/SPC/NetBSD/usr/src/sys/kern >make init_sysent.c >cd../dev/pci >make -f Makefile.pcidevs Kernel config file provided: –/usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/SPC_24MB >cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf >config SPC_24MB >cd../compile/SPC_24MB >make depend

July 7-8, 2000 Kits Workshop 16 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2A - Building a kernel (con’t) Three shell scripts are provided: mk.1, mk.2, mk.3 Its easier for this part if you have two windows open –root window in directory /usr/SPC/root_wdir –regular user window in directory: /usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB –Stage 1: mk.1: run as regular user performs make to build kernel may take several minutes: –take a look at mk.1, mk.2, mk.3 while you are waiting –where does the kernel file end up when it is ready for download? >cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB/ >../mk.1

July 7-8, 2000 Kits Workshop 17 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2A - Building a kernel (con’t) –Stage 2: mk.2 run as root inserts symbol-only copy of /netbsd into filesystem >cd /usr/SPC/root_wdir >./mk.2 –Stage 3: mk.3 run as regular user copies filesystem into netbsd kernel file takes a couple of minutes... >cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB/ >../mk.3 Download bootloader and kernel again –don’t forget to reset SPC –and if you use resetSPC, don’t forget to kill it with

July 7-8, 2000 Kits Workshop 18 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2B - Modifying a kernel Modify the kernel –Change kernel so it boots to multi-user mode look in sys/arch/i386/i386/md_root.c (search for SINGLE) –Change timing of clock in system FPGA look at: sys/arch/i386/isa/timerreg.h (#define TIMER_FREQ) –add a kernel config file option to change TIMER_FREQ If needed, configure switch as in Exercise 1 Rebuild your kernel –You will need to do a ‘> make clean ‘ in the compile/SPC_24MB directory If needed, configure switch as in Exercise 1 If needed, send reset control cell to SPC Download bootloader and kernel Log in via serial port –Check the file system and verify that it is what you built

July 7-8, 2000 Kits Workshop 19 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2C - Modify a file system Modify the SPC Filesystem (as root, be careful...) >cd /usr/SPC/root_wdir >vnconfig -t spc24MB -v -c /dev/vnd0d SPC24MB.fs >mount /dev/vnd0d /mnt >cd /mnt/etc –Put something in it that will tell you that it is yours e.g. a file in /mnt/etc/ –Change console to other TTY port modify /mnt/etc/ttys (in SPC filesystem) –also change default terminal type for non-console tty to vt100 also need to change the kernel config file and re-run config >umount /mnt >vnconfig -u /dev/vnd0d

July 7-8, 2000 Kits Workshop 20 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2C - Modify a file system (con’t) Rebuild your kernel If needed, configure switch as in Exercise 1 If needed, send reset control cell to SPC Download bootloader and kernel Log in via serial port –Check the file system and verify that it is what you built

July 7-8, 2000 Kits Workshop 21 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2D - Build a file system Build your own Filesystem from scratch –Edit /etc/disktab to create a new sized disk: 18MB. spc18MB|spc18MBkernSPC Kernel FileSystem Disk:\ :ty=simulated:se#512:nt#16:rm#300:ns#27:nc#80:\ :pa#34560:oa#0:ba#4096:fa#512:ta=4.2BSD:\ :pb#34560:ob#0:\ :pc#34560:oc#0: >cd /usr/SPC/root_wdir –Use dd(1) to create a file >dd if=/dev/zero of= count= –Use vnconfig(1) to configure the file as a vnode disk >vnconfig -t -v -c /dev/vnd0d –Use disklabel (1) to give it a label >disklabel -rw /dev/vnd0d >disklabel -r /dev/vnd0d –Use newfs(1) to put a filesystem on the disk >newfs -m 0 -o space -i c 54 -T /dev/rvnd0a

July 7-8, 2000 Kits Workshop 22 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2D - Build a file system (con’t) –Mount it. >mount /dev/vnd0a /mnt –Populate the disk: Use mknod(1) to make devices in /dev directory –see next page for details! Mount the original SPC filesystem we used before on /mnt1 >vnconfig -t spc24MB -v -c /dev/vnd1d >mount /dev/vnd1d /mnt1 if /mnt1 does not exists you may have to create it with ‘mkdir’ Look through /mnt1/* and populate your filesystem from there Useful command, this does a tar(1) from directory to directory: >cd ; tar cf -. | (cd ; tar xfBp -) –Unmount >umount /mnt >vnconfig -u /dev/vnd0d >umount /mnt1 >vnconfig -u /dev/vnd1d

July 7-8, 2000 Kits Workshop 23 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2D - (con’t) : mknod(1) Devices >cd /mnt >mkdir dev >cd dev >mknod bpf0 c 23 0 >mknod bpf1 c 23 1 >mknod bpf2 c 23 2 >mknod bpf3 c 23 3 >mknod console c 0 0 >mknod drum c 4 0 >mknod io c 2 14 >mknod klog c 7 0 >mknod kmem c 2 1 >mknod lkm c 28 0 >mknod mem c 2 0 >mknod null c 2 2 >mknod ptyp0 c 6 0 >mknod ptyp1 c 6 1 >mknod ptyp2 c 6 2 >mknod ptyp3 c 6 3 >mknod rmd0a c 24 0 >mknod rmd0b c 24 1 >mknod rmd0c c 24 2 >mknod stderr c 22 2 >mknod stdin c 22 0 >mknod stdout c 22 1 >mknod tty c 1 0 >mknod tty00 c 8 0 >mknod tty01 c 8 1 >mknod ttyp0 c 5 0 >mknod ttyp1 c 5 1 >mknod ttyp2 c 5 2 >mknod ttyp3 c 5 3 >mknod zero c 2 12 >mknod md0a b 17 0 >mknod md0b b 17 1 >mknod md0c b 17 2 >chmod 600 console >chmod 640 drum >chmod 640 io >chmod 600 klog >chmod 640 kmem >chmod 640 mem >chmod 666 null >chmod 640 md0a >chmod 640 md0b >chmod 640 md0c >chmod 640 rmd0a >chmod 640 rmd0b >chmod 640 rmd0c >chmod 666 stderr >chmod 666 stdin >chmod 666 stdout >chmod 666 tty >chmod 644 tty00 >chmod 644 tty01 >chmod 666 zero

July 7-8, 2000 Kits Workshop 24 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2D - (con’t) : mknod(1) Devices >cd /mnt >mkdir dev >cd dev >mknod bpf0 c 23 0 >mknod bpf1 c 23 1 >mknod bpf2 c 23 2 >mknod bpf3 c 23 3 >mknod console c 0 0 >mknod drum c 4 0 >mknod io c 2 14 >mknod klog c 7 0 >mknod kmem c 2 1 >mknod lkm c 28 0 >mknod mem c 2 0 >mknod null c 2 2 >mknod ptyp0 c 6 0 >mknod ptyp1 c 6 1 >mknod ptyp2 c 6 2 >mknod ptyp3 c 6 3 >mknod rmd0a c 24 0 >mknod rmd0b c 24 1 >mknod rmd0c c 24 2 >mknod stderr c 22 2 >mknod stdin c 22 0 >mknod stdout c 22 1 >mknod tty c 1 0 >mknod tty00 c 8 0 >mknod tty01 c 8 1 >mknod ttyp0 c 5 0 >mknod ttyp1 c 5 1 >mknod ttyp2 c 5 2 >mknod ttyp3 c 5 3 >mknod zero c 2 12 >mknod md0a b 17 0 >mknod md0b b 17 1 >mknod md0c b 17 2 >chmod 600 console >chmod 640 drum >chmod 640 io >chmod 600 klog >chmod 640 kmem >chmod 640 mem >chmod 666 null >chmod 640 md0a >chmod 640 md0b >chmod 640 md0c >chmod 640 rmd0a >chmod 640 rmd0b >chmod 640 rmd0c >chmod 666 stderr >chmod 666 stdin >chmod 666 stdout >chmod 666 tty >chmod 644 tty00 >chmod 644 tty01 >chmod 666 zero OK, OK. There is a script for this. /usr/SPC/bin/mknods

July 7-8, 2000 Kits Workshop 25 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 2D - Build a file system (con’t) Rebuild your kernel –Change the kernel config file –Look through the make scripts to change filesystem –Re-config and re-make the kernel If needed, configure switch as in Exercise 1 If needed, send reset control cell to SPC Download bootloader and kernel Log in via serial port –Check the file system and verify that it is what you built –Keep it running while you go on to next exercise...

Exercise 3: IP Configuration

July 7-8, 2000 Kits Workshop 27 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 3A - IP Between SPC and PC Save any changes you make for the SPC in your filesystem Configure an IP address for the SPC’s APIC >ifconfig Configure an IP address for the PC’s APIC Configure an APIC connection on each >atm_ifconfig Add routes on PC and SPC: >route add... Configure the necessary WUGS connection(s)

July 7-8, 2000 Kits Workshop 28 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 3A - IP Address Assignments > cat filename | telnet postscript.cs 9100

July 7-8, 2000 Kits Workshop 29 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 3A - IP Between SPC and PC Try to ping –Iterate until you can ping from PC to SPC –Tools for debugging the IP connection: netstat APIC readCounts utility –/usr/local/bin/readCounts on both PC and SPC WUGS cell counters (Jammer) Turn on telnetd –edit /etc/inetd.conf on SPC –kill -HUP –iterate until you can telnet from PC to SPC –Be patient on some machines telnet seems to take a while to connect. Not sure why yet...

July 7-8, 2000 Kits Workshop 30 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 3A - IP Between SPC and PC Make sure you have saved all your changes Reset switch Rebuild kernel with your newly modified filesystem and re-download. Does it still work...

July 7-8, 2000 Kits Workshop 31 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 3B - IP Between Neighbor SPCs Use SPCs as routers: PC Domain PC Domain... SPC

July 7-8, 2000 Kits Workshop 32 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 3B - IP Between Neighbor SPCs Save any changes you make for the SPC in your filesystem Configure an IP address alias for the SPC’s APIC to put it in the neighboring SPC’s domain >ifconfig …alias Remove the network route that the above caused: >route delete -net Configure an APIC connection on each SPC >atm_ifconfig Add routes on PC and SPC: >route add... Configure the necessary WUGS connection(s)

July 7-8, 2000 Kits Workshop 33 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 3B - IP Between Neighbor SPCs Try to ping –Iterate until you can ping from PC to neighbor PC –Tools for debugging the IP connection: netstat APIC readCounts utility (/usr/local/bin/readCounts on SPC) WUGS cell counters Make sure you have saved all your changes Reset switch Rebuild kernel with your newly modified filesystem and re-download. Does it still work...

Exercise 4: Hardware Installation

July 7-8, 2000 Kits Workshop 35 Washington WASHINGTON UNIVERSITY IN ST LOUIS Exercise 4 - Installing an SPC in a Switch Power Off Open Switch Removing Glink Line Card from Port 4 or 5 Install SPC Install Serial Cables, feed in through front panel Re-Install Glink Line Card on top of SPC Close Switch Connect serial cables to PC Power On Download and Boot Kernel Tom will provide a Slide or two

Open Exercises

July 7-8, 2000 Kits Workshop 37 Washington WASHINGTON UNIVERSITY IN ST LOUIS Open Exercises: SPC Mods for NetBSD Do this first if you are interested in SPC/Linux Compare kernel config files: –/usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/SPC_24MB –/usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/APIC_1_4_1 –Note the major differences –Find the network devices in both. Anything interesting? Compare source files: –/usr/SPC/NetBSD/usr.ORIG –/usr/SPC/NetBSD/usr –Directories where there are differences: sys/arch/i386/i386sys/... sys/arch/i386/isa sys/dev/ic sys/dev/pci

July 7-8, 2000 Kits Workshop 38 Washington WASHINGTON UNIVERSITY IN ST LOUIS Open Exercises: AAL5 Download Program currently we download via APIC control cells –16 bytes of data per cell –send one cell, wait for response cell before sending next one –slow: 4-5 minutes for a 24MB kernel Preferred method would be AAL5 –Use APIC Control cells to configure AAL5 descriptors –Send Kernel in AAL5 frames to SPC –Read it back to verify?

July 7-8, 2000 Kits Workshop 39 Washington WASHINGTON UNIVERSITY IN ST LOUIS Open Exercises: Linux Kernel for the SPC ONLY if you are already familiar with NetBSD/SPC mods ONLY if you are already VERY familiar with Linux Kernel Changes: –Memory Disk –Serial port console –APIC Configuration by APIC driver System FPGA impact: –Timer configuration –RTC requirements –Serial port configuration –Static IRQ assignment Download and Boot –Boot loader –Other things that Linux expects from the BIOS or bootloader?