1. January 8, 2001 SPC Tutorial 1 Washington WASHINGTON UNIVERSITY IN ST LOUIS Agenda 9:00 SPC Hardware -- John DeHart 9:45 SPC Software -- John DeHart 10:30 SPC Utilities -- Ed Spitznagel 11:00 Exercises (lets get started!!) - All 12:00 Lunch (brought in) 1:00-6:00 Exercises -- All

2. January 8, 2001 SPC Tutorial 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS A Smart Port Card Tutorial --- Hardware John DeHart Washington University jdd@arl.wustl.edu http://www.arl.wustl.edu/~jdd

3. January 8, 2001 SPC Tutorial 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS References: New Links from Kits References Page Intel Embedded Module: –Data Sheet –Design Guide 430HX Chipset –NorthBridge –SouthBridge System FPGA Memory Mobile Pentium with MMX –Software Developer Manuals 1,2,3 –Datasheet APIC Cache

4. January 8, 2001 SPC Tutorial 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS Motivation Active Networking Network Probe High performance router architectures –PC as router is VERY limited –(Gigabit/s + Processing) on each port –MSR: Multi-Service multiport Router

5. January 8, 2001 SPC Tutorial 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS The Smart Port Card Hardware: –SPC as a PC How do they each boot? –SPC Hardware Components What roles do they play?

6. January 8, 2001 SPC Tutorial 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS Typical Pentium PC CPU North- Bridge CacheDRAM SouthBridge (PIIX3) (PIC, PIT, …) PCI Bus ISA Bus PCI Devices ISA Devices BIOS Super-IO BIOS RTC Uarts Kbd/Mse Floppy Parallel... Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT CPU/Memory Bus

7. January 8, 2001 SPC Tutorial 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS How NetBSD Boots on a PC Components: –Pentium –Boot ROM (replaced by BIOS in modern systems?) –BIOS –Bootloader –Kernel

8. January 8, 2001 SPC Tutorial 8 Washington WASHINGTON UNIVERSITY IN ST LOUIS Sketch of How a PC Boots Pentium after Reset: –fetches its first instruction from location 0xFFFFFFF0 Boot Code must be located at 0xFFFFFFF0 Boot Code jumps to BIOS located in ROM –Boot Code may actually be part of the BIOS... BIOS copies itself into memory (Shadow) BIOS remaps memory –future accesses to BIOS addresses go to memory instead of ROM. BIOS performs system configuration (some proprietary) –Motherboard Details –Pentium Details –NB/SB Chipset Details –Device configuration: IRQs, Memory maps,... … at least what I understand…

9. January 8, 2001 SPC Tutorial 9 Washington WASHINGTON UNIVERSITY IN ST LOUIS How a PC Boots (continued…) BIOS loads bootloader into memory (from disk…) BIOS jumps to bootloader Bootloader performs some more configuration: –Pentium control registers –Cache configuration –Memory/Page model Bootloader determines what to run next. Bootloader may have to do some device configuration. –e.g. to get OS from a disk. Bootloader loads OS kernel into memory Bootloader jumps to start of OS kernel

10. January 8, 2001 SPC Tutorial 10 Washington WASHINGTON UNIVERSITY IN ST LOUIS How a PC Boots (continued…) Kernel does some OS-specific configuration: –for NetBSD look in: sys/arch/i386/i386/locore.s –Determines what CPU it has (“cpuid” instruction) –Paging –Virtual Memory

11. January 8, 2001 SPC Tutorial 11 Washington WASHINGTON UNIVERSITY IN ST LOUIS Typical Pentium PC (Again…) CPU North- Bridge CacheDRAM SouthBridge (PIIX3) (PIC, PIT, …) PCI Bus ISA Bus PCI Devices ISA Devices BIOS Super-IO BIOS RTC Uarts Kbd/Mse Floppy Parallel... Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT CPU/Memory Bus

12. January 8, 2001 SPC Tutorial 12 Washington WASHINGTON UNIVERSITY IN ST LOUIS What SPC Needs CPU North- Bridge CacheDRAM SouthBridge (PIC, PIT, …) PCI Bus APIC BIOS Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT RTC Uarts CPU/Memory Bus

13. January 8, 2001 SPC Tutorial 13 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA Intel Embedded Module SPC Architecture CPU North- Bridge CacheDRAM PCI Bus APIC Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT PITPICRTC’ BIOS ROM UART1 Interface UART2 Interface UART1 UART2 Link Interface Switch Interface

14. January 8, 2001 SPC Tutorial 14 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC Photo Tour Switch Interface Link Interface Serial Ports APIC CPU Module PCI Bus System FPGA DRAM

15. January 8, 2001 SPC Tutorial 15 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC Components APIC PCI Bus Master Pentium Embedded Module –166 MHz MMX Pentium Processor L1 Cache: 16KB Data, 16KB Code –L2 cache: 512 KB –NorthBridge - 33 MHz, 32 bit PCI Bus PCI Bus Master System FPGA PCI Bus Slave –Xilinx XC4020XLA-1 FPGA –20K Equivalent Gates –~ 75% used

16. January 8, 2001 SPC Tutorial 16 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC Components (continued) Memory –EDO DRAM –64MB (Max for current design) –SO DIMM Switch Interface - 1 Gb Utopia Link Interface - 1 Gb Utopia UART –Two Serial Ports NetBSD system console TTY port

17. January 8, 2001 SPC Tutorial 17 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA Coded in VHDL PCI slave device Replaces some of the PIIX3 (south bridge) Replaces some of the BIOS Replaces some of the Super IO Chip Provides reset capability

18. January 8, 2001 SPC Tutorial 18 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: PIIX3 Functionality Programmable Interrupt Controller (PIC) –Four Interrupts supported and statically assigned: PIT (IRQ 0) APIC (IRQ 5) COM1 (IRQ 4) COM2 (IRQ 3) –Static fully-nested interrupt priority structure. –Specific End of Interrupt is the only EOI mode supported Programmable Interval Timer (PIT) –generates a clock interrupt for NetBSD every ~10ms Reset - covered in a later slide

19. January 8, 2001 SPC Tutorial 19 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: BIOS Functionality Interrupt functionality replaced by static values Simple 16 word by 32-bit “ROM” –implements loop waiting for location 0xFFE00 to change value –then jumps to boot loader code Does NOT perform configuration of Northbridge –This will be done by the boot loader Does NOT perform PCI configuration of APIC –This will be done by the APIC Driver

20. January 8, 2001 SPC Tutorial 20 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: Super IO Chip Functionality UART Interface –Two Serial lines supported –Fixed IRQs Real Time Clock –only the register accesses of the RTC are supported –no interrupts supported –i.e. supported only so NetBSD didn’t need to change –i.e. no alarms will be generated

21. January 8, 2001 SPC Tutorial 21 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: Reset SPC needs a reset before every download: –switch reset: causes SPC to be reset causes all connections in switch to be lost –System FPGA reset causes SPC to be reset has no effect on the switch Normal SouthBridge reset: –I/O Register: 0xCF9 –Hard Reset: assert CPURST, PCIRST#, and RSTDRV write 0xCF9 0x02 (00000010b) write 0xCF9 0x06 (00000110b) –Soft Reset: assert INIT write 0xCF9 0x00 (00000000b) write 0xCF9 0x04 (00000100b) bits

22. January 8, 2001 SPC Tutorial 22 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: Reset SPC Reset: –a sequence of two writes to memory addresses –APIC Control cells can write to memory addresses configuration registers NOT I/O Registers! Argh... –To mimic the reset structure of the SB we use: 0xFFFFFFF0 0xFFFFFFF4 –Hard Reset (all we really care about) write 0xFFFFFFF0 0x02 (00000010b) write 0xFFFFFFF4 0x06 (00000110b) bits

23. January 8, 2001 SPC Tutorial 23 Washington WASHINGTON UNIVERSITY IN ST LOUIS Caveats Intel Embedded Module problem –Memory corruption caused by noise on M/A bus –Hopefully it will be fixed before we ship –We work around it with 25 MHz PCI and no HLT This reduces the probability of noise on the bus PCI Bus –33 MHz vs. 25 MHz NetBSD Kernel HLT instruction –if absolutely nothing to do, NetBSD does a “HLT” –this reduces power consumption –also causes large power/current swings –We have removed the HLT instruction for the SPC Serial Cables –RS232 is not necessarily hot-swappable sometimes you can get away with it but not always FIXED