1. June 16, 2002 SPC Tutorial 1 Washington WASHINGTON UNIVERSITY IN ST LOUIS A Smart Port Card (SPC and SPC-II) Tutorial --- Hardware John DeHart Washington University jdd@arl.wustl.edu http://www.arl.wustl.edu/~jdd

2. June 16, 2002 SPC Tutorial 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS Motivation Active Networking Network Probe High performance router architectures –PC as router is VERY limited –(Gigabit/s + Processing) on each port –MSR: Multi-Service multiport Router

3. June 16, 2002 SPC Tutorial 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS The Original Smart Port Card Hardware: –SPC as a PC How do they each boot? –SPC Hardware Components What roles do they play?

4. June 16, 2002 SPC Tutorial 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS Typical Pentium PC CPU North- Bridge CacheDRAM SouthBridge (PIIX3) (PIC, PIT, …) PCI Bus ISA Bus PCI Devices ISA Devices BIOS Super-IO BIOS RTC Uarts Kbd/Mse Floppy Parallel... Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT CPU/Memory Bus

5. June 16, 2002 SPC Tutorial 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS Typical Pentium PC: CPU/Memory Bus CPU North- Bridge CacheDRAM SouthBridge (PIIX3) (PIC, PIT, …) PCI Bus ISA Bus PCI Devices ISA Devices BIOS Super-IO BIOS RTC Uarts Kbd/Mse Floppy Parallel... Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT CPU/Memory Bus

6. June 16, 2002 SPC Tutorial 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS Typical Pentium PC: PCI Bus CPU North- Bridge CacheDRAM SouthBridge (PIIX3) (PIC, PIT, …) PCI Bus PCI Devices ISA Devices BIOS Super-IO BIOS RTC Uarts Kbd/Mse Floppy Parallel... Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT CPU/Memory Bus ISA Bus

7. June 16, 2002 SPC Tutorial 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS Typical Pentium PC: ISA Bus CPU North- Bridge CacheDRAM SouthBridge (PIIX3) (PIC, PIT, …) PCI Bus ISA Bus PCI Devices ISA Devices BIOS Super-IO BIOS RTC Uarts Kbd/Mse Floppy Parallel... Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT CPU/Memory Bus

8. June 16, 2002 SPC Tutorial 8 Washington WASHINGTON UNIVERSITY IN ST LOUIS How NetBSD Boots on a PC Components: –Pentium –Boot ROM (part of BIOS in modern systems?) –BIOS –Bootloader –Kernel

9. June 16, 2002 SPC Tutorial 9 Washington WASHINGTON UNIVERSITY IN ST LOUIS Sketch of How a PC Boots Pentium after Reset: –fetches its first instruction from location 0xFFFFFFF0 Boot Code must be located at 0xFFFFFFF0 Boot Code jumps to BIOS located in ROM –Boot Code may actually be part of the BIOS... BIOS copies itself into memory (Shadow) BIOS remaps memory –future accesses to BIOS addresses go to memory instead of ROM. BIOS performs system configuration (some proprietary) –Motherboard Details –Pentium Details –NB/SB Chipset Details –Device configuration: IRQs, Memory maps,...

10. June 16, 2002 SPC Tutorial 10 Washington WASHINGTON UNIVERSITY IN ST LOUIS How a PC Boots (continued…) BIOS loads bootloader into memory (from disk…) BIOS jumps to bootloader Bootloader performs some more configuration: –Pentium control registers –Cache configuration –Memory/Page model Bootloader determines what to run next. Bootloader may have to do some device configuration. –e.g. to get OS from a disk. Bootloader loads OS kernel into memory Bootloader jumps to start of OS kernel Kernel does some OS-specific configuration: –for NetBSD look in: sys/arch/i386/i386/locore.s –Determines what CPU it has (“cpuid” instruction) –Paging –Virtual Memory

11. June 16, 2002 SPC Tutorial 11 Washington WASHINGTON UNIVERSITY IN ST LOUIS Typical Pentium PC (Again…) CPU North- Bridge CacheDRAM SouthBridge (PIIX3) (PIC, PIT, …) PCI Bus ISA Bus PCI Devices ISA Devices BIOS Super-IO BIOS RTC Uarts Kbd/Mse Floppy Parallel... Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT CPU/Memory Bus

12. June 16, 2002 SPC Tutorial 12 Washington WASHINGTON UNIVERSITY IN ST LOUIS What SPC Needs CPU North- Bridge CacheDRAM SouthBridge (PIC, PIT, …) PCI Bus APIC BIOS Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT RTC Uarts CPU/Memory Bus

13. June 16, 2002 SPC Tutorial 13 Washington WASHINGTON UNIVERSITY IN ST LOUIS What SPC Needs: CPU/Memory Bus CPU North- Bridge CacheDRAM SouthBridge (PIC, PIT, …) PCI Bus APIC BIOS Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT RTC Uarts CPU/Memory Bus

14. June 16, 2002 SPC Tutorial 14 Washington WASHINGTON UNIVERSITY IN ST LOUIS What SPC Needs: PCI Bus CPU North- Bridge CacheDRAM SouthBridge (PIC, PIT, …) PCI Bus APIC BIOS Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT RTC Uarts CPU/Memory Bus

15. June 16, 2002 SPC Tutorial 15 Washington WASHINGTON UNIVERSITY IN ST LOUIS What SPC Needs: ISA Bus? CPU North- Bridge CacheDRAM SouthBridge (PIC, PIT, …) PCI Bus APIC BIOS Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT RTC Uarts CPU/Memory Bus

16. June 16, 2002 SPC Tutorial 16 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA Intel Embedded Module SPC Architecture CPU North- Bridge CacheDRAM PCI Bus APIC Addr/DataCtrl Addr/Data/Ctrl Intr NMI INIT PITPICRTC’ BIOS ROM UART1 Interface UART2 Interface UART1 UART2 Link Interface Switch Interface

17. June 16, 2002 SPC Tutorial 17 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC Components APIC Pentium Embedded Module –166 MHz MMX Pentium Processor L1 Cache: 16KB Data, 16KB Code –L2 cache: 512 KB –NorthBridge - 33 MHz, 32 bit PCI Bus System FPGA –Xilinx XC4020XLA-1 FPGA Memory –EDO DRAM: 64MB (Max for current design) –SO DIMM Switch and Link Interfaces – Each 1 Gb Utopia UART supports two Serial Ports: NetBSD system console TTY port

18. June 16, 2002 SPC Tutorial 18 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA Coded in VHDL PCI slave device Replaces some of the PIIX3 (south bridge) Replaces some of the BIOS Replaces some of the Super IO Chip Provides reset capability

19. June 16, 2002 SPC Tutorial 19 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: PIIX3 Functionality Programmable Interrupt Controller (PIC) –Four Interrupts supported and statically assigned: PIT (IRQ 0) APIC (IRQ 5) COM1 (IRQ 4) COM2 (IRQ 3) –Static fully-nested interrupt priority structure. –Specific End of Interrupt is the only EOI mode supported Programmable Interval Timer (PIT) –generates a clock interrupt for NetBSD every ~10ms Reset - covered in a later slide

20. June 16, 2002 SPC Tutorial 20 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: BIOS Functionality Interrupt functionality replaced by static values Simple 16 word by 32-bit “ROM” –implements loop waiting for location 0xFFE00 to change value –then jumps to boot loader code Does NOT perform configuration of Northbridge –This will be done by the boot loader Does NOT perform PCI configuration of APIC –This will be done by the APIC Driver

21. June 16, 2002 SPC Tutorial 21 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: Super IO Chip Functionality UART Interface –Two Serial lines supported –Fixed IRQs Real Time Clock –only the register accesses of the RTC are supported –no interrupts supported –i.e. supported only so NetBSD didn’t need to change –i.e. no alarms will be generated

22. June 16, 2002 SPC Tutorial 22 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: Reset SPC needs a reset before every download: –switch reset: causes SPC to be reset causes all connections in switch to be lost –System FPGA reset causes SPC to be reset has no effect on the switch Normal SouthBridge reset: –I/O Register: 0xCF9 –Hard Reset: assert CPURST, PCIRST#, and RSTDRV write 0xCF9 0x02 (00000010b) write 0xCF9 0x06 (00000110b) –Soft Reset: assert INIT write 0xCF9 0x00 (00000000b) write 0xCF9 0x04 (00000100b) bits

23. June 16, 2002 SPC Tutorial 23 Washington WASHINGTON UNIVERSITY IN ST LOUIS System FPGA: Reset SPC Reset: –a sequence of two writes to memory addresses –APIC Control cells can write to memory addresses configuration registers NOT I/O Registers! Argh... –To mimic the reset structure of the SB we use: 0xFFFFFFF0 0xFFFFFFF4 –Hard Reset (all we really care about) write 0xFFFFFFF0 0x02 (00000010b) write 0xFFFFFFF4 0x06 (00000110b) bits

24. June 16, 2002 SPC Tutorial 24 Washington WASHINGTON UNIVERSITY IN ST LOUIS References: SPC-I (on Kits Web Pages) Intel Embedded Module: –Data Sheet –Design Guide 430HX Chipset –NorthBridge –SouthBridge System FPGA Memory Mobile Pentium with MMX –Software Developer Manuals 1,2,3 –Datasheet APIC Cache

25. June 16, 2002 SPC Tutorial 25 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC-II Motivation –Faster Processor –More Memory –Faster Memory Bus –Simpler Real BIOS No System FPGA to build a fake Southbridge –Interchangeable modules 700 MHz already available Other low-power modules also available You can buy new/faster CPU modules if you want… More details to be given by Dave Richard on Tuesday at the Workshop

26. June 16, 2002 SPC Tutorial 26 Washington WASHINGTON UNIVERSITY IN ST LOUIS ETX Embedded Module P-III CPU North- Bridge VideoDRAM PCI Bus FPGA Video DB15 Link InterfaceSwitch Interface SPC-II Architecture South- Bridge BIOS ISA Bus I/O AGP Bus FLASH DISK IDE Bus Keyboard Mouse Serial Ports APIC

27. June 16, 2002 SPC Tutorial 27 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC-II Architecture TI IPPOPP SWITCH Link 256 MB SDRAM 500/700 MHz Pentium-III 256 KB L2 Cache Intel BX North Bridge Port 1 APIC Port 0 SPC-II 16/32 bit 32 bit PCI BUS South Bridge ISA Bus ISA Devices Super-IO BIOS FPGA FPX 16 bit 100 MHz Memory Bus

28. June 16, 2002 SPC Tutorial 28 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC-II Components –Intel Module 500 MHz Pentium III 256 KB L2 Cache Northbridge and Southbridge 33 MHz, 32 bit PCI Bus 100 MHz memory bus BIOS Video, Mouse, Keyboard –APIC –FPGA: for ATM level routing –Extra PCI Slot: Debugging possibilities. –256 MB of Memory –32 MB IDE DOM (Disk On Module) right now holds binary boot code could later hold actual kernel larger ones are possible

29. June 16, 2002 SPC Tutorial 29 Washington WASHINGTON UNIVERSITY IN ST LOUIS Booting an SPC-II IDE DOM contains boot code –modified Stage2 from AAL5_download configured to boot in place of a kernel –No special BIOS tricks to play contains APIC code to put kernel into memory kernel downloaded via AAL5 frames –uses Stage3 from AAL5_download kernel booted when download complete

30. June 16, 2002 SPC Tutorial 30 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC II FPGA Architecture FPXLC Switch APIC PCI Bus Port Port 1Port 0 SPC-II FPGA VPI[0]=1 VPI[0]=0 64<=VCI<=127??? Reset VPI[0]=1 VCI = 38 OSC

31. June 16, 2002 SPC Tutorial 31 Washington WASHINGTON UNIVERSITY IN ST LOUIS SPC-II References and more info APIC References the same as before Other references to come soon Dave Richard will talk more about SPC-II on Wednesday at the Workshop

32. June 16, 2002 SPC Tutorial 32 Washington WASHINGTON UNIVERSITY IN ST LOUIS The rest of the slides are a Gallery of SPC Photos