12/4/2002 Lab Board A description of the board you will do measurements on

2 12/4/2002Introduction Lab Board Description  Code name is ‘King’s Canyon’ Referred to hereafter as ‘KC’ You will do measurements on this board in the lab  Background for KC Low-end server board Inexpensive board for customers who want servers  Server Computer which serves data to others Supports more memory, more processing power, and more I/O (i.e., network bandwidth, hard drives) than desktop computers More reliable (crashes less and recovers better) than desktop computers

3 12/4/2002Introduction Server in a network

4 12/4/2002Introduction Inside the Server

12/4/2002 Circuit Board Block Diagram Front Side Bus (FSB) Hub Interface SS SS – Source Synchronous CC- Common Clock CC 4 PCI-X busses Low speed 33MHz PCI bus CPU 0 CPU 1 MCH Memory Two (2) DDR (Double Data Rate) Memory Busses ICH Video P64H2 HI to PCI bridge Details to Follow ….

12/4/2002 Components on KC ‘the chips’

7 12/4/2002Introduction CPU  CPUs –Central Processing Units The brains of the computer that ‘run’ your program Runs the Operating System (e.g. windows) and all the software Speed of CPU has large effect on speed of system Currently measured in GHz (e.g., 3.0 GHz) Consumer pc’s typically have 1 cpu inside the box – called ‘UP’ KC has 2 cpus – this is called ‘DP’, i.e. dual processor Some boards have 4+ cpus, they are called ‘MP’, i.e. multi-processor KC uses Intel Xeon CPU’s Currently at 3GHz Similar to Pentium 4  Bigger cache (superfast memory inside CPU)  Can operate with other CPUs on same bus The Silicon die (2) is under this metal heat spreader (1). The die is significantly smaller than the heat spreader. A ‘Fansink’ mounts to the heat spreader to keep die cool.

8 12/4/2002Introduction MCH  MCH – Memory Controller Hub Helps CPU communicate with rest of system The ‘gatekeeper’ to the outside world from perspective of CPU Talks to memory, CPU, and I/O  I/O means busses and chips and boards which ultimately connect to things like hard drives, networks, keyboards, monitor, etc.. Maintains memory tells it when to refresh, etc.. (dynamic memory has to be refreshed or it forgets the data it holds) Detects errors in what is stored in memory (ECC) and periodically corrects these errors KC uses Intel E7501 MCH Actual Die: Passive (no fan) Heatsink mounts on top. Package decoupling capacitors. Try to hold voltage rails (vcc) steady. Closer to the die the better.

9 12/4/2002Introduction ICH  ICH – Input/Output Controller Hub I/O device that deals with critical but slow speed devices  Video controller (basic 2-d video on servers, not meant for 3- d graphics)  Talks to Keyboard, mouse via ‘Super IO’ chip  BIOS (contains software which computer runs when it’s first powered on. BIOS gets computer up and running and then passes control to the OS)  IDE Hard drive (slow hard drive that OS boots off of. The big capacity hard drives do not connect to ICH) Helps with initial bootup of system Talks to mch over a slow version of the Hub Interface bus KC uses Intel 82801CA ICH Still uses wire-bond technology. Doesn’t require heatsink.

10 12/4/2002Introduction Memory Modules  Memory – High speed storage place for code and data used by CPU Much faster than hard drive storage, but not as massive in quantity 2-16 Gigabytes vs. 100’s of Gigabytes Slower than CPU cache (small amount of memory internal to CPU) Memory die’s are inside wire-bond packages called ‘DRAMs’ Dynamic Random Access Memory Many DRAMs solder to a memory board called the ‘DIMM’ Dual In-line Memory Module Anywhere from 9 to 36 DRAMs on one memory board 2 types of memory boards: Unbuffered and Registered. Unbuffered is used in desktop. KC (and most servers) only uses Registered DIMMs. Memory board plugs into KC board via a connector. KC can support up to 8 DIMMs Many different DIMM manufacturers Micron, Samsung, Elpida, Infineon, … DRAM 2 DIMMs

11 12/4/2002Introduction PCI-bridge  P64H2 – PCIX 64-bit to Hub Interface 2 bridge Gateway to High Speed I/O devices such as hard drive arrays adapters or ethernet adapters Talks to MCH on one side Via high speed variey of ‘Hub Interface’ Bus Talks to 2 separate pci-busses on other side Several connectors connect to each pci bus User can slide a card into each of these connectors Typically Hard Disk Controller Cards  Each Provides several SCSI or SATA connection to hard drive  Can end up with quite a few hard drives and large storage capacity Or Network Adapater Cards  Provide several 100Mb/s ethernet connections, which go to a switch or router  Servers tend to read data from the hard drive into memory, process the data with the CPUs, and then send it out to the requesting computer via the network (e.g., web server) ‘P64H2’ is the code name for the Intel 82870P2 P64H2 die, may require passive heatsink

12 12/4/2002Introduction Identifying parts on the board  CPUs there are two of them and they have really huge fansinks. NOTE: If the fansink doesn’t run, and the system is on, the heatsink will get VERY hot (burn skin) and system will eventually shut itself off.  Memory memory consists of 8 wide connectors all in a row. At least 2 DIMMs must be plugged in (‘populated’) for system to boot up. There are rules regarding what kind of dimms to use and which order to plug them into the connectors.  MCH It’s the chip between the cpus and the memory Probably has a passive heatsink (a small heatsink w/ no fan)  ICH & P64H2’s Hard to find, but it looks like the chip pictured in the above slide There is one ICH and 2 P64H2’s (second one not always populated, i.e. sometimes it’s missing)  PCI Slots Larger connectors on opposite side of board from DIMM connectors. One slot is from ICH and is slow speed legacy PCI bus (33MHz, 32-bit) The rest are high speed (66-133MHz, 64-bit – i.e. 64 data bits wide)

12/4/2002 Interconnects on KC ‘the wires that connect the chips together’

14 12/4/2002Introduction Front Side Bus (FSB)  Function: CPUs talk to each other and to the MCH. This is how the CPUs get data and code from memory, and communicate with rest of the system.  Technology: Source synchronous 533MBit/second/wire (266MHz signals) This means each data wire on this bus can send 533 million bits of data in one second 64 data bits wide So the whole bus can transfer 64*533 million bits of data per second Topology: Multi-drop bus. CPU1 has to listen to when CPU0 acceses memory because CPU1 may have that memory data in it’s cache ‘Cache Coherency’ Each cpu has a small superfast memory inside it called the ‘cpu cache’. The cache holds recently accessed memory data. If one cpu holds the data for a memory address that another cpu needs, it has to let it know not to get that data from main memory. Termination: Parallel termination at both ends of bus (MCH and CPU0)

15 12/4/2002Introduction FSB Topology

16 12/4/2002Introduction FSB Signal Groups  Data Signal Group HD[63:0]# – 64 bits of data DINV[3:0]# - dynamic inversion HDSTBP[3:0]# - P strobes HDSTBN[3:0]# - N strobes Example, HDSTBP[0]# & HDSTBN[0]# are used to strobe in HD[0- 16]# & DINV[0]#  E.g., two strobe signals are used to clock in a set of data bits, more on this later  Address Signal Group Switches at ½ the rate of the FSB Data Signals HAB[35:3]# - Address Bits. E.g., what memory address is being requested HADSTB[1:0]# - Strobes for address signal group HADSTB[0]# strobes A[16:3]#, HADSTB[1]# strobes the rest  Common Clock Signals There a whole lot of common clock signals HCLKINP, HCLKINN – Common clock for all signals in the ‘Host Clock Domain’, i.e. all common clock signals, and outer loop timings for strobes (also called BCLK)

17 12/4/2002Introduction FSB Timings – 4X Source Synchronous Timings not published for 533 MHz FSB, 400 MHz timing diagrams shown

18 12/4/2002Introduction FSB Timings – 2X Source Synchronous Timings not published for 533 MHz FSB, 400 MHz timing diagrams shown

19 12/4/2002Introduction DDR (Double Data Rate) Memory Bus  Function: MCH reads and writes data from memory over this high speed bus.  Signaling Technology: Source Synchronous 266Mbit/sec/wire (133Mhz signals) A DDR bus is 64 data bits wide KC’s MCH uses 2 DDR busses in parallel, so it likes having a 128-bit wide bus  Bandwidth is 128*266 Million bits per second Topology:Multi-Drop Bus. Each of the two separate instances of this bus on this system go to four (4) DIMM slots.  Each DIMM can have up to 2GBytes of memory, for a max total of 16GB. Topology and Timings The subject of the next class

20 12/4/2002Introduction Memory speed and size within a system When we study DDR signaling and timings, it will be clear the overhead and latency that main memory exhibits, and thus the need for local L1-L3 CPU cache. Hard Drives

21 12/4/2002Introduction HI (Hub Interface) Bus  Function: Intel proprietary bus to provide flexible high speed data transfer from MCH to various downstream components  Signaling Technology: Source Synchronous HI1: From MCH to ICH: 266Mbit/sec/wire (133Mhz signals) 8 data bits wide, bandwidth = 8*266 Million bits per second HI2: From MCH to P64H2’s: 533Mbit/sec/wire (266Mhz signals) 16 data bits wide, bandwidth = 16*533 Million bits per second Topology: Point to Point. Termination: There are parallel terminations at both ends of the bus

22 12/4/2002Introduction Hub Interface Topology

23 12/4/2002Introduction Hub Interface Signal Groups  Data Bits HI[15:0] – Data Signals PSTRB_[1:0], PSTRB_[1:0]# – Strobe Signals PSTRB_[0] & PSTRB_[0]# strobe in HI[0:7] PSTRB_[1] & PSTRB_[1]# strobe in HI[15:8]  Other HI[18:16] – Command Signals Common Clock signals  GCLKIN 66MHz common clock for HI[18:16] and outer loops of the PSTRB’s  Timings not published for Hub Interface

24 12/4/2002Introduction PCI and PCI-X  Function: Industry standard bus for plug-in peripherals In servers, quite often plug-in hard drive adapters or network adapters are connected to the pci bus  Technology: Common Clock PCI-X: From P64H2 – 133Mbit/sec/wire, clocks are 133MHz signals, data is 66MHz 64 data bits wide, bandwidth = 64*133 Million bits per second PCI: From ICH – 33Mbit/sec/sire, clocks are 33MHz, data is 16MHz 32 data bits wide, bandwidth = 32*33 Million bits per second This speed and width are common in current desktop pc pci slots Topology: Multi-Drop Bus. The bus consists of devices soldered down on the board and slots for plug-in devices. They all dangle off the PCI bus. Termination: Series terminated inside the transmitting agent. This signaling technolgy depends on a doubling of the waveform at the end of the bus (due to open-circuit) in order to function.

25 12/4/2002Introduction PCIX slot Topology x x x

26 12/4/2002Introduction PCI-X Signal Groups  Data and Address use same signals: AD[63:0] Common Clocked  Control Signals DEVSEL, TRDY, … a bunch more Common Clocked  Asynchronous Signals REQ, GNT – Arbitration signals Interrupt Signals  Clocks PCLK[6:0] – goes to every device that hangs off the pci bus including the bridge itself

27 12/4/2002Introduction PCI-X Data to Common Clock Timings ns

28 12/4/2002Introduction PCI-X Common Clock

29 12/4/2002Introduction KC Clock Distribution

30 12/4/2002Introduction Magic Decoder Ring  Names for Signals given so far are from the device sheets  KC Board uses it’s own names Board’s have to name signals clearly and uniquely across the whole board Devices only have to name then uniquely across the device  Decoder Ring: FSB: HD[63:0]#  FSB_HD63_N Rule is append FSB_ in front and replace ‘#’ with ‘_N’ in the device signal name HI: HI[15:0]  P64H2_1_HI[15:0] Rule is append P64H2_1_ before the device signal name Note that the ‘1’ in P64H2_1_ may be a ‘1’ or a ‘2’ because KC has two P64H2 devices PCI-X: AD[63:0]  P64H2_1_PB_AD[63:0] Rule is append P64H2_1_PB to beginning of device signal name Note that the ‘1’ in P64H2_1_PD.. Can be a ‘2’ or ‘3’ … depending on which pci bus the signal belongs too DDR: Given Later