Dezső Sima September 2008 (Ver. 1.0)  Sima Dezső, Overall design space of main memories

Contents 1. Design space of MMs 2. Underlying principles of the implementation of MMs 3. Performance considerations 4. References

Underlying principle of operation 1. Design space of MMs () Instruction Set Architecture (ISA) Micro- architecture Underlying principles of implementation Principles of attaching memory and I/O Von Neumann computational model Figure: Design space of processors

Underlying principle of operation 1. Design space of MMs () Control Set Architecture (CSA) Micro- architecture of the MM Underlying principles of implementation Figure: Design space of main memories (MM)

1. Design space of MMs () Figure: Underlying principle of operation of DRAM devices Underlying principle of operation Basic operation Refreshing (not discussed)

1. Design space of MMs () Basic operation of DRAM devices (Assuming device/bank/row/column addressing) Figure: Basic operation of DRAM devices ReadsActivateReadPrecharge C: Command AD: Device address AB: Bank address AR: Raw address AC: Column address Read data (RD) C AD AB AR C AD AB AC C AD AR AB t t RCD t CL Activate C AD AB AR t RP WritesActivateWritePrecharge Write data (WD) C AD AB AR C AD AB AC C AD AR AB t t RCD t CL t WR Activate C AD AB AR t RP

2. Underlying principles of the implementation of MMs () Underlying principles of the implementation of MMs One/two level implementation Managing the DRAM status Principle of communication Bus topology Type of signaling Type of synchronisation Figure: Main dimensions of the design space of the underlying principles of implementation of MMs Signal grouping for communication

2. Underlying principles of the implementation of MMs () One/two-level implementation One-level implementation Two-level implementation MM is built up of DRAM devices Figure: One/two level implementation of main memories

Figure: One level implementaqtion of the main memory (XDR memory of the Playstation 3 [1] 2. Underlying principles of the implementation of MMs ()

One/two-level implementation One-level implementation Two-level implementation MM is built up of DRAM devices MM is built up of modules, modules consist of DRAM devices Figure: One/two level implementation of main memories

Figure: Two level memory implementation (DDR2 modules on an MSI motherboard [2] 2. Underlying principles of the implementation of MMs ()

One/two-level implementation One-level implementation Two-level implementation MM is built up of DRAM devices MM is built up of modules, modules consist of DRAM devices Type of mounting Expandability Board space needed Signal integrity Typically solderedTypically socketed Not expandable Easily expandable Large boardspace Small boardspace Good signal integrityUnfavorable signal integrity Figure: One/two level implementation of main memories (Earliest PC main memories) XDR memories All other types of main memories E.g.

2. Underlying principles of the implementation of MMs () All other types of main memories (via a second dedicated interface) RDRAM XDR Figure: Options to manage DRAM status This dimension of the design space is not discussed. Managing DRAM status Along with the basic operation Detached from the basic operation

2. Underlying principles of the implementation of MMs () Figure: Principles of communication used in main memories Principle of communication Parallel bus based Packet-based Signals are transferred over a parallel bus in one cycle Signals are transferred over a serial bus in a number of cycles 01 E.g: 16 cycles/packet on a 1-bit wide bus 11 E.g: 4 cycles/packet on a 4-bit wide bus 01 MC t t t E.g: 64 bits in each cycle

2. Underlying principles of the implementation of MMs () Signal grouping for communication Line multiplexingPacket concept In case of parallel bus based transmission In case of packet based transmission

2. Underlying principles of the implementation of MMs () are unidiredctional (they flow in one direction, from the MC to the MM) Data Due to the basic operation Commands and addresses row and column addresses are not used at the same time, that is they may be multiplexed without any performance penalty. Line multiplexing In order to avoid performance impediments data are transferred on a private bus, instead of being multiplexed with commands and addresses. is bidirectional (read data flow from the MC to the MM, write data from the MM to the MC). Assumptions for line multiplexing Additional addresses needed (such as device or bank addresses) are transferred. along with row and column addresses. Read and write data may be multiplexed to reduce cost with a low performance penalty.

2. Underlying principles of the implementation of MMs () Row address/Column address multiplexing Multiplexed Not multiplexed Multiplexed (unidirectional) Read/write data multiplexing Not multiplexed (bi-directional) Asyncr. DRAMs (from the MK4096 on) Synchr. SDRAMs First asynchr. DRAMs (before Mostel’s MK4086) Figure: Multiplexing row and column addresses vs read and write data Multiplexing row addresses/column addresses vs read and write data Line multiplexing

2. Underlying principles of the implementation of MMs () Packet concept Different packet concepts for RDRAM, XDR and FB-DIMM memories.

Figure: RDRAM: Different packet types for Activate/Precharge and Read/Write commands [9] 2. Underlying principles of the implementation of MMs () Activate Read The packet concept of RDRAM memories (1)

Data packets over the bidirectional data bus (DQA/DQB) Row packets over the ROW bus Column packets over the COL bus CR R/W packets over the Serial bus Figure: RDRAM: Memory access packets [10] Row packets Column packets Data packets CR: Control Register R/W: Read/Write 2. Underlying principles of the implementation of MMs () The packet concept of RDRAM memories (2)

2. Underlying principles of the implementation of MMs () Packet concept Different packet concepts for RDRAM, XDR and FB-DIMM memories. RDRAM Different packet types (transferred via different buses) for Activate/Precharge and Read/Write commands Bidirectional data packets Control register read/write packets.

Figure: The packet concept of XDR memories [11] Data packets Request packets CR R/W packets 2. Underlying principles of the implementation of MMs () The packet concept of XDR memories

2. Underlying principles of the implementation of MMs () Packet concept Different packet concepts for RDRAM, XDR and FB-DIMM memories. RDRAM Different packet types (transferred via different buses) for Activate/Precharge and Read/Write commands Bidirectional data packets Control register read/write packets. XDR Unified packet type for memory accesses (called the Request packets) (Activate/Precharge and Read/Write commands) Bidirectional data packets Control register read/write packets.

Southbound packets Memory controller M. module Commands Write data M. module Northbound packets Read data Status Figure: The packet concept of FB-DIMM memories 2. Underlying principles of the implementation of MMs () The packet concept of FB-DIMM memories

2. Underlying principles of the implementation of MMs () Packet concept Different packet concepts for RDRAM, XDR and FB-DIMM memories. RDRAM Different packet types (transferred via different buses) for Activate/Precharge and Read/Write commands Bidirectional data packets Control register read/write packets. XDR Unified packet type for memory accesses (called the Request packets) (Activate/Precharge and Read/Write commands) Bidirectional data packets Control register read/write packets. FB_DIMM Unified packet type for all commands (called the Southbound packets) including memory accesses and control register reads/writes, containing up to 3 commands, or write data and a single command. Unidirectional read data (and status) packets (called the Northbound packets)

2. Underlying principles of the implementation of MMs () Bus topology Multi-drop busPoint-to-point connection Stub-busFly-byDaisy-chained Figure: Bus topologies used to connect DRAM devices or modules to the memory controller Allows to connect more than one devices/modules to the bus Allows to interconnect two units (e.g. a mem. controller and a module) Connection via slots (sockets) E.g. MC DIMMDIMM DIMMDIMM

Figure: Stub bus topology [3] 2. Underlying principles of the implementation of MMs ()

Bus topology Multi-drop busPoint-to-point connection Stub-busFly-byDaisy-chained Figure: Bus topologies used to connect DRAM devices or modules to the memory controller Allows to connect more than one devices/modules to the bus Allows to interconnect two units (e.g. a mem. controller and a module) Connection via slots (sockets) Connection via soldering E.g. MC DIMMDIMM DIMMDIMM DRAMDRAM DRAMDRAM

Figure: Fly-by topology of the RQ bus in a two-channel XDR memory with two XDR devices/channel [5] Fly-by topology 2. Underlying principles of the implementation of MMs ()

Bus topology Multi-drop busPoint-to-point connection Stub-busFly-byDaisy-chained Figure: Bus topologies used to connect DRAM devices or modules to the memory controller Allows to connect more than one devices/modules to the bus Allows to interconnect two units (e.g. a mem. controller and a module) Connection via slots (sockets) Connection via soldering Connecting units to each other (outputs to inputs) E.g. MC DIMMDIMM DIMMDIMM DRAMDRAM DRAMDRAM DIMMDIMM DIMMDIMM

(There are two Command/Address buses (C/A) to reduce loading coming from 9 to 36 DRAMs mounted on the module) Figure: Daisy chained topology of connecting AMBs in FB-DIMM memories [4] 2. Underlying principles of the implementation of MMs ()

Bus topology Multi-drop busPoint-to-point connection Stub-busFly-byDaisy-chained Figure: Bus topologies used to connect DRAM devices or modules to the memory controller Allows to connect more than one devices/modules to the bus Allows to interconnect two units (e.g. a mem. controller and a module) Connection via slots (sockets) Connection via soldering Connecting units to each other (outputs to inputs) E.g. MC DIMMDIMM DIMMDIMM DRAMDRAM DRAMDRAM DIMMDIMM DIMMDIMM DRAMDRAM DRAMDRAM

Figure: Point-to point topology of the data bus (DQ) in a two-channel XDR memory with two XDR devices/channel [5] Point-to-point 2. Underlying principles of the implementation of MMs ()

Bus topology Multi-drop busPoint-to-point bus Stub-busFly-by busDaisy-chained bus Connecting DRAM devices to the MC MC DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM Figure: Overview of bus topologies connecting DRAM devices to the memory controller MC DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM Used inVery early PCs RDRAMs (except the Serial bus) XDR/XDR2 (memory requests, control register reads/writes) XDR/XDR2 (read/write data)

2. Underlying principles of the implementation of MMs () Figure: Overview of bus topologies connecting DRAM modules to the memory controller MC DIMMDIMM DIMMDIMM Used in Parallel connected main memories RIMMs (with fly-by device connection on the module) MC (FPM/EDO/SDRAM, DDR/DDR2/DDR3) Bus topology Multi-drop busPoint-to-point bus Stub-busFly-by busDaisy-chained bus Connecting DIMMs to the MC

2. Underlying principles of the implementation of MMs () Figure: Contrasting the interconnection of RIMM modules with that of DIMMs [12]

2. Underlying principles of the implementation of MMs () Connecting DIMMs to the MC MC DIMMDIMM Figure: Overview of bus topologies connecting DRAM modules to the memory controller MC DIMMDIMM DIMMDIMM DIMMDIMM DIMMDIMM Used in FB-DIMMs (to connect AMBs) Parallel connected main memories MC DIMMDIMM DIMMDIMM RIMMs (with fly-by device connection on the module) MC (FPM/EDO/SDRAM, DDR/DDR2/DDR3) Not feasible Bus topology Multi-drop busPoint-to-point bus Stub-busFly-by busDaisy-chained bus

2. Underlying principles of the implementation of MMs () Figure: Assessing bus topologies connecting DRAM devices/modules to the memory controller Bus topology Multi-drop busPoint-to-point bus Stub-busFly-by busDaisy-chained bus Attaching DRAM devices to the MC Attaching DIMMs to the MC Unfavorable (due to TL discontinuities) BetterGoodExcellent Signal integrity Peak transfer rate (recently) Up to 16 Gb/s (with increasingly sophisticated termination) Up to 4.8 Gb/s MC DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DIMMDIMM DIMMDIMM DRAMDRAM DRAMDRAM DIMMDIMM DIMMDIMM

Bus topologies of parallel connected synchronous MMs (Summary 1) 2. Underlying principles of the implementation of MMs () Synchronous DRAMs (except DDR3) read/write data commands addresses Stub bus DDR3 read/write data commands addresses Stub bus Fly-by DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0] DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0] I/O I I/O I Bus topology Bus designation Buses

Bus topologies of serial connected MMs (Summary 2) XDR read/write data memory requests) control register (CR) reads control register (CR) writes FB-DIMM (AMBs - memory controller) read data/device status memory requests/ write data/CR reads or writes Bus topology Point-to-point Fly-by Daisy-chained DQ [15:0] RQ [11:0] SDI SDO PN [13:0] PS [9:0] 2. Underlying principles of the implementation of MMs () Bus designation Read/write data Row bus Column bus Serial if. CMD, SIO1, SIO0 Fly-by Daisy-chained Buses RDRAM DQA [8:0], DQB [8:0] ROW [2:0] COL [4:0] CMD SIOI, SIOO I/O I I/O I/O I O I O I

2. Underlying principles of the implementation of MMs () Figure: Bus topologies of current MMs to connect DRAM devices or modules to the memory controller Address/control bus Multi-drop bus Point-to-point Stub-busFly-by busDaisy-chained Data bus Multi-drop bus P2P Stub-bus Fly-by Daisy-chnd SDRAM DDR DDR2 (modules) Devices on the modules DDR3 RDRAM (devices, modules) XDR XDR2 (devices) FB-DIMM (AMBs on modules) TBI (devices)

2. Underlying principles of the implementation of MMs () Figure: Signal types used in MMs for control, address and data signals Signals Voltage referenced Single ended Differential LVDS: Low Voltage Differential Signaling LVTTL: Low Voltage TTL (D)RSL: (Differential) Rambus Signaling Level SSTL: Stub Series Terminated Logic V CM : Common Mode Voltage V REF : Reference Voltage t t V REF LVTTL (3.3 V) FPM/EDO SDRAM TTL (5 V) FPM/EDO SSTL SSTL2 (DDR) SSTL1.8 (DDR2) SSTL1.5 (DDR3) RSL (RDRAM) LVDS FB-DIMMs t S+ S- V CM Smaller voltage swings Typ.voltage swings mV DRSL XDR (data) mV V Used in

shorter signal rise/fall times higher speed grades but lower voltage budgethigher requirements for signal integrity 2. Underlying principles of the implementation of MMs () Smaller voltage swings Q = C in x V = I x t t R ~ C in x V/I Q: Charge on the input capacitance of the line (C in ) C in : Input capacitance of the line V: Voltage I: Current strength of the driver t R : Rise time Voltage swing vs signal rise/fall time

Bus topologies and signaling of parallel connected MMs (Summary 1) 2. Underlying principles of the implementation of MMs () Synchronous DRAMs (except DDR3) read/write data commands addresses Stub bus DDR3 read/write data commands addresses Stub bus Fly-by DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0] DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0] I/O I I/O I Bus topology Bus designation Buses Signaling Volt. ref.

Bus topologies and signaling of serial connected MMs (Summary 2) XDR read/write data memory requests) control register (CR) reads control register (CR) writes FB-DIMM (AMBs - memory controller) read data/device status memory requests/ write data/CR reads or writes Bus topology Point-to-point Fly-by Daisy-chained DQ [15:0] RQ [11:0] SDI SDO PN [13:0] PS [9:0] 2. Underlying principles of the implementation of MMs () Bus designation Read/write data Row bus Column bus Serial if. CMD, SIO1, SIO0 Fly-by Daisy-chained Buses RDRAM DQA [8:0], DQB [8:0] ROW [2:0] COL [4:0] CMD SIOI, SIOO I/O I I/O I/O I O I OIOI Differential Volt. ref. Differential Signaling Volt ref. Volt. ref. CMOS Volt. ref.

2. Underlying principles of the implementation of MMs () Capturing control/address information Central synchronization Source synchronization Mesochronous synchronization The sourcing device (MC or DRAM) sends a strobe signal along with the signals sent A central clock signal is used to latch the signals Keeps the clock frequency between the sender reference clock and the receiver reference clock but not the phase relationship. Synchronisation

Figure: Central clocking of address, command and data lines in an SDRAM device while writing random data [6] Address, command and data lines are latched by the rising edge of the central clock (CLK) Central clocking (SDRAM) 2. Underlying principles of the implementation of MMs ()

Figure: Source synchronous clocking (DDR SDRAMs) of the data lines in a DDR device while writing random data [7] (T DOSS : Write command to first DQS latching transition) Command and address lines are latched by the differential clock (CK, CK#) but write data are latched by the rising edge of the source synchronous data strobe (DQS) Source synchronous clocking of the data lines (DDR) 2. Underlying principles of the implementation of MMs ()

Mesochronous clocking (FB-DIMM) Figure: Mesochronous clocking used to synchronise AMBs in FB-DIMM memories [8] 2. Underlying principles of the implementation of MMs ()

Figure: Synchronisation alternatives SDRAM RDRAM XDR 3 XDR2 3 Capturing control/address information Central synchronization Source synchronization Mesochronous synchronization Capturing data Source synch. Mesochron. synch. FBDIMM Central synch. DDR 1 DDR2 2 DDR3 2 1 : Phase alignement for data reads/writes 2 : Phase alignement for data reads/writes by read/write leveling 3 : Phase alignement for all signals by FlexPhase DDRX?

3. Performance considerations (1) Figure: Peak memory size vs peak bandwidth (BW) of particular DRAM technologies in Intel’s desktop chipsets

, BW GB/s Mem. Size GB QS21 (2D) x P4 Servers, QS (2) x P4 Servers x x 7520 (2) 7501 (2) QS22 (2) (2D) x 7520 (2) x x 840 (2) x 7300 (4) FB-DIMM DDR-2 Core 2 Servers, T2 DDR2 (reg) DDR (reg) RDRAM P4 Servers XDR Servers x 860 (2) x QS20 (2D) ,75 1,061,63,2 4,2 6,48,510,612,821,225,6 Figure: Peak memory size vs peak bandwidth (BW) of particular DRAM technologies in Intel’s server chipsets, IBM’s QS2x blades and Sun’s T2 x SunT2 (4D) 50 51,2 3. Performance considerations (2)

4. References References [1]: Yeung P., „Solving System Engineering Challenges in High Speed Memory Designs,” Rambus Design Seminar, Feb , [2]: [3]: Reddy A., „XDR and XDR2 Overview,”, RDF, Oct , Taiwan, [4]: McTague M. & David H., „ Fully Buffered DIMM (FB-DIMM) Design Considerations,” Febr. 18, 2004, Intel Developer Forum, [5]: Yoshitomi Y., „Elpida DRAM Solutions to Advanced Digital Consumer Electronic Systems,” Rambus Design Seminar, Hsinchu, Elpida, June , [6]: Micron Synchronous DRAM, 64 Mbit, MT48LC16M4A2, MT48LC16M8A2, MT48LC16M16A2, Micron Technology, Inc. Oct [7]: Double Data Rate (DDR) SDRAM MT46V128M4, MT46V64M8, MT46V32M16, Micron Techn. Inc, 2000, [8]: FBDIMM Specification: High Speed Differential Point-to-Point Link at 1.5 V, JESD8-18, Sept. 2006, JEDEC

4. References [9]: Direct Rambus Architecture and Measurements, MindShare, [10]: Crisp R., Direct Rambus Technology: The New Main Memory Standard,”, IEEE Micro, Nov./Dez. 1997, pp [11]: Ishikawa T., „Elpida XDRAM,”, RDF Oct , Taiwan [12]: DeMone P., „Direct Rambus Memory,” Real Word Technologies, ,