1. Dezső Sima September 2008 (Ver. 1.0)  Sima Dezső, 2008 3. Overall design space of main memories

2. Underlying principle of operation Instruction Set Architecture (ISA) Micro- architecture Underlying principles of implementation Principles of attaching memory and I/O Von Neumann computational model Figure: Design Space of processors

3. Underlying principle of operation Control Set Architecture (CSA) Micro- architecture of the MM Underlying principles of implementation Figure: Design Space of main memories (MM)

4. Figure: Underlying principle of operation of DRAM devices Underlying principle of operation Basic operation Refreshing (not discussed)

5. Basic operation of DRAM devices (Assuming device/bank/row/column addressing) ReadsActivateReadPrecharge C: Command AD: Device address AB: Bank address AR: Raw address AC: Column address Basic operation of DRAM devices Read data (RD) C AD AB AR C AD AB AC C AD AR AB t t RAC WritesActivateWritePrecharge Write data (WD) C AD AB AR C AD AB AC C AD AR AB t t RAC

6. Underlying principles of the implementation of MMs One/two level implementation Managing the DRAM status Multiplexing commands, addresses and data Principle of communication Bus topology Bus width Type of signaling Type of synchronisation Figure: Main dimensions of the design space of the underlying principles of implementation of MMs

7. One/two-level implementation One-level implementation Two-level implementation MM is built up of DRAM devices MM is built up of modules, modules are built up of DRAM devices Type of mounting Expandability Board space requirement Signal integrity Typically solderedTypically socketed Not expandable Easily expandable Large boardspace Small boardspace Good signal integrityUnfavorable signal integrity Figure: One/two level implementation of main memories (Earliest PC main memories) XDR memories All other types of main memories E.g.

8. All other types of main memories (via a second dedicated interface) RDRAM XDR Figure: Options to manage DRAM status This dimension of the design space is not discussed. Managing DRAM status Along with the basic operation Detached from the basic operation

9. are unidiredctional (they flow in one direction, from the MC to the MM) they are transferred on the same communication principle b)Data is bidirectional (read data flow from the MC to the MM, write data from the MM to the MC) is transferred separately from the addresses/commands Assumptions for multiplexing commands, addresses and data Commands and addresses

10. AR/AC multiplexing AR/AC multiplexed AR/AC separate DW/DR multiplexed DW/DR separate (unidirectional) DW/DR Not multiplexed (bi-directional) DRAM (asynchr.) (from the MK4096 on) Synchr. SDRAMs First DRAMs (before the MK4086) Figure: Multiplexing row and column addresses (AR/AC) vs read and write data (RA/WA)

11. Figure: Principles of communication used in main memories Principle of communication Via a parallel bus in a single cycle Packet-based in a number of cycles 0 01 E.g: 16 cycles 11 Packet transfer over a one bit wide data path E.g: 4 cycles 01 1 cycle 0101

12. Figure: Principles of communication used in main memories Principle of communication Via a parallel bus in a single cycle Packet-based in a number of cycles 01 E.g: 16 cycles 11 Packet transfer over a one bit wide data path E.g: 4 cycles 01 MC t t t 1 0 1 1 0 0

13. Bus topology Multi-dropPoint-to-point Stub-busFly-byDaisy-chained Attaching DRAM devices (soldered) Attaching DIMMs Unfavorable (due to TL discontinuities) BetterGoodExcellent Signal integrity Peak transfer rate (recently) Up to 16 Gb/s (with increasingly sophisticated termination) Up to 4.8 Gb/s Up to --- Gb/s MC DIMMDIMM DIMMDIMM DIMMDIMM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM (socketed) Figure: Bus topologies used to connect RQAM devices or modules to the memory controller MC DRAMDRAM DRAMDRAM DIMMDIMM DIMMDIMM

14. Bus width Parallel busPentium 32 64 Serial bus Width of serial bus Transmission Parallel Parallel-based

15. Data bus Multi-drop Point-to-point Stub-busFly-byDaisy-chained Address/control bus Multi-drop P2P Stub-bus Fly-by Daisy-chnd Modules SDRAM DDR DDR2 Devices on the module DDR3 Devices RDRAM Devices ? Devices XDR XDR2 Modules FB-DIMM Figure: Bus topologies used to attach DRAM devices or DIMMs TBI

16. SDRAM DDR/2/3 CRDRAM RDRAM XDR ? XDR2 ? TBI ? Figure: Synchronisation alternatives Capturing control/address information Central synchronization Source synchronization Mesochronous synchronization Capturing control/address information Central synch. Source synch. Mesochron. synch.

17. Signals Voltage referenced Open ended Differential LVTTL: Low Voltage TTL LVDS: Low Voltage Differential Signaling HVDS: High Voltage Differential Signaling SSTL: Stub Series Terminated Logic V REF : Reference Voltage V CM : Common Mode Voltage t t V REF LVTTL (3.3 V) SDRAM PCI PCI-X AGP1.0 TTL (5 V) PCI SSTL SSTL2 (DDR) SSTL1.8 (DDR2) SSTL1.5 (DDR3) AGP2.0 (1.5 V) AGP3.0 (0.8 V) LVDS Hypertransport SATA Ultra-2 SCSI and later PCI-E HVDS SCSI-1 t S+ S- V CM Higher data rates Figure: Different kinds of signals used in buses or interfaces

18. DevicesRIMMs Aimed at: Desktop (PIII/P4) Consumer (PS3) Servers (QS20/21) Servers (Intel’s 5000/7000, Sun’s Niagara II) Produced by Samsung Toshiba Qimonda Elpida Samsung Elpida Qimonda Nanya Hynix Figure: Use and production of serial connected DRAMs Serial connected XDRs RDRAMs FB-DIMMs Devices DIMMs Consumer (PS2) Micron

19. MM-Overall DS

21. Principle of operation (1) data, memory requests and control register (CR) read/writes. Data bus: DQ/N [15:0] Request bus: RQ [11:0] CR reads/writes: serial bus (SCK, CMD, SDI, SDO, RST) The set of buses defined Own buses for Designation of the buses

22. Comparison: The set and direction of buses defined in major memory types XDR read/write data (I/O), DQ [15:0] memory requests (I) and RQ [11:0] control register (CR) reads (O), SDI control register (CR) writes (I). SDO FB-DIMM (between the AMBs and the memory controller) read data/device status (O), PN [13:0] memory requests/write data/CR reads or writes (I) PS [9:0] (Direction is interpreted from the point of view of the memory device/module) Synchronous DRAMs read/write data (I/O), DQ [3:0/7:0/15:0] commands (set of individual command lines (I) CS, RAS. CAS, WE addresses (bank address/address within a bank) (I) BA [7:0], A [N:0]

23. Point-to-point topology for the data bus, Fly-by topology for the system clock, request bus and serial bus. Principle of operation (1) Topology of the buses interconnecting the memory controller and the XDR devices

24. 1/1 Point-to-point topology for the data bus rather than a multidrop or daisy chained topology. Principle of operation (1) Data packets Control packets CC R/W packets Point-to-point data bus Data from only one device can be accessed Small memory size Good signal integrity High data rate of 3.2...4.8 Gb/s Figure: Point-to-point implementation of the data bus [4]

25. Figure: Implementation of a two-channel XDR memory with two XDR devices/channel [6] Point-to-point

26. XDR Figure: Contrasting the point-to-point and daisy chained bus implementations of the data bus [4] Data packets Control packets CC R/W packets FB-DIMM Memory controller M. module Point-to-point connection Daisy chained connection Data from multiple modules can be accessed High memory size Good signal integrity Data from only one device can be accessed Small memory size Good signal integrity High data rate of 3.2...4.8 Gb/s High data rate of 3.2...4.8 Gb/s

27. (There are two Command/Address buses (C/A) to reduce loading coming from 9 to 36 DRAMs mounted on the module) Figure: Daisy chained connection of the AMBs in FB-dIMMs [7]

28. Note Concerning the point of termination the daisy chained connection appears like a point-to-point connection, since in this case the controller „sees” only the first memory device/module whereas further devices/modules are hidden from the controller via the repeater chain feature of the daisy chain topology and vice versa.

29. ODT terninated DQ, DQS/#, DM signals Flying-by Command-, Address-, Control-, and CK, CK# signals Point-to-point connection Stub-bus connection Data from only one device can be accessed Good signal integrity Small memory size High data rate of 3.2...4.8 Gb/s Large memory size Low data rate of 0.8...1.6 Gb/s Data from multiple modules can be accessed Unfavourable signal integrity Figure: Contrasting the point-to-point and multidrop bus implementations of the data bus

30. Stub bus topology

31. Principle of operation (1/2) request and CR read/write buses Request bus: RQ [11:0] CR reads/writes: SDI, SDO Fly-by topology for the

32. Comparison: Bus topologies chosen for the major memory types XDR read/write data (I/O) memory requests (I) control register (CR) reads (O) control register (CR) writes (I) FB-DIMM (AMBs - memory controller) read data/device status (O) memory requests/ write data/CR reads or writes (I) Synchronous DRAMs (except DDR3) read/write data (I/O) commands (I) addresses (I) Bus topology Point-to-point Fly-by Daisy-chained Stub bus DDR3 read/write data (I/O) commands (I) addresses (I) Stub bus Fly-by DQ [15:0] RQ [11:0] SDI SDO PN [13:0] PS [9:0] DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0] DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0]

33. XDR read/write data (I/O), DQ [15:0] memory requests (I) and RQ [11:0] control register (CR) reads (O) SDI control register (CR) writes (I) SDO Bus topology Point-to-point Fly-by Signaling Differetial Volt. ref.

35. Comparison: Signaling chosen for the major memory types DQ [15:0] RQ [11:0] SDI SDO PN [13:0] PS [9:0] DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0] DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0]

36. Comparison: Signaling chosen for the major memory types XDR read/write data (I/O), DQ [15:0] memory requests (I) and RQ [11:0] control register (CR) reads (O) SDI control register (CR) writ(I) SDO FB-DIMM (AMBs - memory controller) read data/device status (O) PN [13:0] memory requests/ PS [9:0] write data/CR reads or writes (I) Synchronous DRAMs (except DDR3) read/write data (I/O) DQ [3:0/7:0/15:0] commands (I) CS, RAS. CAS, WE addresses (I) BA [7:0], A [N:0] Bus topology Point-to-point Fly-by Daisy-chained Stub bus DDR3 read/write data (I/O) DQ [3:0/7:0/15:0] commands (I) CS, RAS. CAS, WE addresses (I) BA [7:0], A [N:0] Stub bus Fly-by Signaling Differential Volt. ref. Differential Volt. ref.

37. Comparison: Signaling in the major memory types XDR read/write data (I/O) memory requests (I) control register (CR) reads (O) control register (CR) writes (I) FB-DIMM (AMBs - memory controller) read data/device status (O) memory requests/ write data/CR reads or writes (I) Synchronous DRAMs (except DDR3) read/write data (I/O) commands (I) addresses (I) Bus topology Point-to-point Fly-by Daisy-chained Stub bus DDR3 read/write data (I/O) commands (I) addresses (I) Stub bus Fly-by DQ [15:0] RQ [11:0] SDI SDO PN [13:0] PS [ 9:0] DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0] DQ [3:0/7:0/15:0] CS, RAS. CAS, WE BA [7:0], A [N:0] Signaling Differential Volt. ref. Differential Volt. ref. Buses

38. ODT terninated DQ, DQS/#, DM signals Flying-by Command-, Address-, Control-, and CK, CK# signals Figure: Contrasting communication and synchronisation in XDR and DDR3 memories [4], [9] XDR DDR3 Data PTP, differential (DRSL) Bus, voltage ref. (SSTL) Comm./Addr. Bus, fly-by, volt. ref. (RSL) Bus, fly-by, volt. ref. (SSTL) Clock Fly-by, diff. (DRSL)Fly-by, diff. ( diff. SSTL) Synchron. FlexPhase Read/write leveling Contr. reg. manip. Serial 1-bit, volt. ref. (RSL) n.a. Comm. principle Packet based Parallel bus based Signaling

39. Figure: Principle of operation [4] Principle of operation (2) Packet based communication between the memory controller and the XDR devices (like in FB-DIMM modules) Data packets over the DQ/N lines Request packets over the RQ lines CC R/W packets over the serial if. Interface lines DQ/N [15:0]: Data lines RQ [11:0]: Request lines CFM/N: Clock From Master SCK... Serial interface Packets CC: Control Register R/W: Read/Write /N: Negative signal Data packets Request packets CC R/W packets

40. ODT terninated DQ, DQS/#, DM signals Flying-by Command-, Address-, Control-, and CK, CK# signals Figure: Contrasting communication and synchronisation in XDR and DDR3 memories [4], [9] XDR DDR3 Data PTP, differential (DRSL) Bus, voltage ref. (SSTL) Comm./Addr. Bus, fly-by, volt. ref. (RSL) Bus, fly-by, volt. ref. (SSTL) Clock Fly-by, diff. (DRSL)Fly-by, diff. ( diff. SSTL) Synchron. FlexPhase Read/write leveling Contr. reg. manip. Serial 1-bit, volt. ref. (RSL) n.a. Comm. principle Packet based Parallel bus based Signaling

42. Southbound packets Memory controller M. module Commands Write data M. module Northbound packets Read data Status XDR FB-DIMM Figure: Contrasting the packet concepts of XDR and FB-DIMM memories (1) [4] Remark Data packets Control packets CC R/W packets

43. Both XDR and FB-DIMM memories use packet based communication between the memory controller and the XDR devices. „Clean” packets of memory access and maintenace commands (termed request packets), data, control register read/write commands. FB-DIMMs read data or status packets (termed as northbound packets). Differences in the packet policies XDRs „Clean” packets of commands and write data (termed as southbound packets). Mixed packets of Contrasting the packet concepts of XDR and FB-DIMM memories (2)

45. Principle of operation (2) The memory controller sends request packets to the XDR devices, the XDR devices satisfy these requests, e.g. by sending read data packets to the memory controller. Basic command sequence the same as for synchronous DRAMs Activate – Read/Write - Precharge

46. Example 1 Operation Activate Bank a, Row a Read Bank a, Column a1 Read Bank a, Column a2 Precharge Bank a Read data packet Q(a1) Read data packet Q(a2) Figure: Example for reading from the XDR device [3]

47. Operation Activate Bank a, Row a Write Bank a, Column a1 Write Bank a, Column a2 Precharge Bank a Read data packet Q(a1) Read data packet Q(a2) Figure: Example for writing to the XDR device [3] Example 2

48. 1 100 2 2 20 5 0,5 1020 10 1 30 200 50 5 500 BW GB/s Mem. Size GB QS21 (2D) x P4 Servers, QS22 5100 (2) x DDR2 P35 (2) x 925X (2) P4 Servers x x x 7520 (2) 7501 (2) QS22 (2) (2D) x 7520 (2) x 845 (1) 850E (1) 820 (1) 850 (1) x 875 (2) x DDR x x 845 (1) 840 (2) x 7300 (4) FB-DIMM DDR-2 Core 2 Servers, T2 DDR2 (reg) DDR (reg) Core 2 Desktop RDRAM P4 Servers XDR Servers RDRAM P4 Desktops 192 512 32 48 xx 860 (2) x P4 Desktops x QS20 (2D) SDRAM 16 8 4 3 0,75 1,061,63,2 4,2 6,48,510,612,821,225,6 Figure: Peak memory size vs peak bandwidth (BW) of particular DRAM technologies in Intel’s chipsets, IBM’s QS2x blades and Sun’s T2 x SunT2 (4D) 50 51,2