1. Temporal Memory Streaming
Babak Falsafi
Team: Mike Ferdman, Brian Gold, Nikos Hardavellas,
Jangwoo Kim, Stephen Somogyi, Tom Wenisch
Collaborator: Anastassia Ailamaki & Andreas Moshovos
STEMS
Computer Architecture Lab Carnegie Mellon

2. The Memory Wall Logic/DRAM speed gap continues to increase! VAX/1980
Database performance on modern processors hurts because of the processor/memory speed gap.
The problems is due to (a) memories getting bigger (b) storage managers are smart and hide disk I/O
And (c) DBMS have too many knobs, benchmarks too complex to be used in early processor design stages
Database instruction streams contain many data accesses (=accesses to memory hierarchy)
And many dependencies (=>instruction-level parallelism is hard)
So on a 1980 machine (VAX 11/750) the access time to memory was about as long as the time to execute one instruction, whereas the access time to memory in 2000 (Pentium II) was equivalent to the time needed to execute 1000’s of instructions. The trend is going even faster; On the Pentium 4 the memory latency is 300 cycles, which is one more order of magnitude in only 2 years time. We project that soon the caches will play the role of memory (on-chip memory), the main memory will become the disk and the disk will be tertiary storage.
This means that unless the cache usage (cache=fast on-chip memory) is optimal, memory latency will dominate, and no matter how fast the processor is, database applications will be slow.
Therefore the cache-aware database research.
VAX/1980
PPro/1996
2010+
Logic/DRAM speed gap continues to increase!

3. Current Approach Cache hierarchies: Trade off capacity for speed
Exploit “reuse”
But, in modern servers
Only 50% utilization of one proc. [Ailamaki, VLDB’99]
Much bigger problem in MPs
What is wrong?
Demand fetch/repl. data
CPU
1000 clk
100 clk
10 clk
1 clk
L1 64K
L M
The electronic world is encompassing much of our daily activities
electronic commerce and search engines are two examples of daily
activities.
People use client devices such as -- handheld, desktop computers -- to interface
exchange information through the internet.
The backbone of the e-world consists
of compute servers that process and store information and make them available
to e-world’s users.
My talk will primarily focus designing servers. While there are several
issues associated with server design, in this talk I will focus on two
key aspects: performance
and
programmability
L M
DRAM
100G

4. Prior Work (SW-Transparent)
Prefetching [Joseph 97] [Roth 96] [Nesbit 04] [Gracia Pérez 04]
Simple patterns or low accuracy
Large Exec. Windows / Runahead [Mutlu 03]
Fetch dependent addresses serially
Coherence Optimizations [Stenström 93] [Lai 00] [Huh 04]
Limited applicability (e.g., migratory)
Need solutions for arbitrary access patterns

5. Our Solution: Spatio-Temporal Memory Streaming
CPU
STEMS
Observation:
Data spatially/temporally correlated
Arbitrary, yet repetitive, patterns
Approach  Memory Streaming
Extract spat./temp. patterns
Stream data to/from CPU
Manage resources for multiple blocks
Break dependence chains
In HW, SW or both L1 A C D B .. X
The electronic world is encompassing much of our daily activities
electronic commerce and search engines are two examples of daily
activities.
People use client devices such as -- handheld, desktop computers -- to interface
exchange information through the internet.
The backbone of the e-world consists
of compute servers that process and store information and make them available
to e-world’s users.
My talk will primarily focus designing servers. While there are several
issues associated with server design, in this talk I will focus on two
key aspects: performance
and
programmability
replace
stream W
fetch Z
DRAM or Other CPUs

6. Contribution #1: Temporal Shared-Memory Streaming
Recent coherence miss sequences recur
50% misses closely follow previous sequence
Large opportunity to exploit MLP
Temporal streaming engine
Ordered streams allow practical HW
Performance improvement:
7%-230% in scientific apps.
6%-21% in commercial Web & OLTP apps.

7. Contribution #2: Last-touch Correlated Data Streaming
Last-touch prefetchers
Cache block deadtime >> livetime
Fetch on a predicted “last touch”
But, designs impractical (> 200MB on-chip)
Last-touch correlated data streaming
Miss order ~ last-touch order
Stream table entries from off-chip
Eliminates 75% of all L1 misses with ~200KB

8. Outline STEMS Overview Example Temporal Streaming Summary
Temporal Shared-Memory Streaming
Last-Touch Correlated Data Streaming
Summary

9. Temporal Shared-Memory Streaming [ISCA’05]
Record sequences of memory accesses
Transfer data sequences ahead of requests
Baseline System
Streaming System
Miss A
CPU
Mem
CPU
Mem
Fill A
Miss A
Miss B
Fill A,B,C,…
Fill B
Accelerates arbitrary access patterns
Parallelizes critical path of pointer-chasing

10. Relationship Between Misses
Intuition: Miss sequences repeat
Because code sequences repeat
Observed for uniprocessors in [Chilimbi’02]
Temporal Address Correlation
Same miss addresses repeat in the same order
Correlated miss sequence = stream …
Miss seq. Q W A B C D E R T A B C D E Y

11. Relationship Between Streams
Intuition: Streams exhibit temporal locality
Because working set exhibits temporal locality
For shared data, repetition often across nodes
Temporal Stream Locality
Recent streams likely to recur
Node 1 Q W A B C D E R
Node 2 T A B C D E Y
Addr. correlation + stream locality = temporal correlation

12. Memory Level Parallelism
Streams create MLP for dependent misses
Not possible with larger windows / runahead
Temporal streaming breaks dependence chains
Baseline
Temporal Streaming A A
CPU
CPU B B C C
Must wait to follow pointers
Fetch in parallel

13. Temporal Streaming   Record Node 1 Directory Node 2 Miss A Miss B
Miss C
Miss D
 Record

14. Temporal Streaming   Record Node 1 Directory Node 2 Miss A Miss B
Miss C
Miss D
Req. A
Miss A
Fill A
 Record

15. Temporal Streaming    Record  Locate Node 1 Directory Node 2
Miss A
Miss B
Miss C
Miss D 
Req. A
Miss A
Locate A
Fill A
Stream B, C, D
 Record
 Locate

16. Temporal Streaming    Record   Locate  Stream Node 1 Directory
Miss A
Miss B
Miss C
Miss D 
Req. A
Miss A
Locate A
Fill A
Stream B, C, D
 Record
 Locate
 Stream 
Fetch B, C, D
Hit B
Hit C

17. Temporal Streaming Engine
 Record
Coherence Miss Order Buffer (CMOB)
~1.5MB circular buffer per node
In local memory
Addresses only
Coalesced accesses
CPU $
Fill E
Local Memory
CMOB Q W A B C D E R T Y

18. Temporal Streaming Engine
 Locate
Annotate directory
Already has coherence info for every block
CMOB append  send pointer to directory
Coherence miss  forward stream request
Directory A
shared
Node CMOB[23] B
modified
Node CMOB[401]

19. Temporal Streaming Engine
Fetch data to match use rate
Addresses in FIFO stream queue
Fetch into streamed value buffer
CPU
L1 $
Node i: stream {A,B,C…}
Streamed Value Buffer A
data
Stream Queue F E D C B
Fetch A
~32 entries

20. Practical HW Mechanisms
Streams recorded/followed in order
FIFO stream queues
~32-entry streamed value buffer
Coalesced cache-block size CMOB appends
Predicts many misses from one request
More lookahead
Allows off-chip stream storage
Leverages existing directory lookup

21. Methodology: Infrastructure
SimFlex [SIGMETRICS’04]
Statistically sampling → uArch sim. in minutes
Full-system MP simulation (boots Linux & Solaris)
Uni, CMP, DSM timing models
Real server software (e.g., DB2 & Oracle)
Component-based  FPGA board interface for hybrid simulation/prototyping
Publicly available at

22. Methodology: Benchmarks & Parameters
Benchmark Applications
Scientific
em3d, moldyn, ocean
OLTP: TPC-C WH
IBM DB2 7.2
Oracle 10g
SPECweb99 w/ 16K con.
Apache 2.0
Zeus 4.3
Model Parameters
16 4GHz SPARC CPUs
8-wide OoO; 8-stage pipe
256-entry ROB/LSQ
64K L1, 8MB L2
TSO w/ speculation

23. TSE Coverage Comparison
TSE outperforms Stride and GHB for coherence misses

24. Stream Lengths Comm: Short streams; low base MLP (1.2-1.3)
Sci: Long streams; high base MLP ( )
Temporal Streaming addresses both cases

25. TSE Performance Impact
Time Breakdown
Speedup
95% CI
em3d moldyn ocean Apache DB2 Oracle Zeus
TSE eliminates 25%-95% of coherent read stalls
6% to 230% performance improvement

26. TSE Conclusions Temporal Streaming Temporal Streaming Engine
Intuition: Recent coherence miss sequences recur
Impact: Eliminates % of coherence misses
Temporal Streaming Engine
Intuition: In-order streams enable practical HW
Impact: Performance improvement
7%-230% in scientific apps.
6%-21% in commercial Web & OLTP apps.

27. Outline Big Picture Example Streaming Techniques Summary
Temporal Shared Memory Streaming
Last-Touch Correlated Data Streaming
Summary

28. Enhancing Lookahead Observation [Mendelson, Wood&Hill]: Dead sets
Time T1
Observation [Mendelson, Wood&Hill]:
Few live sets
Use until last “hit”
Data reuse  high hit rate
~80% dead frames!
Exploit for lookahead:
Predict last “touch” prior to “death”
Evict, predict and fetch next line
Dead sets
Time T2
Live sets

29. How Much Lookahead?
L2 latency
DRAM latency
Frame Deadtimes (cycles)
Predicting last-touches will eliminate all latency!

30. Dead-Block Prediction [ISCA’00 & ’01] Accesses to a block frame
Per-block trace of memory accesses to a block
Predicts repetitive last-touch events
PC0: load/store A0
(hit)
PC1: load/store A1
(miss)
First touch
Accesses to a block frame
PC3: load/store A1
(hit)
PC3: load/store A1
(hit)
Last touch
PC5: load/store A3
(miss)
Trace = A1  (PC1,PC3, PC3)

31. Dead-Block Prefetcher (DBCP)
History & correlation tables
History ~ L1 tag array
Correlation ~ memory footprint
Encoding  truncated addition
Two bit saturating counter
History Table (HT)
Correlation Table
Latest
PC1,PC3 
A1,PC1,PC3,PC3 A3 11 A1
PC3
Evict A1
Fetch A3
Current Access

32. DBCP Coverage with Unlimited Table Storage
Say something more intelligent
High average L1 miss coverage
Low misprediction (2-bit counters)

33. Impractical On-Chip Storage Size
Needs over 150MB to achieve full potential!

34. Our Observation: Signatures are Temporally Correlated
Signatures need not reside on chip
Last-touch sequences recur
Much as cache miss sequences recur [Chilimbi’02]
Often due to large structure traversals
Last-touch order ~ cache miss order
Off by at most L1 cache capacity
Key implications:
Can record last touches in miss order
Store & stream signatures from off-chip

35. Last-Touch Correlated Data Streaming (LT-CORDS)
Streaming signatures on chip
Keep all sigs. in sequences in off-chip DRAM
Retain sequence “heads” on chip
“Head” signals a stream fetch
Small (~200KB) on-chip stream cache
Tolerate order mismatch
Lookahead for stream startup
DBCP coverage with moderate on-chip storage!

36. DBCP Mechanisms Core L1 L2 DRAM HT
Sigs. (160MB)
All signatures in random-access on-chip table

37. Signatures stored off-chip
What LT-CORDS Does
Core L1 L2
DRAM HT
… and only in order
“Head” as cue for the “stream”
Only a subset needed at a time
Signatures stored off-chip

38. LT-CORDS Mechanisms Core L1 L2 DRAM HT SC
Heads (10K)
Sigs. (200K)
On-chip storage independent of footprint

39. Methodology SimpleScalar CPU model with Alpha ISA
SPEC CPU2000 & Olden benchmarks
8-wide out-of-order processor
2 cycle L1, 16 cycle L2, 180 cycle DRAM
FU latencies similar to Alpha EV8
64KB 2-way L1D, 1MB 8-way L2
LT-CORDS with 214KB on-chip storage
Apps. with significant memory stalls

40. LT-CORDS vs. DBCP Coverage
Olden, Int, FP
LT-CORDS reaches infinite DBCP coverage

41. LT-CORDS hides large fraction of memory latency
LT-CORDS Speedup
LT-CORDS hides large fraction of memory latency

42. LT-CORDS Conclusions Intuition: Signatures temporally correlated
Cache miss & last-touch sequences recur
Miss order ~ last-touch order
Impact: eliminates 75% of all misses
Retains DBCP coverage, lookahead, accuracy
On-chip storage indep. of footprint
2x less memory stalls over best prior work

43. For more information
Visit our website: