1. Computer Architecture Lecture 14a: Emerging Memory Technologies II
Prof. Onur Mutlu
ETH Zürich
Fall 2018
1 November 2018

2. Emerging Memory Technologies

3. Other Opportunities with Emerging Technologies
Merging of memory and storage
e.g., a single interface to manage all data
New applications
e.g., ultra-fast checkpoint and restore
More robust system design
e.g., reducing data loss
Processing tightly-coupled with memory
e.g., enabling efficient search and filtering

4. Unified Memory and Storage with NVM
Goal: Unify memory and storage management in a single unit to eliminate wasted work to locate, transfer, and translate data
Improves both energy and performance
Simplifies programming model as well
Unified Memory/Storage
Persistent Memory
Manager
Processor
and caches
Load/Store
Feedback
Persistent (e.g., Phase-Change) Memory
Meza+, “A Case for Efficient Hardware-Software Cooperative Management of Storage and Memory,” WEED 2013.

5. Provides an opportunity to manipulate persistent data directly
PERSISTENT MEMORY
CPU
Ld/St
PERSISTENTMEMORY
NVM
NVM provides an unique opportunity to manipulate persistent data directly in the main memory.
Provides an opportunity to manipulate persistent data directly

6. The Persistent Memory Manager (PMM)
Persistent objects
And to tie everything together, here is how a persistent memory would operate. Code allocates persistent objects and accesses them using loads and stores – no different than persistent memory in today’s machines. Optionally, the software can provide hints as to the types of access patterns or requirements of the software.
Then, a Persistent Memory Manager uses this access information and software hints to allocate, locate, migrate, and access data among the heterogeneous array of devices present on the system, attempting to strike a balance between application-level guarantees, like persistence and security, and performance and energy efficiency.
PMM uses access and hint information to allocate, locate, migrate and access data in the heterogeneous array of devices

7. On Persistent Memory Benefits & Challenges
Justin Meza, Yixin Luo, Samira Khan, Jishen Zhao, Yuan Xie, and Onur Mutlu, "A Case for Efficient Hardware-Software Cooperative Management of Storage and Memory" Proceedings of the 5th Workshop on Energy-Efficient Design (WEED), Tel-Aviv, Israel, June Slides (pptx) Slides (pdf)

8. Challenge and Opportunity
Combined Memory & Storage

9. Challenge and Opportunity
A Unified Interface to All Data

10. One Key Challenge in Persistent Memory
How to ensure consistency of system/data if all memory is persistent?
Two extremes
Programmer transparent: Let the system handle it
Programmer only: Let the programmer handle it
Many alternatives in-between…

11. CRASH CONSISTENCY PROBLEM
Add a node to a linked list
2. Link to prev
1. Link to next
Let’s see an example of crash consistency problem in persistent memory systems. When we add a node in the linked list, there are two operations, first, we need to link the new node to next node and then link the previous node to the new node. Unfortunately due to the write ordering, we cannot guarantee which write will reach the memory first. If there is a power failure, depending on which write persisted, we might have a broken linked list. As a result, after a crash we can have an inconsistent memory state.
System crash can result in
inconsistent memory state

12. CURRENT SOLUTIONS Burden on the programmers
Explicit interfaces to manage consistency
NV-Heaps [ASPLOS’11], BPFS [SOSP’09], Mnemosyne [ASPLOS’11]
AtomicBegin {
Insert a new node;
} AtomicEnd;
Limits adoption of NVM
Have to rewrite code with clear partition
between volatile and non-volatile data
Burden on the programmers

13. update a node in a persistent hash table
CURRENT SOLUTIONS
Explicit interfaces to manage consistency
NV-Heaps [ASPLOS’11], BPFS [SOSP’09], Mnemosyne [ASPLOS’11]
Example Code
update a node in a persistent hash table
void hashtable_update(hashtable_t* ht,
void *key, void *data) {
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) list_find(chain,
&updatePair);
pair->second = data; }
Current solution propose to use explicit interfaces to protect data from inconsistency.
Let’s see a real code that updates a node in a persistent hash table.

14. CURRENT SOLUTIONS void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
The code looks like this after we use the transactional interface. However, there are several issues in the code.

15. Manual declaration of persistent components
CURRENT SOLUTIONS
Manual declaration of persistent components
void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
First, the programmer has to manually declare the persistent object.

16. Manual declaration of persistent components Need a new implementation
CURRENT SOLUTIONS
Manual declaration of persistent components
void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
Need a new implementation
Second, the programmers need to rewrite the functions so that they correctly manipulate the persistent data

17. Manual declaration of persistent components Need a new implementation
CURRENT SOLUTIONS
Manual declaration of persistent components
void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
Need a new implementation
Third, third party code needs to be consistent
Third party code
can be inconsistent

18. CURRENT SOLUTIONS Burden on the programmers
Manual declaration of persistent components
void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
Need a new implementation
Programmers need to make sure there is no volatile to non-volatile pointers. Clearly writing code for persistent memory is a burden on the general programmers.
Prohibited
Operation
Third party code
can be inconsistent
Burden on the programmers

19. OUR APPROACH: ThyNVM Goal: Software transparent consistency in
persistent memory systems
Key Idea:
Periodically checkpoint state;
recover to previous checkpt on crash

20. ThyNVM: Summary A new hardware-based checkpointing mechanism
Checkpoints at multiple granularities to reduce both checkpointing latency and metadata overhead
Overlaps checkpointing and execution to reduce checkpointing latency
Adapts to DRAM and NVM characteristics
Performs within 4.9% of an idealized DRAM with zero cost consistency

21. OUTLINE Crash Consistency Problem Current Solutions ThyNVM Evaluation
Here is the outline of the rest of the talk. Next I will talk about the crash consistency problem in detail and show the challenges of the current solutions. Then I will present our design ThyNVM, show the results and finally will conclude.
Conclusion

22. CRASH CONSISTENCY PROBLEM
Add a node to a linked list
2. Link to prev
1. Link to next
Let’s see an example of crash consistency problem in persistent memory systems. When we add a node in the linked list, there are two operations, first, we need to link the new node to next node and then link the previous node to the new node. Unfortunately due to the write ordering, we cannot guarantee which write will reach the memory first. If there is a power failure, depending on which write persisted, we might have a broken linked list. As a result, after a crash we can have an inconsistent memory state.
System crash can result in
inconsistent memory state

23. OUTLINE Crash Consistency Problem Current Solutions ThyNVM Evaluation
So how did the prior works solved the crash consistency problem?
Conclusion

24. update a node in a persistent hash table
CURRENT SOLUTIONS
Explicit interfaces to manage consistency
NV-Heaps [ASPLOS’11], BPFS [SOSP’09], Mnemosyne [ASPLOS’11]
Example Code
update a node in a persistent hash table
void hashtable_update(hashtable_t* ht,
void *key, void *data) {
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) list_find(chain,
&updatePair);
pair->second = data; }
Current solution propose to use explicit interfaces to protect data from inconsistency.
Let’s see a real code that updates a node in a persistent hash table.

25. CURRENT SOLUTIONS void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
The code looks like this after we use the transactional interface. However, there are several issues in the code.

26. Manual declaration of persistent components
CURRENT SOLUTIONS
Manual declaration of persistent components
void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
First, the programmer has to manually declare the persistent object.

27. Manual declaration of persistent components Need a new implementation
CURRENT SOLUTIONS
Manual declaration of persistent components
void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
Need a new implementation
Second, the programmers need to rewrite the functions so that they correctly manipulate the persistent data

28. Manual declaration of persistent components Need a new implementation
CURRENT SOLUTIONS
Manual declaration of persistent components
void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
Need a new implementation
Third, third party code needs to be consistent
Third party code
can be inconsistent

29. CURRENT SOLUTIONS Burden on the programmers
Manual declaration of persistent components
void TMhashtable_update(TMARCGDECL
hashtable_t* ht, void *key, void*data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) TMLIST_FIND(chain,
&updatePair);
pair->second = data; }
Need a new implementation
Programmers need to make sure there is no volatile to non-volatile pointers. Clearly writing code for persistent memory is a burden on the general programmers.
Prohibited
Operation
Third party code
can be inconsistent
Burden on the programmers

30. OUTLINE Crash Consistency Problem Current Solutions ThyNVM Evaluation
In this work we present thynvm
Conclusion

31. Software transparent consistency in persistent memory systems
OUR GOAL
Software transparent consistency
in persistent memory systems
Execute legacy applications
Reduce burden on programmers
Enable easier integration of NVM
The goal of thynvm is to provide software transparent memory consistent such that we can execute legacy applications, reduce the burden on the programmers and enable easier integration of NVM

32. NO MODIFICATION IN THE CODE
void hashtable_update(hashtable_t* ht,
void *key, void *data) {
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) list_find(chain,
&updatePair);
pair->second = data; }
So we will take the exact same code

33. RUN THE EXACT SAME CODE…
void hashtable_update(hashtable_t* ht,
void *key, void *data){
list_t* chain = get_chain(ht, key);
pair_t* pair;
pair_t updatePair;
updatePair.first = key;
pair = (pair_t*) list_find(chain,
&updatePair);
pair->second = data; }
Persistent Memory System
And run that in a persistent memory system, and thynvm will provide memory crash consistency
Software transparent
memory crash consistency

34. Periodic checkpointing of data Transparent to application and system
ThyNVM APPROACH
Periodic checkpointing of data
managed by hardware
time
Running
Checkpointing
Running
Checkpointing
Epoch 0
Epoch 1
Thynvm provides this consistency support by taking periodic checkpoint of data at hardware. So if there is a crash in the system, it rolls back to the last checkpointed state.
Transparent to application and system

35. CHECKPOINTING OVERHEAD
1. Metadata overhead
Metadata Table
Working location
Checkpoint location X X’ Y Y’
time
Running
Checkpointing
Running
Checkpointing
STALLED
STALLED
However, we notice that there are two major overhead of checkpointing. First, we need to track the location of current working copy and the last checkpoint copy and this metadata can have a huge space overhead
Second, during the checkpoint, the programs get stalled, so the checkpointing latency can have a performance impact on the applications.
We make two new observations to solve these challenges.
Epoch 0
Epoch 1
2. Checkpointing latency

36. 1. METADATA AND CHECKPOINTING GRANULARITY
Working location
Checkpoint location X X’ Y Y’
PAGE
CACHE BLOCK
First, we observe that there is a trade off between the metadata overhead and checkpointing granularity. If we checkpoint at page granularity, we need one entry per page, so it will have s small metadata overhead.
If we checkpoint at block granularity, we will need one entry per block leading to a huge metadata overhead
PAGE GRANULARITY
BLOCK GRANULARITY
One Entry Per Page
Small Metadata
One Entry Per Block
Huge Metadata

37. Long latency of writing back data to NVM
2. LATENCY AND LOCATION
DRAM-BASED WRITEBACK W
2. Update the metadata table
Working location
Checkpoint location X X’
1. Writeback data from DRAM
DRAM
NVM
Second, we observe that there is a relation between the location of data and checkpoint latency. We can have the working copy either in DRAM on NVM. Let’s look at DRAM first. Checkpointing data when it is stored in DRAM has two steps. Since data in DRAM is not persistent, any working copy needs to be written back to NVM during checkpointing time to make it persistent. Then we need to update the meta table with the location of the checkpoint copy.
So there is a long latency of writing back data from DRAM to NVM
Long latency of writing back data to NVM

38. Short latency in NVM-based remapping
2. LATENCY AND LOCATION
NVM-BASED REMAPPING
2. Update the metadata table
Working location
Checkpoint location Y X
3. Write in a
new location
No copying
of data
DRAM
NVM
However, if the current working copy of the data is stored in NVM, it is already persistent and there is no need to copy data during the checkpointing time. We only need to persistent the location of the data in the metadata table and when we there is a new write to that location, we can just remap the write to a new location in NVM.
Since there is no copy involved, NVM-based remapping has a short latency.
Short latency in NVM-based remapping

39. ThyNVM KEY MECHANISMS Checkpointing granularity Latency and location
Small granularity: large metadata
Large granularity: small metadata
Latency and location
Writeback from DRAM: long latency
Remap in NVM: short latency
Based on these, we propose two key mechanisms
So to summarize, small granularity checkpointing leads to large metadata and large granularity checkpointing leads to small metadata overhead. On the other hand, writeback from DRAM introduces long stall time but remapping in NVM is very fast.
Based on these two observations, ThyNVM has two key mechanisms to reduce metadata overhead and checkpointing latency,
Dual granularity checkpointing
Overlapping of execution and checkpointing
1. Dual granularity checkpointing
2. Overlap of execution and checkpointing

40. 1. DUAL GRANULARITY CHECKPOINTING
Page Writeback
in DRAM
Block Remapping
in NVM
DRAM
NVM
GOOD FOR
STREAMING WRITES
GOOD FOR
RANDOM WRITES
For dual granularity checkpointing, thynvm uses page granularity writeback scheme for DRAM and block remapping scheme for NVM. Thynvm places pages with high write locality in DRAM and low locality writes in NVM. Placing high write locality pages in DRAM is good for streaming writes, As DRAM coalesces the writes and minimizes the write traffic during the checkpointing. On the other hand, nvm just remaps blocks and is good for random writes, if these pages were placed in DRAM,DRAM had to writeback whole pages with only one or two updates.
High write locality pages in DRAM,
low write locality pages in NVM

41. TRADEOFF SPACE

42. 2. OVERLAPPING CHECKPOINTING AND EXECUTION
time
Running
Checkpointing
Running
Checkpointing
Epoch 0
Epoch 1
time
Epoch 0
Epoch 1
Epoch 2
Running
Checkpointing
Even with the dual granularity checkpointing, checkpointing latency is dominated by the slow DRAM write back. In order to reduce the latency we propose to overlap checkpointing with execution. Instead of stalling when DRAM pages are getting written back to memory, we let the application start the next epoch. This hides the long latency of page writeback scheme.
Hides the long latency of Page Writeback

43. OUTLINE Crash Consistency Problem Current Solutions ThyNVM Evaluation
Now that we have described our key mechanisms let’s take a look at the results
Conclusion

44. SYSTEM ARCHITECTURE

45. MEMORY ADDRESS SPACE

46. METHODOLOGY Cycle accurate x86 simulator Gem5 Comparison Points:
Ideal DRAM: DRAM-based, no cost for consistency
Lowest latency system
Ideal NVM: NVM-based, no cost for consistency
NVM has higher latency than DRAM
Journaling: Hybrid, commit dirty cache blocks
Leverages DRAM to buffer dirty blocks
Shadow Paging: Hybrid, copy-on-write pages
Leverages DRAM to buffer dirty pages
We have evaluated thynvm using cycle accurate x86 simulator GEM5. We compare thynvm with four different systems. Ideal DRAM is a DRAM only system that provides crash consistency support at no cost. As DRAM has lower latency NVM, this is the lowest latency system.
Ideal NVM is an NVM-only system with zero cost for consistency. This system will have higher latency than ideal DRAM.
We also compare thynvm with journaling and shadow paging. Both of them has both DRAM and NVM, similar to ThyNVM. Journaling commits updates in cache block granularity and shadow paging uses copy-on-write at page granularity.

47. ADAPTIVITY TO ACCESS PATTERN
RANDOM SEQUENTIAL
BETTER
LOW
LOW
LOW
LOW
So let’s see how good thynvm can adapt to access patterns. We evaluate two microbenchmarks with random and sequential access patterns. In the x axis we compare thynvm with journaling and shadow paging. In the y axis we have write traffic normalized to journaling, so lower is better.
If we compare jounaling and shaow paging, jounaling is
Journaling is better for Random and
Shadow paging is better for Sequential
ThyNVM adapts to both access patterns

48. OVERLAPPING CHECKPOINTING AND EXECUTION
RANDOM SEQUENTIAL
BETTER
45%
35%
Can spend 35-45% of the execution
on checkpointing
ThyNVM spends only 2.4%/5.5% of the execution time on checkpointing
Stalls the application for a negligible time

49. PERFORMANCE OF LEGACY CODE
Within -4.9%/+2.7% of an
idealized DRAM/NVM system
Provides consistency without
significant performance overhead

50. KEY-VALUE STORE TX THROUGHPUT
Storage throughput close to Ideal DRAM

51. OUTLINE Crash Consistency Problem Current Solutions ThyNVM Evaluation
Conclusion

52. with no programming effort
ThyNVM
A new hardware-based
checkpointing mechanism,
with no programming effort
Checkpoints at multiple granularities to minimize both latency and metadata
Overlaps checkpointing and execution
Adapts to DRAM and NVM characteristics
Can enable widespread adoption
of persistent memory

53. Source Code and More Available at
ThyNVM
Enabling Software-transparent Crash Consistency In Persistent Memory Systems

54. More About ThyNVM
Jinglei Ren, Jishen Zhao, Samira Khan, Jongmoo Choi, Yongwei Wu, and Onur Mutlu, "ThyNVM: Enabling Software-Transparent Crash Consistency in Persistent Memory Systems" Proceedings of the 48th International Symposium on Microarchitecture (MICRO), Waikiki, Hawaii, USA, December 2015.  [Slides (pptx) (pdf)] [Lightning Session Slides (pptx) (pdf)] [Poster (pptx) (pdf)]  [Source Code] 

55. Another Key Challenge in Persistent Memory
Programming Ease to Exploit Persistence

56. Tools/Libraries to Help Programmers
Himanshu Chauhan, Irina Calciu, Vijay Chidambaram, Eric Schkufza, Onur Mutlu, and Pratap Subrahmanyam, "NVMove: Helping Programmers Move to Byte-Based Persistence" Proceedings of the 4th Workshop on Interactions of NVM/Flash with Operating Systems and Workloads (INFLOW), Savannah, GA, USA, November 2016.  [Slides (pptx) (pdf)] 

57. The Future of Emerging Technologies is Bright
Regardless of challenges
in underlying technology and overlying problems/requirements
Problem
Can enable:
- Orders of magnitude
improvements
- New applications and
computing systems
Yet, we have to
- Think across the stack
- Design enabling systems
Algorithm
Program/Language
System Software
SW/HW Interface
Micro-architecture
Logic
Devices
Electrons

58. If In Doubt, Refer to Flash Memory
A very “doubtful” emerging technology
for at least two decades
Proceedings of the IEEE, Sept. 2017

59. Computer Architecture Lecture 14a: Emerging Memory Technologies II
Prof. Onur Mutlu
ETH Zürich
Fall 2018
1 November 2018