1. ThyNVM Enabling Software-Transparent Crash Consistency In Persistent Memory Systems Jinglei Ren, Jishen Zhao, Samira Khan, Jongmoo Choi, Yongwei Wu, and Onur Mutlu Jinglei Ren, Jishen Zhao, Samira Khan, Jongmoo Choi, Yongwei Wu, and Onur Mutlu

2. TWO-LEVEL STORAGE MODEL CPU MEMORY STORAGE VOLATILE FAST BYTE ADDR NONVOLATILE SLOW BLOCK ADDR Ld/St FILE I/O DRAM 2

3. TWO-LEVEL STORAGE MODEL CPU MEMORY STORAGE VOLATILE FAST BYTE ADDR NONVOLATILE SLOW BLOCK ADDR Ld/St FILE I/O DRAM 3 PCM, STT-RAM NVM Non-volatile memories combine characteristics of memory and storage

4. PERSISTENT MEMORY CPU PERSISTENT MEMORY Provides an opportunity to manipulate persistent data directly Ld/St NVM 4

5. CHALLENGE: CRASH CONSISTENCY System crash can result in permanent data corruption in NVM 5 Persistent Memory System

6. CURRENT SOLUTIONS Explicit interfaces to manage consistency – NV-Heaps [ASPLOS’11], BPFS [SOSP’09], Mnemosyne [ASPLOS’11] AtomicBegin { Insert a new node; } AtomicEnd; Limits adoption of NVM Have to rewrite code with clear partition between volatile and non-volatile data Burden on the programmers 6

7. OUR APPROACH: ThyNVM 7 Goal: Software transparent consistency in persistent memory systems

8. ThyNVM: Summary 8 Checkpoints at multiple granularities to reduce both checkpointing latency and metadata overhead Overlaps checkpointing and execution to reduce checkpointing latency Adapts to DRAM and NVM characteristics Performs within 4.9% of an idealized DRAM with zero cost consistency A new hardware-based checkpointing mechanism A new hardware-based checkpointing mechanism

9. OUTLINE 9 Crash Consistency Problem Current Solutions ThyNVM Evaluation Conclusion

10. CRASH CONSISTENCY PROBLEM 10 Add a node to a linked list 1. Link to next 2. Link to prev System crash can result in inconsistent memory state

11. OUTLINE 11 Crash Consistency Problem Current Solutions ThyNVM Evaluation Conclusion

12. CURRENT SOLUTIONS 12 void hashtable_update(hashtable_t* ht, void *key, void *data) { list_t* chain = get_chain(ht, key); pair_t* pair; pair_t updatePair; updatePair.first = key; pair = (pair_t*) list_find(chain, &updatePair); pair->second = data; } Example Code update a node in a persistent hash table Example Code update a node in a persistent hash table Explicit interfaces to manage consistency – NV-Heaps [ASPLOS’11], BPFS [SOSP’09], Mnemosyne [ASPLOS’11]

13. CURRENT SOLUTIONS 13 void TMhashtable_update(TMARCGDECL hashtable_t* ht, void *key, void*data){ list_t* chain = get_chain(ht, key); pair_t* pair; pair_t updatePair; updatePair.first = key; pair = (pair_t*) TMLIST_FIND (chain, &updatePair); pair->second = data; }

14. CURRENT SOLUTIONS 14 void TMhashtable_update(TMARCGDECL hashtable_t* ht, void *key, void*data){ list_t* chain = get_chain(ht, key); pair_t* pair; pair_t updatePair; updatePair.first = key; pair = (pair_t*) TMLIST_FIND (chain, &updatePair); pair->second = data; } Manual declaration of persistent components

15. CURRENT SOLUTIONS 15 void TMhashtable_update(TMARCGDECL hashtable_t* ht, void *key, void*data){ list_t* chain = get_chain(ht, key); pair_t* pair; pair_t updatePair; updatePair.first = key; pair = (pair_t*) TMLIST_FIND (chain, &updatePair); pair->second = data; } Manual declaration of persistent components Need a new implementation

16. CURRENT SOLUTIONS 16 void TMhashtable_update(TMARCGDECL hashtable_t* ht, void *key, void*data){ list_t* chain = get_chain(ht, key); pair_t* pair; pair_t updatePair; updatePair.first = key; pair = (pair_t*) TMLIST_FIND (chain, &updatePair); pair->second = data; } Manual declaration of persistent components Need a new implementation Third party code can be inconsistent

17. CURRENT SOLUTIONS 17 void TMhashtable_update(TMARCGDECL hashtable_t* ht, void *key, void*data){ list_t* chain = get_chain(ht, key); pair_t* pair; pair_t updatePair; updatePair.first = key; pair = (pair_t*) TMLIST_FIND (chain, &updatePair); pair->second = data; } Manual declaration of persistent components Need a new implementation Third party code can be inconsistent Prohibited Operation Burden on the programmers

18. OUTLINE 18 Crash Consistency Problem Current Solutions ThyNVM Evaluation Conclusion

19. OUR GOAL 19 Software transparent consistency in persistent memory systems Execute legacy applications Reduce burden on programmers Enable easier integration of NVM

20. NO MODIFICATION IN THE CODE void hashtable_update(hashtable_t* ht, void *key, void *data) { list_t* chain = get_chain(ht, key); pair_t* pair; pair_t updatePair; updatePair.first = key; pair = (pair_t*) list_find(chain, &updatePair); pair->second = data; }

21. RUN THE EXACT SAME CODE… Persistent Memory System Software transparent memory crash consistency 21 void hashtable_update(hashtable_t* ht, void *key, void *data){ list_t* chain = get_chain(ht, key); pair_t* pair; pair_t updatePair; updatePair.first = key; pair = (pair_t*) list_find(chain, &updatePair); pair->second = data; }

22. ThyNVM APPROACH Running Epoch 0 Epoch 1 time Checkpointing Running Checkpointing 22 Periodic checkpointing of data managed by hardware Periodic checkpointing of data managed by hardware Transparent to application and system

23. CHECKPOINTING OVERHEAD Running Epoch 0 Epoch 1 time Checkpointing Running Checkpointing 23 1. Metadata overhead Working location Checkpoint location X X X’ Y Y Y’ Metadata Table 2. Checkpointing latency

24. 1. METADATA AND CHECKPOINTING GRANULARITY PAGE CACHE BLOCK One Entry Per Page Small Metadata One Entry Per Page Small Metadata One Entry Per Block Huge Metadata One Entry Per Block Huge Metadata 24 PAGE GRANULARITY BLOCK GRANULARITY Working location Checkpoint location X X X’ Y Y Y’

25. W 2. LATENCY AND LOCATION 25 DRAM-BASED WRITEBACK Long latency of writing back data to NVM DRAM NVM Working location Checkpoint location X X X’ 1. Writeback data from DRAM 2. Update the metadata table

26. 2. LATENCY AND LOCATION 26 NVM-BASED REMAPPING Short latency in NVM-based remapping DRAM NVM Working location Checkpoint location Y Y X X 1.No copying of data 1.No copying of data 2. Update the metadata table 3. Write in a new location 3. Write in a new location

27. ThyNVM KEY MECHANISMS Checkpointing granularity Small granularity: large metadata Large granularity: small metadata Latency and location Writeback from DRAM: long latency Remap in NVM: short latency Checkpointing granularity Small granularity: large metadata Large granularity: small metadata Latency and location Writeback from DRAM: long latency Remap in NVM: short latency Based on these we propose two key mechanisms 1. Dual granularity checkpointing 2. Overlap of execution and checkpointing 1. Dual granularity checkpointing 2. Overlap of execution and checkpointing

28. DRAM NVM 1. DUAL GRANULARITY CHECKPOINTING High write locality pages in DRAM, low write locality pages in NVM Page Writeback in DRAM Page Writeback in DRAM Block Remapping in NVM Block Remapping in NVM 28 GOOD FOR STREAMING WRITES GOOD FOR STREAMING WRITES GOOD FOR RANDOM WRITES GOOD FOR RANDOM WRITES

29. Running time Checkpointing Running Checkpointing time Epoch 0 Epoch 1 Epoch 2 Epoch 0 Epoch 1 Running Checkpointing Running Checkpointing Running Checkpointing Epoch 0 Epoch 1 2. OVERLAPPING CHECKOINTING AND EXECUTION Hides the long latency of Page Writeback

30. OUTLINE 30 Crash Consistency Problem Current Solutions ThyNVM Evaluation Conclusion

31. METHODOLOGY 31 Cycle accurate x86 simulator Gem5 Comparison Points: Ideal DRAM: DRAM-based, no cost for consistency – Lowest latency system Ideal NVM: NVM-based, no cost for consistency – NVM has higher latency than DRAM Journaling: Hybrid, commit dirty cache blocks – Leverages DRAM to buffer dirty blocks Shadow Paging: Hybrid, copy-on-write pages – Leverages DRAM to buffer dirty pages

32. ADAPTIVITY TO ACCESS PATTERN 32 RANDOM SEQUENTIAL ThyNVM adapts to both access patterns Journaling is better for Random and Shadow paging is better for Sequential Journaling is better for Random and Shadow paging is better for Sequential BETTERBETTER BETTERBETTER

33. OVERLAPPING CHECKPOINTING AND EXECUTION 33 RANDOM SEQUENTIAL Stalls the application for a negligible time ThyNVM spends only 2.4%/5.5% of the execution time on checkpointing Can spend 35-45% of the execution on checkpointing Can spend 35-45% of the execution on checkpointing BETTERBETTER BETTERBETTER

34. PERFORMANCE OF LEGACY CODE 34 Provides consistency without significant performance overhead Within -4.9%/+2.7% of an idealized DRAM/NVM system Within -4.9%/+2.7% of an idealized DRAM/NVM system

35. OUTLINE 35 Crash Consistency Problem Current Solutions ThyNVM Evaluation Conclusion

36. ThyNVM 36 Checkpoints at multiple granularities to minimize both latency and metadata Overlaps checkpointing and execution Adapts to DRAM and NVM characteristics Can enable widespread adoption of persistent memory A new hardware-based checkpointing mechanism, with no programming effort A new hardware-based checkpointing mechanism, with no programming effort

37. ThyNVM Enabling Software-transparent Crash Consistency In Persistent Memory Systems Jinglei Ren, Jishen Zhao, Samira Khan, Jongmoo Choi, Yongwei Wu, and Onur Mutlu Jinglei Ren, Jishen Zhao, Samira Khan, Jongmoo Choi, Yongwei Wu, and Onur Mutlu Available at http://persper.com/thynvm Available at http://persper.com/thynvm