Snapshots in a Flash with ioSnap TM Sriram Subramanian, Swami Sundararaman, Nisha Talagala, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau Copyright © 2014 Fusion-io, Inc. All rights reserved.

Snapshots in a Flash with ioSnap TM (patent pending technology) 10 Snapshots Overview Point-in-time representation of the state of a storage device Used for backup/recovery, audit trails Implemented at different levels Device, block layer, LVM, file systems, applications, databases Use Copy-on-Write (CoW) or Redirect-on-Write (RoW) Copyright © 2014 Fusion-io, Inc. All rights reserved.2 10 Existing Data Update Data Existing Data Update Data

Snapshots in a Flash with ioSnap TM (patent pending technology) Why Rethink Snapshots for Flash? Flash is revolutionizing storage systems Accelerating data centers, enterprise apps, desktops Natural fit for supporting snapshots Redirect-on-Write: never overwrite existing data Log-structuring: data ordered on their creation time (almost) Rate of data-change is higher for Flash devices e.g., multi-threaded 8KB IOs, TB capacity device Flash: 30K IOPS and device fills in ~1 hour HDD : 500 IOPS and device fills in ~3 days Copyright © 2014 Fusion-io, Inc. All rights reserved.3

Snapshots in a Flash with ioSnap TM (patent pending technology) Block Driver Why Implement Snapshots in Block Layer? Block Driver Disk-based Systems Free spaceNamespacejournalingMapping File Systems Free spaceAtomic writeMapping Dynamic provisioning Flash-based Systems Block Driver Free spaceNamespacejournalingMapping File Systems Copyright © 2014 Fusion-io, Inc. All rights reserved.4

Snapshots in a Flash with ioSnap TM (patent pending technology) Snapshots in Flash Challenges Users are sensitive to performance variability Need predictable performance all the time NAND flash has low endurance & inefficient in-place writes In-place updates of reference counts not possible Data could be moved due to a variety of reasons Cannot waste storage space for storing snapshot metadata $/GB is high and need to keep costs low Copyright © 2014 Fusion-io, Inc. All rights reserved.5

Snapshots in a Flash with ioSnap TM (patent pending technology) ioSnap TM Overview First flash-aware snapshotting system Leverages RoW in FTL to support snapshot operations Proposes usage of epochs in FTL to maintain log-time ordering Supports large number of writable snapshots (2 16 ) Embraces rate-limiting to minimize performance implications Performance Results (prototype built into Fusion-io VSL driver) Instantaneous snapshot creation/deletion (~50usec, 4k metadata) Matches vanilla read/write performance numbers Provides predictable performance for foreground IOs Copyright © 2014 Fusion-io, Inc. All rights reserved.6

Snapshots in a Flash with ioSnap TM (patent pending technology) Outline Introduction ioSnap TM Design Evaluation Conclusions Copyright © 2014 Fusion-io, Inc. All rights reserved.7

Snapshots in a Flash with ioSnap TM (patent pending technology) Design Goals and Assumptions Goals Negligible impact on foreground performance Predictable foreground performance Minimal space overheads for snapshot metadata Integrate with existing FTLs Assumptions Snapshots are primarily used for backup, disaster recovery Creates/deletes are common operations Accesses/activates are rare operations Copyright © 2014 Fusion-io, Inc. All rights reserved.8

Snapshots in a Flash with ioSnap TM (patent pending technology) Relevant components of Fusion-io VSL TM Organizes NAND Flash into segments & maintains a log NAND Flash Modules Mapping Layer (FTL) LBA NAND ADDR Mapping Layer (FTL) LBA NAND ADDR Free Space Mgmt. Validity of blocks Segment cleaner Segment 1Segment N Validity Bitmap Segment N Segment 1 Virtual Storage Layer (VSL TM ) Crash Recovery Reconstruct forward and validity map Crash Recovery Reconstruct forward and validity map Copyright © 2014 Fusion-io, Inc. All rights reserved.9

Snapshots in a Flash with ioSnap TM (patent pending technology) Segment Boundary Creating / Deleting Snapshots in Flash Key concept: Infinite size logS1S1 S2S2 S3S3 SNSN Snapshot Deletion is similar to creation – simply write a deletion note Creation is just writing a snapshot create note in the log Copyright © 2014 Fusion-io, Inc. All rights reserved.10 Leverage time ordering of data in a log to create snapshots Segment 1Segment 2Segment N Snapshot creation/deletion is fast & negligible (fixed) metadata

Snapshots in a Flash with ioSnap TM (patent pending technology) Well What About Segment Cleaner? S1 Active Segment Boundary Snapshot Active 70 S1 Active S1 Active Active S1 Active S1 Active Epoch 1Epoch 2 LBA Epoch Epoch: notion of period of time Copyright © 2014 Fusion-io, Inc. All rights reserved.11 Epochs help preserve log-time ordering

Snapshots in a Flash with ioSnap TM (patent pending technology) Managing Liveness of blocks Issue: snapshots indirectly increase the reference count Validity bitmap with a single bit doesn’t work Possible solutions: Maintain more bits/block ( 2 16 snapshot implies 16x increase in bitmaps ) Selectively maintain per sub-segment bitmap for snapshots Only create a bitmap if a snapshot has (or modified) data in it Insight: determine if a given block has at least one reference to it Copyright © 2014 Fusion-io, Inc. All rights reserved.12

Snapshots in a Flash with ioSnap TM (patent pending technology) Preserving Liveness Via CoW Validity Bitmaps Epoch 1 Segment Boundary Snapshot Epoch Bits needed to be flipped Validity Map CoW Epoch 2 Epoch 1 Epoch 2 (step1) Epoch 2 (step2) Copyright © 2014 Fusion-io, Inc. All rights reserved.13

Snapshots in a Flash with ioSnap TM (patent pending technology) Snapshot-Aware Segment Cleaner Epoch Epoch Epoch 2 Merge Bitmaps Valid Block Invalid Block Epoch 2 Epoch Epoch Epoch Cleaned Segment Epoch 1 Copyright © 2014 Fusion-io, Inc. All rights reserved.14 Segment Boundary Snapshot Segment cleaner preserves log-time ordering

Snapshots in a Flash with ioSnap TM (patent pending technology) Design Summary Leverage implicit time ordering in the Log Epochs preserve log-time ordering even with a cleaner Sub-segment-level bitmaps to track validity of blocks Lazily update reference counts for snapshotted blocks Snapshot-aware cleaner preserves log-time and block validity Snapshot management Background snapshot activation Rate-limiting to provide predictable foreground performance Copyright © 2014 Fusion-io, Inc. All rights reserved.15

Snapshots in a Flash with ioSnap TM (patent pending technology) Outline Introduction ioSnap TM Design Evaluation Conclusions Copyright © 2014 Fusion-io, Inc. All rights reserved.16

Snapshots in a Flash with ioSnap TM (patent pending technology) Evaluation What’s the impact on user IOs in the absence of snapshots? Snapshot creation/deletion time? Implications on user IO? How does it compare with existing snapshotting systems? Implications of a snapshot-aware segment cleaner? How long does it talk to activate a snapshot? Implications on the crash recovery mechanism? Setup: quad core Intel i7 processor, 1.2TB NAND flash, 12GB RAM, Linux , older generation of Fusion-io VSL driver, 4K Block sizes Copyright © 2014 Fusion-io, Inc. All rights reserved.17

Snapshots in a Flash with ioSnap TM (patent pending technology) Performance of Regular operations Experiment: 4K read/writes using two threads. 16GB of data read/written. Asynchronous writes to device. Snapshot support does not impact regular operations Copyright © 2014 Fusion-io, Inc. All rights reserved.18 Bandwidth (MB/s)

Snapshots in a Flash with ioSnap TM (patent pending technology) Snapshot creation / Deletion Always 50us latency and 4K metadata (independent of data) Impact on foreground operations? Write 512 byte blocks to arbitrary logical addresses (total data of 3GB). Create a snapshot and then continue writing data to random locations (of 8 MB). Copyright © 2014 Fusion-io, Inc. All rights reserved.19 Validity bitmap CoW

Snapshots in a Flash with ioSnap TM (patent pending technology) Comparison with BTRFS (1) Impact on foreground latency upon snapshot creation Around 8 GB of sequential writes followed by a random workload interspersed by a snapshot every 5 sec Snapshots in ioSnap does not impact foreground latency Copyright © 2014 Fusion-io, Inc. All rights reserved.20

Snapshots in a Flash with ioSnap TM (patent pending technology) Comparison with BTRFS (2) Impact of snapshots on sustained bandwidth Around 200 GB of sequential writes followed by a random workload interspersed by a snapshot once every 15 sec Snapshots in ioSnap does not impact sustained bandwidth Copyright © 2014 Fusion-io, Inc. All rights reserved.21

Snapshots in a Flash with ioSnap TM (patent pending technology) Outline Introduction ioSnap TM Design Evaluation Conclusions Copyright © 2014 Fusion-io, Inc. All rights reserved.22

Snapshots in a Flash with ioSnap TM (patent pending technology) Conclusions “Make everything as simple as possible, but not simpler.” – Albert Einstein Flash is revolutionizing the storage industry Rethink data services to leverage flash’s capability & performance ioSnap: first flash-aware snapshotting system Leverages RoW capability in FTLs to implement snapshots Proposes usage of epochs to preserve log-time ordering Low-overhead instantaneous snapshots (performance & storage) Embraces rate-limiting to minimize impact to foreground traffic Activations are slow & can be 10s of sec for a TB size snapshot Copyright © 2014 Fusion-io, Inc. All rights reserved.23

fusionio.com | DELIVERING THE WORLD’S DATA. FASTER. Thank You