X-RAY: A Non-Invasive Exclusive Caching Mechanism for RAIDs Lakshmi N. Bairavasundaram Muthian Sivathanu Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau ADvanced Systems Laboratory Computer Sciences Department University of Wisconsin – Madison

Introduction  Caching in modern systems Multiple levels Storage: 2-level hierarchy  Level 1: File system (FS) cache Software-managed Main memory of host/client LRU-like cache replacement  Level 2: RAID cache Firmware-managed Memory inside RAID system Usually LRU replacement File system cache RAID cache RAID Application Host

Introduction – contd.  LRU Replace LRU block Cache placement on read Read Block no. 10 LRUMRU Read Block no …… ……..4523

Introduction – contd.  LRU Replace LRU block Cache placement on read  2 levels of LRU Redundant contents …….. Read Block no MRU LRU 10 LRU10 MRU LRU …. FS Cache RAID Cache

Introduction – contd.  LRU Cache placement on read Replace LRU block  2 levels of LRU Redundant contents  Goal: Exclusive caching 10 LRU10 MRU LRU …. FS Cache RAID Cache

Improved RAID Caching  Multi-Queue (Zhou et al. 2001) Add frequency component to cache policy Not strictly exclusive!  DEMOTE (Wong and Wilkes 2002) Change interface to disk File system issues “cache place” command Has perfect information and hence perfectly exclusive caches Interface changes – difficult to deploy

Ideal RAID Cache  Exclusive caching File system and RAID caches should have different contents  Global LRU Known to work well RAID cache should be a victim cache  No interface changes …. …… FS Cache RAID Cache Block Read Victim Block LRU MRU

X-RAY  Observes disk traffic Reads and writes to data and metadata  Builds a model of the FS cache Uses semantic knowledge Predicts size and contents of FS cache  Identifies set of exclusive blocks Recent victims of the FS cache  Reads blocks from disk into cache  Result A nearly exclusive cache without interface changes File system cache RAID cache RAID Host Model of FS cache X-RAY

Talk Outline  Introduction  File Systems  Information and Inferences  X-RAY Cache Design  Results  Conclusion

File System Operation  Applications perform file reads and writes  File system (Unix) Translates file accesses to disk block requests Metadata  To maintain application data on disk and manage disk blocks  Periodically written to disk  Examples: inodes, bitmap blocks

File System Operation  Inode  Pointers to data blocks  File access information Inode Data Blocks Latest access time Pointers to data blocks File

File System Operation  File access Use inode to obtain pointers to disk data blocks Read corresponding blocks from disk if they are not in FS cache Update the access time information in inode  Metadata updates Periodically check for “dirty” inodes and write to disk

The Problem  To observe disk traffic and infer the contents of FS cache  Why difficult? FS cache size changes over time  Shares main memory with virtual memory system

The Problem  To observe disk traffic and infer the contents of FS cache  Why difficult? FS cache size changes over time Disk cannot observe all FS-level accesses Read block: 10 Disk Read LRU MRU FS Cache FS Cache Model RAID

The Problem  To observe disk traffic and infer the contents of FS cache  Why difficult? FS cache size changes over time Disk cannot observe all FS-level accesses Read block: 10 Disk Read LRU MRU 13 FS Cache FS Cache Model RAID

The Problem Read block: LRU MRU FS Cache FS Cache Model RAID  To observe disk traffic and infer the contents of FS cache  Why difficult? FS cache size changes over time Disk cannot observe all FS-level accesses

The Problem Read block: LRU MRU FS Cache FS Cache Model RAID  To observe disk traffic and infer the contents of FS cache  Why difficult? FS cache size changes over time Disk cannot observe all FS-level accesses  Key observation We need information about accesses that hit in FS cache File system maintains access information in inodes

Talk Outline  Introduction  File Systems  Information and Inferences  X-RAY Cache Design  Results  Conclusion

Information  Obtain information from observing disk traffic  Knowledge of file system structures and operations File system maintains time of last access in inodes Periodic inode writes Assuming whole file access, all blocks are in FS cache  Assume file system cache policy is LRU

Inferences  Read for data block Block will be placed in file system cache (MRU block)  Read for previously read data block Block became victim in file system cache Blocks with an earlier access time should also be victims  Inode write: new access time, no disk read observed All blocks belonging to file are in FS cache Other blocks with later access time should also be present

Talk Outline  Introduction  File Systems  Information and Inferences  X-RAY Cache Design  Results  Conclusion

Design  Recency list (R-list) List of data blocks ordered by access time  Cache Begin (CB) pointer Divides R-list into inclusive and exclusive regions  RAID Cache contents Subset of blocks in exclusive region LRU MRU A, 1 B, 1 D, 3C, 2 F, 5 E, 3 CB Inclusive region Exclusive region Block numberAccess time Blocks the RAID should cache Blocks expected to be in FS cache

Disk Read LRUMRU A, 1B, 1C, 2D, 3 E, 3F, 4 CB Inclusive region Exclusive region Read Block ‘D’ ; time = 6

Disk Read LRUMRU A, 1B, 1C, 2D, 3 E, 3F, 4 CB Inclusive region Exclusive region Read Block ‘D’ ; time = 6

Disk Read LRUMRU A, 1B, 1C, 2D, 6 E, 3F, 4 CB Inclusive region Exclusive region Read Block ‘D’ ; time = 6

Inode Write – Access time change LRUMRU A, 1B, 1C, 2D, 3 E, 4F, 5 CB Inclusive region Exclusive region G, 7 Inode “23” : access time = 6 Semantic knowledge Inode “23” == blocks D & E Blocks D, E : access time = 6

Inode Write – Access time change LRUMRU A, 1B, 1C, 2D, 3 E, 4F, 5 CB Inclusive region Exclusive region G, 7 Blocks D, E : access time = 6 Inode “23” : access time = 6

Inode Write – Access time change LRUMRU A, 1B, 1C, 2F, 5 D, 6E, 6 CB Inclusive region Exclusive region G, 7 Blocks D, E : access time = 6 Inode “23” : access time = 6

X-RAY Cache LRUMRU A, 1B, 1 C, 2F, 5D, 6E, 6 CB Inclusive region Exclusive region G, 7 RAID Cache (size = 2 blocks)  Keep track of additions to window in exclusive region

X-RAY Cache  Read newly-added blocks from disk Replace blocks no longer in the window Additional disk bandwidth Idle time, extra internal bandwidth, freeblock scheduling LRUMRU A, 1B, 1 C, 2F, 5D, 6E, 6 CB Inclusive region Exclusive region G, 7 RAID Cache (size = 2 blocks)

Talk Outline  Introduction  File Systems  Information and Inferences  X-RAY Cache Design  Results Tracking FS Cache Contents RAID Cache Performance  Conclusion

Results – Tracking  Accurate size and content prediction  Highly responsive to FS cache size changes  Tolerates changes in inode write interval  Partial file reads X-RAY performs well if percentage of partially accessed files is < 40% (typical traces have less than 30%)

Results – Cache Performance  Performs better than LRU and Multi-Queue  Close to DEMOTE, in spite of imperfect information  Hit rate advantage translates to lower read latency

Additional Results  File system cache policy is not LRU Clock, 2Q X-RAY performs nearly as well as before It performs better than both LRU and Multi-Queue  Idle time requirements X-RAY reads blocks into cache only during idle time It performs well if idle time is greater than one-third of actual idle time observed in the trace  More in the paper …

Conclusion  Easy deployment is an important goal in developing technology Avoid interface changes – use non-invasive mechanisms  Higher-level systems maintain various pieces of information about data they manage Provide low-level systems with basic semantic knowledge  Semantic intelligence for managing RAID caches Use access information in metadata to track file system cache contents and cache exclusive blocks In spite of imperfect information, X-RAY performs nearly as well as changing the interface  Semantically-smart Disk Systems Availability, security and performance improvements

Questions ? ADvanced Systems Laboratory (ADSL) Computer Sciences, University of Wisconsin-Madison