Conquest: Better Performance Through A Disk/Persistent-RAM Hybrid File System USENIX 2002 An-I Andy Wang Peter Reiher Gerald Popek University of California, Los Angeles Geoffrey Kuenning Harvey Mudd College

2 Conquest Overview File systems are optimized for disks Performance problem Complexity Now we have tons of inexpensive RAM What can we do with that RAM?

3 Conquest Approach Combine disk and persistent RAM (e.g., battery-backed RAM) in a novel way Simplification > 20% fewer semicolons than ext2, reiserfs, and SGI XFS Performance (under popular benchmarks) 24% to 1900% faster than LRU disk caching

4 Motivation Most file systems are built for disks Problems with the disk assumption: Performance Complexity Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

5 Hardware Evolution KHz 1 MHz 1 GHz CPU (50% /yr) Memory (50% /yr) Disk (15% /yr) Accesses Per Second (Log Scale) (1 sec : 6 days)(1 sec : 3 months) Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

6 Inside the Pandora’s Box Disk arm Disk platters Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion Access time = seek time (disk arm) + rotational delay (disk platter) + transfer time

7 Disk Optimization Methods Disk arm scheduling Group information on disk Disk readahead Buffered writes Disk caching Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion Data mirroring Hardware parallelism

8 Complexity Bytes synchronization predictive readahead cache replacement elevator algorithm data clustering data consistency asynchronous write Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

[Caceres et al., 1993; Hillyer et al., 1996; Qualstar 1998; Tanisys 1999; Micron Semiconductor Products 2000; Quantum 2000]9 Storage Media Alternatives accesses/sec (log) $/MB (log) persistent RAM Magnetic RAM? (write once) flash memory disk tape battery-backed DRAM Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

[Grochowski 2000]10 Price Trend of Persistent RAM Year $/MB (log) paper/film 3.5” HDD 2.5” HDD 1” HDD Persistent RAM Booming of digital photography 4 to 10 GB of persistent RAM Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

11 Old Order; New World Disk staying around Cost, capacity, power, heat RAM as a viable storage alternative PDAs, digital cameras, MP3 players More architectural changes due to RAM A big assumption change from disk Rethink data structures, interface, applications Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

12 What does it take to design and build a system that assumes ample persistent RAM as the primary storage medium? Getting a Fresh Start Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

13 Conquest Design and build a disk/persistent-RAM hybrid file system Deliver all file system services from memory, with the exception of high-capacity storage Benefits: Simplicity Performance Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

14 Simplicity Remove disk-related complexities for most files Make things simpler for disk as well Less complexity Fewer bugs Easier maintenance Shorter data path Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

15 Overall All management performed in memory Memory data path No disk-related overhead Disk data path Faster speed due to simpler access models Performance Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

16 Conquest Components Media management Metadata management Allocation service Persistence support Resiliency support Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

[Iram 1993; Douceur et al., 1999; Roselli et al., 2000]17 User Access Patterns Small files Take little space (10%) Represent most accesses (90%) Large files Take most space Mostly sequential accesses Except database applications Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

18 Files Stored in Persistent RAM Small files (< 1MB) No seek time or rotational delays Fast byte-level accesses Contiguous allocation Metadata Fast synchronous update No dual representations Executables and shared libraries In-place execution Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

19 Memory Data Path of Conquest Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion Conventional file systems IO buffer Disk management Storage requests IO buffer management Disk Persistence support Conquest Memory Data Path Storage requests Persistence support Battery-backed RAM Small file and metadata storage

[Devlinux.com 2000]20 Large-File-Only Disk Storage Allocate in big chunks Lower access overhead Reduced management overhead No fragmentation management No tricks for small files Storing data in metadata No elaborate data structures Wrapping a balanced tree onto disk cylinders Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

21 Sequential-Access Large Files Sequential disk accesses Near-raw bandwidth Well-defined readahead semantics Read-mostly Little synchronization overhead (between memory and disk) Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

22 Disk Data Path of Conquest Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion Conventional file systems IO buffer Disk management Storage requests IO buffer management Disk Persistence support Conquest Disk Data Path IO buffer management IO buffer Storage requests Disk management Disk Battery-backed RAM Small file and metadata storage Large-file-only file system

23 Random-Access Large Files Random access? Common definition: nonsequential access A typical movie has 150 scene changes MP3 stores the title at the end of the files Near Sequential access? Simplify large-file metadata representation significantly Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

24 Logical File Representation File Name(s) i-node File attributes Data Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

25 Physical File Representation File Name(s) i-node File attributes Data locations Data blocks Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

26 Ext2 Data Representation data block location index block location data block location index block location data block location i-node 10 data block location index block location Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

27 Problems with Ext2 Design -Designed for disk storage -Optimization for small files makes things complex -Random-access data structure for large files that are accessed mostly sequentially -Data access time dependent on the byte position in a file -Maximum file size is limited Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

28 Conquest Representation Persistent RAM Hash(file name) = location of data Offset(location of data) Disk storage Per-file, doubly linked list of disk block segments (stored in persistent RAM) Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

29 Conquest Design + Direct data access for in-core files + Worse case: sequential memory search for infrequent random accesses to on-disk files + Maximum file size limited by physical storage Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

30 Implementation Status Kernel module under Linux Fully functional and POSIX compliant Modified memory manager to support Conquest persistence Preparing for office-wide deployment Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

31 Conquest Evaluation Architectural simplification Feature count Performance improvement Memory-only workload Memory and disk workload Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

32 Conventional Data Path Buffer allocation management Buffer garbage collection Data caching Metadata caching Predictive readahead Write behind Cache replacement Metadata allocation Metadata placement Metadata translation Disk layout Fragmentation management Conventional file systems IO buffer Disk management Storage requests IO buffer management Disk Persistence support Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

33 Memory Path of Conquest Buffer allocation management Buffer garbage collection Data caching Metadata caching Predictive readahead Write behind Cache replacement Metadata allocation Metadata placement Metadata translation Disk layout Fragmentation management Conquest Memory Data Path Storage requests Persistence support Battery-backed RAM Small file and metadata storage Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion Memory manager encapsulation

34 Disk Path of Conquest Buffer allocation management Buffer garbage collection Data caching Metadata caching Predictive readahead Write behind Cache replacement Metadata allocation Metadata placement Metadata translation Disk layout Fragmentation management Conquest Disk Data Path IO buffer management IO buffer Storage requests Disk management Disk Battery-backed RAM Small file and metadata storage Large-file-only file system Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

[Katcher 1997; Sweeney et al., 1996; Card et al., 1999; Namesys 2002]35 Conquest is comparable to ramfs At least 24% faster than the LRU disk cache ISP workload ( s, web-based transactions) PostMark Benchmark Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion 250 MB working set with 2 GB physical RAM

36 When both memory and disk components are exercised, Conquest can be several times faster than ext2fs, reiserfs, and SGI XFS PostMark Benchmark Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion 10,000 files, 3.5 GB working set with 2 GB physical RAM > RAM<= RAM

37 When working set > RAM, Conquest is 1.4 to 2 times faster than ext2fs, reiserfs, and SGI XFS PostMark Benchmark Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion 10,000 files, 3.5 GB working set with 2 GB physical RAM

38 Lessons Learned Faster than LRU caching, unexpected Heavyweight disk handling Severe penalty for accesses to content Matching user access patterns to storage media offers considerable simplification and better performance Not an automatic result Need careful design Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

39 Conclusion Conquest demonstrates how rethinking changes in underlying assumptions can lead to significant architectural and performance improvements Radical changes in hardware, applications, and user expectations in the past decade should lead us to rethink other aspects of OS as well. Motivation – Conquest Design – Conquest Components – Performance Evaluation – Conclusion

40 Questions... Conquest: Andy Wang: