1. HyLog: A High Performance Approach to Managing Disk Layout Wenguang Wang Yanping Zhao Rick Bunt Department of Computer Science University of Saskatchewan Saskatoon, Canada USENIX FAST 2004April 1, 2004 Wenguang Wang

2. April 1, 2004 USENIX FAST 2004 Wenguang Wang 2 Background The write performance of a storage system is impacted by –the disk characteristics Disk positioning time Transfer bandwidth –the strategy for writing Overwrite LFS (Log-structured File System)

3. April 1, 2004 USENIX FAST 2004 Wenguang Wang 3 Overwrite Idea: new data are overwritten on top of old data Problems: lots of time lost in disk arm positioning in workloads with small writes scattered over the disk

4. April 1, 2004 USENIX FAST 2004 Wenguang Wang 4 LFS Idea: new data are accumulated and written to new disk locations in large sequential transfers Assumptions of the disk characteristics: large sequential transfers are more efficient than small block transfers

5. April 1, 2004 USENIX FAST 2004 Wenguang Wang 5 LFS (cont.) Advantages –good write performance –no small write penalty on RAID-5 –fast recovery –easy to support snapshot and versioning (WAFL) Problems: segment cleaning is expensive –For a year 1991 disk, TPC-B workload, and 50% disk space utilization, cleaning overhead reduces overall system throughput by 33% (Seltzer et al. USENIX95)

6. April 1, 2004 USENIX FAST 2004 Wenguang Wang 6 Motivation Observation: disk sequential transfer bandwidth has improved 10x more than positioning time Question: how are Overwrite and LFS affected by this trend? DEC RZ26 (year 1991) Cheetah X15 36LP (year 2003) Diff. Positioning time15ms5.6ms2.7x Transfer B/w2.3MB/s61MB/s27x

7. April 1, 2004 USENIX FAST 2004 Wenguang Wang 7 Objective Revisit the performance of LFS under modern and future disks Evaluate the performance of LFS under disk arrays and concurrent users Attempt to perform better than LFS and Overwrite

8. April 1, 2004 USENIX FAST 2004 Wenguang Wang 8 Outline Background, Motivation, and Objective The analysis of LFS and Overwrite The design of HyLog Experimental methodology and results Conclusions and future work

9. April 1, 2004 USENIX FAST 2004 Wenguang Wang 9 Experimental Parameters Three SCSI disks –DEC RZ26 (year 1991) –Quantum atlas10k (year 1999) –Cheetah X15 36LP (year 2003) Page size: 8KB Workload: uniformly distributed random update (TPC-B)

10. April 1, 2004 USENIX FAST 2004 Wenguang Wang 10 In Overwrite: In LFS: T 1 > T 2 Segment I/O Efficiency = T 1 / T 2 Modeling Write Performance Time to write N pages is T 1 Time to write a segment containing N pages is T 2

11. April 1, 2004 USENIX FAST 2004 Wenguang Wang 11 A Simple Scenario Assume the segments to be cleaned are always 80% utilized (cleaning space utilization = 80%) LFS requires 5 seg. reads and 4 seg. writes to reclaim a free segment LFS requires 10 seg. I/Os (9 seg. for cleaning, 1 seg. for new data) to write a segment If Segment I/O Efficiency > 10, LFS is still faster than Overwrite!

12. April 1, 2004 USENIX FAST 2004 Wenguang Wang 12 Segment I/O Efficiency 33

13. April 1, 2004 USENIX FAST 2004 Wenguang Wang 13 Overwrite vs. LFS Overwrite LFS cleaning LFS hole- plugging 1999 disk 0.88

14. April 1, 2004 USENIX FAST 2004 Wenguang Wang 14 Overwrite vs. LFS The crossing point where LFS has the same performance as Overwrite Year of Disk Cleaning Space Utilization Disk Space Utilization 19910.520.74 19990.880.94 20030.940.97

15. April 1, 2004 USENIX FAST 2004 Wenguang Wang 15 Disk Access Characteristics In most workloads, most writes are to a small number of pages (hot pages) Impact of skewness on LFS performance –Most of the cleaning cost comes from cold pages –Most of the good write performance comes from accumulating the writes to hot pages

16. April 1, 2004 USENIX FAST 2004 Wenguang Wang 16 HyLog (Hybrid Log-structured Approach) Separates the disk into two partitions: hot partition and cold partition Uses log-structured approach to manage the hot partition Uses overwrite to manage the cold partition

17. April 1, 2004 USENIX FAST 2004 Wenguang Wang 17 Performance Potential of HyLog Write Cost Disk: year 2003, workload: 80% references are in 20% pages Overwrite LFS HyLog Overwrite LFS HyLog

18. April 1, 2004 USENIX FAST 2004 Wenguang Wang 18 Design of HyLog Key design issue: page separating algorithm –Collects page write frequencies –Finds the hot page proportion to minimize expected write cost –Determines the threshold of write frequency from the desired hot page proportion –Uses the threshold to distinguish hot pages from cold pages

19. April 1, 2004 USENIX FAST 2004 Wenguang Wang 19 Evaluation Methodology Trace driven simulation –Overwrite, LFS, WOLF, and HyLog are implemented –TPC-C, Email, and OLTP traces –Year 1999, 2003, and 2008 disk models –No think time between requests Metrics –Throughput: # I/O requests finished per second

20. April 1, 2004 USENIX FAST 2004 Wenguang Wang 20 HyLog adjusts the hot page proportion between 35-45% Results – HyLog Page Separating Algorithm TPC-C trace with 20 users and 4 disks, 98% disk space utilization

21. April 1, 2004 USENIX FAST 2004 Wenguang Wang 21 Results – Disk Space Utilization TPC-C trace with 20 users and 4 disks 1999 disk

22. April 1, 2004 USENIX FAST 2004 Wenguang Wang 22 Results – Disk Type TPC-C trace with 20 users and 4 disks Overwrite (year03 disk) LFS/HyLog (year03 disk) 1999 disk 2003 disk

23. April 1, 2004 USENIX FAST 2004 Wenguang Wang 23 Results – Disk Type (cont.) TPC-C trace with 20 users and 4 disks Overwrite (year03 disk) LFS/HyLog (year03 disk) Overwrite (year08 disk) LFS/HyLog (year08 disk) 1999 disk 2003 disk 2008 disk

24. April 1, 2004 USENIX FAST 2004 Wenguang Wang 24 Results – Number of Users TPC-C trace, disk space utilization 98%, year 1999 disk

25. April 1, 2004 USENIX FAST 2004 Wenguang Wang 25 Results – Number of Disks TPC-C trace, disk space utilization 98%, year 1999 disk

26. April 1, 2004 USENIX FAST 2004 Wenguang Wang 26 Results – RAID-5 TPC-C trace, year 1999 disk, 8-disk RAID-0, 9-disk RAID-5

27. April 1, 2004 USENIX FAST 2004 Wenguang Wang 27 Results – Other Traces Year 1999 disk

28. April 1, 2004 USENIX FAST 2004 Wenguang Wang 28 Conclusions On modern and future disks, LFS significantly outperforms Overwrite unless the disk space utilization is very high HyLog performs comparably to the best of Overwrite, LFS, and WOLF

29. April 1, 2004 USENIX FAST 2004 Wenguang Wang 29 Future Work Add fast recovery support in HyLog –All meta-data are considered as hot pages Stabilize NetBSD LFS implementation and measure its performance Implement and evaluate HyLog in NetBSD

30. April 1, 2004 USENIX FAST 2004 Wenguang Wang 30

31. April 1, 2004 USENIX FAST 2004 Wenguang Wang 31 Results – # Users and Disks TPC-C trace, disk space utilization 98%, year 1999 disk HyLog LFS Number of Users Number of Disks