1. PARALLEL DATA LABORATORY Carnegie Mellon University A framework for implementing IO-bound maintenance applications Eno Thereska

2. Eno Thereska November 2004http://www.pdl.cmu.edu/2 Disk maintenance applications Lots of disk maintenance apps data protection (backup) storage optimization (defrag, load bal.) caching (write-backs) Important for system robustness Background activities should eventually complete ideally without interfering with primary apps

3. Eno Thereska November 2004http://www.pdl.cmu.edu/3 Current approaches Implement maintenance application as foreground application competes for bandwidth or off-hours only Client workload Backup throughput MB/s

4. Eno Thereska November 2004http://www.pdl.cmu.edu/4 Current approaches Trickle maintenance activity periodically lost opportunities due to inadequate scheduling decisions Client workload Cache write- back throughput MB/s

5. Eno Thereska November 2004http://www.pdl.cmu.edu/5 Real support for background applications Push maintenance activities in the background priorities and explicit support for them APIs allow application expressiveness Storage subsystem does the scheduling using idle time, if there is any using otherwise-wasted rotational latency in a busy system

6. Eno Thereska November 2004http://www.pdl.cmu.edu/6 Outline Motivation and overview The freeblock subsystem Background application interfaces Example applications Conclusions

7. Eno Thereska November 2004http://www.pdl.cmu.edu/7 The freeblock subsystem Disk scheduling subsystem supporting new APIs with explicit background requests Finds time for background activities by detecting idle time (short and long bursts) by utilizing otherwise-wasted rotational latency in a busy system

8. Eno Thereska November 2004http://www.pdl.cmu.edu/8 After reading blue sector After BLUE read

9. Eno Thereska November 2004http://www.pdl.cmu.edu/9 Red request scheduled next After BLUE read

10. Eno Thereska November 2004http://www.pdl.cmu.edu/10 Seek to Red’s track After BLUE readSeek for RED

11. Eno Thereska November 2004http://www.pdl.cmu.edu/11 Wait for Red sector to reach head After BLUE readSeek for REDRotational latency

12. Eno Thereska November 2004http://www.pdl.cmu.edu/12 Read Red sector After BLUE readSeek for REDRotational latencyAfter RED read

13. Eno Thereska November 2004http://www.pdl.cmu.edu/13 Traditional service time components Rotational latency is wasted time After BLUE readSeek for REDRotational latencyAfter RED read

14. Eno Thereska November 2004http://www.pdl.cmu.edu/14 Rotational latency gap utilization After BLUE read

15. Eno Thereska November 2004http://www.pdl.cmu.edu/15 Seek to Third track After BLUE readSeek to Third SEEK

16. Eno Thereska November 2004http://www.pdl.cmu.edu/16 Free transfer After BLUE readSeek to ThirdFree transfer SEEKFREE TRANSFER

17. Eno Thereska November 2004http://www.pdl.cmu.edu/17 Seek to Red’s track After BLUE readSeek to ThirdSeek to REDFree transfer SEEK FREE TRANSFER

18. Eno Thereska November 2004http://www.pdl.cmu.edu/18 Read Red sector After BLUE readSeek to ThirdSeek to REDAfter RED readFree transfer SEEK FREE TRANSFER

19. Eno Thereska November 2004http://www.pdl.cmu.edu/19 Steady background I/O progress 0 5 10 15 20 25 30 35 40 0102030405060708090100 % disk utilization by foreground (random 4KB) reads/writes “Free” MB/s from idle time from rotational gaps

20. Eno Thereska November 2004http://www.pdl.cmu.edu/20 The freeblock subsystem (cont…) Implemented in FreeBSD Efficient scheduling low CPU and memory utilizations Minimal impact on foreground workloads < 2% See refs for more details

21. Eno Thereska November 2004http://www.pdl.cmu.edu/21 Outline Motivation and overview The freeblock subsystem Background application interfaces Example applications Conclusions

22. Eno Thereska November 2004http://www.pdl.cmu.edu/22 Application programming interface (API) goals Work exposed but done opportunistically all disk accesses are asynchronous Minimized memory-induced constraints late binding of memory buffers late locking of memory buffers “Block size” can be application-specific Support for speculative tasks Support for rate control

23. Eno Thereska November 2004http://www.pdl.cmu.edu/23 API description: task registration fb_read (addr_range, blksize,…) fb_write (addr_range, blksize,…) Foreground Background foreground scheduler background scheduler application

24. Eno Thereska November 2004http://www.pdl.cmu.edu/24 API description: task completion callback_fn (addr, buffer, flag, …) Foreground Background foreground scheduler background scheduler application

25. Eno Thereska November 2004http://www.pdl.cmu.edu/25 API description: late locking of buffers buffer = getbuffer_fn (addr, …) Foreground Background foreground scheduler background scheduler application

26. Eno Thereska November 2004http://www.pdl.cmu.edu/26 API description: aborting/promoting tasks fb_abort (addr_range, …) fb_promote (addr_range, …) Foreground Background foreground scheduler background scheduler application

27. Eno Thereska November 2004http://www.pdl.cmu.edu/27 Complete API

28. Eno Thereska November 2004http://www.pdl.cmu.edu/28 Designing disk maintenance applications APIs talk in terms of logical blocks (LBNs) Some applications need structured version as presented by file system or database Example consistency issues application wants to read file “foo” registers task for inode’s blocks by time blocks read, file may not exist anymore!

29. Eno Thereska November 2004http://www.pdl.cmu.edu/29 Application does not care about structure scrubbing, data migration, array reconstruction Coordinate with file system/database cache write-backs, LFS cleaner, index generation Utilize snapshots backup, background fsck Designing disk maintenance applications

30. Eno Thereska November 2004http://www.pdl.cmu.edu/30 Outline Motivation and overview The freeblock subsystem Background application interfaces Example applications Conclusions

31. Eno Thereska November 2004http://www.pdl.cmu.edu/31 Example #1: Physical backup Backup done using snapshots backup application freeblock subsystem snapshot subsystem (in FS) getblks() sys_fb_read() sys_fb_getrecord()

32. Eno Thereska November 2004http://www.pdl.cmu.edu/32 Example #1: Physical backup Experimental setup 18 GB Seagate Cheetah 36ES FreeBSD in-kernel implementation PIII with 384MB of RAM 3 benchmarks used: Synthetic, TPC-C, Postmark snapshot includes 12GB of disk GOAL: read whole snapshot for free

33. Eno Thereska November 2004http://www.pdl.cmu.edu/33 Backup completed for free 0 10 20 30 40 50 60 70 80 90 Idle systemSyntheticTPC-CPostmark Backup time (mins) < 2% impact on foreground workload

34. Eno Thereska November 2004http://www.pdl.cmu.edu/34 Example #2: Cache write-backs Must flush dirty buffers for space reclamation for persistence (if memory is not NVRAM) Simple cache manager extensions fb_write(dirty_buffer,…) getbuffer_fn(dirty_buffer,…) fb_promote(dirty_buffer,…) fb_abort(dirty_buffer,…) when blocks become dirty when flush is forced when block dies in cache calling back to lock

35. Eno Thereska November 2004http://www.pdl.cmu.edu/35 Example #2: Cache w rite-backs Experimental setup 18 GB Seagate Cheetah 36ES PIII with 384MB of RAM controlled experiments with synthetic workload benchmarks (same as used before) in FreeBSD syncer daemon wakes up every 1 sec and flushes entries that have been dirty > 30secs GOAL: write back dirty buffers for free

36. Eno Thereska November 2004http://www.pdl.cmu.edu/36 Foreground read:write has impact 0 20 40 60 80 100 01:21:12:1 read-write ratio % dirty buffers cleaned for free 0 20 40 60 80 100 01:21:12:1 % improvement in avg. resp. time read-write ratio

37. Eno Thereska November 2004http://www.pdl.cmu.edu/37 95% of NVRAM cleaned for free 0 20 40 60 80 100 LRU+SyncerLRU only % dirty buffers cleaned for free 0 20 40 60 80 100 LRU+SyncerLRU only % improvement in avg. resp. time

38. Eno Thereska November 2004http://www.pdl.cmu.edu/38 10-20% improvement in overall perf. 0 10 20 30 40 50 % dirty buffers cleaned for free 0 10 20 30 40 50 % improvement in app. throughput Synthetic TPC-C Postmark

39. Eno Thereska November 2004http://www.pdl.cmu.edu/39 Example #3: Layout reorganizer Layout reorganization improves access latencies defragmentation is a type of reorganization typical example of background activity Our experiment: disk used is 18GB we want to defrag up to 20% of it goal: defrag for free

40. Eno Thereska November 2004http://www.pdl.cmu.edu/40 Disk Layout Reorganized for Free! Random Circular Track Shuffle 0 100 200 300 400 500 600 1%10%20%1%10%20% 8MB64MB Reorganizer buffer size (MB) Reorganization time (mins)

41. Eno Thereska November 2004http://www.pdl.cmu.edu/41 Other maintenance applications Virus scanner LFS cleaner Disk scrubber Data mining Data migration

42. Eno Thereska November 2004http://www.pdl.cmu.edu/42 Summary Framework for building background storage applications Asynchronous interfaces applications describe what they need storage subsystem satisfies their needs Works well for real applications http://www.pdl.cmu.edu/Freeblock/