1. DADA – Dynamic Allocation of Disk Area
Jayaram Bobba
Vivek Shrivastava
Department of Computer Sciences, University of Wisconsin Madison

2. Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction
Existing Framework
Implementation
Issues
Results
Conclusions
Department of Computer Sciences, University of Wisconsin Madison

3. Department of Computer Sciences, University of Wisconsin Madison
Problem
Disk Volumes
Allocate physical space on creation
Multiple Disk Volumes on a physical disk
Unused space
Disk space not a premium
What If?
Department of Computer Sciences, University of Wisconsin Madison

4. Department of Computer Sciences, University of Wisconsin Madison
Motivation
Virtual Machine Environment
Multiple OS may create pressure on disk space
Access patterns may warrant dynamic relocation of disk blocks
Storage Area Networks
Storage area is a bottleneck resource
Dynamic Physical Allocation (LFS style)
Group writes from various volumes into a single physical write.
Department of Computer Sciences, University of Wisconsin Madison

5. Department of Computer Sciences, University of Wisconsin Madison
Related Work
HP AutoRAID
Dynamically change redundancy levels
Hot Data – Mirrored
Cold Data – RAID 5
Dynamic migration of data
Department of Computer Sciences, University of Wisconsin Madison

6. Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction
Existing Framework
Implementation
Issues
Results
Conclusions
Department of Computer Sciences, University of Wisconsin Madison

7. Department of Computer Sciences, University of Wisconsin Madison
Existing Framework
Logical Volume Manager (LVM)
User space tool that enables creation of logical volumes of a logical partition.
Supports resizing of volumes.
Device Mapper (DM)
Kernel driver that provides a level of indirection in address translation
Department of Computer Sciences, University of Wisconsin Madison

8. Department of Computer Sciences, University of Wisconsin Madison
LVM design
Physical Volume (PV)
Volume Group (VG)
Logical Volume (LV)
Physical Extent (PE)
Logical Extent (LE)
Department of Computer Sciences, University of Wisconsin Madison

9. Department of Computer Sciences, University of Wisconsin Madison
LVM design
Physical Device
Department of Computer Sciences, University of Wisconsin Madison

10. Department of Computer Sciences, University of Wisconsin Madison
LVM design PV PE
Department of Computer Sciences, University of Wisconsin Madison

11. Department of Computer Sciences, University of Wisconsin Madison
LVM design
LV 1 PV PE
Department of Computer Sciences, University of Wisconsin Madison

12. Department of Computer Sciences, University of Wisconsin Madison
LVM design LE
LV 1
LV2 PV PE
Department of Computer Sciences, University of Wisconsin Madison

13. Department of Computer Sciences, University of Wisconsin Madison
LVM design LE
LV 1
LV2 PV PE
Department of Computer Sciences, University of Wisconsin Madison

14. Department of Computer Sciences, University of Wisconsin Madison
LVM design LE VG
LV 1
LV2 PV PE
Department of Computer Sciences, University of Wisconsin Madison

15. Department of Computer Sciences, University of Wisconsin Madison
LVM – DM interaction
LVM2
libdevmapper
Userspace
Kernel
Device Mapper
Department of Computer Sciences, University of Wisconsin Madison

16. Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction
Existing Framework
Implementation
Issues
Results
Conclusions
Department of Computer Sciences, University of Wisconsin Madison

17. Department of Computer Sciences, University of Wisconsin Madison
Modifications
A trap mechanism from kernel driver to user-level LVM code.
Support for multiple segments in LV
Support for virtual mappings in driver
Department of Computer Sciences, University of Wisconsin Madison

18. Department of Computer Sciences, University of Wisconsin Madison
Initial State
LVM daemon
LVM
User space
Kernel space
Device Mapper
Unallocated Disk Space
Department of Computer Sciences, University of Wisconsin Madison

19. Department of Computer Sciences, University of Wisconsin Madison
Volume Group Creation
LVM daemon
LVM
User space
Kernel space
Device Mapper
PE (4MB)
25 Extents
Unallocated Disk Space
Department of Computer Sciences, University of Wisconsin Madison

20. Current Implementation
LVM daemon
Create LV
32MB
User A
LVM
User space
Kernel space
Device Mapper
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

21. Current Implementation
LVM daemon
User A
LVM
Map 0-7 extents for user A
User space
Kernel space
Device Mapper
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

22. Current Implementation
LVM daemon
User A
LVM
Map 0-7 extents for user A
User space
Kernel space
Device Mapper
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

23. Current Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
User space
Kernel space
Device Mapper
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

24. Current Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
map 8-11 extents for B
User space
Kernel space
Device Mapper
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

25. Current Implementation
LVM daemon
User A
LVM
User B
map 8-11 extents for B
User space
Kernel space
Device Mapper
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

26. Current Implementation
LVM daemon
User A
LVM
User B
Read/write
User space
Read/write
Kernel space
Device Mapper
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

27. Department of Computer Sciences, University of Wisconsin Madison
Volume Group Creation
LVM daemon
LVM
User space
Kernel space
Device Mapper
PE (4MB)
25 Extents
Unallocated Disk Space
Department of Computer Sciences, University of Wisconsin Madison

28. Modified Implementation
LVM daemon
Create LV1
32MB
User A
LVM
User space
Kernel space
Device Mapper
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

29. Modified Implementation
LVM daemon
User A
LVM
Virtually map 0-7 extents for user A
User space
Kernel space
Device Mapper
{0-7 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

30. Modified Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
User space
Kernel space
Device Mapper
{LV A 0-7 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

31. Modified Implementation
LVM daemon
User A
LVM
Create LV2
User B
16 MB
Virtually map 0-3 extents for B
User space
Kernel space
Device Mapper
{LV A 0-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

32. Modified Implementation
LVM daemon
User A
LVM
Write 6MB
User B
User space
Kernel space
Device Mapper
{LV A 0-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

33. Modified Implementation
LVM daemon
User A
LVM
Write 6MB
User B
trap
User space
Kernel space
Device Mapper
{LV A 0-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

34. Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
User B
trap
User space
Kernel space
Device Mapper
{LV A 0-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

35. Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
User B
Allocate 0-1 extents (8MB) LV A
User space
Kernel space
Device Mapper
{LV A 0-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

36. Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
User B
Return
User space
Kernel space
Device Mapper
{LV A 0-1 linear,2-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

37. Modified Implementation
LVM daemon
User A
LVM
User B
Write
10MB
User space
Kernel space
Device Mapper
{LV A 0-1 linear,2-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

38. Modified Implementation
LVM daemon
User A
LVM
User B
Write 10 MB
trap
User space
Kernel space
Device Mapper
{LV A 0-1 linear,2-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

39. Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
User B
Write 10 MB
trap
User space
Kernel space
Device Mapper
{LV A 0-1 linear,2-7 error}
{LV B 0-3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

40. Modified Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
User B
LV B Allocate 8-11 extents (12 MB)
Write 10 MB
User space
Kernel space
Device Mapper
{LV A 0-1 linear,2-7 error}
{LV B 0-2 linear, 3 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

41. Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction
Existing Framework
Implementation
Issues
Results
Conclusions
Department of Computer Sciences, University of Wisconsin Madison

42. Department of Computer Sciences, University of Wisconsin Madison
Issues
LVM implementation not efficient
LVM2
libdevmapper
Userspace
Kernel
Device Mapper
Department of Computer Sciences, University of Wisconsin Madison

43. Issues LVM implementation not efficient LVM2
Costly kernel-user space crossing
libdevmapper
Userspace
Kernel
Device Mapper
Department of Computer Sciences, University of Wisconsin Madison

44. Department of Computer Sciences, University of Wisconsin Madison
Issues
LVM implementation not efficient
LVM2
Normal function call
libdevmapper
Userspace
Device Mapper
Kernel
Department of Computer Sciences, University of Wisconsin Madison

45. Department of Computer Sciences, University of Wisconsin Madison
More Issues
On a trap
How much to allocate?
On demand
Pre-allocate
Where to allocate?
Temporal locality
Spatial locality
Free unused space
Department of Computer Sciences, University of Wisconsin Madison

46. Department of Computer Sciences, University of Wisconsin Madison
How much?
On-demand
Allocate only as many blocks as needed
Pre-allocate
Pre-allocate physical space for blocks likely
to be accessed in the future.
How much?
Where?
Department of Computer Sciences, University of Wisconsin Madison

47. Department of Computer Sciences, University of Wisconsin Madison
How much?
Tradeoff – Performance vs. Disk Space
Depends on the workload
I/O on critical path
Asynchronous I/O
Multithreaded Applications
Department of Computer Sciences, University of Wisconsin Madison

48. Department of Computer Sciences, University of Wisconsin Madison
Pre-Allocation
Think: Filesystem Block Allocation
Dumb Allocation
Stride Allocation
Multi-Strided Allocation
Department of Computer Sciences, University of Wisconsin Madison

49. On Demand allocation of disk area
Allocation Strategy
On Demand allocation of disk area
LVM daemon
User A
LVM
Write
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
{0-1 linear}{2-7 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

50. On Demand allocation of disk area
Allocation Strategy
On Demand allocation of disk area
LVM daemon
User A
LVM
Write
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
{0-1 linear}{2-7 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

51. On Demand allocation of disk area
Dumb Allocation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
{0-1 linear}{2-7 error}
Free Disk Space
Department of Computer Sciences, University of Wisconsin Madison

52. Strided Allocation (Motivation)
Department of Computer Sciences, University of Wisconsin Madison

53. On Demand allocation of disk area
Stride Allocation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write
Allocate 4 extents (16MB)
trap
User space
Kernel space
Device Mapper
{0-1 linear}{2-7 error}
Free Disk Space
PreAllocated Extent
Department of Computer Sciences, University of Wisconsin Madison

54. Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction
Existing Framework
Implementation
Issues
Results
Conclusions
Department of Computer Sciences, University of Wisconsin Madison

55. Department of Computer Sciences, University of Wisconsin Madison
Results
MicroBenchmarks
mkfs
Asynchronous io (aio) based workload
Trace based analysis
Trace collected by HP Research Labs used
in the D-GRAID paper
Department of Computer Sciences, University of Wisconsin Madison

56. Department of Computer Sciences, University of Wisconsin Madison
Micro Benchmark : mkfs
Does the # traps taken affect performance ?
Department of Computer Sciences, University of Wisconsin Madison

57. Micro Benchmark : Asynchronous I/O
Does the # traps taken affect performance ?
Department of Computer Sciences, University of Wisconsin Madison

58. Department of Computer Sciences, University of Wisconsin Madison
Trace Based Analysis
Dumb Allocation – What is the tradeoff in choosing extent size?
Department of Computer Sciences, University of Wisconsin Madison

59. Department of Computer Sciences, University of Wisconsin Madison
Trace Based Analysis
Dumb Allocation – What is the tradeoff in choosing extent size?
Department of Computer Sciences, University of Wisconsin Madison

60. Department of Computer Sciences, University of Wisconsin Madison
Trace Based Analysis
Strided Allocation – What is the tradeoff in choosing extent size?
Department of Computer Sciences, University of Wisconsin Madison

61. Department of Computer Sciences, University of Wisconsin Madison
Trace Based Analysis
Strided Allocation – What is the tradeoff in choosing extent size?
Department of Computer Sciences, University of Wisconsin Madison

62. Trace Based Analysis : Dumb vs. Stride (ext2)
Department of Computer Sciences, University of Wisconsin Madison

63. Trace Based Analysis : Dumb vs. Stride (ext3)
Department of Computer Sciences, University of Wisconsin Madison

64. Department of Computer Sciences, University of Wisconsin Madison
Outline
Introduction
Existing Framework
Implementation
Issues
Results
Conclusions
Department of Computer Sciences, University of Wisconsin Madison

65. Department of Computer Sciences, University of Wisconsin Madison
Conclusions
Tradeoff between performance penalty and space
Useful in environments where disk space is the bottleneck
Techniques like Pre Allocation can be used to reduce the performance penalty.
For practical implementation, LVM must be placed in kernel address space
Department of Computer Sciences, University of Wisconsin Madison

66. Department of Computer Sciences, University of Wisconsin Madison
Acknowledgments
Vijayan
Prof. Remzi
Department of Computer Sciences, University of Wisconsin Madison