DADA

1
DADA Dynamic Allocation of Disk Area

• Jayaram Bobba
• Vivek Shrivastava

2
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

3
Problem

• Disk Volumes
• Allocate physical space on creation
• Multiple Disk Volumes on a physical disk
• Unused space
• Disk space not a premium
• What If?

4
Motivation

• Virtual Machine Environment
• Add Some points
• Storage Area Networks
• Storage area is the bo
• Dynamic Physical Allocation (LFS style)

5
Related Work

• HP AutoRAID
• Dynamically change redundancy levels
• Hot Data Mirrored
• Cold Data RAID 5
• Dynamic migration of data

6
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

7
Existing Framework

• Logical Volume Manager (LVM)
• User space tool that enables creation of logical
  volumes of a logical partition.
• Supports resizing of volumes.
• Device Mapper (DM)
• Kernel driver that provides a level of
  indirection in address translation

8
LVM design

• Volume Group (VG)
• Physical Volume (PV)
• Logical Volume (LV)
• Physical Extent (PE)
• Logical Extent (LE)

9
LVM design
Physical Device
10
LVM design
PV
PE
11
LVM design
PV
PE
12
LVM design
LV 1
PV
PE
13
LVM design
LE
LV 1
LV2
PV
PE
14
LVM design
LE
LV 1
LV2
PV
PE
15
LVM design
LE
VG
LV 1
LV2
PV
PE
16
DM design
Userspace Applications
filesystem interface
ioctl interface
control interface
block interface
core device-mapper
mapping/target interface
linear
striped
snapshot
multipath
log
kcopyd
path selectors
hardware handlers
emc
round-robin
17
LVM DM interaction
Device Mapper
Kernel
Userspace
libdevmapper
LVM2
18
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

19
Modifications

• A trap mechanism from kernel driver to user-level
  LVM code.
• Support for multiple segments in LV
• Support for virtual mappings in driver

20
Initial State
LVM daemon
LVM
User space
Kernel space
Device Mapper
Unallocated Disk Space
21
Volume Group Creation
LVM daemon
LVM
User space
Kernel space
Device Mapper
PE (4MB)
25 Extents
Unallocated Disk Space
22
Current Implementation
LVM daemon
Create LV 32MB
User A
LVM
User space
Kernel space
Device Mapper
Free Disk Space
23
Current Implementation
LVM daemon
User A
LVM
Map 0-7 extents for user A
User space
Kernel space
Device Mapper
Free Disk Space
24
Current Implementation
LVM daemon
User A
LVM
Map 0-7 extents for user A
User space
Kernel space
Device Mapper
Free Disk Space
25
Current Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
User space
Kernel space
Device Mapper
Free Disk Space
26
Current Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
map 8-11 extents for B
User space
Kernel space
Device Mapper
Free Disk Space
27
Current Implementation
LVM daemon
User A
LVM
User B
map 8-11 extents for B
User space
Kernel space
Device Mapper
Free Disk Space
28
Current Implementation
LVM daemon
User A
LVM
User B
Read/write
User space
Read/write
Kernel space
Device Mapper
Free Disk Space
29
Volume Group Creation
LVM daemon
LVM
User space
Kernel space
Device Mapper
PE (4MB)
25 Extents
Unallocated Disk Space
30
Current Implementation
LVM daemon
Create LV1 32MB
User A
LVM
User space
Kernel space
Device Mapper
Free Disk Space
31
Current Implementation
LVM daemon
User A
LVM
Virtually map 0-7 extents for user A
User space
Kernel space
Device Mapper
0-7 error
Free Disk Space
32
Current Implementation
LVM daemon
User A
LVM
Create LV
User B
16 MB
User space
Kernel space
Device Mapper
0-7 error
Free Disk Space
33
Current Implementation
LVM daemon
User A
LVM
Create LV2
User B
16 MB
Virtually map 8-11 extents for B
User space
Kernel space
Device Mapper
0-7 error
8-11 error
Free Disk Space
34
Current Implementation
LVM daemon
User A
LVM
Write 6MB
User B
User space
Kernel space
Device Mapper
0-7 error
8-11 error
Free Disk Space
35
Current Implementation
LVM daemon
User A
LVM
User B
write
trap
User space
Kernel space
Device Mapper
0-7 error
8-11 error
Free Disk Space
36
Current Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
User B
write
trap
User space
Kernel space
Device Mapper
0-7 error
8-11 error
Free Disk Space
37
Current Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
User B
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
0-7 error
8-11 error
Free Disk Space
38
Current Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
User B
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
0-1 linear2-7 error
8-11 error
Free Disk Space
39
Current Implementation
LVM daemon
User A
LVM
User B
Write 10MB
User space
Kernel space
Device Mapper
0-1 linear2-7 error
8-11 error
Free Disk Space
40
Current Implementation
LVM daemon
User A
LVM
User B
Write 10 MB
trap
User space
Kernel space
Device Mapper
(0-1 linear0-7 error
8-11 error
Free Disk Space
41
Current Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
User B
write
trap
User space
Kernel space
Device Mapper
0-7 error
8-11 error
Free Disk Space
42
Current Implementation
On Demand allocation of disk area
LVM daemon
User A
LVM
User B
Allocate 3 extents (12 MB)
Write 10 MB
trap
User space
Kernel space
Device Mapper
0-1 linear2-7 error
11 error
8-10 linear
Free Disk Space
43
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

44
Issues

• Not the most efficient implementation
• On a trap,
• How much to allocate?
• On demand
• Pre-allocate
• Where to allocate?
• Temporal locality
• Spatial locality
• Free unused space

45
How much?

• On-demand
• Allocate only as many blocks as needed
• Pre-allocate
• Pre-allocate physical space for blocks likely
• to be accessed in the future.
• How much?
• Where?

46
How much?

• Tradeoff Performance vs. Disk Space
• Depends on the workload
• I/O on critical path
• Asynchronous I/O
• Multithreaded Applications

47
Pre-Allocation

• Think Filesystem Block Allocation
• Dumb Allocation
• Stride Allocation
• Multi-Strided Allocation

48
Dumb Allocation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
Allocate 2 extents (8MB)
trap
User space
Kernel space
Device Mapper
0-1 linear2-7 error
Free Disk Space
49
Stride Allocation
On Demand allocation of disk area
LVM daemon
User A
LVM
Write 6MB
Allocate 4 extents (16MB)
trap
User space
Kernel space
Device Mapper
0-1 linear2-7 error
Free Disk Space
PreAllocated Extent
50
Ext2 access pattern
51
Ext3 access pattern
52
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

53
Results

• Micro Benchmarks
• mkfs
• Trace based analysis
• Trace collected by HP Research Lab

54
Micro Benchmark mkfs
55
Micro Benchmark Asynchronous I/O
56
Micro Benchmark Asynchronous I/O
57
Trace Based Analysis
58
Trace Based Analysis
59
Outline

• Introduction
• Existing Framework
• Implementation
• Issues
• Results
• Conclusions

60
Conclusions

• Tradeoff between performance penalty and space
• Useful in environments where space is the
  bottleneck resource
• Techniques like Pre Allocation reduce the
  performance penalty to some extent
• For practical implementation, LVM must be placed
  in kernel address space

61
Conclusions

• mkfs performance degrades substantially due to
  traps.
• No degradation in performance for CPU intensive
  applications with asynchronous I/O
• Extent size in the range of 512K-1M provide best
  results for the trace
• This evaluation will be more realistic, if LVM is
  moved into Kernel Address space.