1 ================================ 1 ================================
2 Device-mapper "unstriped" target 2 Device-mapper "unstriped" target
3 ================================ 3 ================================
4 4
5 Introduction 5 Introduction
6 ============ 6 ============
7 7
8 The device-mapper "unstriped" target provides 8 The device-mapper "unstriped" target provides a transparent mechanism to
9 unstripe a device-mapper "striped" target to a 9 unstripe a device-mapper "striped" target to access the underlying disks
10 without having to touch the true backing block 10 without having to touch the true backing block-device. It can also be
11 used to unstripe a hardware RAID-0 to access b 11 used to unstripe a hardware RAID-0 to access backing disks.
12 12
13 Parameters: 13 Parameters:
14 <number of stripes> <chunk size> <stripe #> <d 14 <number of stripes> <chunk size> <stripe #> <dev_path> <offset>
15 15
16 <number of stripes> 16 <number of stripes>
17 The number of stripes in the RAID 0. 17 The number of stripes in the RAID 0.
18 18
19 <chunk size> 19 <chunk size>
20 The amount of 512B sectors in the chun 20 The amount of 512B sectors in the chunk striping.
21 21
22 <dev_path> 22 <dev_path>
23 The block device you wish to unstripe. 23 The block device you wish to unstripe.
24 24
25 <stripe #> 25 <stripe #>
26 The stripe number within the device th 26 The stripe number within the device that corresponds to physical
27 drive you wish to unstripe. This must 27 drive you wish to unstripe. This must be 0 indexed.
28 28
29 29
30 Why use this module? 30 Why use this module?
31 ==================== 31 ====================
32 32
33 An example of undoing an existing dm-stripe 33 An example of undoing an existing dm-stripe
34 ------------------------------------------- 34 -------------------------------------------
35 35
36 This small bash script will setup 4 loop devic 36 This small bash script will setup 4 loop devices and use the existing
37 striped target to combine the 4 devices into o 37 striped target to combine the 4 devices into one. It then will use
38 the unstriped target on top of the striped dev !! 38 the unstriped target ontop of the striped device to access the
39 individual backing loop devices. We write dat 39 individual backing loop devices. We write data to the newly exposed
40 unstriped devices and verify the data written 40 unstriped devices and verify the data written matches the correct
41 underlying device on the striped array:: 41 underlying device on the striped array::
42 42
43 #!/bin/bash 43 #!/bin/bash
44 44
45 MEMBER_SIZE=$((128 * 1024 * 1024)) 45 MEMBER_SIZE=$((128 * 1024 * 1024))
46 NUM=4 46 NUM=4
47 SEQ_END=$((${NUM}-1)) 47 SEQ_END=$((${NUM}-1))
48 CHUNK=256 48 CHUNK=256
49 BS=4096 49 BS=4096
50 50
51 RAID_SIZE=$((${MEMBER_SIZE}*${NUM}/512)) 51 RAID_SIZE=$((${MEMBER_SIZE}*${NUM}/512))
52 DM_PARMS="0 ${RAID_SIZE} striped ${NUM} ${CH 52 DM_PARMS="0 ${RAID_SIZE} striped ${NUM} ${CHUNK}"
53 COUNT=$((${MEMBER_SIZE} / ${BS})) 53 COUNT=$((${MEMBER_SIZE} / ${BS}))
54 54
55 for i in $(seq 0 ${SEQ_END}); do 55 for i in $(seq 0 ${SEQ_END}); do
56 dd if=/dev/zero of=member-${i} bs=${MEMBER 56 dd if=/dev/zero of=member-${i} bs=${MEMBER_SIZE} count=1 oflag=direct
57 losetup /dev/loop${i} member-${i} 57 losetup /dev/loop${i} member-${i}
58 DM_PARMS+=" /dev/loop${i} 0" 58 DM_PARMS+=" /dev/loop${i} 0"
59 done 59 done
60 60
61 echo $DM_PARMS | dmsetup create raid0 61 echo $DM_PARMS | dmsetup create raid0
62 for i in $(seq 0 ${SEQ_END}); do 62 for i in $(seq 0 ${SEQ_END}); do
63 echo "0 1 unstriped ${NUM} ${CHUNK} ${i} / 63 echo "0 1 unstriped ${NUM} ${CHUNK} ${i} /dev/mapper/raid0 0" | dmsetup create set-${i}
64 done; 64 done;
65 65
66 for i in $(seq 0 ${SEQ_END}); do 66 for i in $(seq 0 ${SEQ_END}); do
67 dd if=/dev/urandom of=/dev/mapper/set-${i} 67 dd if=/dev/urandom of=/dev/mapper/set-${i} bs=${BS} count=${COUNT} oflag=direct
68 diff /dev/mapper/set-${i} member-${i} 68 diff /dev/mapper/set-${i} member-${i}
69 done; 69 done;
70 70
71 for i in $(seq 0 ${SEQ_END}); do 71 for i in $(seq 0 ${SEQ_END}); do
72 dmsetup remove set-${i} 72 dmsetup remove set-${i}
73 done 73 done
74 74
75 dmsetup remove raid0 75 dmsetup remove raid0
76 76
77 for i in $(seq 0 ${SEQ_END}); do 77 for i in $(seq 0 ${SEQ_END}); do
78 losetup -d /dev/loop${i} 78 losetup -d /dev/loop${i}
79 rm -f member-${i} 79 rm -f member-${i}
80 done 80 done
81 81
82 Another example 82 Another example
83 --------------- 83 ---------------
84 84
85 Intel NVMe drives contain two cores on the phy 85 Intel NVMe drives contain two cores on the physical device.
86 Each core of the drive has segregated access t 86 Each core of the drive has segregated access to its LBA range.
87 The current LBA model has a RAID 0 128k chunk 87 The current LBA model has a RAID 0 128k chunk on each core, resulting
88 in a 256k stripe across the two cores:: 88 in a 256k stripe across the two cores::
89 89
90 Core 0: Core 1: 90 Core 0: Core 1:
91 __________ __________ 91 __________ __________
92 | LBA 512| | LBA 768| 92 | LBA 512| | LBA 768|
93 | LBA 0 | | LBA 256| 93 | LBA 0 | | LBA 256|
94 ---------- ---------- 94 ---------- ----------
95 95
96 The purpose of this unstriping is to provide b 96 The purpose of this unstriping is to provide better QoS in noisy
97 neighbor environments. When two partitions are 97 neighbor environments. When two partitions are created on the
98 aggregate drive without this unstriping, reads 98 aggregate drive without this unstriping, reads on one partition
99 can affect writes on another partition. This 99 can affect writes on another partition. This is because the partitions
100 are striped across the two cores. When we uns 100 are striped across the two cores. When we unstripe this hardware RAID 0
101 and make partitions on each new exposed device 101 and make partitions on each new exposed device the two partitions are now
102 physically separated. 102 physically separated.
103 103
104 With the dm-unstriped target we're able to seg 104 With the dm-unstriped target we're able to segregate an fio script that
105 has read and write jobs that are independent o 105 has read and write jobs that are independent of each other. Compared to
106 when we run the test on a combined drive with 106 when we run the test on a combined drive with partitions, we were able
107 to get a 92% reduction in read latency using t 107 to get a 92% reduction in read latency using this device mapper target.
108 108
109 109
110 Example dmsetup usage 110 Example dmsetup usage
111 ===================== 111 =====================
112 112
113 unstriped on top of Intel NVMe device that has !! 113 unstriped ontop of Intel NVMe device that has 2 cores
114 ---------------------------------------------- !! 114 -----------------------------------------------------
115 115
116 :: 116 ::
117 117
118 dmsetup create nvmset0 --table '0 512 unstri 118 dmsetup create nvmset0 --table '0 512 unstriped 2 256 0 /dev/nvme0n1 0'
119 dmsetup create nvmset1 --table '0 512 unstri 119 dmsetup create nvmset1 --table '0 512 unstriped 2 256 1 /dev/nvme0n1 0'
120 120
121 There will now be two devices that expose Inte 121 There will now be two devices that expose Intel NVMe core 0 and 1
122 respectively:: 122 respectively::
123 123
124 /dev/mapper/nvmset0 124 /dev/mapper/nvmset0
125 /dev/mapper/nvmset1 125 /dev/mapper/nvmset1
126 126
127 unstriped on top of striped with 4 drives usin !! 127 unstriped ontop of striped with 4 drives using 128K chunk size
128 ---------------------------------------------- !! 128 --------------------------------------------------------------
129 129
130 :: 130 ::
131 131
132 dmsetup create raid_disk0 --table '0 512 uns 132 dmsetup create raid_disk0 --table '0 512 unstriped 4 256 0 /dev/mapper/striped 0'
133 dmsetup create raid_disk1 --table '0 512 uns 133 dmsetup create raid_disk1 --table '0 512 unstriped 4 256 1 /dev/mapper/striped 0'
134 dmsetup create raid_disk2 --table '0 512 uns 134 dmsetup create raid_disk2 --table '0 512 unstriped 4 256 2 /dev/mapper/striped 0'
135 dmsetup create raid_disk3 --table '0 512 uns 135 dmsetup create raid_disk3 --table '0 512 unstriped 4 256 3 /dev/mapper/striped 0'
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.