1 .. SPDX-License-Identifier: GPL-2.0-only 1 .. SPDX-License-Identifier: GPL-2.0-only
2 2
3 ======== 3 ========
4 dm-clone 4 dm-clone
5 ======== 5 ========
6 6
7 Introduction 7 Introduction
8 ============ 8 ============
9 9
10 dm-clone is a device mapper target which produ 10 dm-clone is a device mapper target which produces a one-to-one copy of an
11 existing, read-only source device into a writa 11 existing, read-only source device into a writable destination device: It
12 presents a virtual block device which makes al 12 presents a virtual block device which makes all data appear immediately, and
13 redirects reads and writes accordingly. 13 redirects reads and writes accordingly.
14 14
15 The main use case of dm-clone is to clone a po 15 The main use case of dm-clone is to clone a potentially remote, high-latency,
16 read-only, archival-type block device into a w 16 read-only, archival-type block device into a writable, fast, primary-type device
17 for fast, low-latency I/O. The cloned device i 17 for fast, low-latency I/O. The cloned device is visible/mountable immediately
18 and the copy of the source device to the desti 18 and the copy of the source device to the destination device happens in the
19 background, in parallel with user I/O. 19 background, in parallel with user I/O.
20 20
21 For example, one could restore an application 21 For example, one could restore an application backup from a read-only copy,
22 accessible through a network storage protocol 22 accessible through a network storage protocol (NBD, Fibre Channel, iSCSI, AoE,
23 etc.), into a local SSD or NVMe device, and st 23 etc.), into a local SSD or NVMe device, and start using the device immediately,
24 without waiting for the restore to complete. 24 without waiting for the restore to complete.
25 25
26 When the cloning completes, the dm-clone table 26 When the cloning completes, the dm-clone table can be removed altogether and be
27 replaced, e.g., by a linear table, mapping dir 27 replaced, e.g., by a linear table, mapping directly to the destination device.
28 28
29 The dm-clone target reuses the metadata librar 29 The dm-clone target reuses the metadata library used by the thin-provisioning
30 target. 30 target.
31 31
32 Glossary 32 Glossary
33 ======== 33 ========
34 34
35 Hydration 35 Hydration
36 The process of filling a region of the de 36 The process of filling a region of the destination device with data from
37 the same region of the source device, i.e 37 the same region of the source device, i.e., copying the region from the
38 source to the destination device. 38 source to the destination device.
39 39
40 Once a region gets hydrated we redirect all I/ 40 Once a region gets hydrated we redirect all I/O regarding it to the destination
41 device. 41 device.
42 42
43 Design 43 Design
44 ====== 44 ======
45 45
46 Sub-devices 46 Sub-devices
47 ----------- 47 -----------
48 48
49 The target is constructed by passing three dev 49 The target is constructed by passing three devices to it (along with other
50 parameters detailed later): 50 parameters detailed later):
51 51
52 1. A source device - the read-only device that 52 1. A source device - the read-only device that gets cloned and source of the
53 hydration. 53 hydration.
54 54
55 2. A destination device - the destination of t 55 2. A destination device - the destination of the hydration, which will become a
56 clone of the source device. 56 clone of the source device.
57 57
58 3. A small metadata device - it records which 58 3. A small metadata device - it records which regions are already valid in the
59 destination device, i.e., which regions hav 59 destination device, i.e., which regions have already been hydrated, or have
60 been written to directly, via user I/O. 60 been written to directly, via user I/O.
61 61
62 The size of the destination device must be at 62 The size of the destination device must be at least equal to the size of the
63 source device. 63 source device.
64 64
65 Regions 65 Regions
66 ------- 66 -------
67 67
68 dm-clone divides the source and destination de 68 dm-clone divides the source and destination devices in fixed sized regions.
69 Regions are the unit of hydration, i.e., the m 69 Regions are the unit of hydration, i.e., the minimum amount of data copied from
70 the source to the destination device. 70 the source to the destination device.
71 71
72 The region size is configurable when you first 72 The region size is configurable when you first create the dm-clone device. The
73 recommended region size is the same as the fil 73 recommended region size is the same as the file system block size, which usually
74 is 4KB. The region size must be between 8 sect 74 is 4KB. The region size must be between 8 sectors (4KB) and 2097152 sectors
75 (1GB) and a power of two. 75 (1GB) and a power of two.
76 76
77 Reads and writes from/to hydrated regions are 77 Reads and writes from/to hydrated regions are serviced from the destination
78 device. 78 device.
79 79
80 A read to a not yet hydrated region is service 80 A read to a not yet hydrated region is serviced directly from the source device.
81 81
82 A write to a not yet hydrated region will be d 82 A write to a not yet hydrated region will be delayed until the corresponding
83 region has been hydrated and the hydration of 83 region has been hydrated and the hydration of the region starts immediately.
84 84
85 Note that a write request with size equal to r 85 Note that a write request with size equal to region size will skip copying of
86 the corresponding region from the source devic 86 the corresponding region from the source device and overwrite the region of the
87 destination device directly. 87 destination device directly.
88 88
89 Discards 89 Discards
90 -------- 90 --------
91 91
92 dm-clone interprets a discard request to a ran 92 dm-clone interprets a discard request to a range that hasn't been hydrated yet
93 as a hint to skip hydration of the regions cov 93 as a hint to skip hydration of the regions covered by the request, i.e., it
94 skips copying the region's data from the sourc 94 skips copying the region's data from the source to the destination device, and
95 only updates its metadata. 95 only updates its metadata.
96 96
97 If the destination device supports discards, t 97 If the destination device supports discards, then by default dm-clone will pass
98 down discard requests to it. 98 down discard requests to it.
99 99
100 Background Hydration 100 Background Hydration
101 -------------------- 101 --------------------
102 102
103 dm-clone copies continuously from the source t 103 dm-clone copies continuously from the source to the destination device, until
104 all of the device has been copied. 104 all of the device has been copied.
105 105
106 Copying data from the source to the destinatio 106 Copying data from the source to the destination device uses bandwidth. The user
107 can set a throttle to prevent more than a cert 107 can set a throttle to prevent more than a certain amount of copying occurring at
108 any one time. Moreover, dm-clone takes into ac 108 any one time. Moreover, dm-clone takes into account user I/O traffic going to
109 the devices and pauses the background hydratio 109 the devices and pauses the background hydration when there is I/O in-flight.
110 110
111 A message `hydration_threshold <#regions>` can 111 A message `hydration_threshold <#regions>` can be used to set the maximum number
112 of regions being copied, the default being 1 r 112 of regions being copied, the default being 1 region.
113 113
114 dm-clone employs dm-kcopyd for copying portion 114 dm-clone employs dm-kcopyd for copying portions of the source device to the
115 destination device. By default, we issue copy 115 destination device. By default, we issue copy requests of size equal to the
116 region size. A message `hydration_batch_size < 116 region size. A message `hydration_batch_size <#regions>` can be used to tune the
117 size of these copy requests. Increasing the hy 117 size of these copy requests. Increasing the hydration batch size results in
118 dm-clone trying to batch together contiguous r 118 dm-clone trying to batch together contiguous regions, so we copy the data in
119 batches of this many regions. 119 batches of this many regions.
120 120
121 When the hydration of the destination device f 121 When the hydration of the destination device finishes, a dm event will be sent
122 to user space. 122 to user space.
123 123
124 Updating on-disk metadata 124 Updating on-disk metadata
125 ------------------------- 125 -------------------------
126 126
127 On-disk metadata is committed every time a FLU 127 On-disk metadata is committed every time a FLUSH or FUA bio is written. If no
128 such requests are made then commits will occur 128 such requests are made then commits will occur every second. This means the
129 dm-clone device behaves like a physical disk t 129 dm-clone device behaves like a physical disk that has a volatile write cache. If
130 power is lost you may lose some recent writes. 130 power is lost you may lose some recent writes. The metadata should always be
131 consistent in spite of any crash. 131 consistent in spite of any crash.
132 132
133 Target Interface 133 Target Interface
134 ================ 134 ================
135 135
136 Constructor 136 Constructor
137 ----------- 137 -----------
138 138
139 :: 139 ::
140 140
141 clone <metadata dev> <destination dev> <sou 141 clone <metadata dev> <destination dev> <source dev> <region size>
142 [<#feature args> [<feature arg>]* [<# 142 [<#feature args> [<feature arg>]* [<#core args> [<core arg>]*]]
143 143
144 ================ ============================ 144 ================ ==============================================================
145 metadata dev Fast device holding the pers 145 metadata dev Fast device holding the persistent metadata
146 destination dev The destination device, wher 146 destination dev The destination device, where the source will be cloned
147 source dev Read only device containing 147 source dev Read only device containing the data that gets cloned
148 region size The size of a region in sect 148 region size The size of a region in sectors
149 149
150 #feature args Number of feature arguments 150 #feature args Number of feature arguments passed
151 feature args no_hydration or no_discard_p 151 feature args no_hydration or no_discard_passdown
152 152
153 #core args An even number of arguments 153 #core args An even number of arguments corresponding to key/value pairs
154 passed to dm-clone 154 passed to dm-clone
155 core args Key/value pairs passed to dm 155 core args Key/value pairs passed to dm-clone, e.g. `hydration_threshold
156 256` 156 256`
157 ================ ============================ 157 ================ ==============================================================
158 158
159 Optional feature arguments are: 159 Optional feature arguments are:
160 160
161 ==================== ======================== 161 ==================== =========================================================
162 no_hydration Create a dm-clone instan 162 no_hydration Create a dm-clone instance with background hydration
163 disabled 163 disabled
164 no_discard_passdown Disable passing down dis 164 no_discard_passdown Disable passing down discards to the destination device
165 ==================== ======================== 165 ==================== =========================================================
166 166
167 Optional core arguments are: 167 Optional core arguments are:
168 168
169 ================================ ============ 169 ================================ ==============================================
170 hydration_threshold <#regions> Maximum numb 170 hydration_threshold <#regions> Maximum number of regions being copied from
171 the source t 171 the source to the destination device at any
172 one time, du 172 one time, during background hydration.
173 hydration_batch_size <#regions> During backg 173 hydration_batch_size <#regions> During background hydration, try to batch
174 together con 174 together contiguous regions, so we copy data
175 from the sou 175 from the source to the destination device in
176 batches of t 176 batches of this many regions.
177 ================================ ============ 177 ================================ ==============================================
178 178
179 Status 179 Status
180 ------ 180 ------
181 181
182 :: 182 ::
183 183
184 <metadata block size> <#used metadata block 184 <metadata block size> <#used metadata blocks>/<#total metadata blocks>
185 <region size> <#hydrated regions>/<#total r 185 <region size> <#hydrated regions>/<#total regions> <#hydrating regions>
186 <#feature args> <feature args>* <#core args 186 <#feature args> <feature args>* <#core args> <core args>*
187 <clone metadata mode> 187 <clone metadata mode>
188 188
189 ======================= ===================== 189 ======================= =======================================================
190 metadata block size Fixed block size for 190 metadata block size Fixed block size for each metadata block in sectors
191 #used metadata blocks Number of metadata bl 191 #used metadata blocks Number of metadata blocks used
192 #total metadata blocks Total number of metad 192 #total metadata blocks Total number of metadata blocks
193 region size Configurable region s 193 region size Configurable region size for the device in sectors
194 #hydrated regions Number of regions tha 194 #hydrated regions Number of regions that have finished hydrating
195 #total regions Total number of regio 195 #total regions Total number of regions to hydrate
196 #hydrating regions Number of regions cur 196 #hydrating regions Number of regions currently hydrating
197 #feature args Number of feature arg 197 #feature args Number of feature arguments to follow
198 feature args Feature arguments, e. 198 feature args Feature arguments, e.g. `no_hydration`
199 #core args Even number of core a 199 #core args Even number of core arguments to follow
200 core args Key/value pairs for t 200 core args Key/value pairs for tuning the core, e.g.
201 `hydration_threshold 201 `hydration_threshold 256`
202 clone metadata mode ro if read-only, rw i 202 clone metadata mode ro if read-only, rw if read-write
203 203
204 In serious cases wher 204 In serious cases where even a read-only mode is deemed
205 unsafe no further I/O 205 unsafe no further I/O will be permitted and the status
206 will just contain the 206 will just contain the string 'Fail'. If the metadata
207 mode changes, a dm ev 207 mode changes, a dm event will be sent to user space.
208 ======================= ===================== 208 ======================= =======================================================
209 209
210 Messages 210 Messages
211 -------- 211 --------
212 212
213 `disable_hydration` 213 `disable_hydration`
214 Disable the background hydration of the 214 Disable the background hydration of the destination device.
215 215
216 `enable_hydration` 216 `enable_hydration`
217 Enable the background hydration of the d 217 Enable the background hydration of the destination device.
218 218
219 `hydration_threshold <#regions>` 219 `hydration_threshold <#regions>`
220 Set background hydration threshold. 220 Set background hydration threshold.
221 221
222 `hydration_batch_size <#regions>` 222 `hydration_batch_size <#regions>`
223 Set background hydration batch size. 223 Set background hydration batch size.
224 224
225 Examples 225 Examples
226 ======== 226 ========
227 227
228 Clone a device containing a file system 228 Clone a device containing a file system
229 --------------------------------------- 229 ---------------------------------------
230 230
231 1. Create the dm-clone device. 231 1. Create the dm-clone device.
232 232
233 :: 233 ::
234 234
235 dmsetup create clone --table "0 1048576000 235 dmsetup create clone --table "0 1048576000 clone $metadata_dev $dest_dev \
236 $source_dev 8 1 no_hydration" 236 $source_dev 8 1 no_hydration"
237 237
238 2. Mount the device and trim the file system. 238 2. Mount the device and trim the file system. dm-clone interprets the discards
239 sent by the file system and it will not hyd 239 sent by the file system and it will not hydrate the unused space.
240 240
241 :: 241 ::
242 242
243 mount /dev/mapper/clone /mnt/cloned-fs 243 mount /dev/mapper/clone /mnt/cloned-fs
244 fstrim /mnt/cloned-fs 244 fstrim /mnt/cloned-fs
245 245
246 3. Enable background hydration of the destinat 246 3. Enable background hydration of the destination device.
247 247
248 :: 248 ::
249 249
250 dmsetup message clone 0 enable_hydration 250 dmsetup message clone 0 enable_hydration
251 251
252 4. When the hydration finishes, we can replace 252 4. When the hydration finishes, we can replace the dm-clone table with a linear
253 table. 253 table.
254 254
255 :: 255 ::
256 256
257 dmsetup suspend clone 257 dmsetup suspend clone
258 dmsetup load clone --table "0 1048576000 l 258 dmsetup load clone --table "0 1048576000 linear $dest_dev 0"
259 dmsetup resume clone 259 dmsetup resume clone
260 260
261 The metadata device is no longer needed and 261 The metadata device is no longer needed and can be safely discarded or reused
262 for other purposes. 262 for other purposes.
263 263
264 Known issues 264 Known issues
265 ============ 265 ============
266 266
267 1. We redirect reads, to not-yet-hydrated regi 267 1. We redirect reads, to not-yet-hydrated regions, to the source device. If
268 reading the source device has high latency 268 reading the source device has high latency and the user repeatedly reads from
269 the same regions, this behaviour could degr 269 the same regions, this behaviour could degrade performance. We should use
270 these reads as hints to hydrate the relevan 270 these reads as hints to hydrate the relevant regions sooner. Currently, we
271 rely on the page cache to cache these regio 271 rely on the page cache to cache these regions, so we hopefully don't end up
272 reading them multiple times from the source 272 reading them multiple times from the source device.
273 273
274 2. Release in-core resources, i.e., the bitmap 274 2. Release in-core resources, i.e., the bitmaps tracking which regions are
275 hydrated, after the hydration has finished. 275 hydrated, after the hydration has finished.
276 276
277 3. During background hydration, if we fail to 277 3. During background hydration, if we fail to read the source or write to the
278 destination device, we print an error messa 278 destination device, we print an error message, but the hydration process
279 continues indefinitely, until it succeeds. 279 continues indefinitely, until it succeeds. We should stop the background
280 hydration after a number of failures and em 280 hydration after a number of failures and emit a dm event for user space to
281 notice. 281 notice.
282 282
283 Why not...? 283 Why not...?
284 =========== 284 ===========
285 285
286 We explored the following alternatives before 286 We explored the following alternatives before implementing dm-clone:
287 287
288 1. Use dm-cache with cache size equal to the s 288 1. Use dm-cache with cache size equal to the source device and implement a new
289 cloning policy: 289 cloning policy:
290 290
291 * The resulting cache device is not a one-t 291 * The resulting cache device is not a one-to-one mirror of the source device
292 and thus we cannot remove the cache devic 292 and thus we cannot remove the cache device once cloning completes.
293 293
294 * dm-cache writes to the source device, whi 294 * dm-cache writes to the source device, which violates our requirement that
295 the source device must be treated as read 295 the source device must be treated as read-only.
296 296
297 * Caching is semantically different from cl 297 * Caching is semantically different from cloning.
298 298
299 2. Use dm-snapshot with a COW device equal to 299 2. Use dm-snapshot with a COW device equal to the source device:
300 300
301 * dm-snapshot stores its metadata in the CO 301 * dm-snapshot stores its metadata in the COW device, so the resulting device
302 is not a one-to-one mirror of the source 302 is not a one-to-one mirror of the source device.
303 303
304 * No background copying mechanism. 304 * No background copying mechanism.
305 305
306 * dm-snapshot needs to commit its metadata 306 * dm-snapshot needs to commit its metadata whenever a pending exception
307 completes, to ensure snapshot consistency 307 completes, to ensure snapshot consistency. In the case of cloning, we don't
308 need to be so strict and can rely on comm 308 need to be so strict and can rely on committing metadata every time a FLUSH
309 or FUA bio is written, or periodically, l 309 or FUA bio is written, or periodically, like dm-thin and dm-cache do. This
310 improves the performance significantly. 310 improves the performance significantly.
311 311
312 3. Use dm-mirror: The mirror target has a back 312 3. Use dm-mirror: The mirror target has a background copying/mirroring
313 mechanism, but it writes to all mirrors, th 313 mechanism, but it writes to all mirrors, thus violating our requirement that
314 the source device must be treated as read-o 314 the source device must be treated as read-only.
315 315
316 4. Use dm-thin's external snapshot functionali 316 4. Use dm-thin's external snapshot functionality. This approach is the most
317 promising among all alternatives, as the th 317 promising among all alternatives, as the thinly-provisioned volume is a
318 one-to-one mirror of the source device and 318 one-to-one mirror of the source device and handles reads and writes to
319 un-provisioned/not-yet-cloned areas the sam 319 un-provisioned/not-yet-cloned areas the same way as dm-clone does.
320 320
321 Still: 321 Still:
322 322
323 * There is no background copying mechanism, 323 * There is no background copying mechanism, though one could be implemented.
324 324
325 * Most importantly, we want to support arbi 325 * Most importantly, we want to support arbitrary block devices as the
326 destination of the cloning process and no 326 destination of the cloning process and not restrict ourselves to
327 thinly-provisioned volumes. Thin-provisio 327 thinly-provisioned volumes. Thin-provisioning has an inherent metadata
328 overhead, for maintaining the thin volume 328 overhead, for maintaining the thin volume mappings, which significantly
329 degrades performance. 329 degrades performance.
330 330
331 Moreover, cloning a device shouldn't force 331 Moreover, cloning a device shouldn't force the use of thin-provisioning. On
332 the other hand, if we wish to use thin prov 332 the other hand, if we wish to use thin provisioning, we can just use a thin
333 LV as dm-clone's destination device. 333 LV as dm-clone's destination device.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.