1 RAID arrays
2 ===========
3
4 Boot time assembly of RAID arrays
5 ---------------------------------
6
7 Tools that manage md devices can be found at
8 https://www.kernel.org/pub/linux/utils/raid
9
10
11 You can boot with your md device with the foll
12 lines:
13
14 for old raid arrays without persistent superbl
15
16 md=<md device no.>,<raid level>,<chunk size
17
18 for raid arrays with persistent superblocks::
19
20 md=<md device no.>,dev0,dev1,...,devn
21
22 or, to assemble a partitionable array::
23
24 md=d<md device no.>,dev0,dev1,...,devn
25
26 ``md device no.``
27 +++++++++++++++++
28
29 The number of the md device
30
31 ================= =========
32 ``md device no.`` device
33 ================= =========
34 0 md0
35 1 md1
36 2 md2
37 3 md3
38 4 md4
39 ================= =========
40
41 ``raid level``
42 ++++++++++++++
43
44 level of the RAID array
45
46 =============== =============
47 ``raid level`` level
48 =============== =============
49 -1 linear mode
50 0 striped mode
51 =============== =============
52
53 other modes are only supported with persistent
54
55 ``chunk size factor``
56 +++++++++++++++++++++
57
58 (raid-0 and raid-1 only)
59
60 Set the chunk size as 4k << n.
61
62 ``fault level``
63 +++++++++++++++
64
65 Totally ignored
66
67 ``dev0`` to ``devn``
68 ++++++++++++++++++++
69
70 e.g. ``/dev/hda1``, ``/dev/hdc1``, ``/dev/sda1
71
72 A possible loadlin line (Harald Hoyer <HarryH@R
73
74 e:\loadlin\loadlin e:\zimage root=/dev
75
76
77 Boot time autodetection of RAID arrays
78 --------------------------------------
79
80 When md is compiled into the kernel (not as mo
81 type 0xfd are scanned and automatically assemb
82 This autodetection may be suppressed with the
83 ``raid=noautodetect``. As of kernel 2.6.9, on
84 superblock can be autodetected and run at boot
85
86 The kernel parameter ``raid=partitionable`` (o
87 that all auto-detected arrays are assembled as
88
89 Boot time assembly of degraded/dirty arrays
90 -------------------------------------------
91
92 If a raid5 or raid6 array is both dirty and de
93 undetectable data corruption. This is because
94 ``dirty`` means that the parity cannot be trus
95 is degraded means that some datablocks are mis
96 be reconstructed (due to no parity).
97
98 For this reason, md will normally refuse to st
99 requires the sysadmin to take action to explic
100 despite possible corruption. This is normally
101
102 mdadm --assemble --force ....
103
104 This option is not really available if the arr
105 filesystem on it. In order to support this bo
106 array, md supports a module parameter ``start_
107 when set to 1, bypassed the checks and will al
108 arrays to be started.
109
110 So, to boot with a root filesystem of a dirty
111
112 md-mod.start_dirty_degraded=1
113
114
115 Superblock formats
116 ------------------
117
118 The md driver can support a variety of differe
119 Currently, it supports superblock formats ``0.
120 introduced in the 2.5 development series.
121
122 The kernel will autodetect which format superb
123
124 Superblock format ``0`` is treated differently
125 reasons - it is the original superblock format
126
127
128 General Rules - apply for all superblock forma
129 ----------------------------------------------
130
131 An array is ``created`` by writing appropriate
132 devices.
133
134 It is ``assembled`` by associating each of the
135 particular md virtual device. Once it is comp
136 be accessed.
137
138 An array should be created by a user-space too
139 superblocks to all devices. It will usually m
140 ``unclean``, or with some devices missing so t
141 can create appropriate redundancy (copying in
142 calculation in raid 4/5).
143
144 When an array is assembled, it is first initia
145 SET_ARRAY_INFO ioctl. This contains, in parti
146 version number. The major version number sele
147 format is to be used. The minor number might
148 of the format, such as suggesting where on eac
149 superblock.
150
151 Then each device is added using the ADD_NEW_DI
152 provides, in particular, a major and minor num
153 device to add.
154
155 The array is started with the RUN_ARRAY ioctl.
156
157 Once started, new devices can be added. They
158 appropriate superblock written to them, and th
159 ADD_NEW_DISK.
160
161 Devices that have failed or are not yet active
162 array using HOT_REMOVE_DISK.
163
164
165 Specific Rules that apply to format-0 super bl
166 ----------------------------------------------
167
168 An array can be ``created`` by describing the
169 etc) in a SET_ARRAY_INFO ioctl. This must hav
170 ``raid_disks != 0``.
171
172 Then uninitialized devices can be added with A
173 structure passed to ADD_NEW_DISK must specify
174 and its role in the array.
175
176 Once started with RUN_ARRAY, uninitialized spa
177 HOT_ADD_DISK.
178
179
180 MD devices in sysfs
181 -------------------
182
183 md devices appear in sysfs (``/sys``) as regul
184 e.g.::
185
186 /sys/block/md0
187
188 Each ``md`` device will contain a subdirectory
189 contains further md-specific information about
190
191 All md devices contain:
192
193 level
194 a text file indicating the ``raid level``
195 raid5, linear, multipath, faulty.
196 If no raid level has been set yet (array
197 assembled), the value will reflect whatev
198 to it, which may be a name like the above
199 such as ``0``, ``5``, etc.
200
201 raid_disks
202 a text file with a simple number indicati
203 in a fully functional array. If this is
204 will be empty. If an array is being resi
205 the new number of devices.
206 Some raid levels allow this value to be s
207 active. This will reconfigure the array.
208 be set while assembling an array.
209 A change to this attribute will not be pe
210 reduce the size of the array. To reduce
211 in an e.g. raid5, the array size must fir
212 setting the ``array_size`` attribute.
213
214 chunk_size
215 This is the size in bytes for ``chunks``
216 raid levels that involve striping (0,4,5,
217 of the array is conceptually divided into
218 chunks are striped onto neighbouring devi
219 The size should be at least PAGE_SIZE (4k
220 of 2. This can only be set while assembl
221
222 layout
223 The ``layout`` for the array for the part
224 simply a number that is interpreted diffe
225 levels. It can be written while assembli
226
227 array_size
228 This can be used to artificially constrai
229 the array to be less than is actually ava
230 devices. Writing a number (in Kilobytes)
231 the available size will set the size. An
232 array (e.g. adding devices) will not caus
233 Writing the word ``default`` will cause t
234 array to be whatever size is actually ava
235 ``level``, ``chunk_size`` and ``component
236
237 This can be used to reduce the size of th
238 the number of devices in a raid4/5/6, or
239 metadata formats which mandate such clipp
240
241 reshape_position
242 This is either ``none`` or a sector numbe
243 the array where ``reshape`` is up to. If
244 attributes mentioned above (raid_disks, c
245 potentially have 2 values, an old and a n
246 values differ, reading the attribute retu
247
248 new (old)
249
250 and writing will effect the ``new`` value
251 unchanged.
252
253 component_size
254 For arrays with data redundancy (i.e. not
255 multipath), all components must be the sa
256 there must a size that they all provide s
257 part or the geometry of the array. It is
258 and can be read from here. Writing to th
259 the array if the personality supports it
260 and if the component drives are large eno
261
262 metadata_version
263 This indicates the format that is being u
264 about the array. It can be 0.90 (traditi
265 1.2 (newer format in varying locations) o
266 the kernel isn't managing metadata at all
267 Alternately it can be ``external:`` follo
268 is set by user-space. This indicates tha
269 by a user-space program. Any device fail
270 requires a metadata update will cause arr
271 suspended until the event is acknowledged
272
273 resync_start
274 The point at which resync should start.
275 this will be a very large number (or ``no
276 array creation it will default to 0, thou
277 ``clean`` will set it much larger.
278
279 new_dev
280 This file can be written but not read. T
281 be a block device number as major:minor.
282 This will cause that device to be attache
283 available. It will then appear at md/dev
284 name of the device) and further configura
285
286 safe_mode_delay
287 When an md array has seen no write reques
288 of time, it will be marked as ``clean``.
289 request arrives, the array is marked as `
290 commences. This is known as ``safe_mode`
291 The ``certain period`` is controlled by t
292 period as a number of seconds. The defau
293 Writing a value of 0 disables safemode.
294
295 array_state
296 This file contains a single word which de
297 state of the array. In many cases, the s
298 writing the word for the desired state, h
299 cannot be explicitly set, and some transi
300
301 Select/poll works on this file. All chan
302 Active_idle and active (which can be freq
303 very interesting) are notified. active->
304 reported if the metadata is externally ma
305
306 clear
307 No devices, no size, no level
308
309 Writing is equivalent to STOP_ARRAY i
310
311 inactive
312 May have some settings, but array is
313 all IO results in error
314
315 When written, doesn't tear down array
316
317 suspended (not supported yet)
318 All IO requests will block. The array
319
320 Writing this, if accepted, will block
321
322 readonly
323 no resync can happen. no superblocks
324
325 Write requests fail
326
327 read-auto
328 like readonly, but behaves like ``cle
329
330 clean
331 no pending writes, but otherwise acti
332
333 When written to inactive array, start
334
335 If a write request arrives then
336 if metadata is known, mark ``dirty``
337 if not known, block and switch to wri
338
339 If written to an active array that ha
340 active
341 fully active: IO and resync can be ha
342 When written to inactive array, start
343
344 write-pending
345 clean, but writes are blocked waiting
346
347 active-idle
348 like active, but no writes have been
349
350 bitmap/location
351 This indicates where the write-intent bit
352 stored.
353
354 It can be one of ``none``, ``file`` or ``
355 ``file`` may later be extended to ``file:
356 ``[+-]N`` means that many sectors from th
357
358 This is replicated on all devices. For a
359 managed metadata, the offset is from the
360 device.
361
362 bitmap/chunksize
363 The size, in bytes, of the chunk which wi
364 single bit. For RAID456, it is a portion
365 device. For RAID10, it is a portion of th
366 is both (they come to the same thing).
367
368 bitmap/time_base
369 The time, in seconds, between looking for
370 be cleared. In the current implementation
371 between 2 and 3 times ``time_base`` after
372 are known to be in-sync.
373
374 bitmap/backlog
375 When write-mostly devices are active in a
376 to those devices proceed in the backgroun
377 other user of the device) does not have t
378 ``backlog`` sets a limit on the number of
379 writes. If there are more than this, new
380 synchronous.
381
382 bitmap/metadata
383 This can be either ``internal`` or ``exte
384
385 ``internal``
386 is the default and means the metadata f
387 is stored in the first 256 bytes of the
388 managed by the md module.
389
390 ``external``
391 means that bitmap metadata is managed e
392 the kernel (i.e. by some userspace prog
393
394 bitmap/can_clear
395 This is either ``true`` or ``false``. If
396 bitmap will be cleared when the correspon
397 to be in-sync. If ``false``, bits will n
398 This is automatically set to ``false`` if
399 degraded array, or if the array becomes d
400 When metadata is managed externally, it s
401 once the array becomes non-degraded, and
402 recorded in the metadata.
403
404 consistency_policy
405 This indicates how the array maintains co
406 shutdown. It can be:
407
408 none
409 Array has no redundancy information, e.
410
411 resync
412 Full resync is performed and all redund
413 array is started after unclean shutdown
414
415 bitmap
416 Resync assisted by a write-intent bitma
417
418 journal
419 For raid4/5/6, journal device is used t
420 after unclean shutdown.
421
422 ppl
423 For raid5 only, Partial Parity Log is u
424 eliminate resync.
425
426 The accepted values when writing to this
427 used to enable and disable PPL.
428
429 uuid
430 This indicates the UUID of the array in t
431 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
432
433
434 As component devices are added to an md array,
435 directory as new directories named::
436
437 dev-XXX
438
439 where ``XXX`` is a name that the kernel knows
440 Each directory contains:
441
442 block
443 a symlink to the block device in /sys/
444
445 /sys/block/md0/md/dev-hdb1/block
446
447 super
448 A file containing an image of the supe
449 written to, that device.
450
451 state
452 A file recording the current state of
453 which can be a comma separated list of
454
455 faulty
456 device has been kicked
457 a detected fault, or i
458 blocks
459
460 in_sync
461 device is a fully in-s
462
463 writemostly
464 device will only be su
465 requests if there are
466
467 This applies only to r
468
469 blocked
470 device has failed, and
471 acknowledged yet by th
472
473 Writes that would writ
474 it were not faulty are
475
476 spare
477 device is working, but
478
479 This includes spares t
480 of being recovered to
481
482 write_error
483 device has ever seen a
484
485 want_replacement
486 device is (mostly) wor
487 should be replaced, ei
488 due to user request.
489
490 replacement
491 device is a replacemen
492 device with same raid_
493
494
495 This list may grow in future.
496
497 This can be written to.
498
499 Writing ``faulty`` simulates a failur
500
501 Writing ``remove`` removes the device
502
503 Writing ``writemostly`` sets the write
504
505 Writing ``-writemostly`` clears the wr
506
507 Writing ``blocked`` sets the ``blocked
508
509 Writing ``-blocked`` clears the ``bloc
510 to complete and possibly simulates an
511
512 Writing ``in_sync`` sets the in_sync f
513
514 Writing ``write_error`` sets writeerro
515
516 Writing ``-write_error`` clears writee
517
518 Writing ``want_replacement`` is allowe
519 replacement device or a spare. It set
520
521 Writing ``-want_replacement`` is allow
522 the flag.
523
524 Writing ``replacement`` or ``-replacem
525 starting the array. It sets or clears
526
527
528 This file responds to select/poll. Any
529 or ``blocked`` causes an event.
530
531 errors
532 An approximate count of read errors th
533 this device but have not caused the de
534 the array (either because they were co
535 happened while the array was read-only
536 metadata, this value persists across r
537
538 This value can be written while assemb
539 providing an ongoing count for arrays
540 userspace.
541
542 slot
543 This gives the role that the device ha
544 either be ``none`` if the device is no
545 (i.e. is a spare or has failed) or an
546 ``raid_disks`` number for the array in
547 it currently fills. This can only be
548 array. A device for which this is set
549
550 offset
551 This gives the location in the device
552 start) where data from the array will
553 the device before this offset is not t
554 used for storing metadata (Formats 1.1
555
556 size
557 The amount of the device, after the of
558 for storage of data. This will normal
559 component_size. This can be written w
560 array. If a value less than the curre
561 written, it will be rejected.
562
563 recovery_start
564 When the device is not ``in_sync``, th
565 sectors from the start of the device w
566 correct. This is normally zero, but d
567 operation it will steadily increase, a
568 interrupted, restoring this value can
569 avoid repeating the earlier blocks. W
570 value is saved and restored automatica
571
572 This can be set whenever the device is
573 the array, either before the array is
574 the ``slot`` is set.
575
576 Setting this to ``none`` is equivalent
577 Setting to any other value also clears
578
579 bad_blocks
580 This gives the list of all known bad b
581 start address and length (in sectors r
582 is too big to fit in a page, it will b
583 ``sector length`` to this file adds ne
584 recorded to disk safely) bad blocks.
585
586 unacknowledged_bad_blocks
587 This gives the list of known-but-not-y
588 blocks in the same form of ``bad_block
589 to fit in a page, it will be truncated
590 adds bad blocks without acknowledging
591 for testing.
592
593 ppl_sector, ppl_size
594 Location and size (in sectors) of the
595 on this device.
596
597
598 An active md device will also contain an entry
599 in the array. These are named::
600
601 rdNN
602
603 where ``NN`` is the position in the array, sta
604 So for a 3 drive array there will be rd0, rd1,
605 These are symbolic links to the appropriate ``
606 Thus, for example::
607
608 cat /sys/block/md*/md/rd*/state
609
610 will show ``in_sync`` on every line.
611
612
613
614 Active md devices for levels that support data
615 also have
616
617 sync_action
618 a text file that can be used to monitor a
619 process. It contains one word which can
620
621 resync
622 redundancy is being recalculat
623 shutdown or creation
624
625 recover
626 a hot spare is being built to
627 failed/missing device
628
629 idle
630 nothing is happening
631 check
632 A full check of redundancy was
633 happening. This reads all blo
634 them. A repair may also happen
635 levels.
636
637 repair
638 A full check and repair is hap
639 similar to ``resync``, but was
640 user, and the write-intent bit
641 optimise the process.
642
643 This file is writable, and each of the s
644 read are meaningful for writing.
645
646 ``idle`` will stop an active resync/re
647 guarantee that another resync/recovery
648 started again, though some event will
649 this.
650
651 ``resync`` or ``recovery`` can be used
652 corresponding operation if it was stop
653
654 ``check`` and ``repair`` will start th
655 providing the current state is ``idle`
656
657 This file responds to select/poll. Any
658 triggers a poll event. Sometimes the va
659 ``recover`` if a recovery seems to be ne
660 achieved. In that case, the transition t
661 notified, but the transition away is.
662
663 degraded
664 This contains a count of the number of d
665 arrays is degraded. So an optimal array
666 single failed/missing drive will show ``
667
668 This file responds to select/poll, any i
669 in the count of missing devices will tri
670
671 mismatch_count
672 When performing ``check`` and ``repair``
673 performing ``resync``, md will count the
674 found. The count in ``mismatch_cnt`` is
675 that were re-written, or (for ``check``)
676 re-written. As most raid levels work in
677 than sectors, this may be larger than th
678 by a factor of the number of sectors in
679
680 bitmap_set_bits
681 If the array has a write-intent bitmap,
682 attribute can set bits in the bitmap, in
683 would need to check the corresponding bl
684 numbers or start-end pairs can be writte
685 can be separated by a space.
686
687 Note that the numbers are ``bit`` number
688 They should be scaled by the bitmap_chun
689
690 sync_speed_min, sync_speed_max
691 This are similar to ``/proc/sys/dev/raid/
692 however they only apply to the particular
693
694 If no value has been written to these, or
695 is written, then the system-wide value is
696 in kibibytes-per-second is written, then
697
698 When the files are read, they show the cu
699 followed by ``(local)`` or ``(system)`` d
700 a locally set or system-wide value.
701
702 sync_completed
703 This shows the number of sectors that hav
704 whatever the current sync_action is, foll
705 sectors in total that could need to be pr
706 numbers are separated by a ``/`` thus ef
707 value, a fraction of the process that is
708
709 A ``select`` on this attribute will retur
710 when it reaches the current sync_max (bel
711 other times.
712
713 sync_speed
714 This shows the current actual speed, in K
715 sync_action. It is averaged over the las
716
717 suspend_lo, suspend_hi
718 The two values, given as numbers of secto
719 within the array where IO will be blocked
720 only supported for raid4/5/6.
721
722 sync_min, sync_max
723 The two values, given as numbers of secto
724 within the array where ``check``/``repair
725 a multiple of chunk_size. When it reaches
726 pause, rather than complete.
727 You can use ``select`` or ``poll`` on ``s
728 that number to reach sync_max. Then you
729 ``sync_max``, or can write ``idle`` to ``
730
731 The value of ``max`` for ``sync_max`` eff
732 When a resync is active, the value can on
733 never decreased.
734 The value of ``0`` is the minimum for ``s
735
736
737
738 Each active md device may also have attributes
739 personality module that manages it.
740 These are specific to the implementation of th
741 change substantially if the implementation cha
742
743 These currently include:
744
745 stripe_cache_size (currently raid5 only)
746 number of entries in the stripe cache.
747 there are upper and lower limits (32768,
748
749 strip_cache_active (currently raid5 only)
750 number of active entries in the stripe c
751
752 preread_bypass_threshold (currently raid5 on
753 number of times a stripe requiring prere
754 a stripe that does not require preread.
755 to 1. Setting this to 0 disables bypass
756 requires preread stripes to wait until a
757 writes are complete. Valid values are 0
758
759 journal_mode (currently raid5 only)
760 The cache mode for raid5. raid5 could in
761 caching. The mode can be "write-throuth"
762 default is "write-through".
763
764 ppl_write_hint
765 NVMe stream ID to be set for each PPL wr
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.