1 ==================
2 Memory Hot(Un)Plug
3 ==================
4
5 This document describes generic Linux support for memory hot(un)plug with
6 a focus on System RAM, including ZONE_MOVABLE support.
7
8 .. contents:: :local:
9
10 Introduction
11 ============
12
13 Memory hot(un)plug allows for increasing and decreasing the size of physical
14 memory available to a machine at runtime. In the simplest case, it consists of
15 physically plugging or unplugging a DIMM at runtime, coordinated with the
16 operating system.
17
18 Memory hot(un)plug is used for various purposes:
19
20 - The physical memory available to a machine can be adjusted at runtime, up- or
21 downgrading the memory capacity. This dynamic memory resizing, sometimes
22 referred to as "capacity on demand", is frequently used with virtual machines
23 and logical partitions.
24
25 - Replacing hardware, such as DIMMs or whole NUMA nodes, without downtime. One
26 example is replacing failing memory modules.
27
28 - Reducing energy consumption either by physically unplugging memory modules or
29 by logically unplugging (parts of) memory modules from Linux.
30
31 Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also
32 used to expose persistent memory, other performance-differentiated memory and
33 reserved memory regions as ordinary system RAM to Linux.
34
35 Linux only supports memory hot(un)plug on selected 64 bit architectures, such as
36 x86_64, arm64, ppc64 and s390x.
37
38 Memory Hot(Un)Plug Granularity
39 ------------------------------
40
41 Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the
42 physical memory address space into chunks of the same size: memory sections. The
43 size of a memory section is architecture dependent. For example, x86_64 uses
44 128 MiB and ppc64 uses 16 MiB.
45
46 Memory sections are combined into chunks referred to as "memory blocks". The
47 size of a memory block is architecture dependent and corresponds to the smallest
48 granularity that can be hot(un)plugged. The default size of a memory block is
49 the same as memory section size, unless an architecture specifies otherwise.
50
51 All memory blocks have the same size.
52
53 Phases of Memory Hotplug
54 ------------------------
55
56 Memory hotplug consists of two phases:
57
58 (1) Adding the memory to Linux
59 (2) Onlining memory blocks
60
61 In the first phase, metadata, such as the memory map ("memmap") and page tables
62 for the direct mapping, is allocated and initialized, and memory blocks are
63 created; the latter also creates sysfs files for managing newly created memory
64 blocks.
65
66 In the second phase, added memory is exposed to the page allocator. After this
67 phase, the memory is visible in memory statistics, such as free and total
68 memory, of the system.
69
70 Phases of Memory Hotunplug
71 --------------------------
72
73 Memory hotunplug consists of two phases:
74
75 (1) Offlining memory blocks
76 (2) Removing the memory from Linux
77
78 In the first phase, memory is "hidden" from the page allocator again, for
79 example, by migrating busy memory to other memory locations and removing all
80 relevant free pages from the page allocator After this phase, the memory is no
81 longer visible in memory statistics of the system.
82
83 In the second phase, the memory blocks are removed and metadata is freed.
84
85 Memory Hotplug Notifications
86 ============================
87
88 There are various ways how Linux is notified about memory hotplug events such
89 that it can start adding hotplugged memory. This description is limited to
90 systems that support ACPI; mechanisms specific to other firmware interfaces or
91 virtual machines are not described.
92
93 ACPI Notifications
94 ------------------
95
96 Platforms that support ACPI, such as x86_64, can support memory hotplug
97 notifications via ACPI.
98
99 In general, a firmware supporting memory hotplug defines a memory class object
100 HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI
101 driver will hotplug the memory to Linux.
102
103 If the firmware supports hotplug of NUMA nodes, it defines an object _HID
104 "ACPI0004", "PNP0A05", or "PNP0A06". When notified about an hotplug event, all
105 assigned memory devices are added to Linux by the ACPI driver.
106
107 Similarly, Linux can be notified about requests to hotunplug a memory device or
108 a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory
109 blocks, and, if successful, hotunplug the memory from Linux.
110
111 Manual Probing
112 --------------
113
114 On some architectures, the firmware may not be able to notify the operating
115 system about a memory hotplug event. Instead, the memory has to be manually
116 probed from user space.
117
118 The probe interface is located at::
119
120 /sys/devices/system/memory/probe
121
122 Only complete memory blocks can be probed. Individual memory blocks are probed
123 by providing the physical start address of the memory block::
124
125 % echo addr > /sys/devices/system/memory/probe
126
127 Which results in a memory block for the range [addr, addr + memory_block_size)
128 being created.
129
130 .. note::
131
132 Using the probe interface is discouraged as it is easy to crash the kernel,
133 because Linux cannot validate user input; this interface might be removed in
134 the future.
135
136 Onlining and Offlining Memory Blocks
137 ====================================
138
139 After a memory block has been created, Linux has to be instructed to actually
140 make use of that memory: the memory block has to be "online".
141
142 Before a memory block can be removed, Linux has to stop using any memory part of
143 the memory block: the memory block has to be "offlined".
144
145 The Linux kernel can be configured to automatically online added memory blocks
146 and drivers automatically trigger offlining of memory blocks when trying
147 hotunplug of memory. Memory blocks can only be removed once offlining succeeded
148 and drivers may trigger offlining of memory blocks when attempting hotunplug of
149 memory.
150
151 Onlining Memory Blocks Manually
152 -------------------------------
153
154 If auto-onlining of memory blocks isn't enabled, user-space has to manually
155 trigger onlining of memory blocks. Often, udev rules are used to automate this
156 task in user space.
157
158 Onlining of a memory block can be triggered via::
159
160 % echo online > /sys/devices/system/memory/memoryXXX/state
161
162 Or alternatively::
163
164 % echo 1 > /sys/devices/system/memory/memoryXXX/online
165
166 The kernel will select the target zone automatically, depending on the
167 configured ``online_policy``.
168
169 One can explicitly request to associate an offline memory block with
170 ZONE_MOVABLE by::
171
172 % echo online_movable > /sys/devices/system/memory/memoryXXX/state
173
174 Or one can explicitly request a kernel zone (usually ZONE_NORMAL) by::
175
176 % echo online_kernel > /sys/devices/system/memory/memoryXXX/state
177
178 In any case, if onlining succeeds, the state of the memory block is changed to
179 be "online". If it fails, the state of the memory block will remain unchanged
180 and the above commands will fail.
181
182 Onlining Memory Blocks Automatically
183 ------------------------------------
184
185 The kernel can be configured to try auto-onlining of newly added memory blocks.
186 If this feature is disabled, the memory blocks will stay offline until
187 explicitly onlined from user space.
188
189 The configured auto-online behavior can be observed via::
190
191 % cat /sys/devices/system/memory/auto_online_blocks
192
193 Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
194 ``online_movable`` to that file, like::
195
196 % echo online > /sys/devices/system/memory/auto_online_blocks
197
198 Similarly to manual onlining, with ``online`` the kernel will select the
199 target zone automatically, depending on the configured ``online_policy``.
200
201 Modifying the auto-online behavior will only affect all subsequently added
202 memory blocks only.
203
204 .. note::
205
206 In corner cases, auto-onlining can fail. The kernel won't retry. Note that
207 auto-onlining is not expected to fail in default configurations.
208
209 .. note::
210
211 DLPAR on ppc64 ignores the ``offline`` setting and will still online added
212 memory blocks; if onlining fails, memory blocks are removed again.
213
214 Offlining Memory Blocks
215 -----------------------
216
217 In the current implementation, Linux's memory offlining will try migrating all
218 movable pages off the affected memory block. As most kernel allocations, such as
219 page tables, are unmovable, page migration can fail and, therefore, inhibit
220 memory offlining from succeeding.
221
222 Having the memory provided by memory block managed by ZONE_MOVABLE significantly
223 increases memory offlining reliability; still, memory offlining can fail in
224 some corner cases.
225
226 Further, memory offlining might retry for a long time (or even forever), until
227 aborted by the user.
228
229 Offlining of a memory block can be triggered via::
230
231 % echo offline > /sys/devices/system/memory/memoryXXX/state
232
233 Or alternatively::
234
235 % echo 0 > /sys/devices/system/memory/memoryXXX/online
236
237 If offlining succeeds, the state of the memory block is changed to be "offline".
238 If it fails, the state of the memory block will remain unchanged and the above
239 commands will fail, for example, via::
240
241 bash: echo: write error: Device or resource busy
242
243 or via::
244
245 bash: echo: write error: Invalid argument
246
247 Observing the State of Memory Blocks
248 ------------------------------------
249
250 The state (online/offline/going-offline) of a memory block can be observed
251 either via::
252
253 % cat /sys/devices/system/memory/memoryXXX/state
254
255 Or alternatively (1/0) via::
256
257 % cat /sys/devices/system/memory/memoryXXX/online
258
259 For an online memory block, the managing zone can be observed via::
260
261 % cat /sys/devices/system/memory/memoryXXX/valid_zones
262
263 Configuring Memory Hot(Un)Plug
264 ==============================
265
266 There are various ways how system administrators can configure memory
267 hot(un)plug and interact with memory blocks, especially, to online them.
268
269 Memory Hot(Un)Plug Configuration via Sysfs
270 ------------------------------------------
271
272 Some memory hot(un)plug properties can be configured or inspected via sysfs in::
273
274 /sys/devices/system/memory/
275
276 The following files are currently defined:
277
278 ====================== =========================================================
279 ``auto_online_blocks`` read-write: set or get the default state of new memory
280 blocks; configure auto-onlining.
281
282 The default value depends on the
283 CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel configuration
284 option.
285
286 See the ``state`` property of memory blocks for details.
287 ``block_size_bytes`` read-only: the size in bytes of a memory block.
288 ``probe`` write-only: add (probe) selected memory blocks manually
289 from user space by supplying the physical start address.
290
291 Availability depends on the CONFIG_ARCH_MEMORY_PROBE
292 kernel configuration option.
293 ``uevent`` read-write: generic udev file for device subsystems.
294 ``crash_hotplug`` read-only: when changes to the system memory map
295 occur due to hot un/plug of memory, this file contains
296 '1' if the kernel updates the kdump capture kernel memory
297 map itself (via elfcorehdr and other relevant kexec
298 segments), or '0' if userspace must update the kdump
299 capture kernel memory map.
300
301 Availability depends on the CONFIG_MEMORY_HOTPLUG kernel
302 configuration option.
303 ====================== =========================================================
304
305 .. note::
306
307 When the CONFIG_MEMORY_FAILURE kernel configuration option is enabled, two
308 additional files ``hard_offline_page`` and ``soft_offline_page`` are available
309 to trigger hwpoisoning of pages, for example, for testing purposes. Note that
310 this functionality is not really related to memory hot(un)plug or actual
311 offlining of memory blocks.
312
313 Memory Block Configuration via Sysfs
314 ------------------------------------
315
316 Each memory block is represented as a memory block device that can be
317 onlined or offlined. All memory blocks have their device information located in
318 sysfs. Each present memory block is listed under
319 ``/sys/devices/system/memory`` as::
320
321 /sys/devices/system/memory/memoryXXX
322
323 where XXX is the memory block id; the number of digits is variable.
324
325 A present memory block indicates that some memory in the range is present;
326 however, a memory block might span memory holes. A memory block spanning memory
327 holes cannot be offlined.
328
329 For example, assume 1 GiB memory block size. A device for a memory starting at
330 0x100000000 is ``/sys/devices/system/memory/memory4``::
331
332 (0x100000000 / 1Gib = 4)
333
334 This device covers address range [0x100000000 ... 0x140000000)
335
336 The following files are currently defined:
337
338 =================== ============================================================
339 ``online`` read-write: simplified interface to trigger onlining /
340 offlining and to observe the state of a memory block.
341 When onlining, the zone is selected automatically.
342 ``phys_device`` read-only: legacy interface only ever used on s390x to
343 expose the covered storage increment.
344 ``phys_index`` read-only: the memory block id (XXX).
345 ``removable`` read-only: legacy interface that indicated whether a memory
346 block was likely to be offlineable or not. Nowadays, the
347 kernel return ``1`` if and only if it supports memory
348 offlining.
349 ``state`` read-write: advanced interface to trigger onlining /
350 offlining and to observe the state of a memory block.
351
352 When writing, ``online``, ``offline``, ``online_kernel`` and
353 ``online_movable`` are supported.
354
355 ``online_movable`` specifies onlining to ZONE_MOVABLE.
356 ``online_kernel`` specifies onlining to the default kernel
357 zone for the memory block, such as ZONE_NORMAL.
358 ``online`` let's the kernel select the zone automatically.
359
360 When reading, ``online``, ``offline`` and ``going-offline``
361 may be returned.
362 ``uevent`` read-write: generic uevent file for devices.
363 ``valid_zones`` read-only: when a block is online, shows the zone it
364 belongs to; when a block is offline, shows what zone will
365 manage it when the block will be onlined.
366
367 For online memory blocks, ``DMA``, ``DMA32``, ``Normal``,
368 ``Movable`` and ``none`` may be returned. ``none`` indicates
369 that memory provided by a memory block is managed by
370 multiple zones or spans multiple nodes; such memory blocks
371 cannot be offlined. ``Movable`` indicates ZONE_MOVABLE.
372 Other values indicate a kernel zone.
373
374 For offline memory blocks, the first column shows the
375 zone the kernel would select when onlining the memory block
376 right now without further specifying a zone.
377
378 Availability depends on the CONFIG_MEMORY_HOTREMOVE
379 kernel configuration option.
380 =================== ============================================================
381
382 .. note::
383
384 If the CONFIG_NUMA kernel configuration option is enabled, the memoryXXX/
385 directories can also be accessed via symbolic links located in the
386 ``/sys/devices/system/node/node*`` directories.
387
388 For example::
389
390 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9
391
392 A backlink will also be created::
393
394 /sys/devices/system/memory/memory9/node0 -> ../../node/node0
395
396 Command Line Parameters
397 -----------------------
398
399 Some command line parameters affect memory hot(un)plug handling. The following
400 command line parameters are relevant:
401
402 ======================== =======================================================
403 ``memhp_default_state`` configure auto-onlining by essentially setting
404 ``/sys/devices/system/memory/auto_online_blocks``.
405 ``movable_node`` configure automatic zone selection in the kernel when
406 using the ``contig-zones`` online policy. When
407 set, the kernel will default to ZONE_MOVABLE when
408 onlining a memory block, unless other zones can be kept
409 contiguous.
410 ======================== =======================================================
411
412 See Documentation/admin-guide/kernel-parameters.txt for a more generic
413 description of these command line parameters.
414
415 Module Parameters
416 ------------------
417
418 Instead of additional command line parameters or sysfs files, the
419 ``memory_hotplug`` subsystem now provides a dedicated namespace for module
420 parameters. Module parameters can be set via the command line by predicating
421 them with ``memory_hotplug.`` such as::
422
423 memory_hotplug.memmap_on_memory=1
424
425 and they can be observed (and some even modified at runtime) via::
426
427 /sys/module/memory_hotplug/parameters/
428
429 The following module parameters are currently defined:
430
431 ================================ ===============================================
432 ``memmap_on_memory`` read-write: Allocate memory for the memmap from
433 the added memory block itself. Even if enabled,
434 actual support depends on various other system
435 properties and should only be regarded as a
436 hint whether the behavior would be desired.
437
438 While allocating the memmap from the memory
439 block itself makes memory hotplug less likely
440 to fail and keeps the memmap on the same NUMA
441 node in any case, it can fragment physical
442 memory in a way that huge pages in bigger
443 granularity cannot be formed on hotplugged
444 memory.
445
446 With value "force" it could result in memory
447 wastage due to memmap size limitations. For
448 example, if the memmap for a memory block
449 requires 1 MiB, but the pageblock size is 2
450 MiB, 1 MiB of hotplugged memory will be wasted.
451 Note that there are still cases where the
452 feature cannot be enforced: for example, if the
453 memmap is smaller than a single page, or if the
454 architecture does not support the forced mode
455 in all configurations.
456
457 ``online_policy`` read-write: Set the basic policy used for
458 automatic zone selection when onlining memory
459 blocks without specifying a target zone.
460 ``contig-zones`` has been the kernel default
461 before this parameter was added. After an
462 online policy was configured and memory was
463 online, the policy should not be changed
464 anymore.
465
466 When set to ``contig-zones``, the kernel will
467 try keeping zones contiguous. If a memory block
468 intersects multiple zones or no zone, the
469 behavior depends on the ``movable_node`` kernel
470 command line parameter: default to ZONE_MOVABLE
471 if set, default to the applicable kernel zone
472 (usually ZONE_NORMAL) if not set.
473
474 When set to ``auto-movable``, the kernel will
475 try onlining memory blocks to ZONE_MOVABLE if
476 possible according to the configuration and
477 memory device details. With this policy, one
478 can avoid zone imbalances when eventually
479 hotplugging a lot of memory later and still
480 wanting to be able to hotunplug as much as
481 possible reliably, very desirable in
482 virtualized environments. This policy ignores
483 the ``movable_node`` kernel command line
484 parameter and isn't really applicable in
485 environments that require it (e.g., bare metal
486 with hotunpluggable nodes) where hotplugged
487 memory might be exposed via the
488 firmware-provided memory map early during boot
489 to the system instead of getting detected,
490 added and onlined later during boot (such as
491 done by virtio-mem or by some hypervisors
492 implementing emulated DIMMs). As one example, a
493 hotplugged DIMM will be onlined either
494 completely to ZONE_MOVABLE or completely to
495 ZONE_NORMAL, not a mixture.
496 As another example, as many memory blocks
497 belonging to a virtio-mem device will be
498 onlined to ZONE_MOVABLE as possible,
499 special-casing units of memory blocks that can
500 only get hotunplugged together. *This policy
501 does not protect from setups that are
502 problematic with ZONE_MOVABLE and does not
503 change the zone of memory blocks dynamically
504 after they were onlined.*
505 ``auto_movable_ratio`` read-write: Set the maximum MOVABLE:KERNEL
506 memory ratio in % for the ``auto-movable``
507 online policy. Whether the ratio applies only
508 for the system across all NUMA nodes or also
509 per NUMA nodes depends on the
510 ``auto_movable_numa_aware`` configuration.
511
512 All accounting is based on present memory pages
513 in the zones combined with accounting per
514 memory device. Memory dedicated to the CMA
515 allocator is accounted as MOVABLE, although
516 residing on one of the kernel zones. The
517 possible ratio depends on the actual workload.
518 The kernel default is "301" %, for example,
519 allowing for hotplugging 24 GiB to a 8 GiB VM
520 and automatically onlining all hotplugged
521 memory to ZONE_MOVABLE in many setups. The
522 additional 1% deals with some pages being not
523 present, for example, because of some firmware
524 allocations.
525
526 Note that ZONE_NORMAL memory provided by one
527 memory device does not allow for more
528 ZONE_MOVABLE memory for a different memory
529 device. As one example, onlining memory of a
530 hotplugged DIMM to ZONE_NORMAL will not allow
531 for another hotplugged DIMM to get onlined to
532 ZONE_MOVABLE automatically. In contrast, memory
533 hotplugged by a virtio-mem device that got
534 onlined to ZONE_NORMAL will allow for more
535 ZONE_MOVABLE memory within *the same*
536 virtio-mem device.
537 ``auto_movable_numa_aware`` read-write: Configure whether the
538 ``auto_movable_ratio`` in the ``auto-movable``
539 online policy also applies per NUMA
540 node in addition to the whole system across all
541 NUMA nodes. The kernel default is "Y".
542
543 Disabling NUMA awareness can be helpful when
544 dealing with NUMA nodes that should be
545 completely hotunpluggable, onlining the memory
546 completely to ZONE_MOVABLE automatically if
547 possible.
548
549 Parameter availability depends on CONFIG_NUMA.
550 ================================ ===============================================
551
552 ZONE_MOVABLE
553 ============
554
555 ZONE_MOVABLE is an important mechanism for more reliable memory offlining.
556 Further, having system RAM managed by ZONE_MOVABLE instead of one of the
557 kernel zones can increase the number of possible transparent huge pages and
558 dynamically allocated huge pages.
559
560 Most kernel allocations are unmovable. Important examples include the memory
561 map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations
562 can only be served from the kernel zones.
563
564 Most user space pages, such as anonymous memory, and page cache pages are
565 movable. Such allocations can be served from ZONE_MOVABLE and the kernel zones.
566
567 Only movable allocations are served from ZONE_MOVABLE, resulting in unmovable
568 allocations being limited to the kernel zones. Without ZONE_MOVABLE, there is
569 absolutely no guarantee whether a memory block can be offlined successfully.
570
571 Zone Imbalances
572 ---------------
573
574 Having too much system RAM managed by ZONE_MOVABLE is called a zone imbalance,
575 which can harm the system or degrade performance. As one example, the kernel
576 might crash because it runs out of free memory for unmovable allocations,
577 although there is still plenty of free memory left in ZONE_MOVABLE.
578
579 Usually, MOVABLE:KERNEL ratios of up to 3:1 or even 4:1 are fine. Ratios of 63:1
580 are definitely impossible due to the overhead for the memory map.
581
582 Actual safe zone ratios depend on the workload. Extreme cases, like excessive
583 long-term pinning of pages, might not be able to deal with ZONE_MOVABLE at all.
584
585 .. note::
586
587 CMA memory part of a kernel zone essentially behaves like memory in
588 ZONE_MOVABLE and similar considerations apply, especially when combining
589 CMA with ZONE_MOVABLE.
590
591 ZONE_MOVABLE Sizing Considerations
592 ----------------------------------
593
594 We usually expect that a large portion of available system RAM will actually
595 be consumed by user space, either directly or indirectly via the page cache. In
596 the normal case, ZONE_MOVABLE can be used when allocating such pages just fine.
597
598 With that in mind, it makes sense that we can have a big portion of system RAM
599 managed by ZONE_MOVABLE. However, there are some things to consider when using
600 ZONE_MOVABLE, especially when fine-tuning zone ratios:
601
602 - Having a lot of offline memory blocks. Even offline memory blocks consume
603 memory for metadata and page tables in the direct map; having a lot of offline
604 memory blocks is not a typical case, though.
605
606 - Memory ballooning without balloon compaction is incompatible with
607 ZONE_MOVABLE. Only some implementations, such as virtio-balloon and
608 pseries CMM, fully support balloon compaction.
609
610 Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be
611 disabled. In that case, balloon inflation will only perform unmovable
612 allocations and silently create a zone imbalance, usually triggered by
613 inflation requests from the hypervisor.
614
615 - Gigantic pages are unmovable, resulting in user space consuming a
616 lot of unmovable memory.
617
618 - Huge pages are unmovable when an architectures does not support huge
619 page migration, resulting in a similar issue as with gigantic pages.
620
621 - Page tables are unmovable. Excessive swapping, mapping extremely large
622 files or ZONE_DEVICE memory can be problematic, although only really relevant
623 in corner cases. When we manage a lot of user space memory that has been
624 swapped out or is served from a file/persistent memory/... we still need a lot
625 of page tables to manage that memory once user space accessed that memory.
626
627 - In certain DAX configurations the memory map for the device memory will be
628 allocated from the kernel zones.
629
630 - KASAN can have a significant memory overhead, for example, consuming 1/8th of
631 the total system memory size as (unmovable) tracking metadata.
632
633 - Long-term pinning of pages. Techniques that rely on long-term pinnings
634 (especially, RDMA and vfio/mdev) are fundamentally problematic with
635 ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside
636 on ZONE_MOVABLE as that would turn these pages unmovable. Therefore, they
637 have to be migrated off that zone while pinning. Pinning a page can fail
638 even if there is plenty of free memory in ZONE_MOVABLE.
639
640 In addition, using ZONE_MOVABLE might make page pinning more expensive,
641 because of the page migration overhead.
642
643 By default, all the memory configured at boot time is managed by the kernel
644 zones and ZONE_MOVABLE is not used.
645
646 To enable ZONE_MOVABLE to include the memory present at boot and to control the
647 ratio between movable and kernel zones there are two command line options:
648 ``kernelcore=`` and ``movablecore=``. See
649 Documentation/admin-guide/kernel-parameters.rst for their description.
650
651 Memory Offlining and ZONE_MOVABLE
652 ---------------------------------
653
654 Even with ZONE_MOVABLE, there are some corner cases where offlining a memory
655 block might fail:
656
657 - Memory blocks with memory holes; this applies to memory blocks present during
658 boot and can apply to memory blocks hotplugged via the XEN balloon and the
659 Hyper-V balloon.
660
661 - Mixed NUMA nodes and mixed zones within a single memory block prevent memory
662 offlining; this applies to memory blocks present during boot only.
663
664 - Special memory blocks prevented by the system from getting offlined. Examples
665 include any memory available during boot on arm64 or memory blocks spanning
666 the crashkernel area on s390x; this usually applies to memory blocks present
667 during boot only.
668
669 - Memory blocks overlapping with CMA areas cannot be offlined, this applies to
670 memory blocks present during boot only.
671
672 - Concurrent activity that operates on the same physical memory area, such as
673 allocating gigantic pages, can result in temporary offlining failures.
674
675 - Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap
676 Optimization (HVO) is enabled.
677
678 Offlining code may be able to migrate huge page contents, but may not be able
679 to dissolve the source huge page because it fails allocating (unmovable) pages
680 for the vmemmap, because the system might not have free memory in the kernel
681 zones left.
682
683 Users that depend on memory offlining to succeed for movable zones should
684 carefully consider whether the memory savings gained from this feature are
685 worth the risk of possibly not being able to offline memory in certain
686 situations.
687
688 Further, when running into out of memory situations while migrating pages, or
689 when still encountering permanently unmovable pages within ZONE_MOVABLE
690 (-> BUG), memory offlining will keep retrying until it eventually succeeds.
691
692 When offlining is triggered from user space, the offlining context can be
693 terminated by sending a signal. A timeout based offlining can easily be
694 implemented via::
695
696 % timeout $TIMEOUT offline_block | failure_handling
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.