1 ============================
2 Transparent Hugepage Support
3 ============================
4
5 Objective
6 =========
7
8 Performance critical computing applications de
9 working sets are already running on top of lib
10 hugetlbfs. Transparent HugePage Support (THP)
11 using huge pages for the backing of virtual me
12 that supports the automatic promotion and demo
13 without the shortcomings of hugetlbfs.
14
15 Currently THP only works for anonymous memory
16 But in the future it can expand to other files
17
18 .. note::
19 in the examples below we presume that the b
20 the huge page size is 2M, although the actu
21 depending on the CPU architecture.
22
23 The reason applications are running faster is
24 factors. The first factor is almost completely
25 of significant interest because it'll also hav
26 requiring larger clear-page copy-page in page
27 potentially negative effect. The first factor
28 single page fault for each 2M virtual region t
29 reducing the enter/exit kernel frequency by a
30 only matters the first time the memory is acce
31 a memory mapping. The second long lasting and
32 factor will affect all subsequent accesses to
33 runtime of the application. The second factor
34 components:
35
36 1) the TLB miss will run faster (especially wi
37 nested pagetables but almost always also on
38 virtualization)
39
40 2) a single TLB entry will be mapping a much l
41 memory in turn reducing the number of TLB m
42 virtualization and nested pagetables the TL
43 larger size only if both KVM and the Linux
44 hugepages but a significant speedup already
45 the two is using hugepages just because of
46 going to run faster.
47
48 Modern kernels support "multi-size THP" (mTHP)
49 ability to allocate memory in blocks that are
50 but smaller than traditional PMD-size (as desc
51 increments of a power-of-2 number of pages. mT
52 memory (for example 16K, 32K, 64K, etc). These
53 PTE-mapped, but in many cases can still provid
54 those outlined above: Page faults are signific
55 factor of e.g. 4, 8, 16, etc), but latency spi
56 prominent because the size of each page isn't
57 variant and there is less memory to clear in e
58 architectures also employ TLB compression mech
59 entries in when a set of PTEs are virtually an
60 and approporiately aligned. In this case, TLB
61 often.
62
63 THP can be enabled system wide or restricted t
64 memory ranges inside task's address space. Unl
65 disabled, there is ``khugepaged`` daemon that
66 collapses sequences of basic pages into PMD-si
67
68 The THP behaviour is controlled via :ref:`sysf
69 interface and using madvise(2) and prctl(2) sy
70
71 Transparent Hugepage Support maximizes the use
72 if compared to the reservation approach of hug
73 unused memory to be used as cache or other mov
74 entities). It doesn't require reservation to p
75 allocation failures to be noticeable from user
76 and all other advanced VM features to be avail
77 hugepages. It requires no modifications for ap
78 advantage of it.
79
80 Applications however can be further optimized
81 this feature, like for example they've been op
82 a flood of mmap system calls for every malloc(
83 is by far not mandatory and khugepaged already
84 lived page allocations even for hugepage unawa
85 deals with large amounts of memory.
86
87 In certain cases when hugepages are enabled sy
88 may end up allocating more memory resources. A
89 large region but only touch 1 byte of it, in t
90 be allocated instead of a 4k page for no good.
91 possible to disable hugepages system-wide and
92 MADV_HUGEPAGE madvise regions.
93
94 Embedded systems should enable hugepages only
95 to eliminate any risk of wasting any precious
96 only run faster.
97
98 Applications that gets a lot of benefit from h
99 risk to lose memory by using hugepages, should
100 madvise(MADV_HUGEPAGE) on their critical mmapp
101
102 .. _thp_sysfs:
103
104 sysfs
105 =====
106
107 Global THP controls
108 -------------------
109
110 Transparent Hugepage Support for anonymous mem
111 (mostly for debugging purposes) or only enable
112 regions (to avoid the risk of consuming more m
113 system wide. This can be achieved per-supporte
114
115 echo always >/sys/kernel/mm/transparen
116 echo madvise >/sys/kernel/mm/transpare
117 echo never >/sys/kernel/mm/transparent
118
119 where <size> is the hugepage size being addres
120 for which vary by system.
121
122 For example::
123
124 echo always >/sys/kernel/mm/transparen
125
126 Alternatively it is possible to specify that a
127 will inherit the top-level "enabled" value::
128
129 echo inherit >/sys/kernel/mm/transpare
130
131 For example::
132
133 echo inherit >/sys/kernel/mm/transpare
134
135 The top-level setting (for use with "inherit")
136 one of the following commands::
137
138 echo always >/sys/kernel/mm/transparen
139 echo madvise >/sys/kernel/mm/transpare
140 echo never >/sys/kernel/mm/transparent
141
142 By default, PMD-sized hugepages have enabled="
143 hugepage sizes have enabled="never". If enabli
144 sizes, the kernel will select the most appropr
145 given allocation.
146
147 It's also possible to limit defrag efforts in
148 anonymous hugepages in case they're not immedi
149 regions or to never try to defrag memory and s
150 pages unless hugepages are immediately availab
151 time to defrag memory, we would expect to gain
152 use hugepages later instead of regular pages.
153 guaranteed, but it may be more likely in case
154 MADV_HUGEPAGE region.
155
156 ::
157
158 echo always >/sys/kernel/mm/transparen
159 echo defer >/sys/kernel/mm/transparent
160 echo defer+madvise >/sys/kernel/mm/tra
161 echo madvise >/sys/kernel/mm/transpare
162 echo never >/sys/kernel/mm/transparent
163
164 always
165 means that an application requesting T
166 allocation failure and directly reclai
167 memory in an effort to allocate a THP
168 desirable for virtual machines that be
169 use and are willing to delay the VM st
170
171 defer
172 means that an application will wake ks
173 to reclaim pages and wake kcompactd to
174 THP is available in the near future. I
175 of khugepaged to then install the THP
176
177 defer+madvise
178 will enter direct reclaim and compacti
179 only for regions that have used madvis
180 other regions will wake kswapd in the
181 pages and wake kcompactd to compact me
182 available in the near future.
183
184 madvise
185 will enter direct reclaim like ``alway
186 that are have used madvise(MADV_HUGEPA
187 behaviour.
188
189 never
190 should be self-explanatory.
191
192 By default kernel tries to use huge, PMD-mappa
193 page fault to anonymous mapping. It's possible
194 page by writing 0 or enable it back by writing
195
196 echo 0 >/sys/kernel/mm/transparent_hug
197 echo 1 >/sys/kernel/mm/transparent_hug
198
199 Some userspace (such as a test program, or an
200 allocation library) may want to know the size
201 PMD-mappable transparent hugepage::
202
203 cat /sys/kernel/mm/transparent_hugepag
204
205 All THPs at fault and collapse time will be ad
206 and will therefore be split under memory presu
207 "underused". A THP is underused if the number
208 the THP is above max_ptes_none (see below). It
209 this behaviour by writing 0 to shrink_underuse
210 1 to it::
211
212 echo 0 > /sys/kernel/mm/transparent_hu
213 echo 1 > /sys/kernel/mm/transparent_hu
214
215 khugepaged will be automatically started when
216 (either of the per-size anon control or the to
217 to "always" or "madvise"), and it'll be automa
218 PMD-sized THP is disabled (when both the per-s
219 top-level control are "never")
220
221 Khugepaged controls
222 -------------------
223
224 .. note::
225 khugepaged currently only searches for oppo
226 PMD-sized THP and no attempt is made to col
227 sizes.
228
229 khugepaged runs usually at low frequency so wh
230 invoke defrag algorithms synchronously during
231 should be worth invoking defrag at least in kh
232 also possible to disable defrag in khugepaged
233 defrag in khugepaged by writing 1::
234
235 echo 0 >/sys/kernel/mm/transparent_hug
236 echo 1 >/sys/kernel/mm/transparent_hug
237
238 You can also control how many pages khugepaged
239 pass::
240
241 /sys/kernel/mm/transparent_hugepage/kh
242
243 and how many milliseconds to wait in khugepage
244 can set this to 0 to run khugepaged at 100% ut
245
246 /sys/kernel/mm/transparent_hugepage/kh
247
248 and how many milliseconds to wait in khugepage
249 allocation failure to throttle the next alloca
250
251 /sys/kernel/mm/transparent_hugepage/kh
252
253 The khugepaged progress can be seen in the num
254 that this counter may not be an exact count of
255 collapsed, since "collapsed" could mean multip
256 being replaced by a PMD mapping, or (2) All 4K
257 one 2M hugepage. Each may happen independently
258 the type of memory and the failures that occur
259 be interpreted roughly as a sign of progress,
260 consulted for more accurate accounting)::
261
262 /sys/kernel/mm/transparent_hugepage/kh
263
264 for each pass::
265
266 /sys/kernel/mm/transparent_hugepage/kh
267
268 ``max_ptes_none`` specifies how many extra sma
269 not already mapped) can be allocated when coll
270 of small pages into one large page::
271
272 /sys/kernel/mm/transparent_hugepage/kh
273
274 A higher value leads to use additional memory
275 A lower value leads to gain less thp performan
276 max_ptes_none can waste cpu time very little,
277 ignore it.
278
279 ``max_ptes_swap`` specifies how many pages can
280 swap when collapsing a group of pages into a t
281
282 /sys/kernel/mm/transparent_hugepage/kh
283
284 A higher value can cause excessive swap IO and
285 memory. A lower value can prevent THPs from be
286 collapsed, resulting fewer pages being collaps
287 THPs, and lower memory access performance.
288
289 ``max_ptes_shared`` specifies how many pages c
290 processes. khugepaged might treat pages of THP
291 that THP is shared. Exceeding the number would
292
293 /sys/kernel/mm/transparent_hugepage/kh
294
295 A higher value may increase memory footprint f
296
297 Boot parameters
298 ===============
299
300 You can change the sysfs boot time default for
301 control by passing the parameter ``transparent
302 ``transparent_hugepage=madvise`` or ``transpar
303 kernel command line.
304
305 Alternatively, each supported anonymous THP si
306 passing ``thp_anon=<size>[KMG],<size>[KMG]:<st
307 where ``<size>`` is the THP size (must be a po
308 supported anonymous THP) and ``<state>`` is o
309 ``never`` or ``inherit``.
310
311 For example, the following will set 16K, 32K,
312 set 128K, 512K to ``inherit``, set 256K to ``m
313 to ``never``::
314
315 thp_anon=16K-64K:always;128K,512K:inhe
316
317 ``thp_anon=`` may be specified multiple times
318 required. If ``thp_anon=`` is specified at lea
319 not explicitly configured on the command line
320 ``never``.
321
322 ``transparent_hugepage`` setting only affects
323 ``thp_anon`` is not specified, PMD_ORDER THP w
324 However, if a valid ``thp_anon`` setting is pr
325 PMD_ORDER THP policy will be overridden. If th
326 is not defined within a valid ``thp_anon``, it
327 ``never``.
328
329 Hugepages in tmpfs/shmem
330 ========================
331
332 You can control hugepage allocation policy in
333 ``huge=``. It can have following values:
334
335 always
336 Attempt to allocate huge pages every time
337
338 never
339 Do not allocate huge pages;
340
341 within_size
342 Only allocate huge page if it will be full
343 Also respect fadvise()/madvise() hints;
344
345 advise
346 Only allocate huge pages if requested with
347
348 The default policy is ``never``.
349
350 ``mount -o remount,huge= /mountpoint`` works f
351 ``huge=never`` will not attempt to break up hu
352 from being allocated.
353
354 There's also sysfs knob to control hugepage al
355 shmem mount: /sys/kernel/mm/transparent_hugepa
356 is used for SysV SHM, memfds, shared anonymous
357 MAP_ANONYMOUS), GPU drivers' DRM objects, Ashm
358
359 In addition to policies listed above, shmem_en
360 values:
361
362 deny
363 For use in emergencies, to force the huge
364 all mounts;
365 force
366 Force the huge option on for all - very us
367
368 Shmem can also use "multi-size THP" (mTHP) by
369 control mTHP allocation:
370 '/sys/kernel/mm/transparent_hugepage/hugepages
371 and its value for each mTHP is essentially con
372 setting. An 'inherit' option is added to ensu
373 global settings. Conversely, the options 'for
374 which are rather testing artifacts from the ol
375
376 always
377 Attempt to allocate <size> huge pages ever
378
379 inherit
380 Inherit the top-level "shmem_enabled" valu
381 have enabled="inherit" and all other hugep
382
383 never
384 Do not allocate <size> huge pages;
385
386 within_size
387 Only allocate <size> huge page if it will
388 Also respect fadvise()/madvise() hints;
389
390 advise
391 Only allocate <size> huge pages if request
392
393 Need of application restart
394 ===========================
395
396 The transparent_hugepage/enabled and
397 transparent_hugepage/hugepages-<size>kB/enable
398 option only affect future behavior. So to make
399 to restart any application that could have bee
400 also applies to the regions registered in khug
401
402 Monitoring usage
403 ================
404
405 The number of PMD-sized anonymous transparent
406 system is available by reading the AnonHugePag
407 To identify what applications are using PMD-si
408 pages, it is necessary to read ``/proc/PID/sma
409 fields for each mapping. (Note that AnonHugePa
410 PMD-sized THP for historical reasons and shoul
411 AnonHugePmdMapped).
412
413 The number of file transparent huge pages mapp
414 by reading ShmemPmdMapped and ShmemHugePages f
415 To identify what applications are mapping file
416 is necessary to read ``/proc/PID/smaps`` and c
417 for each mapping.
418
419 Note that reading the smaps file is expensive
420 frequently will incur overhead.
421
422 There are a number of counters in ``/proc/vmst
423 monitor how successfully the system is providi
424
425 thp_fault_alloc
426 is incremented every time a huge page
427 allocated and charged to handle a page
428
429 thp_collapse_alloc
430 is incremented by khugepaged when it h
431 a range of pages to collapse into one
432 successfully allocated a new huge page
433
434 thp_fault_fallback
435 is incremented if a page fault fails t
436 a huge page and instead falls back to
437
438 thp_fault_fallback_charge
439 is incremented if a page fault fails t
440 instead falls back to using small page
441 allocation was successful.
442
443 thp_collapse_alloc_failed
444 is incremented if khugepaged found a r
445 of pages that should be collapsed into
446 the allocation.
447
448 thp_file_alloc
449 is incremented every time a shmem huge
450 allocated (Note that despite being nam
451 measures only shmem).
452
453 thp_file_fallback
454 is incremented if a shmem huge page is
455 but fails and instead falls back to us
456 despite being named after "file", the
457
458 thp_file_fallback_charge
459 is incremented if a shmem huge page ca
460 falls back to using small pages even t
461 successful. (Note that despite being n
462 counter measures only shmem).
463
464 thp_file_mapped
465 is incremented every time a file or sh
466 user address space.
467
468 thp_split_page
469 is incremented every time a huge page
470 pages. This can happen for a variety o
471 reason is that a huge page is old and
472 This action implies splitting all PMD
473
474 thp_split_page_failed
475 is incremented if kernel fails to spli
476 page. This can happen if the page was
477
478 thp_deferred_split_page
479 is incremented when a huge page is put
480 queue. This happens when a huge page i
481 splitting it would free up some memory
482 going to be split under memory pressur
483
484 thp_underused_split_page
485 is incremented when a huge page on the
486 because it was underused. A THP is und
487 zero pages in the THP is above a certa
488 (/sys/kernel/mm/transparent_hugepage/k
489
490 thp_split_pmd
491 is incremented every time a PMD split
492 This can happen, for instance, when ap
493 munmap() on part of huge page. It does
494 page table entry.
495
496 thp_zero_page_alloc
497 is incremented every time a huge zero
498 successfully allocated. Note, it doesn
499 the huge zero page, only its allocatio
500
501 thp_zero_page_alloc_failed
502 is incremented if kernel fails to allo
503 huge zero page and falls back to using
504
505 thp_swpout
506 is incremented every time a huge page
507 piece without splitting.
508
509 thp_swpout_fallback
510 is incremented if a huge page has to b
511 Usually because failed to allocate som
512 for the huge page.
513
514 In /sys/kernel/mm/transparent_hugepage/hugepag
515 also individual counters for each huge page si
516 monitor the system's effectiveness in providin
517 counter has its own corresponding file.
518
519 anon_fault_alloc
520 is incremented every time a huge page
521 allocated and charged to handle a page
522
523 anon_fault_fallback
524 is incremented if a page fault fails t
525 a huge page and instead falls back to
526 lower orders or small pages.
527
528 anon_fault_fallback_charge
529 is incremented if a page fault fails t
530 instead falls back to using huge pages
531 small pages even though the allocation
532
533 swpout
534 is incremented every time a huge page
535 piece without splitting.
536
537 swpout_fallback
538 is incremented if a huge page has to b
539 Usually because failed to allocate som
540 for the huge page.
541
542 shmem_alloc
543 is incremented every time a shmem huge
544 allocated.
545
546 shmem_fallback
547 is incremented if a shmem huge page is
548 but fails and instead falls back to us
549
550 shmem_fallback_charge
551 is incremented if a shmem huge page ca
552 falls back to using small pages even t
553 successful.
554
555 split
556 is incremented every time a huge page
557 smaller orders. This can happen for a
558 common reason is that a huge page is o
559
560 split_failed
561 is incremented if kernel fails to spli
562 page. This can happen if the page was
563
564 split_deferred
565 is incremented when a huge page is put
566 This happens when a huge page is parti
567 it would free up some memory. Pages on
568 be split under memory pressure, if spl
569
570 nr_anon
571 the number of anonymous THP we have in
572 might be currently entirely mapped or h
573 subpages.
574
575 nr_anon_partially_mapped
576 the number of anonymous THP which are l
577 wasting memory, and have been queued fo
578 Note that in corner some cases (e.g., f
579 an anonymous THP as "partially mapped"
580 is not actually partially mapped anymor
581
582 As the system ages, allocating huge pages may
583 system uses memory compaction to copy data aro
584 huge page for use. There are some counters in
585 monitor this overhead.
586
587 compact_stall
588 is incremented every time a process st
589 memory compaction so that a huge page
590
591 compact_success
592 is incremented if the system compacted
593 freed a huge page for use.
594
595 compact_fail
596 is incremented if the system tries to
597 but failed.
598
599 It is possible to establish how long the stall
600 tracer to record how long was spent in __alloc
601 using the mm_page_alloc tracepoint to identify
602 for huge pages.
603
604 Optimizing the applications
605 ===========================
606
607 To be guaranteed that the kernel will map a TH
608 memory region, the mmap region has to be hugep
609 aligned. posix_memalign() can provide that gua
610
611 Hugetlbfs
612 =========
613
614 You can use hugetlbfs on a kernel that has tra
615 support enabled just fine as always. No differ
616 hugetlbfs other than there will be less overal
617 usual features belonging to hugetlbfs are pres
618 unaffected. libhugetlbfs will also work fine a
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.