1 ============================ 1 ============================
2 Transparent Hugepage Support 2 Transparent Hugepage Support
3 ============================ 3 ============================
4 4
5 Objective 5 Objective
6 ========= 6 =========
7 7
8 Performance critical computing applications de 8 Performance critical computing applications dealing with large memory
9 working sets are already running on top of lib 9 working sets are already running on top of libhugetlbfs and in turn
10 hugetlbfs. Transparent HugePage Support (THP) 10 hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of
11 using huge pages for the backing of virtual me 11 using huge pages for the backing of virtual memory with huge pages
12 that supports the automatic promotion and demo 12 that supports the automatic promotion and demotion of page sizes and
13 without the shortcomings of hugetlbfs. 13 without the shortcomings of hugetlbfs.
14 14
15 Currently THP only works for anonymous memory 15 Currently THP only works for anonymous memory mappings and tmpfs/shmem.
16 But in the future it can expand to other files 16 But in the future it can expand to other filesystems.
17 17
18 .. note:: 18 .. note::
19 in the examples below we presume that the b 19 in the examples below we presume that the basic page size is 4K and
20 the huge page size is 2M, although the actu 20 the huge page size is 2M, although the actual numbers may vary
21 depending on the CPU architecture. 21 depending on the CPU architecture.
22 22
23 The reason applications are running faster is 23 The reason applications are running faster is because of two
24 factors. The first factor is almost completely 24 factors. The first factor is almost completely irrelevant and it's not
25 of significant interest because it'll also hav 25 of significant interest because it'll also have the downside of
26 requiring larger clear-page copy-page in page 26 requiring larger clear-page copy-page in page faults which is a
27 potentially negative effect. The first factor 27 potentially negative effect. The first factor consists in taking a
28 single page fault for each 2M virtual region t 28 single page fault for each 2M virtual region touched by userland (so
29 reducing the enter/exit kernel frequency by a 29 reducing the enter/exit kernel frequency by a 512 times factor). This
30 only matters the first time the memory is acce 30 only matters the first time the memory is accessed for the lifetime of
31 a memory mapping. The second long lasting and 31 a memory mapping. The second long lasting and much more important
32 factor will affect all subsequent accesses to 32 factor will affect all subsequent accesses to the memory for the whole
33 runtime of the application. The second factor 33 runtime of the application. The second factor consist of two
34 components: 34 components:
35 35
36 1) the TLB miss will run faster (especially wi 36 1) the TLB miss will run faster (especially with virtualization using
37 nested pagetables but almost always also on 37 nested pagetables but almost always also on bare metal without
38 virtualization) 38 virtualization)
39 39
40 2) a single TLB entry will be mapping a much l 40 2) a single TLB entry will be mapping a much larger amount of virtual
41 memory in turn reducing the number of TLB m 41 memory in turn reducing the number of TLB misses. With
42 virtualization and nested pagetables the TL 42 virtualization and nested pagetables the TLB can be mapped of
43 larger size only if both KVM and the Linux 43 larger size only if both KVM and the Linux guest are using
44 hugepages but a significant speedup already 44 hugepages but a significant speedup already happens if only one of
45 the two is using hugepages just because of 45 the two is using hugepages just because of the fact the TLB miss is
46 going to run faster. 46 going to run faster.
47 47
48 Modern kernels support "multi-size THP" (mTHP) 48 Modern kernels support "multi-size THP" (mTHP), which introduces the
49 ability to allocate memory in blocks that are 49 ability to allocate memory in blocks that are bigger than a base page
50 but smaller than traditional PMD-size (as desc 50 but smaller than traditional PMD-size (as described above), in
51 increments of a power-of-2 number of pages. mT 51 increments of a power-of-2 number of pages. mTHP can back anonymous
52 memory (for example 16K, 32K, 64K, etc). These 52 memory (for example 16K, 32K, 64K, etc). These THPs continue to be
53 PTE-mapped, but in many cases can still provid 53 PTE-mapped, but in many cases can still provide similar benefits to
54 those outlined above: Page faults are signific 54 those outlined above: Page faults are significantly reduced (by a
55 factor of e.g. 4, 8, 16, etc), but latency spi 55 factor of e.g. 4, 8, 16, etc), but latency spikes are much less
56 prominent because the size of each page isn't 56 prominent because the size of each page isn't as huge as the PMD-sized
57 variant and there is less memory to clear in e 57 variant and there is less memory to clear in each page fault. Some
58 architectures also employ TLB compression mech 58 architectures also employ TLB compression mechanisms to squeeze more
59 entries in when a set of PTEs are virtually an 59 entries in when a set of PTEs are virtually and physically contiguous
60 and approporiately aligned. In this case, TLB 60 and approporiately aligned. In this case, TLB misses will occur less
61 often. 61 often.
62 62
63 THP can be enabled system wide or restricted t 63 THP can be enabled system wide or restricted to certain tasks or even
64 memory ranges inside task's address space. Unl 64 memory ranges inside task's address space. Unless THP is completely
65 disabled, there is ``khugepaged`` daemon that 65 disabled, there is ``khugepaged`` daemon that scans memory and
66 collapses sequences of basic pages into PMD-si 66 collapses sequences of basic pages into PMD-sized huge pages.
67 67
68 The THP behaviour is controlled via :ref:`sysf 68 The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
69 interface and using madvise(2) and prctl(2) sy 69 interface and using madvise(2) and prctl(2) system calls.
70 70
71 Transparent Hugepage Support maximizes the use 71 Transparent Hugepage Support maximizes the usefulness of free memory
72 if compared to the reservation approach of hug 72 if compared to the reservation approach of hugetlbfs by allowing all
73 unused memory to be used as cache or other mov 73 unused memory to be used as cache or other movable (or even unmovable
74 entities). It doesn't require reservation to p 74 entities). It doesn't require reservation to prevent hugepage
75 allocation failures to be noticeable from user 75 allocation failures to be noticeable from userland. It allows paging
76 and all other advanced VM features to be avail 76 and all other advanced VM features to be available on the
77 hugepages. It requires no modifications for ap 77 hugepages. It requires no modifications for applications to take
78 advantage of it. 78 advantage of it.
79 79
80 Applications however can be further optimized 80 Applications however can be further optimized to take advantage of
81 this feature, like for example they've been op 81 this feature, like for example they've been optimized before to avoid
82 a flood of mmap system calls for every malloc( 82 a flood of mmap system calls for every malloc(4k). Optimizing userland
83 is by far not mandatory and khugepaged already 83 is by far not mandatory and khugepaged already can take care of long
84 lived page allocations even for hugepage unawa 84 lived page allocations even for hugepage unaware applications that
85 deals with large amounts of memory. 85 deals with large amounts of memory.
86 86
87 In certain cases when hugepages are enabled sy 87 In certain cases when hugepages are enabled system wide, application
88 may end up allocating more memory resources. A 88 may end up allocating more memory resources. An application may mmap a
89 large region but only touch 1 byte of it, in t 89 large region but only touch 1 byte of it, in that case a 2M page might
90 be allocated instead of a 4k page for no good. 90 be allocated instead of a 4k page for no good. This is why it's
91 possible to disable hugepages system-wide and 91 possible to disable hugepages system-wide and to only have them inside
92 MADV_HUGEPAGE madvise regions. 92 MADV_HUGEPAGE madvise regions.
93 93
94 Embedded systems should enable hugepages only 94 Embedded systems should enable hugepages only inside madvise regions
95 to eliminate any risk of wasting any precious 95 to eliminate any risk of wasting any precious byte of memory and to
96 only run faster. 96 only run faster.
97 97
98 Applications that gets a lot of benefit from h 98 Applications that gets a lot of benefit from hugepages and that don't
99 risk to lose memory by using hugepages, should 99 risk to lose memory by using hugepages, should use
100 madvise(MADV_HUGEPAGE) on their critical mmapp 100 madvise(MADV_HUGEPAGE) on their critical mmapped regions.
101 101
102 .. _thp_sysfs: 102 .. _thp_sysfs:
103 103
104 sysfs 104 sysfs
105 ===== 105 =====
106 106
107 Global THP controls 107 Global THP controls
108 ------------------- 108 -------------------
109 109
110 Transparent Hugepage Support for anonymous mem 110 Transparent Hugepage Support for anonymous memory can be entirely disabled
111 (mostly for debugging purposes) or only enable 111 (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE
112 regions (to avoid the risk of consuming more m 112 regions (to avoid the risk of consuming more memory resources) or enabled
113 system wide. This can be achieved per-supporte 113 system wide. This can be achieved per-supported-THP-size with one of::
114 114
115 echo always >/sys/kernel/mm/transparen 115 echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
116 echo madvise >/sys/kernel/mm/transpare 116 echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
117 echo never >/sys/kernel/mm/transparent 117 echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
118 118
119 where <size> is the hugepage size being addres 119 where <size> is the hugepage size being addressed, the available sizes
120 for which vary by system. 120 for which vary by system.
121 121
122 For example:: 122 For example::
123 123
124 echo always >/sys/kernel/mm/transparen 124 echo always >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled
125 125
126 Alternatively it is possible to specify that a 126 Alternatively it is possible to specify that a given hugepage size
127 will inherit the top-level "enabled" value:: 127 will inherit the top-level "enabled" value::
128 128
129 echo inherit >/sys/kernel/mm/transpare 129 echo inherit >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
130 130
131 For example:: 131 For example::
132 132
133 echo inherit >/sys/kernel/mm/transpare 133 echo inherit >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled
134 134
135 The top-level setting (for use with "inherit") 135 The top-level setting (for use with "inherit") can be set by issuing
136 one of the following commands:: 136 one of the following commands::
137 137
138 echo always >/sys/kernel/mm/transparen 138 echo always >/sys/kernel/mm/transparent_hugepage/enabled
139 echo madvise >/sys/kernel/mm/transpare 139 echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
140 echo never >/sys/kernel/mm/transparent 140 echo never >/sys/kernel/mm/transparent_hugepage/enabled
141 141
142 By default, PMD-sized hugepages have enabled=" 142 By default, PMD-sized hugepages have enabled="inherit" and all other
143 hugepage sizes have enabled="never". If enabli 143 hugepage sizes have enabled="never". If enabling multiple hugepage
144 sizes, the kernel will select the most appropr 144 sizes, the kernel will select the most appropriate enabled size for a
145 given allocation. 145 given allocation.
146 146
147 It's also possible to limit defrag efforts in 147 It's also possible to limit defrag efforts in the VM to generate
148 anonymous hugepages in case they're not immedi 148 anonymous hugepages in case they're not immediately free to madvise
149 regions or to never try to defrag memory and s 149 regions or to never try to defrag memory and simply fallback to regular
150 pages unless hugepages are immediately availab 150 pages unless hugepages are immediately available. Clearly if we spend CPU
151 time to defrag memory, we would expect to gain 151 time to defrag memory, we would expect to gain even more by the fact we
152 use hugepages later instead of regular pages. 152 use hugepages later instead of regular pages. This isn't always
153 guaranteed, but it may be more likely in case 153 guaranteed, but it may be more likely in case the allocation is for a
154 MADV_HUGEPAGE region. 154 MADV_HUGEPAGE region.
155 155
156 :: 156 ::
157 157
158 echo always >/sys/kernel/mm/transparen 158 echo always >/sys/kernel/mm/transparent_hugepage/defrag
159 echo defer >/sys/kernel/mm/transparent 159 echo defer >/sys/kernel/mm/transparent_hugepage/defrag
160 echo defer+madvise >/sys/kernel/mm/tra 160 echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag
161 echo madvise >/sys/kernel/mm/transpare 161 echo madvise >/sys/kernel/mm/transparent_hugepage/defrag
162 echo never >/sys/kernel/mm/transparent 162 echo never >/sys/kernel/mm/transparent_hugepage/defrag
163 163
164 always 164 always
165 means that an application requesting T 165 means that an application requesting THP will stall on
166 allocation failure and directly reclai 166 allocation failure and directly reclaim pages and compact
167 memory in an effort to allocate a THP 167 memory in an effort to allocate a THP immediately. This may be
168 desirable for virtual machines that be 168 desirable for virtual machines that benefit heavily from THP
169 use and are willing to delay the VM st 169 use and are willing to delay the VM start to utilise them.
170 170
171 defer 171 defer
172 means that an application will wake ks 172 means that an application will wake kswapd in the background
173 to reclaim pages and wake kcompactd to 173 to reclaim pages and wake kcompactd to compact memory so that
174 THP is available in the near future. I 174 THP is available in the near future. It's the responsibility
175 of khugepaged to then install the THP 175 of khugepaged to then install the THP pages later.
176 176
177 defer+madvise 177 defer+madvise
178 will enter direct reclaim and compacti 178 will enter direct reclaim and compaction like ``always``, but
179 only for regions that have used madvis 179 only for regions that have used madvise(MADV_HUGEPAGE); all
180 other regions will wake kswapd in the 180 other regions will wake kswapd in the background to reclaim
181 pages and wake kcompactd to compact me 181 pages and wake kcompactd to compact memory so that THP is
182 available in the near future. 182 available in the near future.
183 183
184 madvise 184 madvise
185 will enter direct reclaim like ``alway 185 will enter direct reclaim like ``always`` but only for regions
186 that are have used madvise(MADV_HUGEPA 186 that are have used madvise(MADV_HUGEPAGE). This is the default
187 behaviour. 187 behaviour.
188 188
189 never 189 never
190 should be self-explanatory. 190 should be self-explanatory.
191 191
192 By default kernel tries to use huge, PMD-mappa 192 By default kernel tries to use huge, PMD-mappable zero page on read
193 page fault to anonymous mapping. It's possible 193 page fault to anonymous mapping. It's possible to disable huge zero
194 page by writing 0 or enable it back by writing 194 page by writing 0 or enable it back by writing 1::
195 195
196 echo 0 >/sys/kernel/mm/transparent_hug 196 echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
197 echo 1 >/sys/kernel/mm/transparent_hug 197 echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
198 198
199 Some userspace (such as a test program, or an 199 Some userspace (such as a test program, or an optimized memory
200 allocation library) may want to know the size 200 allocation library) may want to know the size (in bytes) of a
201 PMD-mappable transparent hugepage:: 201 PMD-mappable transparent hugepage::
202 202
203 cat /sys/kernel/mm/transparent_hugepag 203 cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
204 204
205 All THPs at fault and collapse time will be ad 205 All THPs at fault and collapse time will be added to _deferred_list,
206 and will therefore be split under memory presu 206 and will therefore be split under memory presure if they are considered
207 "underused". A THP is underused if the number 207 "underused". A THP is underused if the number of zero-filled pages in
208 the THP is above max_ptes_none (see below). It 208 the THP is above max_ptes_none (see below). It is possible to disable
209 this behaviour by writing 0 to shrink_underuse 209 this behaviour by writing 0 to shrink_underused, and enable it by writing
210 1 to it:: 210 1 to it::
211 211
212 echo 0 > /sys/kernel/mm/transparent_hu 212 echo 0 > /sys/kernel/mm/transparent_hugepage/shrink_underused
213 echo 1 > /sys/kernel/mm/transparent_hu 213 echo 1 > /sys/kernel/mm/transparent_hugepage/shrink_underused
214 214
215 khugepaged will be automatically started when 215 khugepaged will be automatically started when PMD-sized THP is enabled
216 (either of the per-size anon control or the to 216 (either of the per-size anon control or the top-level control are set
217 to "always" or "madvise"), and it'll be automa 217 to "always" or "madvise"), and it'll be automatically shutdown when
218 PMD-sized THP is disabled (when both the per-s 218 PMD-sized THP is disabled (when both the per-size anon control and the
219 top-level control are "never") 219 top-level control are "never")
220 220
221 Khugepaged controls 221 Khugepaged controls
222 ------------------- 222 -------------------
223 223
224 .. note:: 224 .. note::
225 khugepaged currently only searches for oppo 225 khugepaged currently only searches for opportunities to collapse to
226 PMD-sized THP and no attempt is made to col 226 PMD-sized THP and no attempt is made to collapse to other THP
227 sizes. 227 sizes.
228 228
229 khugepaged runs usually at low frequency so wh 229 khugepaged runs usually at low frequency so while one may not want to
230 invoke defrag algorithms synchronously during 230 invoke defrag algorithms synchronously during the page faults, it
231 should be worth invoking defrag at least in kh 231 should be worth invoking defrag at least in khugepaged. However it's
232 also possible to disable defrag in khugepaged 232 also possible to disable defrag in khugepaged by writing 0 or enable
233 defrag in khugepaged by writing 1:: 233 defrag in khugepaged by writing 1::
234 234
235 echo 0 >/sys/kernel/mm/transparent_hug 235 echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag
236 echo 1 >/sys/kernel/mm/transparent_hug 236 echo 1 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag
237 237
238 You can also control how many pages khugepaged 238 You can also control how many pages khugepaged should scan at each
239 pass:: 239 pass::
240 240
241 /sys/kernel/mm/transparent_hugepage/kh 241 /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan
242 242
243 and how many milliseconds to wait in khugepage 243 and how many milliseconds to wait in khugepaged between each pass (you
244 can set this to 0 to run khugepaged at 100% ut 244 can set this to 0 to run khugepaged at 100% utilization of one core)::
245 245
246 /sys/kernel/mm/transparent_hugepage/kh 246 /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs
247 247
248 and how many milliseconds to wait in khugepage 248 and how many milliseconds to wait in khugepaged if there's an hugepage
249 allocation failure to throttle the next alloca 249 allocation failure to throttle the next allocation attempt::
250 250
251 /sys/kernel/mm/transparent_hugepage/kh 251 /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs
252 252
253 The khugepaged progress can be seen in the num 253 The khugepaged progress can be seen in the number of pages collapsed (note
254 that this counter may not be an exact count of 254 that this counter may not be an exact count of the number of pages
255 collapsed, since "collapsed" could mean multip 255 collapsed, since "collapsed" could mean multiple things: (1) A PTE mapping
256 being replaced by a PMD mapping, or (2) All 4K 256 being replaced by a PMD mapping, or (2) All 4K physical pages replaced by
257 one 2M hugepage. Each may happen independently 257 one 2M hugepage. Each may happen independently, or together, depending on
258 the type of memory and the failures that occur 258 the type of memory and the failures that occur. As such, this value should
259 be interpreted roughly as a sign of progress, 259 be interpreted roughly as a sign of progress, and counters in /proc/vmstat
260 consulted for more accurate accounting):: 260 consulted for more accurate accounting)::
261 261
262 /sys/kernel/mm/transparent_hugepage/kh 262 /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed
263 263
264 for each pass:: 264 for each pass::
265 265
266 /sys/kernel/mm/transparent_hugepage/kh 266 /sys/kernel/mm/transparent_hugepage/khugepaged/full_scans
267 267
268 ``max_ptes_none`` specifies how many extra sma 268 ``max_ptes_none`` specifies how many extra small pages (that are
269 not already mapped) can be allocated when coll 269 not already mapped) can be allocated when collapsing a group
270 of small pages into one large page:: 270 of small pages into one large page::
271 271
272 /sys/kernel/mm/transparent_hugepage/kh 272 /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
273 273
274 A higher value leads to use additional memory 274 A higher value leads to use additional memory for programs.
275 A lower value leads to gain less thp performan 275 A lower value leads to gain less thp performance. Value of
276 max_ptes_none can waste cpu time very little, 276 max_ptes_none can waste cpu time very little, you can
277 ignore it. 277 ignore it.
278 278
279 ``max_ptes_swap`` specifies how many pages can 279 ``max_ptes_swap`` specifies how many pages can be brought in from
280 swap when collapsing a group of pages into a t 280 swap when collapsing a group of pages into a transparent huge page::
281 281
282 /sys/kernel/mm/transparent_hugepage/kh 282 /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap
283 283
284 A higher value can cause excessive swap IO and 284 A higher value can cause excessive swap IO and waste
285 memory. A lower value can prevent THPs from be 285 memory. A lower value can prevent THPs from being
286 collapsed, resulting fewer pages being collaps 286 collapsed, resulting fewer pages being collapsed into
287 THPs, and lower memory access performance. 287 THPs, and lower memory access performance.
288 288
289 ``max_ptes_shared`` specifies how many pages c 289 ``max_ptes_shared`` specifies how many pages can be shared across multiple
290 processes. khugepaged might treat pages of THP 290 processes. khugepaged might treat pages of THPs as shared if any page of
291 that THP is shared. Exceeding the number would 291 that THP is shared. Exceeding the number would block the collapse::
292 292
293 /sys/kernel/mm/transparent_hugepage/kh 293 /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared
294 294
295 A higher value may increase memory footprint f 295 A higher value may increase memory footprint for some workloads.
296 296
297 Boot parameters 297 Boot parameters
298 =============== 298 ===============
299 299
300 You can change the sysfs boot time default for 300 You can change the sysfs boot time default for the top-level "enabled"
301 control by passing the parameter ``transparent 301 control by passing the parameter ``transparent_hugepage=always`` or
302 ``transparent_hugepage=madvise`` or ``transpar 302 ``transparent_hugepage=madvise`` or ``transparent_hugepage=never`` to the
303 kernel command line. 303 kernel command line.
304 304
305 Alternatively, each supported anonymous THP si 305 Alternatively, each supported anonymous THP size can be controlled by
306 passing ``thp_anon=<size>[KMG],<size>[KMG]:<st 306 passing ``thp_anon=<size>[KMG],<size>[KMG]:<state>;<size>[KMG]-<size>[KMG]:<state>``,
307 where ``<size>`` is the THP size (must be a po 307 where ``<size>`` is the THP size (must be a power of 2 of PAGE_SIZE and
308 supported anonymous THP) and ``<state>`` is o 308 supported anonymous THP) and ``<state>`` is one of ``always``, ``madvise``,
309 ``never`` or ``inherit``. 309 ``never`` or ``inherit``.
310 310
311 For example, the following will set 16K, 32K, 311 For example, the following will set 16K, 32K, 64K THP to ``always``,
312 set 128K, 512K to ``inherit``, set 256K to ``m 312 set 128K, 512K to ``inherit``, set 256K to ``madvise`` and 1M, 2M
313 to ``never``:: 313 to ``never``::
314 314
315 thp_anon=16K-64K:always;128K,512K:inhe 315 thp_anon=16K-64K:always;128K,512K:inherit;256K:madvise;1M-2M:never
316 316
317 ``thp_anon=`` may be specified multiple times 317 ``thp_anon=`` may be specified multiple times to configure all THP sizes as
318 required. If ``thp_anon=`` is specified at lea 318 required. If ``thp_anon=`` is specified at least once, any anon THP sizes
319 not explicitly configured on the command line 319 not explicitly configured on the command line are implicitly set to
320 ``never``. 320 ``never``.
321 321
322 ``transparent_hugepage`` setting only affects 322 ``transparent_hugepage`` setting only affects the global toggle. If
323 ``thp_anon`` is not specified, PMD_ORDER THP w 323 ``thp_anon`` is not specified, PMD_ORDER THP will default to ``inherit``.
324 However, if a valid ``thp_anon`` setting is pr 324 However, if a valid ``thp_anon`` setting is provided by the user, the
325 PMD_ORDER THP policy will be overridden. If th 325 PMD_ORDER THP policy will be overridden. If the policy for PMD_ORDER
326 is not defined within a valid ``thp_anon``, it 326 is not defined within a valid ``thp_anon``, its policy will default to
327 ``never``. 327 ``never``.
328 328
329 Hugepages in tmpfs/shmem 329 Hugepages in tmpfs/shmem
330 ======================== 330 ========================
331 331
332 You can control hugepage allocation policy in 332 You can control hugepage allocation policy in tmpfs with mount option
333 ``huge=``. It can have following values: 333 ``huge=``. It can have following values:
334 334
335 always 335 always
336 Attempt to allocate huge pages every time 336 Attempt to allocate huge pages every time we need a new page;
337 337
338 never 338 never
339 Do not allocate huge pages; 339 Do not allocate huge pages;
340 340
341 within_size 341 within_size
342 Only allocate huge page if it will be full 342 Only allocate huge page if it will be fully within i_size.
343 Also respect fadvise()/madvise() hints; 343 Also respect fadvise()/madvise() hints;
344 344
345 advise 345 advise
346 Only allocate huge pages if requested with 346 Only allocate huge pages if requested with fadvise()/madvise();
347 347
348 The default policy is ``never``. 348 The default policy is ``never``.
349 349
350 ``mount -o remount,huge= /mountpoint`` works f 350 ``mount -o remount,huge= /mountpoint`` works fine after mount: remounting
351 ``huge=never`` will not attempt to break up hu 351 ``huge=never`` will not attempt to break up huge pages at all, just stop more
352 from being allocated. 352 from being allocated.
353 353
354 There's also sysfs knob to control hugepage al 354 There's also sysfs knob to control hugepage allocation policy for internal
355 shmem mount: /sys/kernel/mm/transparent_hugepa 355 shmem mount: /sys/kernel/mm/transparent_hugepage/shmem_enabled. The mount
356 is used for SysV SHM, memfds, shared anonymous 356 is used for SysV SHM, memfds, shared anonymous mmaps (of /dev/zero or
357 MAP_ANONYMOUS), GPU drivers' DRM objects, Ashm 357 MAP_ANONYMOUS), GPU drivers' DRM objects, Ashmem.
358 358
359 In addition to policies listed above, shmem_en 359 In addition to policies listed above, shmem_enabled allows two further
360 values: 360 values:
361 361
362 deny 362 deny
363 For use in emergencies, to force the huge 363 For use in emergencies, to force the huge option off from
364 all mounts; 364 all mounts;
365 force 365 force
366 Force the huge option on for all - very us 366 Force the huge option on for all - very useful for testing;
367 367
368 Shmem can also use "multi-size THP" (mTHP) by 368 Shmem can also use "multi-size THP" (mTHP) by adding a new sysfs knob to
369 control mTHP allocation: 369 control mTHP allocation:
370 '/sys/kernel/mm/transparent_hugepage/hugepages 370 '/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/shmem_enabled',
371 and its value for each mTHP is essentially con 371 and its value for each mTHP is essentially consistent with the global
372 setting. An 'inherit' option is added to ensu 372 setting. An 'inherit' option is added to ensure compatibility with these
373 global settings. Conversely, the options 'for 373 global settings. Conversely, the options 'force' and 'deny' are dropped,
374 which are rather testing artifacts from the ol 374 which are rather testing artifacts from the old ages.
375 375
376 always 376 always
377 Attempt to allocate <size> huge pages ever 377 Attempt to allocate <size> huge pages every time we need a new page;
378 378
379 inherit 379 inherit
380 Inherit the top-level "shmem_enabled" valu 380 Inherit the top-level "shmem_enabled" value. By default, PMD-sized hugepages
381 have enabled="inherit" and all other hugep 381 have enabled="inherit" and all other hugepage sizes have enabled="never";
382 382
383 never 383 never
384 Do not allocate <size> huge pages; 384 Do not allocate <size> huge pages;
385 385
386 within_size 386 within_size
387 Only allocate <size> huge page if it will 387 Only allocate <size> huge page if it will be fully within i_size.
388 Also respect fadvise()/madvise() hints; 388 Also respect fadvise()/madvise() hints;
389 389
390 advise 390 advise
391 Only allocate <size> huge pages if request 391 Only allocate <size> huge pages if requested with fadvise()/madvise();
392 392
393 Need of application restart 393 Need of application restart
394 =========================== 394 ===========================
395 395
396 The transparent_hugepage/enabled and 396 The transparent_hugepage/enabled and
397 transparent_hugepage/hugepages-<size>kB/enable 397 transparent_hugepage/hugepages-<size>kB/enabled values and tmpfs mount
398 option only affect future behavior. So to make 398 option only affect future behavior. So to make them effective you need
399 to restart any application that could have bee 399 to restart any application that could have been using hugepages. This
400 also applies to the regions registered in khug 400 also applies to the regions registered in khugepaged.
401 401
402 Monitoring usage 402 Monitoring usage
403 ================ 403 ================
404 404
405 The number of PMD-sized anonymous transparent 405 The number of PMD-sized anonymous transparent huge pages currently used by the
406 system is available by reading the AnonHugePag 406 system is available by reading the AnonHugePages field in ``/proc/meminfo``.
407 To identify what applications are using PMD-si 407 To identify what applications are using PMD-sized anonymous transparent huge
408 pages, it is necessary to read ``/proc/PID/sma 408 pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
409 fields for each mapping. (Note that AnonHugePa 409 fields for each mapping. (Note that AnonHugePages only applies to traditional
410 PMD-sized THP for historical reasons and shoul 410 PMD-sized THP for historical reasons and should have been called
411 AnonHugePmdMapped). 411 AnonHugePmdMapped).
412 412
413 The number of file transparent huge pages mapp 413 The number of file transparent huge pages mapped to userspace is available
414 by reading ShmemPmdMapped and ShmemHugePages f 414 by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``.
415 To identify what applications are mapping file 415 To identify what applications are mapping file transparent huge pages, it
416 is necessary to read ``/proc/PID/smaps`` and c 416 is necessary to read ``/proc/PID/smaps`` and count the FileHugeMapped fields
417 for each mapping. 417 for each mapping.
418 418
419 Note that reading the smaps file is expensive 419 Note that reading the smaps file is expensive and reading it
420 frequently will incur overhead. 420 frequently will incur overhead.
421 421
422 There are a number of counters in ``/proc/vmst 422 There are a number of counters in ``/proc/vmstat`` that may be used to
423 monitor how successfully the system is providi 423 monitor how successfully the system is providing huge pages for use.
424 424
425 thp_fault_alloc 425 thp_fault_alloc
426 is incremented every time a huge page 426 is incremented every time a huge page is successfully
427 allocated and charged to handle a page 427 allocated and charged to handle a page fault.
428 428
429 thp_collapse_alloc 429 thp_collapse_alloc
430 is incremented by khugepaged when it h 430 is incremented by khugepaged when it has found
431 a range of pages to collapse into one 431 a range of pages to collapse into one huge page and has
432 successfully allocated a new huge page 432 successfully allocated a new huge page to store the data.
433 433
434 thp_fault_fallback 434 thp_fault_fallback
435 is incremented if a page fault fails t 435 is incremented if a page fault fails to allocate or charge
436 a huge page and instead falls back to 436 a huge page and instead falls back to using small pages.
437 437
438 thp_fault_fallback_charge 438 thp_fault_fallback_charge
439 is incremented if a page fault fails t 439 is incremented if a page fault fails to charge a huge page and
440 instead falls back to using small page 440 instead falls back to using small pages even though the
441 allocation was successful. 441 allocation was successful.
442 442
443 thp_collapse_alloc_failed 443 thp_collapse_alloc_failed
444 is incremented if khugepaged found a r 444 is incremented if khugepaged found a range
445 of pages that should be collapsed into 445 of pages that should be collapsed into one huge page but failed
446 the allocation. 446 the allocation.
447 447
448 thp_file_alloc 448 thp_file_alloc
449 is incremented every time a shmem huge 449 is incremented every time a shmem huge page is successfully
450 allocated (Note that despite being nam 450 allocated (Note that despite being named after "file", the counter
451 measures only shmem). 451 measures only shmem).
452 452
453 thp_file_fallback 453 thp_file_fallback
454 is incremented if a shmem huge page is 454 is incremented if a shmem huge page is attempted to be allocated
455 but fails and instead falls back to us 455 but fails and instead falls back to using small pages. (Note that
456 despite being named after "file", the 456 despite being named after "file", the counter measures only shmem).
457 457
458 thp_file_fallback_charge 458 thp_file_fallback_charge
459 is incremented if a shmem huge page ca 459 is incremented if a shmem huge page cannot be charged and instead
460 falls back to using small pages even t 460 falls back to using small pages even though the allocation was
461 successful. (Note that despite being n 461 successful. (Note that despite being named after "file", the
462 counter measures only shmem). 462 counter measures only shmem).
463 463
464 thp_file_mapped 464 thp_file_mapped
465 is incremented every time a file or sh 465 is incremented every time a file or shmem huge page is mapped into
466 user address space. 466 user address space.
467 467
468 thp_split_page 468 thp_split_page
469 is incremented every time a huge page 469 is incremented every time a huge page is split into base
470 pages. This can happen for a variety o 470 pages. This can happen for a variety of reasons but a common
471 reason is that a huge page is old and 471 reason is that a huge page is old and is being reclaimed.
472 This action implies splitting all PMD 472 This action implies splitting all PMD the page mapped with.
473 473
474 thp_split_page_failed 474 thp_split_page_failed
475 is incremented if kernel fails to spli 475 is incremented if kernel fails to split huge
476 page. This can happen if the page was 476 page. This can happen if the page was pinned by somebody.
477 477
478 thp_deferred_split_page 478 thp_deferred_split_page
479 is incremented when a huge page is put 479 is incremented when a huge page is put onto split
480 queue. This happens when a huge page i 480 queue. This happens when a huge page is partially unmapped and
481 splitting it would free up some memory 481 splitting it would free up some memory. Pages on split queue are
482 going to be split under memory pressur 482 going to be split under memory pressure.
483 483
484 thp_underused_split_page 484 thp_underused_split_page
485 is incremented when a huge page on the 485 is incremented when a huge page on the split queue was split
486 because it was underused. A THP is und 486 because it was underused. A THP is underused if the number of
487 zero pages in the THP is above a certa 487 zero pages in the THP is above a certain threshold
488 (/sys/kernel/mm/transparent_hugepage/k 488 (/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none).
489 489
490 thp_split_pmd 490 thp_split_pmd
491 is incremented every time a PMD split 491 is incremented every time a PMD split into table of PTEs.
492 This can happen, for instance, when ap 492 This can happen, for instance, when application calls mprotect() or
493 munmap() on part of huge page. It does 493 munmap() on part of huge page. It doesn't split huge page, only
494 page table entry. 494 page table entry.
495 495
496 thp_zero_page_alloc 496 thp_zero_page_alloc
497 is incremented every time a huge zero 497 is incremented every time a huge zero page used for thp is
498 successfully allocated. Note, it doesn 498 successfully allocated. Note, it doesn't count every map of
499 the huge zero page, only its allocatio 499 the huge zero page, only its allocation.
500 500
501 thp_zero_page_alloc_failed 501 thp_zero_page_alloc_failed
502 is incremented if kernel fails to allo 502 is incremented if kernel fails to allocate
503 huge zero page and falls back to using 503 huge zero page and falls back to using small pages.
504 504
505 thp_swpout 505 thp_swpout
506 is incremented every time a huge page 506 is incremented every time a huge page is swapout in one
507 piece without splitting. 507 piece without splitting.
508 508
509 thp_swpout_fallback 509 thp_swpout_fallback
510 is incremented if a huge page has to b 510 is incremented if a huge page has to be split before swapout.
511 Usually because failed to allocate som 511 Usually because failed to allocate some continuous swap space
512 for the huge page. 512 for the huge page.
513 513
514 In /sys/kernel/mm/transparent_hugepage/hugepag 514 In /sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/stats, There are
515 also individual counters for each huge page si 515 also individual counters for each huge page size, which can be utilized to
516 monitor the system's effectiveness in providin 516 monitor the system's effectiveness in providing huge pages for usage. Each
517 counter has its own corresponding file. 517 counter has its own corresponding file.
518 518
519 anon_fault_alloc 519 anon_fault_alloc
520 is incremented every time a huge page 520 is incremented every time a huge page is successfully
521 allocated and charged to handle a page 521 allocated and charged to handle a page fault.
522 522
523 anon_fault_fallback 523 anon_fault_fallback
524 is incremented if a page fault fails t 524 is incremented if a page fault fails to allocate or charge
525 a huge page and instead falls back to 525 a huge page and instead falls back to using huge pages with
526 lower orders or small pages. 526 lower orders or small pages.
527 527
528 anon_fault_fallback_charge 528 anon_fault_fallback_charge
529 is incremented if a page fault fails t 529 is incremented if a page fault fails to charge a huge page and
530 instead falls back to using huge pages 530 instead falls back to using huge pages with lower orders or
531 small pages even though the allocation 531 small pages even though the allocation was successful.
532 532
533 swpout 533 swpout
534 is incremented every time a huge page 534 is incremented every time a huge page is swapped out in one
535 piece without splitting. 535 piece without splitting.
536 536
537 swpout_fallback 537 swpout_fallback
538 is incremented if a huge page has to b 538 is incremented if a huge page has to be split before swapout.
539 Usually because failed to allocate som 539 Usually because failed to allocate some continuous swap space
540 for the huge page. 540 for the huge page.
541 541
542 shmem_alloc 542 shmem_alloc
543 is incremented every time a shmem huge 543 is incremented every time a shmem huge page is successfully
544 allocated. 544 allocated.
545 545
546 shmem_fallback 546 shmem_fallback
547 is incremented if a shmem huge page is 547 is incremented if a shmem huge page is attempted to be allocated
548 but fails and instead falls back to us 548 but fails and instead falls back to using small pages.
549 549
550 shmem_fallback_charge 550 shmem_fallback_charge
551 is incremented if a shmem huge page ca 551 is incremented if a shmem huge page cannot be charged and instead
552 falls back to using small pages even t 552 falls back to using small pages even though the allocation was
553 successful. 553 successful.
554 554
555 split 555 split
556 is incremented every time a huge page 556 is incremented every time a huge page is successfully split into
557 smaller orders. This can happen for a 557 smaller orders. This can happen for a variety of reasons but a
558 common reason is that a huge page is o 558 common reason is that a huge page is old and is being reclaimed.
559 559
560 split_failed 560 split_failed
561 is incremented if kernel fails to spli 561 is incremented if kernel fails to split huge
562 page. This can happen if the page was 562 page. This can happen if the page was pinned by somebody.
563 563
564 split_deferred 564 split_deferred
565 is incremented when a huge page is put 565 is incremented when a huge page is put onto split queue.
566 This happens when a huge page is parti 566 This happens when a huge page is partially unmapped and splitting
567 it would free up some memory. Pages on 567 it would free up some memory. Pages on split queue are going to
568 be split under memory pressure, if spl 568 be split under memory pressure, if splitting is possible.
569 569
570 nr_anon 570 nr_anon
571 the number of anonymous THP we have in 571 the number of anonymous THP we have in the whole system. These THPs
572 might be currently entirely mapped or h 572 might be currently entirely mapped or have partially unmapped/unused
573 subpages. 573 subpages.
574 574
575 nr_anon_partially_mapped 575 nr_anon_partially_mapped
576 the number of anonymous THP which are l 576 the number of anonymous THP which are likely partially mapped, possibly
577 wasting memory, and have been queued fo 577 wasting memory, and have been queued for deferred memory reclamation.
578 Note that in corner some cases (e.g., f 578 Note that in corner some cases (e.g., failed migration), we might detect
579 an anonymous THP as "partially mapped" 579 an anonymous THP as "partially mapped" and count it here, even though it
580 is not actually partially mapped anymor 580 is not actually partially mapped anymore.
581 581
582 As the system ages, allocating huge pages may 582 As the system ages, allocating huge pages may be expensive as the
583 system uses memory compaction to copy data aro 583 system uses memory compaction to copy data around memory to free a
584 huge page for use. There are some counters in 584 huge page for use. There are some counters in ``/proc/vmstat`` to help
585 monitor this overhead. 585 monitor this overhead.
586 586
587 compact_stall 587 compact_stall
588 is incremented every time a process st 588 is incremented every time a process stalls to run
589 memory compaction so that a huge page 589 memory compaction so that a huge page is free for use.
590 590
591 compact_success 591 compact_success
592 is incremented if the system compacted 592 is incremented if the system compacted memory and
593 freed a huge page for use. 593 freed a huge page for use.
594 594
595 compact_fail 595 compact_fail
596 is incremented if the system tries to 596 is incremented if the system tries to compact memory
597 but failed. 597 but failed.
598 598
599 It is possible to establish how long the stall 599 It is possible to establish how long the stalls were using the function
600 tracer to record how long was spent in __alloc 600 tracer to record how long was spent in __alloc_pages() and
601 using the mm_page_alloc tracepoint to identify 601 using the mm_page_alloc tracepoint to identify which allocations were
602 for huge pages. 602 for huge pages.
603 603
604 Optimizing the applications 604 Optimizing the applications
605 =========================== 605 ===========================
606 606
607 To be guaranteed that the kernel will map a TH 607 To be guaranteed that the kernel will map a THP immediately in any
608 memory region, the mmap region has to be hugep 608 memory region, the mmap region has to be hugepage naturally
609 aligned. posix_memalign() can provide that gua 609 aligned. posix_memalign() can provide that guarantee.
610 610
611 Hugetlbfs 611 Hugetlbfs
612 ========= 612 =========
613 613
614 You can use hugetlbfs on a kernel that has tra 614 You can use hugetlbfs on a kernel that has transparent hugepage
615 support enabled just fine as always. No differ 615 support enabled just fine as always. No difference can be noted in
616 hugetlbfs other than there will be less overal 616 hugetlbfs other than there will be less overall fragmentation. All
617 usual features belonging to hugetlbfs are pres 617 usual features belonging to hugetlbfs are preserved and
618 unaffected. libhugetlbfs will also work fine a 618 unaffected. libhugetlbfs will also work fine as usual.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.