1 ================== 1 ==================
2 HugeTLB Controller 2 HugeTLB Controller
3 ================== 3 ==================
4 4
>> 5 The HugeTLB controller allows to limit the HugeTLB usage per control group and
>> 6 enforces the controller limit during page fault. Since HugeTLB doesn't
>> 7 support page reclaim, enforcing the limit at page fault time implies that,
>> 8 the application will get SIGBUS signal if it tries to access HugeTLB pages
>> 9 beyond its limit. This requires the application to know beforehand how much
>> 10 HugeTLB pages it would require for its use.
>> 11
5 HugeTLB controller can be created by first mou 12 HugeTLB controller can be created by first mounting the cgroup filesystem.
6 13
7 # mount -t cgroup -o hugetlb none /sys/fs/cgro 14 # mount -t cgroup -o hugetlb none /sys/fs/cgroup
8 15
9 With the above step, the initial or the parent 16 With the above step, the initial or the parent HugeTLB group becomes
10 visible at /sys/fs/cgroup. At bootup, this gro 17 visible at /sys/fs/cgroup. At bootup, this group includes all the tasks in
11 the system. /sys/fs/cgroup/tasks lists the tas 18 the system. /sys/fs/cgroup/tasks lists the tasks in this cgroup.
12 19
13 New groups can be created under the parent gro 20 New groups can be created under the parent group /sys/fs/cgroup::
14 21
15 # cd /sys/fs/cgroup 22 # cd /sys/fs/cgroup
16 # mkdir g1 23 # mkdir g1
17 # echo $$ > g1/tasks 24 # echo $$ > g1/tasks
18 25
19 The above steps create a new group g1 and move 26 The above steps create a new group g1 and move the current shell
20 process (bash) into it. 27 process (bash) into it.
21 28
22 Brief summary of control files:: 29 Brief summary of control files::
23 30
24 hugetlb.<hugepagesize>.rsvd.limit_in_bytes !! 31 hugetlb.<hugepagesize>.limit_in_bytes # set/show limit of "hugepagesize" hugetlb usage
25 hugetlb.<hugepagesize>.rsvd.max_usage_in_byte !! 32 hugetlb.<hugepagesize>.max_usage_in_bytes # show max "hugepagesize" hugetlb usage recorded
26 hugetlb.<hugepagesize>.rsvd.usage_in_bytes !! 33 hugetlb.<hugepagesize>.usage_in_bytes # show current usage for "hugepagesize" hugetlb
27 hugetlb.<hugepagesize>.rsvd.failcnt !! 34 hugetlb.<hugepagesize>.failcnt # show the number of allocation failure due to HugeTLB limit
28 hugetlb.<hugepagesize>.limit_in_bytes <<
29 hugetlb.<hugepagesize>.max_usage_in_bytes <<
30 hugetlb.<hugepagesize>.usage_in_bytes <<
31 hugetlb.<hugepagesize>.failcnt <<
32 hugetlb.<hugepagesize>.numa_stat <<
33 35
34 For a system supporting three hugepage sizes ( 36 For a system supporting three hugepage sizes (64k, 32M and 1G), the control
35 files include:: 37 files include::
36 38
37 hugetlb.1GB.limit_in_bytes 39 hugetlb.1GB.limit_in_bytes
38 hugetlb.1GB.max_usage_in_bytes 40 hugetlb.1GB.max_usage_in_bytes
39 hugetlb.1GB.numa_stat <<
40 hugetlb.1GB.usage_in_bytes 41 hugetlb.1GB.usage_in_bytes
41 hugetlb.1GB.failcnt 42 hugetlb.1GB.failcnt
42 hugetlb.1GB.rsvd.limit_in_bytes <<
43 hugetlb.1GB.rsvd.max_usage_in_bytes <<
44 hugetlb.1GB.rsvd.usage_in_bytes <<
45 hugetlb.1GB.rsvd.failcnt <<
46 hugetlb.64KB.limit_in_bytes 43 hugetlb.64KB.limit_in_bytes
47 hugetlb.64KB.max_usage_in_bytes 44 hugetlb.64KB.max_usage_in_bytes
48 hugetlb.64KB.numa_stat <<
49 hugetlb.64KB.usage_in_bytes 45 hugetlb.64KB.usage_in_bytes
50 hugetlb.64KB.failcnt 46 hugetlb.64KB.failcnt
51 hugetlb.64KB.rsvd.limit_in_bytes <<
52 hugetlb.64KB.rsvd.max_usage_in_bytes <<
53 hugetlb.64KB.rsvd.usage_in_bytes <<
54 hugetlb.64KB.rsvd.failcnt <<
55 hugetlb.32MB.limit_in_bytes 47 hugetlb.32MB.limit_in_bytes
56 hugetlb.32MB.max_usage_in_bytes 48 hugetlb.32MB.max_usage_in_bytes
57 hugetlb.32MB.numa_stat <<
58 hugetlb.32MB.usage_in_bytes 49 hugetlb.32MB.usage_in_bytes
59 hugetlb.32MB.failcnt 50 hugetlb.32MB.failcnt
60 hugetlb.32MB.rsvd.limit_in_bytes <<
61 hugetlb.32MB.rsvd.max_usage_in_bytes <<
62 hugetlb.32MB.rsvd.usage_in_bytes <<
63 hugetlb.32MB.rsvd.failcnt <<
64 <<
65 <<
66 1. Page fault accounting <<
67 <<
68 :: <<
69 <<
70 hugetlb.<hugepagesize>.limit_in_bytes <<
71 hugetlb.<hugepagesize>.max_usage_in_bytes <<
72 hugetlb.<hugepagesize>.usage_in_bytes <<
73 hugetlb.<hugepagesize>.failcnt <<
74 <<
75 The HugeTLB controller allows users to limit t <<
76 control group and enforces the limit during pa <<
77 doesn't support page reclaim, enforcing the li <<
78 that, the application will get SIGBUS signal i <<
79 pages beyond its limit. Therefore the applicat <<
80 HugeTLB pages it uses before hand, and the sys <<
81 there are enough available on the machine for <<
82 getting SIGBUS. <<
83 <<
84 <<
85 2. Reservation accounting <<
86 <<
87 :: <<
88 <<
89 hugetlb.<hugepagesize>.rsvd.limit_in_bytes <<
90 hugetlb.<hugepagesize>.rsvd.max_usage_in_byt <<
91 hugetlb.<hugepagesize>.rsvd.usage_in_bytes <<
92 hugetlb.<hugepagesize>.rsvd.failcnt <<
93 <<
94 The HugeTLB controller allows to limit the Hug <<
95 group and enforces the controller limit at res <<
96 HugeTLB memory for which no reservation exists <<
97 enforced at reservation time (on mmap or shget <<
98 the application to get SIGBUS signal if the me <<
99 MAP_NORESERVE allocations, the reservation lim <<
100 limit, enforcing memory usage at fault time an <<
101 receive a SIGBUS if it's crossing its limit. <<
102 <<
103 Reservation limits are superior to page fault <<
104 reservation limits are enforced at reservation <<
105 never causes the application to get SIGBUS sig <<
106 before hand. This allows for easier fallback t <<
107 non-HugeTLB memory for example. In the case of <<
108 hard to avoid processes getting SIGBUS since t <<
109 the HugeTLB usage of all the tasks in the syst <<
110 pages to satisfy all requests. Avoiding tasks <<
111 systems is practically impossible with page fa <<
112 <<
113 <<
114 3. Caveats with shared memory <<
115 <<
116 For shared HugeTLB memory, both HugeTLB reserv <<
117 to the first task that causes the memory to be <<
118 subsequent uses of this reserved or faulted me <<
119 <<
120 Shared HugeTLB memory is only uncharged when i <<
121 This is usually when the HugeTLB file is delet <<
122 caused the reservation or fault has exited. <<
123 <<
124 <<
125 4. Caveats with HugeTLB cgroup offline. <<
126 <<
127 When a HugeTLB cgroup goes offline with some r <<
128 charged to it, the behavior is as follows: <<
129 <<
130 - The fault charges are charged to the parent <<
131 - the reservation charges remain on the offlin <<
132 <<
133 This means that if a HugeTLB cgroup gets offli <<
134 reservations charged to it, that cgroup persis <<
135 reservations are uncharged. HugeTLB reservatio <<
136 the memory controller whose cgroups also persi <<
137 memory is uncharged. Also, the tracking of Hug <<
138 complex compared to the tracking of HugeTLB fa <<
139 harder to reparent reservations at offline tim <<
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.