1 ============== 1 ==============
2 Control Groups 2 Control Groups
3 ============== 3 ==============
4 4
5 Written by Paul Menage <menage@google.com> base 5 Written by Paul Menage <menage@google.com> based on
6 Documentation/admin-guide/cgroup-v1/cpusets.rs 6 Documentation/admin-guide/cgroup-v1/cpusets.rst
7 7
8 Original copyright statements from cpusets.txt 8 Original copyright statements from cpusets.txt:
9 9
10 Portions Copyright (C) 2004 BULL SA. 10 Portions Copyright (C) 2004 BULL SA.
11 11
12 Portions Copyright (c) 2004-2006 Silicon Graph 12 Portions Copyright (c) 2004-2006 Silicon Graphics, Inc.
13 13
14 Modified by Paul Jackson <pj@sgi.com> 14 Modified by Paul Jackson <pj@sgi.com>
15 15
16 Modified by Christoph Lameter <cl@linux.com> 16 Modified by Christoph Lameter <cl@linux.com>
17 17
18 .. CONTENTS: 18 .. CONTENTS:
19 19
20 1. Control Groups 20 1. Control Groups
21 1.1 What are cgroups ? 21 1.1 What are cgroups ?
22 1.2 Why are cgroups needed ? 22 1.2 Why are cgroups needed ?
23 1.3 How are cgroups implemented ? 23 1.3 How are cgroups implemented ?
24 1.4 What does notify_on_release do ? 24 1.4 What does notify_on_release do ?
25 1.5 What does clone_children do ? 25 1.5 What does clone_children do ?
26 1.6 How do I use cgroups ? 26 1.6 How do I use cgroups ?
27 2. Usage Examples and Syntax 27 2. Usage Examples and Syntax
28 2.1 Basic Usage 28 2.1 Basic Usage
29 2.2 Attaching processes 29 2.2 Attaching processes
30 2.3 Mounting hierarchies by name 30 2.3 Mounting hierarchies by name
31 3. Kernel API 31 3. Kernel API
32 3.1 Overview 32 3.1 Overview
33 3.2 Synchronization 33 3.2 Synchronization
34 3.3 Subsystem API 34 3.3 Subsystem API
35 4. Extended attributes usage 35 4. Extended attributes usage
36 5. Questions 36 5. Questions
37 37
38 1. Control Groups 38 1. Control Groups
39 ================= 39 =================
40 40
41 1.1 What are cgroups ? 41 1.1 What are cgroups ?
42 ---------------------- 42 ----------------------
43 43
44 Control Groups provide a mechanism for aggrega 44 Control Groups provide a mechanism for aggregating/partitioning sets of
45 tasks, and all their future children, into hie 45 tasks, and all their future children, into hierarchical groups with
46 specialized behaviour. 46 specialized behaviour.
47 47
48 Definitions: 48 Definitions:
49 49
50 A *cgroup* associates a set of tasks with a se 50 A *cgroup* associates a set of tasks with a set of parameters for one
51 or more subsystems. 51 or more subsystems.
52 52
53 A *subsystem* is a module that makes use of th 53 A *subsystem* is a module that makes use of the task grouping
54 facilities provided by cgroups to treat groups 54 facilities provided by cgroups to treat groups of tasks in
55 particular ways. A subsystem is typically a "r 55 particular ways. A subsystem is typically a "resource controller" that
56 schedules a resource or applies per-cgroup lim 56 schedules a resource or applies per-cgroup limits, but it may be
57 anything that wants to act on a group of proce 57 anything that wants to act on a group of processes, e.g. a
58 virtualization subsystem. 58 virtualization subsystem.
59 59
60 A *hierarchy* is a set of cgroups arranged in 60 A *hierarchy* is a set of cgroups arranged in a tree, such that
61 every task in the system is in exactly one of 61 every task in the system is in exactly one of the cgroups in the
62 hierarchy, and a set of subsystems; each subsy 62 hierarchy, and a set of subsystems; each subsystem has system-specific
63 state attached to each cgroup in the hierarchy 63 state attached to each cgroup in the hierarchy. Each hierarchy has
64 an instance of the cgroup virtual filesystem a 64 an instance of the cgroup virtual filesystem associated with it.
65 65
66 At any one time there may be multiple active h 66 At any one time there may be multiple active hierarchies of task
67 cgroups. Each hierarchy is a partition of all 67 cgroups. Each hierarchy is a partition of all tasks in the system.
68 68
69 User-level code may create and destroy cgroups 69 User-level code may create and destroy cgroups by name in an
70 instance of the cgroup virtual file system, sp 70 instance of the cgroup virtual file system, specify and query to
71 which cgroup a task is assigned, and list the 71 which cgroup a task is assigned, and list the task PIDs assigned to
72 a cgroup. Those creations and assignments only 72 a cgroup. Those creations and assignments only affect the hierarchy
73 associated with that instance of the cgroup fi 73 associated with that instance of the cgroup file system.
74 74
75 On their own, the only use for cgroups is for 75 On their own, the only use for cgroups is for simple job
76 tracking. The intention is that other subsyste 76 tracking. The intention is that other subsystems hook into the generic
77 cgroup support to provide new attributes for c 77 cgroup support to provide new attributes for cgroups, such as
78 accounting/limiting the resources which proces 78 accounting/limiting the resources which processes in a cgroup can
79 access. For example, cpusets (see Documentatio 79 access. For example, cpusets (see Documentation/admin-guide/cgroup-v1/cpusets.rst) allow
80 you to associate a set of CPUs and a set of me 80 you to associate a set of CPUs and a set of memory nodes with the
81 tasks in each cgroup. 81 tasks in each cgroup.
82 82
83 .. _cgroups-why-needed: 83 .. _cgroups-why-needed:
84 84
85 1.2 Why are cgroups needed ? 85 1.2 Why are cgroups needed ?
86 ---------------------------- 86 ----------------------------
87 87
88 There are multiple efforts to provide process 88 There are multiple efforts to provide process aggregations in the
89 Linux kernel, mainly for resource-tracking pur 89 Linux kernel, mainly for resource-tracking purposes. Such efforts
90 include cpusets, CKRM/ResGroups, UserBeanCount 90 include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server
91 namespaces. These all require the basic notion 91 namespaces. These all require the basic notion of a
92 grouping/partitioning of processes, with newly 92 grouping/partitioning of processes, with newly forked processes ending
93 up in the same group (cgroup) as their parent 93 up in the same group (cgroup) as their parent process.
94 94
95 The kernel cgroup patch provides the minimum e 95 The kernel cgroup patch provides the minimum essential kernel
96 mechanisms required to efficiently implement s 96 mechanisms required to efficiently implement such groups. It has
97 minimal impact on the system fast paths, and p 97 minimal impact on the system fast paths, and provides hooks for
98 specific subsystems such as cpusets to provide 98 specific subsystems such as cpusets to provide additional behaviour as
99 desired. 99 desired.
100 100
101 Multiple hierarchy support is provided to allo 101 Multiple hierarchy support is provided to allow for situations where
102 the division of tasks into cgroups is distinct 102 the division of tasks into cgroups is distinctly different for
103 different subsystems - having parallel hierarc 103 different subsystems - having parallel hierarchies allows each
104 hierarchy to be a natural division of tasks, w 104 hierarchy to be a natural division of tasks, without having to handle
105 complex combinations of tasks that would be pr 105 complex combinations of tasks that would be present if several
106 unrelated subsystems needed to be forced into 106 unrelated subsystems needed to be forced into the same tree of
107 cgroups. 107 cgroups.
108 108
109 At one extreme, each resource controller or su 109 At one extreme, each resource controller or subsystem could be in a
110 separate hierarchy; at the other extreme, all 110 separate hierarchy; at the other extreme, all subsystems
111 would be attached to the same hierarchy. 111 would be attached to the same hierarchy.
112 112
113 As an example of a scenario (originally propos 113 As an example of a scenario (originally proposed by vatsa@in.ibm.com)
114 that can benefit from multiple hierarchies, co 114 that can benefit from multiple hierarchies, consider a large
115 university server with various users - student 115 university server with various users - students, professors, system
116 tasks etc. The resource planning for this serv 116 tasks etc. The resource planning for this server could be along the
117 following lines:: 117 following lines::
118 118
119 CPU : "Top cpuset" 119 CPU : "Top cpuset"
120 / \ 120 / \
121 CPUSet1 CPUSet2 121 CPUSet1 CPUSet2
122 | | 122 | |
123 (Professors) (Students) 123 (Professors) (Students)
124 124
125 In addition (system tasks) are 125 In addition (system tasks) are attached to topcpuset (so
126 that they can run anywhere) wit 126 that they can run anywhere) with a limit of 20%
127 127
128 Memory : Professors (50%), Students (30 128 Memory : Professors (50%), Students (30%), system (20%)
129 129
130 Disk : Professors (50%), Students (30%) 130 Disk : Professors (50%), Students (30%), system (20%)
131 131
132 Network : WWW browsing (20%), Network F 132 Network : WWW browsing (20%), Network File System (60%), others (20%)
133 / \ 133 / \
134 Professors (15%) students (5%) 134 Professors (15%) students (5%)
135 135
136 Browsers like Firefox/Lynx go into the WWW net 136 Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd goes
137 into the NFS network class. 137 into the NFS network class.
138 138
139 At the same time Firefox/Lynx will share an ap 139 At the same time Firefox/Lynx will share an appropriate CPU/Memory class
140 depending on who launched it (prof/student). 140 depending on who launched it (prof/student).
141 141
142 With the ability to classify tasks differently 142 With the ability to classify tasks differently for different resources
143 (by putting those resource subsystems in diffe 143 (by putting those resource subsystems in different hierarchies),
144 the admin can easily set up a script which rec 144 the admin can easily set up a script which receives exec notifications
145 and depending on who is launching the browser 145 and depending on who is launching the browser he can::
146 146
147 # echo browser_pid > /sys/fs/cgroup/<resty 147 # echo browser_pid > /sys/fs/cgroup/<restype>/<userclass>/tasks
148 148
149 With only a single hierarchy, he now would pot 149 With only a single hierarchy, he now would potentially have to create
150 a separate cgroup for every browser launched a 150 a separate cgroup for every browser launched and associate it with
151 appropriate network and other resource class. 151 appropriate network and other resource class. This may lead to
152 proliferation of such cgroups. 152 proliferation of such cgroups.
153 153
154 Also let's say that the administrator would li 154 Also let's say that the administrator would like to give enhanced network
155 access temporarily to a student's browser (sin 155 access temporarily to a student's browser (since it is night and the user
156 wants to do online gaming :)) OR give one of 156 wants to do online gaming :)) OR give one of the student's simulation
157 apps enhanced CPU power. 157 apps enhanced CPU power.
158 158
159 With ability to write PIDs directly to resourc 159 With ability to write PIDs directly to resource classes, it's just a
160 matter of:: 160 matter of::
161 161
162 # echo pid > /sys/fs/cgroup/network/<ne 162 # echo pid > /sys/fs/cgroup/network/<new_class>/tasks
163 (after some time) 163 (after some time)
164 # echo pid > /sys/fs/cgroup/network/<or 164 # echo pid > /sys/fs/cgroup/network/<orig_class>/tasks
165 165
166 Without this ability, the administrator would 166 Without this ability, the administrator would have to split the cgroup into
167 multiple separate ones and then associate the 167 multiple separate ones and then associate the new cgroups with the
168 new resource classes. 168 new resource classes.
169 169
170 170
171 171
172 1.3 How are cgroups implemented ? 172 1.3 How are cgroups implemented ?
173 --------------------------------- 173 ---------------------------------
174 174
175 Control Groups extends the kernel as follows: 175 Control Groups extends the kernel as follows:
176 176
177 - Each task in the system has a reference-cou 177 - Each task in the system has a reference-counted pointer to a
178 css_set. 178 css_set.
179 179
180 - A css_set contains a set of reference-count 180 - A css_set contains a set of reference-counted pointers to
181 cgroup_subsys_state objects, one for each c 181 cgroup_subsys_state objects, one for each cgroup subsystem
182 registered in the system. There is no direc 182 registered in the system. There is no direct link from a task to
183 the cgroup of which it's a member in each h 183 the cgroup of which it's a member in each hierarchy, but this
184 can be determined by following pointers thr 184 can be determined by following pointers through the
185 cgroup_subsys_state objects. This is becaus 185 cgroup_subsys_state objects. This is because accessing the
186 subsystem state is something that's expecte 186 subsystem state is something that's expected to happen frequently
187 and in performance-critical code, whereas o 187 and in performance-critical code, whereas operations that require a
188 task's actual cgroup assignments (in partic 188 task's actual cgroup assignments (in particular, moving between
189 cgroups) are less common. A linked list run 189 cgroups) are less common. A linked list runs through the cg_list
190 field of each task_struct using the css_set 190 field of each task_struct using the css_set, anchored at
191 css_set->tasks. 191 css_set->tasks.
192 192
193 - A cgroup hierarchy filesystem can be mounte 193 - A cgroup hierarchy filesystem can be mounted for browsing and
194 manipulation from user space. 194 manipulation from user space.
195 195
196 - You can list all the tasks (by PID) attache 196 - You can list all the tasks (by PID) attached to any cgroup.
197 197
198 The implementation of cgroups requires a few, 198 The implementation of cgroups requires a few, simple hooks
199 into the rest of the kernel, none in performan 199 into the rest of the kernel, none in performance-critical paths:
200 200
201 - in init/main.c, to initialize the root cgro 201 - in init/main.c, to initialize the root cgroups and initial
202 css_set at system boot. 202 css_set at system boot.
203 203
204 - in fork and exit, to attach and detach a ta 204 - in fork and exit, to attach and detach a task from its css_set.
205 205
206 In addition, a new file system of type "cgroup 206 In addition, a new file system of type "cgroup" may be mounted, to
207 enable browsing and modifying the cgroups pres 207 enable browsing and modifying the cgroups presently known to the
208 kernel. When mounting a cgroup hierarchy, you 208 kernel. When mounting a cgroup hierarchy, you may specify a
209 comma-separated list of subsystems to mount as 209 comma-separated list of subsystems to mount as the filesystem mount
210 options. By default, mounting the cgroup file 210 options. By default, mounting the cgroup filesystem attempts to
211 mount a hierarchy containing all registered su 211 mount a hierarchy containing all registered subsystems.
212 212
213 If an active hierarchy with exactly the same s 213 If an active hierarchy with exactly the same set of subsystems already
214 exists, it will be reused for the new mount. I 214 exists, it will be reused for the new mount. If no existing hierarchy
215 matches, and any of the requested subsystems a 215 matches, and any of the requested subsystems are in use in an existing
216 hierarchy, the mount will fail with -EBUSY. Ot 216 hierarchy, the mount will fail with -EBUSY. Otherwise, a new hierarchy
217 is activated, associated with the requested su 217 is activated, associated with the requested subsystems.
218 218
219 It's not currently possible to bind a new subs 219 It's not currently possible to bind a new subsystem to an active
220 cgroup hierarchy, or to unbind a subsystem fro 220 cgroup hierarchy, or to unbind a subsystem from an active cgroup
221 hierarchy. This may be possible in future, but 221 hierarchy. This may be possible in future, but is fraught with nasty
222 error-recovery issues. 222 error-recovery issues.
223 223
224 When a cgroup filesystem is unmounted, if ther 224 When a cgroup filesystem is unmounted, if there are any
225 child cgroups created below the top-level cgro 225 child cgroups created below the top-level cgroup, that hierarchy
226 will remain active even though unmounted; if t 226 will remain active even though unmounted; if there are no
227 child cgroups then the hierarchy will be deact 227 child cgroups then the hierarchy will be deactivated.
228 228
229 No new system calls are added for cgroups - al 229 No new system calls are added for cgroups - all support for
230 querying and modifying cgroups is via this cgr 230 querying and modifying cgroups is via this cgroup file system.
231 231
232 Each task under /proc has an added file named 232 Each task under /proc has an added file named 'cgroup' displaying,
233 for each active hierarchy, the subsystem names 233 for each active hierarchy, the subsystem names and the cgroup name
234 as the path relative to the root of the cgroup 234 as the path relative to the root of the cgroup file system.
235 235
236 Each cgroup is represented by a directory in t 236 Each cgroup is represented by a directory in the cgroup file system
237 containing the following files describing that 237 containing the following files describing that cgroup:
238 238
239 - tasks: list of tasks (by PID) attached to t 239 - tasks: list of tasks (by PID) attached to that cgroup. This list
240 is not guaranteed to be sorted. Writing a 240 is not guaranteed to be sorted. Writing a thread ID into this file
241 moves the thread into this cgroup. 241 moves the thread into this cgroup.
242 - cgroup.procs: list of thread group IDs in t 242 - cgroup.procs: list of thread group IDs in the cgroup. This list is
243 not guaranteed to be sorted or free of dupl 243 not guaranteed to be sorted or free of duplicate TGIDs, and userspace
244 should sort/uniquify the list if this prope 244 should sort/uniquify the list if this property is required.
245 Writing a thread group ID into this file mo 245 Writing a thread group ID into this file moves all threads in that
246 group into this cgroup. 246 group into this cgroup.
247 - notify_on_release flag: run the release age 247 - notify_on_release flag: run the release agent on exit?
248 - release_agent: the path to use for release 248 - release_agent: the path to use for release notifications (this file
249 exists in the top cgroup only) 249 exists in the top cgroup only)
250 250
251 Other subsystems such as cpusets may add addit 251 Other subsystems such as cpusets may add additional files in each
252 cgroup dir. 252 cgroup dir.
253 253
254 New cgroups are created using the mkdir system 254 New cgroups are created using the mkdir system call or shell
255 command. The properties of a cgroup, such as 255 command. The properties of a cgroup, such as its flags, are
256 modified by writing to the appropriate file in 256 modified by writing to the appropriate file in that cgroups
257 directory, as listed above. 257 directory, as listed above.
258 258
259 The named hierarchical structure of nested cgr 259 The named hierarchical structure of nested cgroups allows partitioning
260 a large system into nested, dynamically change 260 a large system into nested, dynamically changeable, "soft-partitions".
261 261
262 The attachment of each task, automatically inh 262 The attachment of each task, automatically inherited at fork by any
263 children of that task, to a cgroup allows orga 263 children of that task, to a cgroup allows organizing the work load
264 on a system into related sets of tasks. A tas 264 on a system into related sets of tasks. A task may be re-attached to
265 any other cgroup, if allowed by the permission 265 any other cgroup, if allowed by the permissions on the necessary
266 cgroup file system directories. 266 cgroup file system directories.
267 267
268 When a task is moved from one cgroup to anothe 268 When a task is moved from one cgroup to another, it gets a new
269 css_set pointer - if there's an already existi 269 css_set pointer - if there's an already existing css_set with the
270 desired collection of cgroups then that group 270 desired collection of cgroups then that group is reused, otherwise a new
271 css_set is allocated. The appropriate existing 271 css_set is allocated. The appropriate existing css_set is located by
272 looking into a hash table. 272 looking into a hash table.
273 273
274 To allow access from a cgroup to the css_sets 274 To allow access from a cgroup to the css_sets (and hence tasks)
275 that comprise it, a set of cg_cgroup_link obje 275 that comprise it, a set of cg_cgroup_link objects form a lattice;
276 each cg_cgroup_link is linked into a list of c 276 each cg_cgroup_link is linked into a list of cg_cgroup_links for
277 a single cgroup on its cgrp_link_list field, a 277 a single cgroup on its cgrp_link_list field, and a list of
278 cg_cgroup_links for a single css_set on its cg 278 cg_cgroup_links for a single css_set on its cg_link_list.
279 279
280 Thus the set of tasks in a cgroup can be liste 280 Thus the set of tasks in a cgroup can be listed by iterating over
281 each css_set that references the cgroup, and s 281 each css_set that references the cgroup, and sub-iterating over
282 each css_set's task set. 282 each css_set's task set.
283 283
284 The use of a Linux virtual file system (vfs) t 284 The use of a Linux virtual file system (vfs) to represent the
285 cgroup hierarchy provides for a familiar permi 285 cgroup hierarchy provides for a familiar permission and name space
286 for cgroups, with a minimum of additional kern 286 for cgroups, with a minimum of additional kernel code.
287 287
288 1.4 What does notify_on_release do ? 288 1.4 What does notify_on_release do ?
289 ------------------------------------ 289 ------------------------------------
290 290
291 If the notify_on_release flag is enabled (1) i 291 If the notify_on_release flag is enabled (1) in a cgroup, then
292 whenever the last task in the cgroup leaves (e 292 whenever the last task in the cgroup leaves (exits or attaches to
293 some other cgroup) and the last child cgroup o 293 some other cgroup) and the last child cgroup of that cgroup
294 is removed, then the kernel runs the command s 294 is removed, then the kernel runs the command specified by the contents
295 of the "release_agent" file in that hierarchy' 295 of the "release_agent" file in that hierarchy's root directory,
296 supplying the pathname (relative to the mount 296 supplying the pathname (relative to the mount point of the cgroup
297 file system) of the abandoned cgroup. This en 297 file system) of the abandoned cgroup. This enables automatic
298 removal of abandoned cgroups. The default val 298 removal of abandoned cgroups. The default value of
299 notify_on_release in the root cgroup at system 299 notify_on_release in the root cgroup at system boot is disabled
300 (0). The default value of other cgroups at cr 300 (0). The default value of other cgroups at creation is the current
301 value of their parents' notify_on_release sett 301 value of their parents' notify_on_release settings. The default value of
302 a cgroup hierarchy's release_agent path is emp 302 a cgroup hierarchy's release_agent path is empty.
303 303
304 1.5 What does clone_children do ? 304 1.5 What does clone_children do ?
305 --------------------------------- 305 ---------------------------------
306 306
307 This flag only affects the cpuset controller. 307 This flag only affects the cpuset controller. If the clone_children
308 flag is enabled (1) in a cgroup, a new cpuset 308 flag is enabled (1) in a cgroup, a new cpuset cgroup will copy its
309 configuration from the parent during initializ 309 configuration from the parent during initialization.
310 310
311 1.6 How do I use cgroups ? 311 1.6 How do I use cgroups ?
312 -------------------------- 312 --------------------------
313 313
314 To start a new job that is to be contained wit 314 To start a new job that is to be contained within a cgroup, using
315 the "cpuset" cgroup subsystem, the steps are s 315 the "cpuset" cgroup subsystem, the steps are something like::
316 316
317 1) mount -t tmpfs cgroup_root /sys/fs/cgroup 317 1) mount -t tmpfs cgroup_root /sys/fs/cgroup
318 2) mkdir /sys/fs/cgroup/cpuset 318 2) mkdir /sys/fs/cgroup/cpuset
319 3) mount -t cgroup -ocpuset cpuset /sys/fs/cg 319 3) mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset
320 4) Create the new cgroup by doing mkdir's and 320 4) Create the new cgroup by doing mkdir's and write's (or echo's) in
321 the /sys/fs/cgroup/cpuset virtual file sys 321 the /sys/fs/cgroup/cpuset virtual file system.
322 5) Start a task that will be the "founding fa 322 5) Start a task that will be the "founding father" of the new job.
323 6) Attach that task to the new cgroup by writ 323 6) Attach that task to the new cgroup by writing its PID to the
324 /sys/fs/cgroup/cpuset tasks file for that 324 /sys/fs/cgroup/cpuset tasks file for that cgroup.
325 7) fork, exec or clone the job tasks from thi 325 7) fork, exec or clone the job tasks from this founding father task.
326 326
327 For example, the following sequence of command 327 For example, the following sequence of commands will setup a cgroup
328 named "Charlie", containing just CPUs 2 and 3, 328 named "Charlie", containing just CPUs 2 and 3, and Memory Node 1,
329 and then start a subshell 'sh' in that cgroup: 329 and then start a subshell 'sh' in that cgroup::
330 330
331 mount -t tmpfs cgroup_root /sys/fs/cgroup 331 mount -t tmpfs cgroup_root /sys/fs/cgroup
332 mkdir /sys/fs/cgroup/cpuset 332 mkdir /sys/fs/cgroup/cpuset
333 mount -t cgroup cpuset -ocpuset /sys/fs/cgro 333 mount -t cgroup cpuset -ocpuset /sys/fs/cgroup/cpuset
334 cd /sys/fs/cgroup/cpuset 334 cd /sys/fs/cgroup/cpuset
335 mkdir Charlie 335 mkdir Charlie
336 cd Charlie 336 cd Charlie
337 /bin/echo 2-3 > cpuset.cpus 337 /bin/echo 2-3 > cpuset.cpus
338 /bin/echo 1 > cpuset.mems 338 /bin/echo 1 > cpuset.mems
339 /bin/echo $$ > tasks 339 /bin/echo $$ > tasks
340 sh 340 sh
341 # The subshell 'sh' is now running in cgroup 341 # The subshell 'sh' is now running in cgroup Charlie
342 # The next line should display '/Charlie' 342 # The next line should display '/Charlie'
343 cat /proc/self/cgroup 343 cat /proc/self/cgroup
344 344
345 2. Usage Examples and Syntax 345 2. Usage Examples and Syntax
346 ============================ 346 ============================
347 347
348 2.1 Basic Usage 348 2.1 Basic Usage
349 --------------- 349 ---------------
350 350
351 Creating, modifying, using cgroups can be done 351 Creating, modifying, using cgroups can be done through the cgroup
352 virtual filesystem. 352 virtual filesystem.
353 353
354 To mount a cgroup hierarchy with all available 354 To mount a cgroup hierarchy with all available subsystems, type::
355 355
356 # mount -t cgroup xxx /sys/fs/cgroup 356 # mount -t cgroup xxx /sys/fs/cgroup
357 357
358 The "xxx" is not interpreted by the cgroup cod 358 The "xxx" is not interpreted by the cgroup code, but will appear in
359 /proc/mounts so may be any useful identifying 359 /proc/mounts so may be any useful identifying string that you like.
360 360
361 Note: Some subsystems do not work without some 361 Note: Some subsystems do not work without some user input first. For instance,
362 if cpusets are enabled the user will have to p 362 if cpusets are enabled the user will have to populate the cpus and mems files
363 for each new cgroup created before that group 363 for each new cgroup created before that group can be used.
364 364
365 As explained in section `1.2 Why are cgroups n 365 As explained in section `1.2 Why are cgroups needed?` you should create
366 different hierarchies of cgroups for each sing 366 different hierarchies of cgroups for each single resource or group of
367 resources you want to control. Therefore, you 367 resources you want to control. Therefore, you should mount a tmpfs on
368 /sys/fs/cgroup and create directories for each 368 /sys/fs/cgroup and create directories for each cgroup resource or resource
369 group:: 369 group::
370 370
371 # mount -t tmpfs cgroup_root /sys/fs/cgroup 371 # mount -t tmpfs cgroup_root /sys/fs/cgroup
372 # mkdir /sys/fs/cgroup/rg1 372 # mkdir /sys/fs/cgroup/rg1
373 373
374 To mount a cgroup hierarchy with just the cpus 374 To mount a cgroup hierarchy with just the cpuset and memory
375 subsystems, type:: 375 subsystems, type::
376 376
377 # mount -t cgroup -o cpuset,memory hier1 /sy 377 # mount -t cgroup -o cpuset,memory hier1 /sys/fs/cgroup/rg1
378 378
379 While remounting cgroups is currently supporte 379 While remounting cgroups is currently supported, it is not recommend
380 to use it. Remounting allows changing bound su 380 to use it. Remounting allows changing bound subsystems and
381 release_agent. Rebinding is hardly useful as i 381 release_agent. Rebinding is hardly useful as it only works when the
382 hierarchy is empty and release_agent itself sh 382 hierarchy is empty and release_agent itself should be replaced with
383 conventional fsnotify. The support for remount 383 conventional fsnotify. The support for remounting will be removed in
384 the future. 384 the future.
385 385
386 To Specify a hierarchy's release_agent:: 386 To Specify a hierarchy's release_agent::
387 387
388 # mount -t cgroup -o cpuset,release_agent="/ 388 # mount -t cgroup -o cpuset,release_agent="/sbin/cpuset_release_agent" \
389 xxx /sys/fs/cgroup/rg1 389 xxx /sys/fs/cgroup/rg1
390 390
391 Note that specifying 'release_agent' more than 391 Note that specifying 'release_agent' more than once will return failure.
392 392
393 Note that changing the set of subsystems is cu 393 Note that changing the set of subsystems is currently only supported
394 when the hierarchy consists of a single (root) 394 when the hierarchy consists of a single (root) cgroup. Supporting
395 the ability to arbitrarily bind/unbind subsyst 395 the ability to arbitrarily bind/unbind subsystems from an existing
396 cgroup hierarchy is intended to be implemented 396 cgroup hierarchy is intended to be implemented in the future.
397 397
398 Then under /sys/fs/cgroup/rg1 you can find a t 398 Then under /sys/fs/cgroup/rg1 you can find a tree that corresponds to the
399 tree of the cgroups in the system. For instanc 399 tree of the cgroups in the system. For instance, /sys/fs/cgroup/rg1
400 is the cgroup that holds the whole system. 400 is the cgroup that holds the whole system.
401 401
402 If you want to change the value of release_age 402 If you want to change the value of release_agent::
403 403
404 # echo "/sbin/new_release_agent" > /sys/fs/c 404 # echo "/sbin/new_release_agent" > /sys/fs/cgroup/rg1/release_agent
405 405
406 It can also be changed via remount. 406 It can also be changed via remount.
407 407
408 If you want to create a new cgroup under /sys/ 408 If you want to create a new cgroup under /sys/fs/cgroup/rg1::
409 409
410 # cd /sys/fs/cgroup/rg1 410 # cd /sys/fs/cgroup/rg1
411 # mkdir my_cgroup 411 # mkdir my_cgroup
412 412
413 Now you want to do something with this cgroup: 413 Now you want to do something with this cgroup:
414 414
415 # cd my_cgroup 415 # cd my_cgroup
416 416
417 In this directory you can find several files:: 417 In this directory you can find several files::
418 418
419 # ls 419 # ls
420 cgroup.procs notify_on_release tasks 420 cgroup.procs notify_on_release tasks
421 (plus whatever files added by the attached s 421 (plus whatever files added by the attached subsystems)
422 422
423 Now attach your shell to this cgroup:: 423 Now attach your shell to this cgroup::
424 424
425 # /bin/echo $$ > tasks 425 # /bin/echo $$ > tasks
426 426
427 You can also create cgroups inside your cgroup 427 You can also create cgroups inside your cgroup by using mkdir in this
428 directory:: 428 directory::
429 429
430 # mkdir my_sub_cs 430 # mkdir my_sub_cs
431 431
432 To remove a cgroup, just use rmdir:: 432 To remove a cgroup, just use rmdir::
433 433
434 # rmdir my_sub_cs 434 # rmdir my_sub_cs
435 435
436 This will fail if the cgroup is in use (has cg 436 This will fail if the cgroup is in use (has cgroups inside, or
437 has processes attached, or is held alive by ot 437 has processes attached, or is held alive by other subsystem-specific
438 reference). 438 reference).
439 439
440 2.2 Attaching processes 440 2.2 Attaching processes
441 ----------------------- 441 -----------------------
442 442
443 :: 443 ::
444 444
445 # /bin/echo PID > tasks 445 # /bin/echo PID > tasks
446 446
447 Note that it is PID, not PIDs. You can only at 447 Note that it is PID, not PIDs. You can only attach ONE task at a time.
448 If you have several tasks to attach, you have 448 If you have several tasks to attach, you have to do it one after another::
449 449
450 # /bin/echo PID1 > tasks 450 # /bin/echo PID1 > tasks
451 # /bin/echo PID2 > tasks 451 # /bin/echo PID2 > tasks
452 ... 452 ...
453 # /bin/echo PIDn > tasks 453 # /bin/echo PIDn > tasks
454 454
455 You can attach the current shell task by echoi 455 You can attach the current shell task by echoing 0::
456 456
457 # echo 0 > tasks 457 # echo 0 > tasks
458 458
459 You can use the cgroup.procs file instead of t 459 You can use the cgroup.procs file instead of the tasks file to move all
460 threads in a threadgroup at once. Echoing the 460 threads in a threadgroup at once. Echoing the PID of any task in a
461 threadgroup to cgroup.procs causes all tasks i 461 threadgroup to cgroup.procs causes all tasks in that threadgroup to be
462 attached to the cgroup. Writing 0 to cgroup.pr 462 attached to the cgroup. Writing 0 to cgroup.procs moves all tasks
463 in the writing task's threadgroup. 463 in the writing task's threadgroup.
464 464
465 Note: Since every task is always a member of e 465 Note: Since every task is always a member of exactly one cgroup in each
466 mounted hierarchy, to remove a task from its c 466 mounted hierarchy, to remove a task from its current cgroup you must
467 move it into a new cgroup (possibly the root c 467 move it into a new cgroup (possibly the root cgroup) by writing to the
468 new cgroup's tasks file. 468 new cgroup's tasks file.
469 469
470 Note: Due to some restrictions enforced by som 470 Note: Due to some restrictions enforced by some cgroup subsystems, moving
471 a process to another cgroup can fail. 471 a process to another cgroup can fail.
472 472
473 2.3 Mounting hierarchies by name 473 2.3 Mounting hierarchies by name
474 -------------------------------- 474 --------------------------------
475 475
476 Passing the name=<x> option when mounting a cg 476 Passing the name=<x> option when mounting a cgroups hierarchy
477 associates the given name with the hierarchy. 477 associates the given name with the hierarchy. This can be used when
478 mounting a pre-existing hierarchy, in order to 478 mounting a pre-existing hierarchy, in order to refer to it by name
479 rather than by its set of active subsystems. 479 rather than by its set of active subsystems. Each hierarchy is either
480 nameless, or has a unique name. 480 nameless, or has a unique name.
481 481
482 The name should match [\w.-]+ 482 The name should match [\w.-]+
483 483
484 When passing a name=<x> option for a new hiera 484 When passing a name=<x> option for a new hierarchy, you need to
485 specify subsystems manually; the legacy behavi 485 specify subsystems manually; the legacy behaviour of mounting all
486 subsystems when none are explicitly specified 486 subsystems when none are explicitly specified is not supported when
487 you give a subsystem a name. 487 you give a subsystem a name.
488 488
489 The name of the subsystem appears as part of t 489 The name of the subsystem appears as part of the hierarchy description
490 in /proc/mounts and /proc/<pid>/cgroups. 490 in /proc/mounts and /proc/<pid>/cgroups.
491 491
492 492
493 3. Kernel API 493 3. Kernel API
494 ============= 494 =============
495 495
496 3.1 Overview 496 3.1 Overview
497 ------------ 497 ------------
498 498
499 Each kernel subsystem that wants to hook into 499 Each kernel subsystem that wants to hook into the generic cgroup
500 system needs to create a cgroup_subsys object. 500 system needs to create a cgroup_subsys object. This contains
501 various methods, which are callbacks from the 501 various methods, which are callbacks from the cgroup system, along
502 with a subsystem ID which will be assigned by 502 with a subsystem ID which will be assigned by the cgroup system.
503 503
504 Other fields in the cgroup_subsys object inclu 504 Other fields in the cgroup_subsys object include:
505 505
506 - subsys_id: a unique array index for the subs 506 - subsys_id: a unique array index for the subsystem, indicating which
507 entry in cgroup->subsys[] this subsystem sho 507 entry in cgroup->subsys[] this subsystem should be managing.
508 508
509 - name: should be initialized to a unique subs 509 - name: should be initialized to a unique subsystem name. Should be
510 no longer than MAX_CGROUP_TYPE_NAMELEN. 510 no longer than MAX_CGROUP_TYPE_NAMELEN.
511 511
512 - early_init: indicate if the subsystem needs 512 - early_init: indicate if the subsystem needs early initialization
513 at system boot. 513 at system boot.
514 514
515 Each cgroup object created by the system has a 515 Each cgroup object created by the system has an array of pointers,
516 indexed by subsystem ID; this pointer is entir 516 indexed by subsystem ID; this pointer is entirely managed by the
517 subsystem; the generic cgroup code will never 517 subsystem; the generic cgroup code will never touch this pointer.
518 518
519 3.2 Synchronization 519 3.2 Synchronization
520 ------------------- 520 -------------------
521 521
522 There is a global mutex, cgroup_mutex, used by 522 There is a global mutex, cgroup_mutex, used by the cgroup
523 system. This should be taken by anything that 523 system. This should be taken by anything that wants to modify a
524 cgroup. It may also be taken to prevent cgroup 524 cgroup. It may also be taken to prevent cgroups from being
525 modified, but more specific locks may be more 525 modified, but more specific locks may be more appropriate in that
526 situation. 526 situation.
527 527
528 See kernel/cgroup.c for more details. 528 See kernel/cgroup.c for more details.
529 529
530 Subsystems can take/release the cgroup_mutex v 530 Subsystems can take/release the cgroup_mutex via the functions
531 cgroup_lock()/cgroup_unlock(). 531 cgroup_lock()/cgroup_unlock().
532 532
533 Accessing a task's cgroup pointer may be done 533 Accessing a task's cgroup pointer may be done in the following ways:
534 - while holding cgroup_mutex 534 - while holding cgroup_mutex
535 - while holding the task's alloc_lock (via tas 535 - while holding the task's alloc_lock (via task_lock())
536 - inside an rcu_read_lock() section via rcu_de 536 - inside an rcu_read_lock() section via rcu_dereference()
537 537
538 3.3 Subsystem API 538 3.3 Subsystem API
539 ----------------- 539 -----------------
540 540
541 Each subsystem should: 541 Each subsystem should:
542 542
543 - add an entry in linux/cgroup_subsys.h 543 - add an entry in linux/cgroup_subsys.h
544 - define a cgroup_subsys object called <name>_ 544 - define a cgroup_subsys object called <name>_cgrp_subsys
545 545
546 Each subsystem may export the following method 546 Each subsystem may export the following methods. The only mandatory
547 methods are css_alloc/free. Any others that ar 547 methods are css_alloc/free. Any others that are null are presumed to
548 be successful no-ops. 548 be successful no-ops.
549 549
550 ``struct cgroup_subsys_state *css_alloc(struct 550 ``struct cgroup_subsys_state *css_alloc(struct cgroup *cgrp)``
551 (cgroup_mutex held by caller) 551 (cgroup_mutex held by caller)
552 552
553 Called to allocate a subsystem state object fo 553 Called to allocate a subsystem state object for a cgroup. The
554 subsystem should allocate its subsystem state 554 subsystem should allocate its subsystem state object for the passed
555 cgroup, returning a pointer to the new object 555 cgroup, returning a pointer to the new object on success or a
556 ERR_PTR() value. On success, the subsystem poi 556 ERR_PTR() value. On success, the subsystem pointer should point to
557 a structure of type cgroup_subsys_state (typic 557 a structure of type cgroup_subsys_state (typically embedded in a
558 larger subsystem-specific object), which will 558 larger subsystem-specific object), which will be initialized by the
559 cgroup system. Note that this will be called a 559 cgroup system. Note that this will be called at initialization to
560 create the root subsystem state for this subsy 560 create the root subsystem state for this subsystem; this case can be
561 identified by the passed cgroup object having 561 identified by the passed cgroup object having a NULL parent (since
562 it's the root of the hierarchy) and may be an 562 it's the root of the hierarchy) and may be an appropriate place for
563 initialization code. 563 initialization code.
564 564
565 ``int css_online(struct cgroup *cgrp)`` 565 ``int css_online(struct cgroup *cgrp)``
566 (cgroup_mutex held by caller) 566 (cgroup_mutex held by caller)
567 567
568 Called after @cgrp successfully completed all 568 Called after @cgrp successfully completed all allocations and made
569 visible to cgroup_for_each_child/descendant_*( 569 visible to cgroup_for_each_child/descendant_*() iterators. The
570 subsystem may choose to fail creation by retur 570 subsystem may choose to fail creation by returning -errno. This
571 callback can be used to implement reliable sta 571 callback can be used to implement reliable state sharing and
572 propagation along the hierarchy. See the comme 572 propagation along the hierarchy. See the comment on
573 cgroup_for_each_live_descendant_pre() for deta !! 573 cgroup_for_each_descendant_pre() for details.
574 574
575 ``void css_offline(struct cgroup *cgrp);`` 575 ``void css_offline(struct cgroup *cgrp);``
576 (cgroup_mutex held by caller) 576 (cgroup_mutex held by caller)
577 577
578 This is the counterpart of css_online() and ca 578 This is the counterpart of css_online() and called iff css_online()
579 has succeeded on @cgrp. This signifies the beg 579 has succeeded on @cgrp. This signifies the beginning of the end of
580 @cgrp. @cgrp is being removed and the subsyste 580 @cgrp. @cgrp is being removed and the subsystem should start dropping
581 all references it's holding on @cgrp. When all 581 all references it's holding on @cgrp. When all references are dropped,
582 cgroup removal will proceed to the next step - 582 cgroup removal will proceed to the next step - css_free(). After this
583 callback, @cgrp should be considered dead to t 583 callback, @cgrp should be considered dead to the subsystem.
584 584
585 ``void css_free(struct cgroup *cgrp)`` 585 ``void css_free(struct cgroup *cgrp)``
586 (cgroup_mutex held by caller) 586 (cgroup_mutex held by caller)
587 587
588 The cgroup system is about to free @cgrp; the 588 The cgroup system is about to free @cgrp; the subsystem should free
589 its subsystem state object. By the time this m 589 its subsystem state object. By the time this method is called, @cgrp
590 is completely unused; @cgrp->parent is still v 590 is completely unused; @cgrp->parent is still valid. (Note - can also
591 be called for a newly-created cgroup if an err 591 be called for a newly-created cgroup if an error occurs after this
592 subsystem's create() method has been called fo 592 subsystem's create() method has been called for the new cgroup).
593 593
594 ``int can_attach(struct cgroup *cgrp, struct c 594 ``int can_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
595 (cgroup_mutex held by caller) 595 (cgroup_mutex held by caller)
596 596
597 Called prior to moving one or more tasks into 597 Called prior to moving one or more tasks into a cgroup; if the
598 subsystem returns an error, this will abort th 598 subsystem returns an error, this will abort the attach operation.
599 @tset contains the tasks to be attached and is 599 @tset contains the tasks to be attached and is guaranteed to have at
600 least one task in it. 600 least one task in it.
601 601
602 If there are multiple tasks in the taskset, th 602 If there are multiple tasks in the taskset, then:
603 - it's guaranteed that all are from the same 603 - it's guaranteed that all are from the same thread group
604 - @tset contains all tasks from the thread g 604 - @tset contains all tasks from the thread group whether or not
605 they're switching cgroups 605 they're switching cgroups
606 - the first task is the leader 606 - the first task is the leader
607 607
608 Each @tset entry also contains the task's old 608 Each @tset entry also contains the task's old cgroup and tasks which
609 aren't switching cgroup can be skipped easily 609 aren't switching cgroup can be skipped easily using the
610 cgroup_taskset_for_each() iterator. Note that 610 cgroup_taskset_for_each() iterator. Note that this isn't called on a
611 fork. If this method returns 0 (success) then 611 fork. If this method returns 0 (success) then this should remain valid
612 while the caller holds cgroup_mutex and it is 612 while the caller holds cgroup_mutex and it is ensured that either
613 attach() or cancel_attach() will be called in 613 attach() or cancel_attach() will be called in future.
614 614
615 ``void css_reset(struct cgroup_subsys_state *c 615 ``void css_reset(struct cgroup_subsys_state *css)``
616 (cgroup_mutex held by caller) 616 (cgroup_mutex held by caller)
617 617
618 An optional operation which should restore @cs 618 An optional operation which should restore @css's configuration to the
619 initial state. This is currently only used on 619 initial state. This is currently only used on the unified hierarchy
620 when a subsystem is disabled on a cgroup throu 620 when a subsystem is disabled on a cgroup through
621 "cgroup.subtree_control" but should remain ena 621 "cgroup.subtree_control" but should remain enabled because other
622 subsystems depend on it. cgroup core makes su 622 subsystems depend on it. cgroup core makes such a css invisible by
623 removing the associated interface files and in 623 removing the associated interface files and invokes this callback so
624 that the hidden subsystem can return to the in 624 that the hidden subsystem can return to the initial neutral state.
625 This prevents unexpected resource control from 625 This prevents unexpected resource control from a hidden css and
626 ensures that the configuration is in the initi 626 ensures that the configuration is in the initial state when it is made
627 visible again later. 627 visible again later.
628 628
629 ``void cancel_attach(struct cgroup *cgrp, stru 629 ``void cancel_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
630 (cgroup_mutex held by caller) 630 (cgroup_mutex held by caller)
631 631
632 Called when a task attach operation has failed 632 Called when a task attach operation has failed after can_attach() has succeeded.
633 A subsystem whose can_attach() has some side-e 633 A subsystem whose can_attach() has some side-effects should provide this
634 function, so that the subsystem can implement 634 function, so that the subsystem can implement a rollback. If not, not necessary.
635 This will be called only about subsystems whos 635 This will be called only about subsystems whose can_attach() operation have
636 succeeded. The parameters are identical to can 636 succeeded. The parameters are identical to can_attach().
637 637
638 ``void attach(struct cgroup *cgrp, struct cgro 638 ``void attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
639 (cgroup_mutex held by caller) 639 (cgroup_mutex held by caller)
640 640
641 Called after the task has been attached to the 641 Called after the task has been attached to the cgroup, to allow any
642 post-attachment activity that requires memory 642 post-attachment activity that requires memory allocations or blocking.
643 The parameters are identical to can_attach(). 643 The parameters are identical to can_attach().
644 644
645 ``void fork(struct task_struct *task)`` 645 ``void fork(struct task_struct *task)``
646 646
647 Called when a task is forked into a cgroup. 647 Called when a task is forked into a cgroup.
648 648
649 ``void exit(struct task_struct *task)`` 649 ``void exit(struct task_struct *task)``
650 650
651 Called during task exit. 651 Called during task exit.
652 652
653 ``void free(struct task_struct *task)`` 653 ``void free(struct task_struct *task)``
654 654
655 Called when the task_struct is freed. 655 Called when the task_struct is freed.
656 656
657 ``void bind(struct cgroup *root)`` 657 ``void bind(struct cgroup *root)``
658 (cgroup_mutex held by caller) 658 (cgroup_mutex held by caller)
659 659
660 Called when a cgroup subsystem is rebound to a 660 Called when a cgroup subsystem is rebound to a different hierarchy
661 and root cgroup. Currently this will only invo 661 and root cgroup. Currently this will only involve movement between
662 the default hierarchy (which never has sub-cgr 662 the default hierarchy (which never has sub-cgroups) and a hierarchy
663 that is being created/destroyed (and hence has 663 that is being created/destroyed (and hence has no sub-cgroups).
664 664
665 4. Extended attribute usage 665 4. Extended attribute usage
666 =========================== 666 ===========================
667 667
668 cgroup filesystem supports certain types of ex 668 cgroup filesystem supports certain types of extended attributes in its
669 directories and files. The current supported 669 directories and files. The current supported types are:
670 670
671 - Trusted (XATTR_TRUSTED) 671 - Trusted (XATTR_TRUSTED)
672 - Security (XATTR_SECURITY) 672 - Security (XATTR_SECURITY)
673 673
674 Both require CAP_SYS_ADMIN capability to set. 674 Both require CAP_SYS_ADMIN capability to set.
675 675
676 Like in tmpfs, the extended attributes in cgro 676 Like in tmpfs, the extended attributes in cgroup filesystem are stored
677 using kernel memory and it's advised to keep t 677 using kernel memory and it's advised to keep the usage at minimum. This
678 is the reason why user defined extended attrib 678 is the reason why user defined extended attributes are not supported, since
679 any user can do it and there's no limit in the 679 any user can do it and there's no limit in the value size.
680 680
681 The current known users for this feature are S 681 The current known users for this feature are SELinux to limit cgroup usage
682 in containers and systemd for assorted meta da 682 in containers and systemd for assorted meta data like main PID in a cgroup
683 (systemd creates a cgroup per service). 683 (systemd creates a cgroup per service).
684 684
685 5. Questions 685 5. Questions
686 ============ 686 ============
687 687
688 :: 688 ::
689 689
690 Q: what's up with this '/bin/echo' ? 690 Q: what's up with this '/bin/echo' ?
691 A: bash's builtin 'echo' command does not ch 691 A: bash's builtin 'echo' command does not check calls to write() against
692 errors. If you use it in the cgroup file 692 errors. If you use it in the cgroup file system, you won't be
693 able to tell whether a command succeeded 693 able to tell whether a command succeeded or failed.
694 694
695 Q: When I attach processes, only the first o 695 Q: When I attach processes, only the first of the line gets really attached !
696 A: We can only return one error code per cal 696 A: We can only return one error code per call to write(). So you should also
697 put only ONE PID. 697 put only ONE PID.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.