1 ============== 1 ==============
2 Control Groups 2 Control Groups
3 ============== 3 ==============
4 4
5 Written by Paul Menage <menage@google.com> base 5 Written by Paul Menage <menage@google.com> based on
6 Documentation/admin-guide/cgroup-v1/cpusets.rs 6 Documentation/admin-guide/cgroup-v1/cpusets.rst
7 7
8 Original copyright statements from cpusets.txt 8 Original copyright statements from cpusets.txt:
9 9
10 Portions Copyright (C) 2004 BULL SA. 10 Portions Copyright (C) 2004 BULL SA.
11 11
12 Portions Copyright (c) 2004-2006 Silicon Graph 12 Portions Copyright (c) 2004-2006 Silicon Graphics, Inc.
13 13
14 Modified by Paul Jackson <pj@sgi.com> 14 Modified by Paul Jackson <pj@sgi.com>
15 15
16 Modified by Christoph Lameter <cl@linux.com> 16 Modified by Christoph Lameter <cl@linux.com>
17 17
18 .. CONTENTS: 18 .. CONTENTS:
19 19
20 1. Control Groups 20 1. Control Groups
21 1.1 What are cgroups ? 21 1.1 What are cgroups ?
22 1.2 Why are cgroups needed ? 22 1.2 Why are cgroups needed ?
23 1.3 How are cgroups implemented ? 23 1.3 How are cgroups implemented ?
24 1.4 What does notify_on_release do ? 24 1.4 What does notify_on_release do ?
25 1.5 What does clone_children do ? 25 1.5 What does clone_children do ?
26 1.6 How do I use cgroups ? 26 1.6 How do I use cgroups ?
27 2. Usage Examples and Syntax 27 2. Usage Examples and Syntax
28 2.1 Basic Usage 28 2.1 Basic Usage
29 2.2 Attaching processes 29 2.2 Attaching processes
30 2.3 Mounting hierarchies by name 30 2.3 Mounting hierarchies by name
31 3. Kernel API 31 3. Kernel API
32 3.1 Overview 32 3.1 Overview
33 3.2 Synchronization 33 3.2 Synchronization
34 3.3 Subsystem API 34 3.3 Subsystem API
35 4. Extended attributes usage 35 4. Extended attributes usage
36 5. Questions 36 5. Questions
37 37
38 1. Control Groups 38 1. Control Groups
39 ================= 39 =================
40 40
41 1.1 What are cgroups ? 41 1.1 What are cgroups ?
42 ---------------------- 42 ----------------------
43 43
44 Control Groups provide a mechanism for aggrega 44 Control Groups provide a mechanism for aggregating/partitioning sets of
45 tasks, and all their future children, into hie 45 tasks, and all their future children, into hierarchical groups with
46 specialized behaviour. 46 specialized behaviour.
47 47
48 Definitions: 48 Definitions:
49 49
50 A *cgroup* associates a set of tasks with a se 50 A *cgroup* associates a set of tasks with a set of parameters for one
51 or more subsystems. 51 or more subsystems.
52 52
53 A *subsystem* is a module that makes use of th 53 A *subsystem* is a module that makes use of the task grouping
54 facilities provided by cgroups to treat groups 54 facilities provided by cgroups to treat groups of tasks in
55 particular ways. A subsystem is typically a "r 55 particular ways. A subsystem is typically a "resource controller" that
56 schedules a resource or applies per-cgroup lim 56 schedules a resource or applies per-cgroup limits, but it may be
57 anything that wants to act on a group of proce 57 anything that wants to act on a group of processes, e.g. a
58 virtualization subsystem. 58 virtualization subsystem.
59 59
60 A *hierarchy* is a set of cgroups arranged in 60 A *hierarchy* is a set of cgroups arranged in a tree, such that
61 every task in the system is in exactly one of 61 every task in the system is in exactly one of the cgroups in the
62 hierarchy, and a set of subsystems; each subsy 62 hierarchy, and a set of subsystems; each subsystem has system-specific
63 state attached to each cgroup in the hierarchy 63 state attached to each cgroup in the hierarchy. Each hierarchy has
64 an instance of the cgroup virtual filesystem a 64 an instance of the cgroup virtual filesystem associated with it.
65 65
66 At any one time there may be multiple active h 66 At any one time there may be multiple active hierarchies of task
67 cgroups. Each hierarchy is a partition of all 67 cgroups. Each hierarchy is a partition of all tasks in the system.
68 68
69 User-level code may create and destroy cgroups 69 User-level code may create and destroy cgroups by name in an
70 instance of the cgroup virtual file system, sp 70 instance of the cgroup virtual file system, specify and query to
71 which cgroup a task is assigned, and list the 71 which cgroup a task is assigned, and list the task PIDs assigned to
72 a cgroup. Those creations and assignments only 72 a cgroup. Those creations and assignments only affect the hierarchy
73 associated with that instance of the cgroup fi 73 associated with that instance of the cgroup file system.
74 74
75 On their own, the only use for cgroups is for 75 On their own, the only use for cgroups is for simple job
76 tracking. The intention is that other subsyste 76 tracking. The intention is that other subsystems hook into the generic
77 cgroup support to provide new attributes for c 77 cgroup support to provide new attributes for cgroups, such as
78 accounting/limiting the resources which proces 78 accounting/limiting the resources which processes in a cgroup can
79 access. For example, cpusets (see Documentatio 79 access. For example, cpusets (see Documentation/admin-guide/cgroup-v1/cpusets.rst) allow
80 you to associate a set of CPUs and a set of me 80 you to associate a set of CPUs and a set of memory nodes with the
81 tasks in each cgroup. 81 tasks in each cgroup.
82 82
83 .. _cgroups-why-needed: <<
84 <<
85 1.2 Why are cgroups needed ? 83 1.2 Why are cgroups needed ?
86 ---------------------------- 84 ----------------------------
87 85
88 There are multiple efforts to provide process 86 There are multiple efforts to provide process aggregations in the
89 Linux kernel, mainly for resource-tracking pur 87 Linux kernel, mainly for resource-tracking purposes. Such efforts
90 include cpusets, CKRM/ResGroups, UserBeanCount 88 include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server
91 namespaces. These all require the basic notion 89 namespaces. These all require the basic notion of a
92 grouping/partitioning of processes, with newly 90 grouping/partitioning of processes, with newly forked processes ending
93 up in the same group (cgroup) as their parent 91 up in the same group (cgroup) as their parent process.
94 92
95 The kernel cgroup patch provides the minimum e 93 The kernel cgroup patch provides the minimum essential kernel
96 mechanisms required to efficiently implement s 94 mechanisms required to efficiently implement such groups. It has
97 minimal impact on the system fast paths, and p 95 minimal impact on the system fast paths, and provides hooks for
98 specific subsystems such as cpusets to provide 96 specific subsystems such as cpusets to provide additional behaviour as
99 desired. 97 desired.
100 98
101 Multiple hierarchy support is provided to allo 99 Multiple hierarchy support is provided to allow for situations where
102 the division of tasks into cgroups is distinct 100 the division of tasks into cgroups is distinctly different for
103 different subsystems - having parallel hierarc 101 different subsystems - having parallel hierarchies allows each
104 hierarchy to be a natural division of tasks, w 102 hierarchy to be a natural division of tasks, without having to handle
105 complex combinations of tasks that would be pr 103 complex combinations of tasks that would be present if several
106 unrelated subsystems needed to be forced into 104 unrelated subsystems needed to be forced into the same tree of
107 cgroups. 105 cgroups.
108 106
109 At one extreme, each resource controller or su 107 At one extreme, each resource controller or subsystem could be in a
110 separate hierarchy; at the other extreme, all 108 separate hierarchy; at the other extreme, all subsystems
111 would be attached to the same hierarchy. 109 would be attached to the same hierarchy.
112 110
113 As an example of a scenario (originally propos 111 As an example of a scenario (originally proposed by vatsa@in.ibm.com)
114 that can benefit from multiple hierarchies, co 112 that can benefit from multiple hierarchies, consider a large
115 university server with various users - student 113 university server with various users - students, professors, system
116 tasks etc. The resource planning for this serv 114 tasks etc. The resource planning for this server could be along the
117 following lines:: 115 following lines::
118 116
119 CPU : "Top cpuset" 117 CPU : "Top cpuset"
120 / \ 118 / \
121 CPUSet1 CPUSet2 119 CPUSet1 CPUSet2
122 | | 120 | |
123 (Professors) (Students) 121 (Professors) (Students)
124 122
125 In addition (system tasks) are 123 In addition (system tasks) are attached to topcpuset (so
126 that they can run anywhere) wit 124 that they can run anywhere) with a limit of 20%
127 125
128 Memory : Professors (50%), Students (30 126 Memory : Professors (50%), Students (30%), system (20%)
129 127
130 Disk : Professors (50%), Students (30%) 128 Disk : Professors (50%), Students (30%), system (20%)
131 129
132 Network : WWW browsing (20%), Network F 130 Network : WWW browsing (20%), Network File System (60%), others (20%)
133 / \ 131 / \
134 Professors (15%) students (5%) 132 Professors (15%) students (5%)
135 133
136 Browsers like Firefox/Lynx go into the WWW net 134 Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd goes
137 into the NFS network class. 135 into the NFS network class.
138 136
139 At the same time Firefox/Lynx will share an ap 137 At the same time Firefox/Lynx will share an appropriate CPU/Memory class
140 depending on who launched it (prof/student). 138 depending on who launched it (prof/student).
141 139
142 With the ability to classify tasks differently 140 With the ability to classify tasks differently for different resources
143 (by putting those resource subsystems in diffe 141 (by putting those resource subsystems in different hierarchies),
144 the admin can easily set up a script which rec 142 the admin can easily set up a script which receives exec notifications
145 and depending on who is launching the browser 143 and depending on who is launching the browser he can::
146 144
147 # echo browser_pid > /sys/fs/cgroup/<resty 145 # echo browser_pid > /sys/fs/cgroup/<restype>/<userclass>/tasks
148 146
149 With only a single hierarchy, he now would pot 147 With only a single hierarchy, he now would potentially have to create
150 a separate cgroup for every browser launched a 148 a separate cgroup for every browser launched and associate it with
151 appropriate network and other resource class. 149 appropriate network and other resource class. This may lead to
152 proliferation of such cgroups. 150 proliferation of such cgroups.
153 151
154 Also let's say that the administrator would li 152 Also let's say that the administrator would like to give enhanced network
155 access temporarily to a student's browser (sin 153 access temporarily to a student's browser (since it is night and the user
156 wants to do online gaming :)) OR give one of 154 wants to do online gaming :)) OR give one of the student's simulation
157 apps enhanced CPU power. 155 apps enhanced CPU power.
158 156
159 With ability to write PIDs directly to resourc 157 With ability to write PIDs directly to resource classes, it's just a
160 matter of:: 158 matter of::
161 159
162 # echo pid > /sys/fs/cgroup/network/<ne 160 # echo pid > /sys/fs/cgroup/network/<new_class>/tasks
163 (after some time) 161 (after some time)
164 # echo pid > /sys/fs/cgroup/network/<or 162 # echo pid > /sys/fs/cgroup/network/<orig_class>/tasks
165 163
166 Without this ability, the administrator would 164 Without this ability, the administrator would have to split the cgroup into
167 multiple separate ones and then associate the 165 multiple separate ones and then associate the new cgroups with the
168 new resource classes. 166 new resource classes.
169 167
170 168
171 169
172 1.3 How are cgroups implemented ? 170 1.3 How are cgroups implemented ?
173 --------------------------------- 171 ---------------------------------
174 172
175 Control Groups extends the kernel as follows: 173 Control Groups extends the kernel as follows:
176 174
177 - Each task in the system has a reference-cou 175 - Each task in the system has a reference-counted pointer to a
178 css_set. 176 css_set.
179 177
180 - A css_set contains a set of reference-count 178 - A css_set contains a set of reference-counted pointers to
181 cgroup_subsys_state objects, one for each c 179 cgroup_subsys_state objects, one for each cgroup subsystem
182 registered in the system. There is no direc 180 registered in the system. There is no direct link from a task to
183 the cgroup of which it's a member in each h 181 the cgroup of which it's a member in each hierarchy, but this
184 can be determined by following pointers thr 182 can be determined by following pointers through the
185 cgroup_subsys_state objects. This is becaus 183 cgroup_subsys_state objects. This is because accessing the
186 subsystem state is something that's expecte 184 subsystem state is something that's expected to happen frequently
187 and in performance-critical code, whereas o 185 and in performance-critical code, whereas operations that require a
188 task's actual cgroup assignments (in partic 186 task's actual cgroup assignments (in particular, moving between
189 cgroups) are less common. A linked list run 187 cgroups) are less common. A linked list runs through the cg_list
190 field of each task_struct using the css_set 188 field of each task_struct using the css_set, anchored at
191 css_set->tasks. 189 css_set->tasks.
192 190
193 - A cgroup hierarchy filesystem can be mounte 191 - A cgroup hierarchy filesystem can be mounted for browsing and
194 manipulation from user space. 192 manipulation from user space.
195 193
196 - You can list all the tasks (by PID) attache 194 - You can list all the tasks (by PID) attached to any cgroup.
197 195
198 The implementation of cgroups requires a few, 196 The implementation of cgroups requires a few, simple hooks
199 into the rest of the kernel, none in performan 197 into the rest of the kernel, none in performance-critical paths:
200 198
201 - in init/main.c, to initialize the root cgro 199 - in init/main.c, to initialize the root cgroups and initial
202 css_set at system boot. 200 css_set at system boot.
203 201
204 - in fork and exit, to attach and detach a ta 202 - in fork and exit, to attach and detach a task from its css_set.
205 203
206 In addition, a new file system of type "cgroup 204 In addition, a new file system of type "cgroup" may be mounted, to
207 enable browsing and modifying the cgroups pres 205 enable browsing and modifying the cgroups presently known to the
208 kernel. When mounting a cgroup hierarchy, you 206 kernel. When mounting a cgroup hierarchy, you may specify a
209 comma-separated list of subsystems to mount as 207 comma-separated list of subsystems to mount as the filesystem mount
210 options. By default, mounting the cgroup file 208 options. By default, mounting the cgroup filesystem attempts to
211 mount a hierarchy containing all registered su 209 mount a hierarchy containing all registered subsystems.
212 210
213 If an active hierarchy with exactly the same s 211 If an active hierarchy with exactly the same set of subsystems already
214 exists, it will be reused for the new mount. I 212 exists, it will be reused for the new mount. If no existing hierarchy
215 matches, and any of the requested subsystems a 213 matches, and any of the requested subsystems are in use in an existing
216 hierarchy, the mount will fail with -EBUSY. Ot 214 hierarchy, the mount will fail with -EBUSY. Otherwise, a new hierarchy
217 is activated, associated with the requested su 215 is activated, associated with the requested subsystems.
218 216
219 It's not currently possible to bind a new subs 217 It's not currently possible to bind a new subsystem to an active
220 cgroup hierarchy, or to unbind a subsystem fro 218 cgroup hierarchy, or to unbind a subsystem from an active cgroup
221 hierarchy. This may be possible in future, but 219 hierarchy. This may be possible in future, but is fraught with nasty
222 error-recovery issues. 220 error-recovery issues.
223 221
224 When a cgroup filesystem is unmounted, if ther 222 When a cgroup filesystem is unmounted, if there are any
225 child cgroups created below the top-level cgro 223 child cgroups created below the top-level cgroup, that hierarchy
226 will remain active even though unmounted; if t 224 will remain active even though unmounted; if there are no
227 child cgroups then the hierarchy will be deact 225 child cgroups then the hierarchy will be deactivated.
228 226
229 No new system calls are added for cgroups - al 227 No new system calls are added for cgroups - all support for
230 querying and modifying cgroups is via this cgr 228 querying and modifying cgroups is via this cgroup file system.
231 229
232 Each task under /proc has an added file named 230 Each task under /proc has an added file named 'cgroup' displaying,
233 for each active hierarchy, the subsystem names 231 for each active hierarchy, the subsystem names and the cgroup name
234 as the path relative to the root of the cgroup 232 as the path relative to the root of the cgroup file system.
235 233
236 Each cgroup is represented by a directory in t 234 Each cgroup is represented by a directory in the cgroup file system
237 containing the following files describing that 235 containing the following files describing that cgroup:
238 236
239 - tasks: list of tasks (by PID) attached to t 237 - tasks: list of tasks (by PID) attached to that cgroup. This list
240 is not guaranteed to be sorted. Writing a 238 is not guaranteed to be sorted. Writing a thread ID into this file
241 moves the thread into this cgroup. 239 moves the thread into this cgroup.
242 - cgroup.procs: list of thread group IDs in t 240 - cgroup.procs: list of thread group IDs in the cgroup. This list is
243 not guaranteed to be sorted or free of dupl 241 not guaranteed to be sorted or free of duplicate TGIDs, and userspace
244 should sort/uniquify the list if this prope 242 should sort/uniquify the list if this property is required.
245 Writing a thread group ID into this file mo 243 Writing a thread group ID into this file moves all threads in that
246 group into this cgroup. 244 group into this cgroup.
247 - notify_on_release flag: run the release age 245 - notify_on_release flag: run the release agent on exit?
248 - release_agent: the path to use for release 246 - release_agent: the path to use for release notifications (this file
249 exists in the top cgroup only) 247 exists in the top cgroup only)
250 248
251 Other subsystems such as cpusets may add addit 249 Other subsystems such as cpusets may add additional files in each
252 cgroup dir. 250 cgroup dir.
253 251
254 New cgroups are created using the mkdir system 252 New cgroups are created using the mkdir system call or shell
255 command. The properties of a cgroup, such as 253 command. The properties of a cgroup, such as its flags, are
256 modified by writing to the appropriate file in 254 modified by writing to the appropriate file in that cgroups
257 directory, as listed above. 255 directory, as listed above.
258 256
259 The named hierarchical structure of nested cgr 257 The named hierarchical structure of nested cgroups allows partitioning
260 a large system into nested, dynamically change 258 a large system into nested, dynamically changeable, "soft-partitions".
261 259
262 The attachment of each task, automatically inh 260 The attachment of each task, automatically inherited at fork by any
263 children of that task, to a cgroup allows orga 261 children of that task, to a cgroup allows organizing the work load
264 on a system into related sets of tasks. A tas 262 on a system into related sets of tasks. A task may be re-attached to
265 any other cgroup, if allowed by the permission 263 any other cgroup, if allowed by the permissions on the necessary
266 cgroup file system directories. 264 cgroup file system directories.
267 265
268 When a task is moved from one cgroup to anothe 266 When a task is moved from one cgroup to another, it gets a new
269 css_set pointer - if there's an already existi 267 css_set pointer - if there's an already existing css_set with the
270 desired collection of cgroups then that group 268 desired collection of cgroups then that group is reused, otherwise a new
271 css_set is allocated. The appropriate existing 269 css_set is allocated. The appropriate existing css_set is located by
272 looking into a hash table. 270 looking into a hash table.
273 271
274 To allow access from a cgroup to the css_sets 272 To allow access from a cgroup to the css_sets (and hence tasks)
275 that comprise it, a set of cg_cgroup_link obje 273 that comprise it, a set of cg_cgroup_link objects form a lattice;
276 each cg_cgroup_link is linked into a list of c 274 each cg_cgroup_link is linked into a list of cg_cgroup_links for
277 a single cgroup on its cgrp_link_list field, a 275 a single cgroup on its cgrp_link_list field, and a list of
278 cg_cgroup_links for a single css_set on its cg 276 cg_cgroup_links for a single css_set on its cg_link_list.
279 277
280 Thus the set of tasks in a cgroup can be liste 278 Thus the set of tasks in a cgroup can be listed by iterating over
281 each css_set that references the cgroup, and s 279 each css_set that references the cgroup, and sub-iterating over
282 each css_set's task set. 280 each css_set's task set.
283 281
284 The use of a Linux virtual file system (vfs) t 282 The use of a Linux virtual file system (vfs) to represent the
285 cgroup hierarchy provides for a familiar permi 283 cgroup hierarchy provides for a familiar permission and name space
286 for cgroups, with a minimum of additional kern 284 for cgroups, with a minimum of additional kernel code.
287 285
288 1.4 What does notify_on_release do ? 286 1.4 What does notify_on_release do ?
289 ------------------------------------ 287 ------------------------------------
290 288
291 If the notify_on_release flag is enabled (1) i 289 If the notify_on_release flag is enabled (1) in a cgroup, then
292 whenever the last task in the cgroup leaves (e 290 whenever the last task in the cgroup leaves (exits or attaches to
293 some other cgroup) and the last child cgroup o 291 some other cgroup) and the last child cgroup of that cgroup
294 is removed, then the kernel runs the command s 292 is removed, then the kernel runs the command specified by the contents
295 of the "release_agent" file in that hierarchy' 293 of the "release_agent" file in that hierarchy's root directory,
296 supplying the pathname (relative to the mount 294 supplying the pathname (relative to the mount point of the cgroup
297 file system) of the abandoned cgroup. This en 295 file system) of the abandoned cgroup. This enables automatic
298 removal of abandoned cgroups. The default val 296 removal of abandoned cgroups. The default value of
299 notify_on_release in the root cgroup at system 297 notify_on_release in the root cgroup at system boot is disabled
300 (0). The default value of other cgroups at cr 298 (0). The default value of other cgroups at creation is the current
301 value of their parents' notify_on_release sett 299 value of their parents' notify_on_release settings. The default value of
302 a cgroup hierarchy's release_agent path is emp 300 a cgroup hierarchy's release_agent path is empty.
303 301
304 1.5 What does clone_children do ? 302 1.5 What does clone_children do ?
305 --------------------------------- 303 ---------------------------------
306 304
307 This flag only affects the cpuset controller. 305 This flag only affects the cpuset controller. If the clone_children
308 flag is enabled (1) in a cgroup, a new cpuset 306 flag is enabled (1) in a cgroup, a new cpuset cgroup will copy its
309 configuration from the parent during initializ 307 configuration from the parent during initialization.
310 308
311 1.6 How do I use cgroups ? 309 1.6 How do I use cgroups ?
312 -------------------------- 310 --------------------------
313 311
314 To start a new job that is to be contained wit 312 To start a new job that is to be contained within a cgroup, using
315 the "cpuset" cgroup subsystem, the steps are s 313 the "cpuset" cgroup subsystem, the steps are something like::
316 314
317 1) mount -t tmpfs cgroup_root /sys/fs/cgroup 315 1) mount -t tmpfs cgroup_root /sys/fs/cgroup
318 2) mkdir /sys/fs/cgroup/cpuset 316 2) mkdir /sys/fs/cgroup/cpuset
319 3) mount -t cgroup -ocpuset cpuset /sys/fs/cg 317 3) mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset
320 4) Create the new cgroup by doing mkdir's and 318 4) Create the new cgroup by doing mkdir's and write's (or echo's) in
321 the /sys/fs/cgroup/cpuset virtual file sys 319 the /sys/fs/cgroup/cpuset virtual file system.
322 5) Start a task that will be the "founding fa 320 5) Start a task that will be the "founding father" of the new job.
323 6) Attach that task to the new cgroup by writ 321 6) Attach that task to the new cgroup by writing its PID to the
324 /sys/fs/cgroup/cpuset tasks file for that 322 /sys/fs/cgroup/cpuset tasks file for that cgroup.
325 7) fork, exec or clone the job tasks from thi 323 7) fork, exec or clone the job tasks from this founding father task.
326 324
327 For example, the following sequence of command 325 For example, the following sequence of commands will setup a cgroup
328 named "Charlie", containing just CPUs 2 and 3, 326 named "Charlie", containing just CPUs 2 and 3, and Memory Node 1,
329 and then start a subshell 'sh' in that cgroup: 327 and then start a subshell 'sh' in that cgroup::
330 328
331 mount -t tmpfs cgroup_root /sys/fs/cgroup 329 mount -t tmpfs cgroup_root /sys/fs/cgroup
332 mkdir /sys/fs/cgroup/cpuset 330 mkdir /sys/fs/cgroup/cpuset
333 mount -t cgroup cpuset -ocpuset /sys/fs/cgro 331 mount -t cgroup cpuset -ocpuset /sys/fs/cgroup/cpuset
334 cd /sys/fs/cgroup/cpuset 332 cd /sys/fs/cgroup/cpuset
335 mkdir Charlie 333 mkdir Charlie
336 cd Charlie 334 cd Charlie
337 /bin/echo 2-3 > cpuset.cpus 335 /bin/echo 2-3 > cpuset.cpus
338 /bin/echo 1 > cpuset.mems 336 /bin/echo 1 > cpuset.mems
339 /bin/echo $$ > tasks 337 /bin/echo $$ > tasks
340 sh 338 sh
341 # The subshell 'sh' is now running in cgroup 339 # The subshell 'sh' is now running in cgroup Charlie
342 # The next line should display '/Charlie' 340 # The next line should display '/Charlie'
343 cat /proc/self/cgroup 341 cat /proc/self/cgroup
344 342
345 2. Usage Examples and Syntax 343 2. Usage Examples and Syntax
346 ============================ 344 ============================
347 345
348 2.1 Basic Usage 346 2.1 Basic Usage
349 --------------- 347 ---------------
350 348
351 Creating, modifying, using cgroups can be done 349 Creating, modifying, using cgroups can be done through the cgroup
352 virtual filesystem. 350 virtual filesystem.
353 351
354 To mount a cgroup hierarchy with all available 352 To mount a cgroup hierarchy with all available subsystems, type::
355 353
356 # mount -t cgroup xxx /sys/fs/cgroup 354 # mount -t cgroup xxx /sys/fs/cgroup
357 355
358 The "xxx" is not interpreted by the cgroup cod 356 The "xxx" is not interpreted by the cgroup code, but will appear in
359 /proc/mounts so may be any useful identifying 357 /proc/mounts so may be any useful identifying string that you like.
360 358
361 Note: Some subsystems do not work without some 359 Note: Some subsystems do not work without some user input first. For instance,
362 if cpusets are enabled the user will have to p 360 if cpusets are enabled the user will have to populate the cpus and mems files
363 for each new cgroup created before that group 361 for each new cgroup created before that group can be used.
364 362
365 As explained in section `1.2 Why are cgroups n 363 As explained in section `1.2 Why are cgroups needed?` you should create
366 different hierarchies of cgroups for each sing 364 different hierarchies of cgroups for each single resource or group of
367 resources you want to control. Therefore, you 365 resources you want to control. Therefore, you should mount a tmpfs on
368 /sys/fs/cgroup and create directories for each 366 /sys/fs/cgroup and create directories for each cgroup resource or resource
369 group:: 367 group::
370 368
371 # mount -t tmpfs cgroup_root /sys/fs/cgroup 369 # mount -t tmpfs cgroup_root /sys/fs/cgroup
372 # mkdir /sys/fs/cgroup/rg1 370 # mkdir /sys/fs/cgroup/rg1
373 371
374 To mount a cgroup hierarchy with just the cpus 372 To mount a cgroup hierarchy with just the cpuset and memory
375 subsystems, type:: 373 subsystems, type::
376 374
377 # mount -t cgroup -o cpuset,memory hier1 /sy 375 # mount -t cgroup -o cpuset,memory hier1 /sys/fs/cgroup/rg1
378 376
379 While remounting cgroups is currently supporte 377 While remounting cgroups is currently supported, it is not recommend
380 to use it. Remounting allows changing bound su 378 to use it. Remounting allows changing bound subsystems and
381 release_agent. Rebinding is hardly useful as i 379 release_agent. Rebinding is hardly useful as it only works when the
382 hierarchy is empty and release_agent itself sh 380 hierarchy is empty and release_agent itself should be replaced with
383 conventional fsnotify. The support for remount 381 conventional fsnotify. The support for remounting will be removed in
384 the future. 382 the future.
385 383
386 To Specify a hierarchy's release_agent:: 384 To Specify a hierarchy's release_agent::
387 385
388 # mount -t cgroup -o cpuset,release_agent="/ 386 # mount -t cgroup -o cpuset,release_agent="/sbin/cpuset_release_agent" \
389 xxx /sys/fs/cgroup/rg1 387 xxx /sys/fs/cgroup/rg1
390 388
391 Note that specifying 'release_agent' more than 389 Note that specifying 'release_agent' more than once will return failure.
392 390
393 Note that changing the set of subsystems is cu 391 Note that changing the set of subsystems is currently only supported
394 when the hierarchy consists of a single (root) 392 when the hierarchy consists of a single (root) cgroup. Supporting
395 the ability to arbitrarily bind/unbind subsyst 393 the ability to arbitrarily bind/unbind subsystems from an existing
396 cgroup hierarchy is intended to be implemented 394 cgroup hierarchy is intended to be implemented in the future.
397 395
398 Then under /sys/fs/cgroup/rg1 you can find a t 396 Then under /sys/fs/cgroup/rg1 you can find a tree that corresponds to the
399 tree of the cgroups in the system. For instanc 397 tree of the cgroups in the system. For instance, /sys/fs/cgroup/rg1
400 is the cgroup that holds the whole system. 398 is the cgroup that holds the whole system.
401 399
402 If you want to change the value of release_age 400 If you want to change the value of release_agent::
403 401
404 # echo "/sbin/new_release_agent" > /sys/fs/c 402 # echo "/sbin/new_release_agent" > /sys/fs/cgroup/rg1/release_agent
405 403
406 It can also be changed via remount. 404 It can also be changed via remount.
407 405
408 If you want to create a new cgroup under /sys/ 406 If you want to create a new cgroup under /sys/fs/cgroup/rg1::
409 407
410 # cd /sys/fs/cgroup/rg1 408 # cd /sys/fs/cgroup/rg1
411 # mkdir my_cgroup 409 # mkdir my_cgroup
412 410
413 Now you want to do something with this cgroup: 411 Now you want to do something with this cgroup:
414 412
415 # cd my_cgroup 413 # cd my_cgroup
416 414
417 In this directory you can find several files:: 415 In this directory you can find several files::
418 416
419 # ls 417 # ls
420 cgroup.procs notify_on_release tasks 418 cgroup.procs notify_on_release tasks
421 (plus whatever files added by the attached s 419 (plus whatever files added by the attached subsystems)
422 420
423 Now attach your shell to this cgroup:: 421 Now attach your shell to this cgroup::
424 422
425 # /bin/echo $$ > tasks 423 # /bin/echo $$ > tasks
426 424
427 You can also create cgroups inside your cgroup 425 You can also create cgroups inside your cgroup by using mkdir in this
428 directory:: 426 directory::
429 427
430 # mkdir my_sub_cs 428 # mkdir my_sub_cs
431 429
432 To remove a cgroup, just use rmdir:: 430 To remove a cgroup, just use rmdir::
433 431
434 # rmdir my_sub_cs 432 # rmdir my_sub_cs
435 433
436 This will fail if the cgroup is in use (has cg 434 This will fail if the cgroup is in use (has cgroups inside, or
437 has processes attached, or is held alive by ot 435 has processes attached, or is held alive by other subsystem-specific
438 reference). 436 reference).
439 437
440 2.2 Attaching processes 438 2.2 Attaching processes
441 ----------------------- 439 -----------------------
442 440
443 :: 441 ::
444 442
445 # /bin/echo PID > tasks 443 # /bin/echo PID > tasks
446 444
447 Note that it is PID, not PIDs. You can only at 445 Note that it is PID, not PIDs. You can only attach ONE task at a time.
448 If you have several tasks to attach, you have 446 If you have several tasks to attach, you have to do it one after another::
449 447
450 # /bin/echo PID1 > tasks 448 # /bin/echo PID1 > tasks
451 # /bin/echo PID2 > tasks 449 # /bin/echo PID2 > tasks
452 ... 450 ...
453 # /bin/echo PIDn > tasks 451 # /bin/echo PIDn > tasks
454 452
455 You can attach the current shell task by echoi 453 You can attach the current shell task by echoing 0::
456 454
457 # echo 0 > tasks 455 # echo 0 > tasks
458 456
459 You can use the cgroup.procs file instead of t 457 You can use the cgroup.procs file instead of the tasks file to move all
460 threads in a threadgroup at once. Echoing the 458 threads in a threadgroup at once. Echoing the PID of any task in a
461 threadgroup to cgroup.procs causes all tasks i 459 threadgroup to cgroup.procs causes all tasks in that threadgroup to be
462 attached to the cgroup. Writing 0 to cgroup.pr 460 attached to the cgroup. Writing 0 to cgroup.procs moves all tasks
463 in the writing task's threadgroup. 461 in the writing task's threadgroup.
464 462
465 Note: Since every task is always a member of e 463 Note: Since every task is always a member of exactly one cgroup in each
466 mounted hierarchy, to remove a task from its c 464 mounted hierarchy, to remove a task from its current cgroup you must
467 move it into a new cgroup (possibly the root c 465 move it into a new cgroup (possibly the root cgroup) by writing to the
468 new cgroup's tasks file. 466 new cgroup's tasks file.
469 467
470 Note: Due to some restrictions enforced by som 468 Note: Due to some restrictions enforced by some cgroup subsystems, moving
471 a process to another cgroup can fail. 469 a process to another cgroup can fail.
472 470
473 2.3 Mounting hierarchies by name 471 2.3 Mounting hierarchies by name
474 -------------------------------- 472 --------------------------------
475 473
476 Passing the name=<x> option when mounting a cg 474 Passing the name=<x> option when mounting a cgroups hierarchy
477 associates the given name with the hierarchy. 475 associates the given name with the hierarchy. This can be used when
478 mounting a pre-existing hierarchy, in order to 476 mounting a pre-existing hierarchy, in order to refer to it by name
479 rather than by its set of active subsystems. 477 rather than by its set of active subsystems. Each hierarchy is either
480 nameless, or has a unique name. 478 nameless, or has a unique name.
481 479
482 The name should match [\w.-]+ 480 The name should match [\w.-]+
483 481
484 When passing a name=<x> option for a new hiera 482 When passing a name=<x> option for a new hierarchy, you need to
485 specify subsystems manually; the legacy behavi 483 specify subsystems manually; the legacy behaviour of mounting all
486 subsystems when none are explicitly specified 484 subsystems when none are explicitly specified is not supported when
487 you give a subsystem a name. 485 you give a subsystem a name.
488 486
489 The name of the subsystem appears as part of t 487 The name of the subsystem appears as part of the hierarchy description
490 in /proc/mounts and /proc/<pid>/cgroups. 488 in /proc/mounts and /proc/<pid>/cgroups.
491 489
492 490
493 3. Kernel API 491 3. Kernel API
494 ============= 492 =============
495 493
496 3.1 Overview 494 3.1 Overview
497 ------------ 495 ------------
498 496
499 Each kernel subsystem that wants to hook into 497 Each kernel subsystem that wants to hook into the generic cgroup
500 system needs to create a cgroup_subsys object. 498 system needs to create a cgroup_subsys object. This contains
501 various methods, which are callbacks from the 499 various methods, which are callbacks from the cgroup system, along
502 with a subsystem ID which will be assigned by 500 with a subsystem ID which will be assigned by the cgroup system.
503 501
504 Other fields in the cgroup_subsys object inclu 502 Other fields in the cgroup_subsys object include:
505 503
506 - subsys_id: a unique array index for the subs 504 - subsys_id: a unique array index for the subsystem, indicating which
507 entry in cgroup->subsys[] this subsystem sho 505 entry in cgroup->subsys[] this subsystem should be managing.
508 506
509 - name: should be initialized to a unique subs 507 - name: should be initialized to a unique subsystem name. Should be
510 no longer than MAX_CGROUP_TYPE_NAMELEN. 508 no longer than MAX_CGROUP_TYPE_NAMELEN.
511 509
512 - early_init: indicate if the subsystem needs 510 - early_init: indicate if the subsystem needs early initialization
513 at system boot. 511 at system boot.
514 512
515 Each cgroup object created by the system has a 513 Each cgroup object created by the system has an array of pointers,
516 indexed by subsystem ID; this pointer is entir 514 indexed by subsystem ID; this pointer is entirely managed by the
517 subsystem; the generic cgroup code will never 515 subsystem; the generic cgroup code will never touch this pointer.
518 516
519 3.2 Synchronization 517 3.2 Synchronization
520 ------------------- 518 -------------------
521 519
522 There is a global mutex, cgroup_mutex, used by 520 There is a global mutex, cgroup_mutex, used by the cgroup
523 system. This should be taken by anything that 521 system. This should be taken by anything that wants to modify a
524 cgroup. It may also be taken to prevent cgroup 522 cgroup. It may also be taken to prevent cgroups from being
525 modified, but more specific locks may be more 523 modified, but more specific locks may be more appropriate in that
526 situation. 524 situation.
527 525
528 See kernel/cgroup.c for more details. 526 See kernel/cgroup.c for more details.
529 527
530 Subsystems can take/release the cgroup_mutex v 528 Subsystems can take/release the cgroup_mutex via the functions
531 cgroup_lock()/cgroup_unlock(). 529 cgroup_lock()/cgroup_unlock().
532 530
533 Accessing a task's cgroup pointer may be done 531 Accessing a task's cgroup pointer may be done in the following ways:
534 - while holding cgroup_mutex 532 - while holding cgroup_mutex
535 - while holding the task's alloc_lock (via tas 533 - while holding the task's alloc_lock (via task_lock())
536 - inside an rcu_read_lock() section via rcu_de 534 - inside an rcu_read_lock() section via rcu_dereference()
537 535
538 3.3 Subsystem API 536 3.3 Subsystem API
539 ----------------- 537 -----------------
540 538
541 Each subsystem should: 539 Each subsystem should:
542 540
543 - add an entry in linux/cgroup_subsys.h 541 - add an entry in linux/cgroup_subsys.h
544 - define a cgroup_subsys object called <name>_ 542 - define a cgroup_subsys object called <name>_cgrp_subsys
545 543
546 Each subsystem may export the following method 544 Each subsystem may export the following methods. The only mandatory
547 methods are css_alloc/free. Any others that ar 545 methods are css_alloc/free. Any others that are null are presumed to
548 be successful no-ops. 546 be successful no-ops.
549 547
550 ``struct cgroup_subsys_state *css_alloc(struct 548 ``struct cgroup_subsys_state *css_alloc(struct cgroup *cgrp)``
551 (cgroup_mutex held by caller) 549 (cgroup_mutex held by caller)
552 550
553 Called to allocate a subsystem state object fo 551 Called to allocate a subsystem state object for a cgroup. The
554 subsystem should allocate its subsystem state 552 subsystem should allocate its subsystem state object for the passed
555 cgroup, returning a pointer to the new object 553 cgroup, returning a pointer to the new object on success or a
556 ERR_PTR() value. On success, the subsystem poi 554 ERR_PTR() value. On success, the subsystem pointer should point to
557 a structure of type cgroup_subsys_state (typic 555 a structure of type cgroup_subsys_state (typically embedded in a
558 larger subsystem-specific object), which will 556 larger subsystem-specific object), which will be initialized by the
559 cgroup system. Note that this will be called a 557 cgroup system. Note that this will be called at initialization to
560 create the root subsystem state for this subsy 558 create the root subsystem state for this subsystem; this case can be
561 identified by the passed cgroup object having 559 identified by the passed cgroup object having a NULL parent (since
562 it's the root of the hierarchy) and may be an 560 it's the root of the hierarchy) and may be an appropriate place for
563 initialization code. 561 initialization code.
564 562
565 ``int css_online(struct cgroup *cgrp)`` 563 ``int css_online(struct cgroup *cgrp)``
566 (cgroup_mutex held by caller) 564 (cgroup_mutex held by caller)
567 565
568 Called after @cgrp successfully completed all 566 Called after @cgrp successfully completed all allocations and made
569 visible to cgroup_for_each_child/descendant_*( 567 visible to cgroup_for_each_child/descendant_*() iterators. The
570 subsystem may choose to fail creation by retur 568 subsystem may choose to fail creation by returning -errno. This
571 callback can be used to implement reliable sta 569 callback can be used to implement reliable state sharing and
572 propagation along the hierarchy. See the comme 570 propagation along the hierarchy. See the comment on
573 cgroup_for_each_live_descendant_pre() for deta !! 571 cgroup_for_each_descendant_pre() for details.
574 572
575 ``void css_offline(struct cgroup *cgrp);`` 573 ``void css_offline(struct cgroup *cgrp);``
576 (cgroup_mutex held by caller) 574 (cgroup_mutex held by caller)
577 575
578 This is the counterpart of css_online() and ca 576 This is the counterpart of css_online() and called iff css_online()
579 has succeeded on @cgrp. This signifies the beg 577 has succeeded on @cgrp. This signifies the beginning of the end of
580 @cgrp. @cgrp is being removed and the subsyste 578 @cgrp. @cgrp is being removed and the subsystem should start dropping
581 all references it's holding on @cgrp. When all 579 all references it's holding on @cgrp. When all references are dropped,
582 cgroup removal will proceed to the next step - 580 cgroup removal will proceed to the next step - css_free(). After this
583 callback, @cgrp should be considered dead to t 581 callback, @cgrp should be considered dead to the subsystem.
584 582
585 ``void css_free(struct cgroup *cgrp)`` 583 ``void css_free(struct cgroup *cgrp)``
586 (cgroup_mutex held by caller) 584 (cgroup_mutex held by caller)
587 585
588 The cgroup system is about to free @cgrp; the 586 The cgroup system is about to free @cgrp; the subsystem should free
589 its subsystem state object. By the time this m 587 its subsystem state object. By the time this method is called, @cgrp
590 is completely unused; @cgrp->parent is still v 588 is completely unused; @cgrp->parent is still valid. (Note - can also
591 be called for a newly-created cgroup if an err 589 be called for a newly-created cgroup if an error occurs after this
592 subsystem's create() method has been called fo 590 subsystem's create() method has been called for the new cgroup).
593 591
594 ``int can_attach(struct cgroup *cgrp, struct c 592 ``int can_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
595 (cgroup_mutex held by caller) 593 (cgroup_mutex held by caller)
596 594
597 Called prior to moving one or more tasks into 595 Called prior to moving one or more tasks into a cgroup; if the
598 subsystem returns an error, this will abort th 596 subsystem returns an error, this will abort the attach operation.
599 @tset contains the tasks to be attached and is 597 @tset contains the tasks to be attached and is guaranteed to have at
600 least one task in it. 598 least one task in it.
601 599
602 If there are multiple tasks in the taskset, th 600 If there are multiple tasks in the taskset, then:
603 - it's guaranteed that all are from the same 601 - it's guaranteed that all are from the same thread group
604 - @tset contains all tasks from the thread g 602 - @tset contains all tasks from the thread group whether or not
605 they're switching cgroups 603 they're switching cgroups
606 - the first task is the leader 604 - the first task is the leader
607 605
608 Each @tset entry also contains the task's old 606 Each @tset entry also contains the task's old cgroup and tasks which
609 aren't switching cgroup can be skipped easily 607 aren't switching cgroup can be skipped easily using the
610 cgroup_taskset_for_each() iterator. Note that 608 cgroup_taskset_for_each() iterator. Note that this isn't called on a
611 fork. If this method returns 0 (success) then 609 fork. If this method returns 0 (success) then this should remain valid
612 while the caller holds cgroup_mutex and it is 610 while the caller holds cgroup_mutex and it is ensured that either
613 attach() or cancel_attach() will be called in 611 attach() or cancel_attach() will be called in future.
614 612
615 ``void css_reset(struct cgroup_subsys_state *c 613 ``void css_reset(struct cgroup_subsys_state *css)``
616 (cgroup_mutex held by caller) 614 (cgroup_mutex held by caller)
617 615
618 An optional operation which should restore @cs 616 An optional operation which should restore @css's configuration to the
619 initial state. This is currently only used on 617 initial state. This is currently only used on the unified hierarchy
620 when a subsystem is disabled on a cgroup throu 618 when a subsystem is disabled on a cgroup through
621 "cgroup.subtree_control" but should remain ena 619 "cgroup.subtree_control" but should remain enabled because other
622 subsystems depend on it. cgroup core makes su 620 subsystems depend on it. cgroup core makes such a css invisible by
623 removing the associated interface files and in 621 removing the associated interface files and invokes this callback so
624 that the hidden subsystem can return to the in 622 that the hidden subsystem can return to the initial neutral state.
625 This prevents unexpected resource control from 623 This prevents unexpected resource control from a hidden css and
626 ensures that the configuration is in the initi 624 ensures that the configuration is in the initial state when it is made
627 visible again later. 625 visible again later.
628 626
629 ``void cancel_attach(struct cgroup *cgrp, stru 627 ``void cancel_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
630 (cgroup_mutex held by caller) 628 (cgroup_mutex held by caller)
631 629
632 Called when a task attach operation has failed 630 Called when a task attach operation has failed after can_attach() has succeeded.
633 A subsystem whose can_attach() has some side-e 631 A subsystem whose can_attach() has some side-effects should provide this
634 function, so that the subsystem can implement 632 function, so that the subsystem can implement a rollback. If not, not necessary.
635 This will be called only about subsystems whos 633 This will be called only about subsystems whose can_attach() operation have
636 succeeded. The parameters are identical to can 634 succeeded. The parameters are identical to can_attach().
637 635
638 ``void attach(struct cgroup *cgrp, struct cgro 636 ``void attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
639 (cgroup_mutex held by caller) 637 (cgroup_mutex held by caller)
640 638
641 Called after the task has been attached to the 639 Called after the task has been attached to the cgroup, to allow any
642 post-attachment activity that requires memory 640 post-attachment activity that requires memory allocations or blocking.
643 The parameters are identical to can_attach(). 641 The parameters are identical to can_attach().
644 642
645 ``void fork(struct task_struct *task)`` 643 ``void fork(struct task_struct *task)``
646 644
647 Called when a task is forked into a cgroup. 645 Called when a task is forked into a cgroup.
648 646
649 ``void exit(struct task_struct *task)`` 647 ``void exit(struct task_struct *task)``
650 648
651 Called during task exit. 649 Called during task exit.
652 650
653 ``void free(struct task_struct *task)`` 651 ``void free(struct task_struct *task)``
654 652
655 Called when the task_struct is freed. 653 Called when the task_struct is freed.
656 654
657 ``void bind(struct cgroup *root)`` 655 ``void bind(struct cgroup *root)``
658 (cgroup_mutex held by caller) 656 (cgroup_mutex held by caller)
659 657
660 Called when a cgroup subsystem is rebound to a 658 Called when a cgroup subsystem is rebound to a different hierarchy
661 and root cgroup. Currently this will only invo 659 and root cgroup. Currently this will only involve movement between
662 the default hierarchy (which never has sub-cgr 660 the default hierarchy (which never has sub-cgroups) and a hierarchy
663 that is being created/destroyed (and hence has 661 that is being created/destroyed (and hence has no sub-cgroups).
664 662
665 4. Extended attribute usage 663 4. Extended attribute usage
666 =========================== 664 ===========================
667 665
668 cgroup filesystem supports certain types of ex 666 cgroup filesystem supports certain types of extended attributes in its
669 directories and files. The current supported 667 directories and files. The current supported types are:
670 668
671 - Trusted (XATTR_TRUSTED) 669 - Trusted (XATTR_TRUSTED)
672 - Security (XATTR_SECURITY) 670 - Security (XATTR_SECURITY)
673 671
674 Both require CAP_SYS_ADMIN capability to set. 672 Both require CAP_SYS_ADMIN capability to set.
675 673
676 Like in tmpfs, the extended attributes in cgro 674 Like in tmpfs, the extended attributes in cgroup filesystem are stored
677 using kernel memory and it's advised to keep t 675 using kernel memory and it's advised to keep the usage at minimum. This
678 is the reason why user defined extended attrib 676 is the reason why user defined extended attributes are not supported, since
679 any user can do it and there's no limit in the 677 any user can do it and there's no limit in the value size.
680 678
681 The current known users for this feature are S 679 The current known users for this feature are SELinux to limit cgroup usage
682 in containers and systemd for assorted meta da 680 in containers and systemd for assorted meta data like main PID in a cgroup
683 (systemd creates a cgroup per service). 681 (systemd creates a cgroup per service).
684 682
685 5. Questions 683 5. Questions
686 ============ 684 ============
687 685
688 :: 686 ::
689 687
690 Q: what's up with this '/bin/echo' ? 688 Q: what's up with this '/bin/echo' ?
691 A: bash's builtin 'echo' command does not ch 689 A: bash's builtin 'echo' command does not check calls to write() against
692 errors. If you use it in the cgroup file 690 errors. If you use it in the cgroup file system, you won't be
693 able to tell whether a command succeeded 691 able to tell whether a command succeeded or failed.
694 692
695 Q: When I attach processes, only the first o 693 Q: When I attach processes, only the first of the line gets really attached !
696 A: We can only return one error code per cal 694 A: We can only return one error code per call to write(). So you should also
697 put only ONE PID. 695 put only ONE PID.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.