1 =====================
2 CFS Bandwidth Control
3 =====================
4
5 .. note::
6 This document only discusses CPU bandwidth
7 The SCHED_RT case is covered in Documentati
8
9 CFS bandwidth control is a CONFIG_FAIR_GROUP_S
10 specification of the maximum CPU bandwidth ava
11
12 The bandwidth allowed for a group is specified
13 each given "period" (microseconds), a task gro
14 microseconds of CPU time. That quota is assign
15 slices as threads in the cgroup become runnabl
16 assigned any additional requests for quota wil
17 throttled. Throttled threads will not be able
18 period when the quota is replenished.
19
20 A group's unassigned quota is globally tracked
21 cfs_quota units at each period boundary. As th
22 is transferred to cpu-local "silos" on a deman
23 within each of these updates is tunable and de
24
25 Burst feature
26 -------------
27 This feature borrows time now against our futu
28 increased interference against the other syste
29
30 Traditional (UP-EDF) bandwidth control is some
31
32 (U = \Sum u_i) <= 1
33
34 This guaranteeds both that every deadline is m
35 stable. After all, if U were > 1, then for eve
36 we'd have to run more than a second of program
37 our deadline, but the next deadline will be fu
38 never time to catch up, unbounded fail.
39
40 The burst feature observes that a workload doe
41 quota; this enables one to describe u_i as a s
42
43 For example, have u_i = {x,e}_i, where x is th
44 (the traditional WCET). This effectively allow
45 increasing the efficiency (we can pack more ta
46 the cost of missing deadlines when all the odd
47 does maintain stability, since every overrun m
48 underrun as long as our x is above the average
49
50 That is, suppose we have 2 tasks, both specify
51 have a p(95)*p(95) = 90.25% chance both tasks
52 everything is good. At the same time we have a
53 both tasks will exceed their quota at the same
54 fail). Somewhere in between there's a threshol
55 the other doesn't underrun enough to compensat
56 specific CDFs.
57
58 At the same time, we can say that the worst ca
59 \Sum e_i; that is, there is a bounded tardines
60 that x+e is indeed WCET).
61
62 The interferenece when using burst is valued b
63 missing the deadline and the average WCET. Tes
64 there many cgroups or CPU is under utilized, t
65 limited. More details are shown in:
66 https://lore.kernel.org/lkml/5371BD36-55AE-4F7
67
68 Management
69 ----------
70 Quota, period and burst are managed within the
71
72 .. note::
73 The cgroupfs files described in this sectio
74 to cgroup v1. For cgroup v2, see
75 :ref:`Documentation/admin-guide/cgroup-v2.r
76
77 - cpu.cfs_quota_us: run-time replenished withi
78 - cpu.cfs_period_us: the length of a period (i
79 - cpu.stat: exports throttling statistics [exp
80 - cpu.cfs_burst_us: the maximum accumulated ru
81
82 The default values are::
83
84 cpu.cfs_period_us=100ms
85 cpu.cfs_quota_us=-1
86 cpu.cfs_burst_us=0
87
88 A value of -1 for cpu.cfs_quota_us indicates t
89 bandwidth restriction in place, such a group i
90 bandwidth group. This represents the tradition
91 CFS.
92
93 Writing any (valid) positive value(s) no small
94 enact the specified bandwidth limit. The minim
95 period is 1ms. There is also an upper bound on
96 Additional restrictions exist when bandwidth l
97 fashion, these are explained in more detail be
98
99 Writing any negative value to cpu.cfs_quota_us
100 and return the group to an unconstrained state
101
102 A value of 0 for cpu.cfs_burst_us indicates th
103 any unused bandwidth. It makes the traditional
104 CFS unchanged. Writing any (valid) positive va
105 cpu.cfs_quota_us into cpu.cfs_burst_us will en
106 accumulation.
107
108 Any updates to a group's bandwidth specificati
109 unthrottled if it is in a constrained state.
110
111 System wide settings
112 --------------------
113 For efficiency run-time is transferred between
114 "silos" in a batch fashion. This greatly reduc
115 on large systems. The amount transferred each
116 is described as the "slice".
117
118 This is tunable via procfs::
119
120 /proc/sys/kernel/sched_cfs_bandwidth_s
121
122 Larger slice values will reduce transfer overh
123 for more fine-grained consumption.
124
125 Statistics
126 ----------
127 A group's bandwidth statistics are exported vi
128
129 cpu.stat:
130
131 - nr_periods: Number of enforcement intervals
132 - nr_throttled: Number of times the group has
133 - throttled_time: The total time duration (in
134 of the group have been throttled.
135 - nr_bursts: Number of periods burst occurs.
136 - burst_time: Cumulative wall-time (in nanosec
137 above quota in respective periods.
138
139 This interface is read-only.
140
141 Hierarchical considerations
142 ---------------------------
143 The interface enforces that an individual enti
144 attainable, that is: max(c_i) <= C. However, o
145 aggregate case is explicitly allowed to enable
146 within a hierarchy:
147
148 e.g. \Sum (c_i) may exceed C
149
150 [ Where C is the parent's bandwidth, and c_i i
151
152
153 There are two ways in which a group may become
154
155 a. it fully consumes its own quota wit
156 b. a parent's quota is fully consumed
157
158 In case b) above, even though the child may ha
159 be allowed to until the parent's runtime is re
160
161 CFS Bandwidth Quota Caveats
162 ---------------------------
163 Once a slice is assigned to a cpu it does not
164 the slice may be returned to the global pool i
165 unrunnable. This is configured at compile time
166 variable. This is a performance tweak that hel
167 the global lock.
168
169 The fact that cpu-local slices do not expire r
170 cases that should be understood.
171
172 For cgroup cpu constrained applications that a
173 relatively moot point because they will natura
174 quota as well as the entirety of each cpu-loca
175 result it is expected that nr_periods roughly
176 cpuacct.usage will increase roughly equal to c
177
178 For highly-threaded, non-cpu bound application
179 allows applications to briefly burst past thei
180 unused slice on each cpu that the task group i
181 1ms per cpu or as defined by min_cfs_rq_runtim
182 applies if quota had been assigned to a cpu an
183 in previous periods. This burst amount will no
184 As a result, this mechanism still strictly lim
185 average usage, albeit over a longer time windo
186 also limits the burst ability to no more than
187 better more predictable user experience for hi
188 small quota limits on high core count machines
189 propensity to throttle these applications whil
190 quota amounts of cpu. Another way to say this,
191 portion of a slice to remain valid across peri
192 possibility of wastefully expiring quota on cp
193 full slice's amount of cpu time.
194
195 The interaction between cpu-bound and non-cpu-
196 should also be considered, especially when sin
197 gave each of these applications half of a cpu-
198 on the same CPU it is theoretically possible t
199 will use up to 1ms additional quota in some pe
200 cpu-bound application from fully using its quo
201 instances it will be up to the CFS algorithm (
202 decide which application is chosen to run, as
203 have remaining quota. This runtime discrepancy
204 periods when the interactive application idles
205
206 Examples
207 --------
208 1. Limit a group to 1 CPU worth of runtime::
209
210 If period is 250ms and quota is also 2
211 1 CPU worth of runtime every 250ms.
212
213 # echo 250000 > cpu.cfs_quota_us /* qu
214 # echo 250000 > cpu.cfs_period_us /* p
215
216 2. Limit a group to 2 CPUs worth of runtime on
217
218 With 500ms period and 1000ms quota, the gro
219 runtime every 500ms::
220
221 # echo 1000000 > cpu.cfs_quota_us /* q
222 # echo 500000 > cpu.cfs_period_us /* p
223
224 The larger period here allows for incr
225
226 3. Limit a group to 20% of 1 CPU.
227
228 With 50ms period, 10ms quota will be equiva
229
230 # echo 10000 > cpu.cfs_quota_us /* quo
231 # echo 50000 > cpu.cfs_period_us /* pe
232
233 By using a small period here we are ensurin
234 response at the expense of burst capacity.
235
236 4. Limit a group to 40% of 1 CPU, and allow ac
237 additionally, in case accumulation has been
238
239 With 50ms period, 20ms quota will be equiva
240 And 10ms burst will be equivalent to 20% of
241
242 # echo 20000 > cpu.cfs_quota_us /* quo
243 # echo 50000 > cpu.cfs_period_us /* pe
244 # echo 10000 > cpu.cfs_burst_us /* bur
245
246 Larger buffer setting (no larger than quota
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.