1 ========================================== 1 ==========================================
2 Reducing OS jitter due to per-cpu kthreads 2 Reducing OS jitter due to per-cpu kthreads
3 ========================================== 3 ==========================================
4 4
5 This document lists per-CPU kthreads in the Li 5 This document lists per-CPU kthreads in the Linux kernel and presents
6 options to control their OS jitter. Note that 6 options to control their OS jitter. Note that non-per-CPU kthreads are
7 not listed here. To reduce OS jitter from non 7 not listed here. To reduce OS jitter from non-per-CPU kthreads, bind
8 them to a "housekeeping" CPU dedicated to such 8 them to a "housekeeping" CPU dedicated to such work.
9 9
10 References 10 References
11 ========== 11 ==========
12 12
13 - Documentation/core-api/irq/irq-affinit 13 - Documentation/core-api/irq/irq-affinity.rst: Binding interrupts to sets of CPUs.
14 14
15 - Documentation/admin-guide/cgroup-v1: 15 - Documentation/admin-guide/cgroup-v1: Using cgroups to bind tasks to sets of CPUs.
16 16
17 - man taskset: Using the taskset comman 17 - man taskset: Using the taskset command to bind tasks to sets
18 of CPUs. 18 of CPUs.
19 19
20 - man sched_setaffinity: Using the sche 20 - man sched_setaffinity: Using the sched_setaffinity() system
21 call to bind tasks to sets of CPUs. 21 call to bind tasks to sets of CPUs.
22 22
23 - /sys/devices/system/cpu/cpuN/online: 23 - /sys/devices/system/cpu/cpuN/online: Control CPU N's hotplug state,
24 writing "0" to offline and "1" to onli 24 writing "0" to offline and "1" to online.
25 25
26 - In order to locate kernel-generated OS 26 - In order to locate kernel-generated OS jitter on CPU N:
27 27
28 cd /sys/kernel/tracing 28 cd /sys/kernel/tracing
29 echo 1 > max_graph_depth # Inc 29 echo 1 > max_graph_depth # Increase the "1" for more detail
30 echo function_graph > current_ 30 echo function_graph > current_tracer
31 # run workload 31 # run workload
32 cat per_cpu/cpuN/trace 32 cat per_cpu/cpuN/trace
33 33
34 kthreads 34 kthreads
35 ======== 35 ========
36 36
37 Name: 37 Name:
38 ehca_comp/%u 38 ehca_comp/%u
39 39
40 Purpose: 40 Purpose:
41 Periodically process Infiniband-related work 41 Periodically process Infiniband-related work.
42 42
43 To reduce its OS jitter, do any of the followi 43 To reduce its OS jitter, do any of the following:
44 44
45 1. Don't use eHCA Infiniband hardware, in 45 1. Don't use eHCA Infiniband hardware, instead choosing hardware
46 that does not require per-CPU kthreads 46 that does not require per-CPU kthreads. This will prevent these
47 kthreads from being created in the fir 47 kthreads from being created in the first place. (This will
48 work for most people, as this hardware 48 work for most people, as this hardware, though important, is
49 relatively old and is produced in rela 49 relatively old and is produced in relatively low unit volumes.)
50 2. Do all eHCA-Infiniband-related work on 50 2. Do all eHCA-Infiniband-related work on other CPUs, including
51 interrupts. 51 interrupts.
52 3. Rework the eHCA driver so that its per 52 3. Rework the eHCA driver so that its per-CPU kthreads are
53 provisioned only on selected CPUs. 53 provisioned only on selected CPUs.
54 54
55 55
56 Name: 56 Name:
57 irq/%d-%s 57 irq/%d-%s
58 58
59 Purpose: 59 Purpose:
60 Handle threaded interrupts. 60 Handle threaded interrupts.
61 61
62 To reduce its OS jitter, do the following: 62 To reduce its OS jitter, do the following:
63 63
64 1. Use irq affinity to force the irq thre 64 1. Use irq affinity to force the irq threads to execute on
65 some other CPU. 65 some other CPU.
66 66
67 Name: 67 Name:
68 kcmtpd_ctr_%d 68 kcmtpd_ctr_%d
69 69
70 Purpose: 70 Purpose:
71 Handle Bluetooth work. 71 Handle Bluetooth work.
72 72
73 To reduce its OS jitter, do one of the followi 73 To reduce its OS jitter, do one of the following:
74 74
75 1. Don't use Bluetooth, in which case the 75 1. Don't use Bluetooth, in which case these kthreads won't be
76 created in the first place. 76 created in the first place.
77 2. Use irq affinity to force Bluetooth-re 77 2. Use irq affinity to force Bluetooth-related interrupts to
78 occur on some other CPU and furthermor 78 occur on some other CPU and furthermore initiate all
79 Bluetooth activity on some other CPU. 79 Bluetooth activity on some other CPU.
80 80
81 Name: 81 Name:
82 ksoftirqd/%u 82 ksoftirqd/%u
83 83
84 Purpose: 84 Purpose:
85 Execute softirq handlers when threaded or wh 85 Execute softirq handlers when threaded or when under heavy load.
86 86
87 To reduce its OS jitter, each softirq vector m 87 To reduce its OS jitter, each softirq vector must be handled
88 separately as follows: 88 separately as follows:
89 89
90 TIMER_SOFTIRQ 90 TIMER_SOFTIRQ
91 ------------- 91 -------------
92 92
93 Do all of the following: 93 Do all of the following:
94 94
95 1. To the extent possible, keep the CPU o 95 1. To the extent possible, keep the CPU out of the kernel when it
96 is non-idle, for example, by avoiding 96 is non-idle, for example, by avoiding system calls and by forcing
97 both kernel threads and interrupts to 97 both kernel threads and interrupts to execute elsewhere.
98 2. Build with CONFIG_HOTPLUG_CPU=y. Afte 98 2. Build with CONFIG_HOTPLUG_CPU=y. After boot completes, force
99 the CPU offline, then bring it back on 99 the CPU offline, then bring it back online. This forces
100 recurring timers to migrate elsewhere. 100 recurring timers to migrate elsewhere. If you are concerned
101 with multiple CPUs, force them all off 101 with multiple CPUs, force them all offline before bringing the
102 first one back online. Once you have 102 first one back online. Once you have onlined the CPUs in question,
103 do not offline any other CPUs, because 103 do not offline any other CPUs, because doing so could force the
104 timer back onto one of the CPUs in que 104 timer back onto one of the CPUs in question.
105 105
106 NET_TX_SOFTIRQ and NET_RX_SOFTIRQ 106 NET_TX_SOFTIRQ and NET_RX_SOFTIRQ
107 --------------------------------- 107 ---------------------------------
108 108
109 Do all of the following: 109 Do all of the following:
110 110
111 1. Force networking interrupts onto other 111 1. Force networking interrupts onto other CPUs.
112 2. Initiate any network I/O on other CPUs 112 2. Initiate any network I/O on other CPUs.
113 3. Once your application has started, pre 113 3. Once your application has started, prevent CPU-hotplug operations
114 from being initiated from tasks that m 114 from being initiated from tasks that might run on the CPU to
115 be de-jittered. (It is OK to force th 115 be de-jittered. (It is OK to force this CPU offline and then
116 bring it back online before you start 116 bring it back online before you start your application.)
117 117
118 BLOCK_SOFTIRQ 118 BLOCK_SOFTIRQ
119 ------------- 119 -------------
120 120
121 Do all of the following: 121 Do all of the following:
122 122
123 1. Force block-device interrupts onto som 123 1. Force block-device interrupts onto some other CPU.
124 2. Initiate any block I/O on other CPUs. 124 2. Initiate any block I/O on other CPUs.
125 3. Once your application has started, pre 125 3. Once your application has started, prevent CPU-hotplug operations
126 from being initiated from tasks that m 126 from being initiated from tasks that might run on the CPU to
127 be de-jittered. (It is OK to force th 127 be de-jittered. (It is OK to force this CPU offline and then
128 bring it back online before you start 128 bring it back online before you start your application.)
129 129
130 IRQ_POLL_SOFTIRQ 130 IRQ_POLL_SOFTIRQ
131 ---------------- 131 ----------------
132 132
133 Do all of the following: 133 Do all of the following:
134 134
135 1. Force block-device interrupts onto som 135 1. Force block-device interrupts onto some other CPU.
136 2. Initiate any block I/O and block-I/O p 136 2. Initiate any block I/O and block-I/O polling on other CPUs.
137 3. Once your application has started, pre 137 3. Once your application has started, prevent CPU-hotplug operations
138 from being initiated from tasks that m 138 from being initiated from tasks that might run on the CPU to
139 be de-jittered. (It is OK to force th 139 be de-jittered. (It is OK to force this CPU offline and then
140 bring it back online before you start 140 bring it back online before you start your application.)
141 141
142 TASKLET_SOFTIRQ 142 TASKLET_SOFTIRQ
143 --------------- 143 ---------------
144 144
145 Do one or more of the following: 145 Do one or more of the following:
146 146
147 1. Avoid use of drivers that use tasklets 147 1. Avoid use of drivers that use tasklets. (Such drivers will contain
148 calls to things like tasklet_schedule( 148 calls to things like tasklet_schedule().)
149 2. Convert all drivers that you must use 149 2. Convert all drivers that you must use from tasklets to workqueues.
150 3. Force interrupts for drivers using tas 150 3. Force interrupts for drivers using tasklets onto other CPUs,
151 and also do I/O involving these driver 151 and also do I/O involving these drivers on other CPUs.
152 152
153 SCHED_SOFTIRQ 153 SCHED_SOFTIRQ
154 ------------- 154 -------------
155 155
156 Do all of the following: 156 Do all of the following:
157 157
158 1. Avoid sending scheduler IPIs to the CP 158 1. Avoid sending scheduler IPIs to the CPU to be de-jittered,
159 for example, ensure that at most one r 159 for example, ensure that at most one runnable kthread is present
160 on that CPU. If a thread that expects 160 on that CPU. If a thread that expects to run on the de-jittered
161 CPU awakens, the scheduler will send a 161 CPU awakens, the scheduler will send an IPI that can result in
162 a subsequent SCHED_SOFTIRQ. 162 a subsequent SCHED_SOFTIRQ.
163 2. CONFIG_NO_HZ_FULL=y and ensure that th 163 2. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be de-jittered
164 is marked as an adaptive-ticks CPU usi 164 is marked as an adaptive-ticks CPU using the "nohz_full="
165 boot parameter. This reduces the numb 165 boot parameter. This reduces the number of scheduler-clock
166 interrupts that the de-jittered CPU re 166 interrupts that the de-jittered CPU receives, minimizing its
167 chances of being selected to do the lo 167 chances of being selected to do the load balancing work that
168 runs in SCHED_SOFTIRQ context. 168 runs in SCHED_SOFTIRQ context.
169 3. To the extent possible, keep the CPU o 169 3. To the extent possible, keep the CPU out of the kernel when it
170 is non-idle, for example, by avoiding 170 is non-idle, for example, by avoiding system calls and by
171 forcing both kernel threads and interr 171 forcing both kernel threads and interrupts to execute elsewhere.
172 This further reduces the number of sch 172 This further reduces the number of scheduler-clock interrupts
173 received by the de-jittered CPU. 173 received by the de-jittered CPU.
174 174
175 HRTIMER_SOFTIRQ 175 HRTIMER_SOFTIRQ
176 --------------- 176 ---------------
177 177
178 Do all of the following: 178 Do all of the following:
179 179
180 1. To the extent possible, keep the CPU o 180 1. To the extent possible, keep the CPU out of the kernel when it
181 is non-idle. For example, avoid syste 181 is non-idle. For example, avoid system calls and force both
182 kernel threads and interrupts to execu 182 kernel threads and interrupts to execute elsewhere.
183 2. Build with CONFIG_HOTPLUG_CPU=y. Once 183 2. Build with CONFIG_HOTPLUG_CPU=y. Once boot completes, force the
184 CPU offline, then bring it back online 184 CPU offline, then bring it back online. This forces recurring
185 timers to migrate elsewhere. If you a 185 timers to migrate elsewhere. If you are concerned with multiple
186 CPUs, force them all offline before br 186 CPUs, force them all offline before bringing the first one
187 back online. Once you have onlined th 187 back online. Once you have onlined the CPUs in question, do not
188 offline any other CPUs, because doing 188 offline any other CPUs, because doing so could force the timer
189 back onto one of the CPUs in question. 189 back onto one of the CPUs in question.
190 190
191 RCU_SOFTIRQ 191 RCU_SOFTIRQ
192 ----------- 192 -----------
193 193
194 Do at least one of the following: 194 Do at least one of the following:
195 195
196 1. Offload callbacks and keep the CPU in 196 1. Offload callbacks and keep the CPU in either dyntick-idle or
197 adaptive-ticks state by doing all of t 197 adaptive-ticks state by doing all of the following:
198 198
199 a. CONFIG_NO_HZ_FULL=y and ensure 199 a. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be
200 de-jittered is marked as an ad 200 de-jittered is marked as an adaptive-ticks CPU using the
201 "nohz_full=" boot parameter. 201 "nohz_full=" boot parameter. Bind the rcuo kthreads to
202 housekeeping CPUs, which can t 202 housekeeping CPUs, which can tolerate OS jitter.
203 b. To the extent possible, keep t 203 b. To the extent possible, keep the CPU out of the kernel
204 when it is non-idle, for examp 204 when it is non-idle, for example, by avoiding system
205 calls and by forcing both kern 205 calls and by forcing both kernel threads and interrupts
206 to execute elsewhere. 206 to execute elsewhere.
207 207
208 2. Enable RCU to do its processing remote 208 2. Enable RCU to do its processing remotely via dyntick-idle by
209 doing all of the following: 209 doing all of the following:
210 210
211 a. Build with CONFIG_NO_HZ=y. 211 a. Build with CONFIG_NO_HZ=y.
212 b. Ensure that the CPU goes idle 212 b. Ensure that the CPU goes idle frequently, allowing other
213 CPUs to detect that it has pas 213 CPUs to detect that it has passed through an RCU quiescent
214 state. If the kernel is built 214 state. If the kernel is built with CONFIG_NO_HZ_FULL=y,
215 userspace execution also allow 215 userspace execution also allows other CPUs to detect that
216 the CPU in question has passed 216 the CPU in question has passed through a quiescent state.
217 c. To the extent possible, keep t 217 c. To the extent possible, keep the CPU out of the kernel
218 when it is non-idle, for examp 218 when it is non-idle, for example, by avoiding system
219 calls and by forcing both kern 219 calls and by forcing both kernel threads and interrupts
220 to execute elsewhere. 220 to execute elsewhere.
221 221
222 Name: 222 Name:
223 kworker/%u:%d%s (cpu, id, priority) 223 kworker/%u:%d%s (cpu, id, priority)
224 224
225 Purpose: 225 Purpose:
226 Execute workqueue requests 226 Execute workqueue requests
227 227
228 To reduce its OS jitter, do any of the followi 228 To reduce its OS jitter, do any of the following:
229 229
230 1. Run your workload at a real-time prior 230 1. Run your workload at a real-time priority, which will allow
231 preempting the kworker daemons. 231 preempting the kworker daemons.
232 2. A given workqueue can be made visible 232 2. A given workqueue can be made visible in the sysfs filesystem
233 by passing the WQ_SYSFS to that workqu 233 by passing the WQ_SYSFS to that workqueue's alloc_workqueue().
234 Such a workqueue can be confined to a 234 Such a workqueue can be confined to a given subset of the
235 CPUs using the ``/sys/devices/virtual/ 235 CPUs using the ``/sys/devices/virtual/workqueue/*/cpumask`` sysfs
236 files. The set of WQ_SYSFS workqueues 236 files. The set of WQ_SYSFS workqueues can be displayed using
237 "ls /sys/devices/virtual/workqueue". 237 "ls /sys/devices/virtual/workqueue". That said, the workqueues
238 maintainer would like to caution peopl 238 maintainer would like to caution people against indiscriminately
239 sprinkling WQ_SYSFS across all the wor 239 sprinkling WQ_SYSFS across all the workqueues. The reason for
240 caution is that it is easy to add WQ_S 240 caution is that it is easy to add WQ_SYSFS, but because sysfs is
241 part of the formal user/kernel API, it 241 part of the formal user/kernel API, it can be nearly impossible
242 to remove it, even if its addition was 242 to remove it, even if its addition was a mistake.
243 3. Do any of the following needed to avoi 243 3. Do any of the following needed to avoid jitter that your
244 application cannot tolerate: 244 application cannot tolerate:
245 245
246 a. Avoid using oprofile, thus avo !! 246 a. Build your kernel with CONFIG_SLUB=y rather than
>> 247 CONFIG_SLAB=y, thus avoiding the slab allocator's periodic
>> 248 use of each CPU's workqueues to run its cache_reap()
>> 249 function.
>> 250 b. Avoid using oprofile, thus avoiding OS jitter from
247 wq_sync_buffer(). 251 wq_sync_buffer().
248 b. Limit your CPU frequency so th !! 252 c. Limit your CPU frequency so that a CPU-frequency
249 governor is not required, poss 253 governor is not required, possibly enlisting the aid of
250 special heatsinks or other coo 254 special heatsinks or other cooling technologies. If done
251 correctly, and if you CPU arch 255 correctly, and if you CPU architecture permits, you should
252 be able to build your kernel w 256 be able to build your kernel with CONFIG_CPU_FREQ=n to
253 avoid the CPU-frequency govern 257 avoid the CPU-frequency governor periodically running
254 on each CPU, including cs_dbs_ 258 on each CPU, including cs_dbs_timer() and od_dbs_timer().
255 259
256 WARNING: Please check your CP 260 WARNING: Please check your CPU specifications to
257 make sure that this is safe on 261 make sure that this is safe on your particular system.
258 c. As of v3.18, Christoph Lameter !! 262 d. As of v3.18, Christoph Lameter's on-demand vmstat workers
259 commit prevents OS jitter due 263 commit prevents OS jitter due to vmstat_update() on
260 CONFIG_SMP=y systems. Before 264 CONFIG_SMP=y systems. Before v3.18, is not possible
261 to entirely get rid of the OS 265 to entirely get rid of the OS jitter, but you can
262 decrease its frequency by writ 266 decrease its frequency by writing a large value to
263 /proc/sys/vm/stat_interval. T 267 /proc/sys/vm/stat_interval. The default value is HZ,
264 for an interval of one second. 268 for an interval of one second. Of course, larger values
265 will make your virtual-memory 269 will make your virtual-memory statistics update more
266 slowly. Of course, you can al 270 slowly. Of course, you can also run your workload at
267 a real-time priority, thus pre 271 a real-time priority, thus preempting vmstat_update(),
268 but if your workload is CPU-bo 272 but if your workload is CPU-bound, this is a bad idea.
269 However, there is an RFC patch 273 However, there is an RFC patch from Christoph Lameter
270 (based on an earlier one from 274 (based on an earlier one from Gilad Ben-Yossef) that
271 reduces or even eliminates vms 275 reduces or even eliminates vmstat overhead for some
272 workloads at https://lore.kern 276 workloads at https://lore.kernel.org/r/00000140e9dfd6bd-40db3d4f-c1be-434f-8132-7820f81bb586-000000@email.amazonses.com.
273 d. If running on high-end powerpc !! 277 e. If running on high-end powerpc servers, build with
274 CONFIG_PPC_RTAS_DAEMON=n. Thi 278 CONFIG_PPC_RTAS_DAEMON=n. This prevents the RTAS
275 daemon from running on each CP 279 daemon from running on each CPU every second or so.
276 (This will require editing Kco 280 (This will require editing Kconfig files and will defeat
277 this platform's RAS functional 281 this platform's RAS functionality.) This avoids jitter
278 due to the rtas_event_scan() f 282 due to the rtas_event_scan() function.
279 WARNING: Please check your CP 283 WARNING: Please check your CPU specifications to
280 make sure that this is safe on 284 make sure that this is safe on your particular system.
281 e. If running on Cell Processor, !! 285 f. If running on Cell Processor, build your kernel with
282 CBE_CPUFREQ_SPU_GOVERNOR=n to 286 CBE_CPUFREQ_SPU_GOVERNOR=n to avoid OS jitter from
283 spu_gov_work(). 287 spu_gov_work().
284 WARNING: Please check your CP 288 WARNING: Please check your CPU specifications to
285 make sure that this is safe on 289 make sure that this is safe on your particular system.
286 f. If running on PowerMAC, build !! 290 g. If running on PowerMAC, build your kernel with
287 CONFIG_PMAC_RACKMETER=n to dis 291 CONFIG_PMAC_RACKMETER=n to disable the CPU-meter,
288 avoiding OS jitter from rackme 292 avoiding OS jitter from rackmeter_do_timer().
289 293
290 Name: 294 Name:
291 rcuc/%u 295 rcuc/%u
292 296
293 Purpose: 297 Purpose:
294 Execute RCU callbacks in CONFIG_RCU_BOOST=y 298 Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
295 299
296 To reduce its OS jitter, do at least one of th 300 To reduce its OS jitter, do at least one of the following:
297 301
298 1. Build the kernel with CONFIG_PREEMPT=n 302 1. Build the kernel with CONFIG_PREEMPT=n. This prevents these
299 kthreads from being created in the fir 303 kthreads from being created in the first place, and also obviates
300 the need for RCU priority boosting. T 304 the need for RCU priority boosting. This approach is feasible
301 for workloads that do not require high 305 for workloads that do not require high degrees of responsiveness.
302 2. Build the kernel with CONFIG_RCU_BOOST 306 2. Build the kernel with CONFIG_RCU_BOOST=n. This prevents these
303 kthreads from being created in the fir 307 kthreads from being created in the first place. This approach
304 is feasible only if your workload neve 308 is feasible only if your workload never requires RCU priority
305 boosting, for example, if you ensure f 309 boosting, for example, if you ensure frequent idle time on all
306 CPUs that might execute within the ker 310 CPUs that might execute within the kernel.
307 3. Build with CONFIG_RCU_NOCB_CPU=y and b 311 3. Build with CONFIG_RCU_NOCB_CPU=y and boot with the rcu_nocbs=
308 boot parameter offloading RCU callback 312 boot parameter offloading RCU callbacks from all CPUs susceptible
309 to OS jitter. This approach prevents 313 to OS jitter. This approach prevents the rcuc/%u kthreads from
310 having any work to do, so that they ar 314 having any work to do, so that they are never awakened.
311 4. Ensure that the CPU never enters the k 315 4. Ensure that the CPU never enters the kernel, and, in particular,
312 avoid initiating any CPU hotplug opera 316 avoid initiating any CPU hotplug operations on this CPU. This is
313 another way of preventing any callback 317 another way of preventing any callbacks from being queued on the
314 CPU, again preventing the rcuc/%u kthr 318 CPU, again preventing the rcuc/%u kthreads from having any work
315 to do. 319 to do.
316 320
317 Name: 321 Name:
318 rcuop/%d and rcuos/%d 322 rcuop/%d and rcuos/%d
319 323
320 Purpose: 324 Purpose:
321 Offload RCU callbacks from the corresponding 325 Offload RCU callbacks from the corresponding CPU.
322 326
323 To reduce its OS jitter, do at least one of th 327 To reduce its OS jitter, do at least one of the following:
324 328
325 1. Use affinity, cgroups, or other mechan 329 1. Use affinity, cgroups, or other mechanism to force these kthreads
326 to execute on some other CPU. 330 to execute on some other CPU.
327 2. Build with CONFIG_RCU_NOCB_CPU=n, whic 331 2. Build with CONFIG_RCU_NOCB_CPU=n, which will prevent these
328 kthreads from being created in the fir 332 kthreads from being created in the first place. However, please
329 note that this will not eliminate OS j 333 note that this will not eliminate OS jitter, but will instead
330 shift it to RCU_SOFTIRQ. 334 shift it to RCU_SOFTIRQ.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.