1 .. SPDX-License-Identifier: GPL-2.0
2
3 ==========================
4 RCU Torture Test Operation
5 ==========================
6
7
8 CONFIG_RCU_TORTURE_TEST
9 =======================
10
11 The CONFIG_RCU_TORTURE_TEST config option is a
12 implementations. It creates an rcutorture ker
13 be loaded to run a torture test. The test per
14 status messages via printk(), which can be exa
15 command (perhaps grepping for "torture"). The
16 when the module is loaded, and stops when the
17
18 Module parameters are prefixed by "rcutorture.
19 Documentation/admin-guide/kernel-parameters.tx
20
21 Output
22 ======
23
24 The statistics output is as follows::
25
26 rcu-torture:--- Start of test: nreader
27 rcu-torture: rtc: (null) ver
28 rcu-torture: Reader Pipe: 727860534 3
29 rcu-torture: Reader Batch: 727877838
30 rcu-torture: Free-Block Circulation:
31 rcu-torture:--- End of test: SUCCESS:
32
33 The command "dmesg | grep torture:" will extra
34 most systems. On more esoteric configurations
35 use other commands to access the output of the
36 the RCU torture test. The printk()s use KERN_
37 be evident. ;-)
38
39 The first and last lines show the rcutorture m
40 last line shows either "SUCCESS" or "FAILURE",
41 automatic determination as to whether RCU oper
42
43 The entries are as follows:
44
45 * "rtc": The hexadecimal address of the
46 to readers.
47
48 * "ver": The number of times since boot
49 has changed the structure visible to r
50
51 * "tfle": If non-zero, indicates that th
52 containing structures to be placed int
53 This condition is important, since it
54 that RCU is working when it is not. :
55
56 * "rta": Number of structures allocated
57
58 * "rtaf": Number of allocations from the
59 failed due to the list being empty. I
60 to be non-zero, but it is bad for it t
61 the value indicated by "rta".
62
63 * "rtf": Number of frees into the tortur
64
65 * "rtmbe": A non-zero value indicates th
66 rcu_assign_pointer() and rcu_dereferen
67 correctly. This value should be zero.
68
69 * "rtbe": A non-zero value indicates tha
70 family of functions is not working cor
71
72 * "rtbke": rcutorture was unable to crea
73 used to force RCU priority inversion.
74
75 * "rtbre": Although rcutorture successfu
76 used to force RCU priority inversion,
77 to the real-time priority level of 1.
78
79 * "rtbf": The number of times that RCU p
80 to resolve RCU priority inversion.
81
82 * "rtb": The number of times that rcutor
83 an RCU priority inversion condition.
84 priority boosting via the "test_boost"
85 value should be non-zero.
86
87 * "nt": The number of times rcutorture r
88 within a timer handler. This value sh
89 if you specified the "irqreader" modul
90
91 * "Reader Pipe": Histogram of "ages" of
92 If any entries past the first two are
93 And rcutorture prints the error flag s
94 you notice. The age of a newly alloca
95 it becomes one when removed from reade
96 incremented once per grace period subs
97 after passing through (RCU_TORTURE_PIP
98
99 The output displayed above was taken f
100 RCU. If you want to see what it looks
101 it yourself. ;-)
102
103 * "Reader Batch": Another histogram of "
104 by readers, but in terms of counter fl
105 than in terms of grace periods. The l
106 entries is again two. The reason for
107 it is sometimes easier to get the thir
108 "Reader Batch" list than in the "Reade
109
110 * "Free-Block Circulation": Shows the nu
111 that have reached a given point in the
112 should closely correspond to the numbe
113 the second to the number that have bee
114 and all but the last remaining to the
115 passes through a grace period. The la
116 as it is only incremented if a torture
117 somehow gets incremented farther than
118
119 Different implementations of RCU can provide i
120 additional information. For example, Tree SRC
121 additional line::
122
123 srcud-torture: Tree SRCU per-CPU(idx=0
124
125 This line shows the per-CPU counter state, in
126 using a dynamically allocated srcu_struct (hen
127 "srcu-"). The numbers in parentheses are the
128 "current" counters for the corresponding CPU.
129 "old" and "current" values to the underlying a
130 debugging. The final "T" entry contains the t
131
132 Usage on Specific Kernel Builds
133 ===============================
134
135 It is sometimes desirable to torture RCU on a
136 for example, when preparing to put that kernel
137 In that case, the kernel should be built with
138 so that the test can be started using modprobe
139
140 For example, the following script may be used
141
142 #!/bin/sh
143
144 modprobe rcutorture
145 sleep 3600
146 rmmod rcutorture
147 dmesg | grep torture:
148
149 The output can be manually inspected for the e
150 One could of course create a more elaborate sc
151 checked for such errors. The "rmmod" command
152 "FAILURE", or "RCU_HOTPLUG" indication to be p
153 two are self-explanatory, while the last indic
154 were no RCU failures, CPU-hotplug problems wer
155
156
157 Usage on Mainline Kernels
158 =========================
159
160 When using rcutorture to test changes to RCU i
161 necessary to build a number of kernels in orde
162 across a broad range of combinations of the re
163 and of the relevant kernel boot parameters. I
164 of modprobe and rmmod can be quite time-consum
165
166 Therefore, the tools/testing/selftests/rcutort
167 script is available for mainline testing for x
168 powerpc. By default, it will run the series o
169 tools/testing/selftests/rcutorture/configs/rcu
170 running for 30 minutes within a guest OS using
171 supplied by an automatically generated initrd.
172 complete, the resulting build products and con
173 for errors and the results of the runs are sum
174
175 On larger systems, rcutorture testing can be a
176 --cpus argument to kvm.sh. For example, on a
177 would use up to 43 CPUs to run tests concurren
178 complete all the scenarios in two batches, red
179 from about eight hours to about one hour (not
180 the sixteen kernels). The "--dryrun sched" ar
181 but rather tell you how the tests would be sch
182 can be useful when working out how many CPUs t
183 argument.
184
185 Not all changes require that all scenarios be
186 to Tree SRCU might run only the SRCU-N and SRC
187 --configs argument to kvm.sh as follows: "--c
188 Large systems can run multiple copies of the f
189 for example, a system with 448 hardware thread
190 of the full set concurrently. To make this ha
191
192 kvm.sh --cpus 448 --configs '5*CFLIST'
193
194 Alternatively, such a system can run 56 concur
195 eight-CPU scenario::
196
197 kvm.sh --cpus 448 --configs '56*TREE04
198
199 Or 28 concurrent instances of each of two eigh
200
201 kvm.sh --cpus 448 --configs '28*TREE03
202
203 Of course, each concurrent instance will use m
204 limited using the --memory argument, which def
205 values for memory may require disabling the ca
206 using the --bootargs parameter discussed below
207
208 Sometimes additional debugging is useful, and
209 parameter to kvm.sh may be used, for example,
210 In addition, there are the --gdb, --kasan, and
211 Note that --gdb limits you to one scenario per
212 that you have another window open from which t
213 by the script.
214
215 Kernel boot arguments can also be supplied, fo
216 rcutorture's module parameters. For example,
217 CPU stall-warning code, use "--bootargs 'rcuto
218 This will of course result in the scripting re
219 the resulting RCU CPU stall warning. As noted
220 require disabling rcutorture's callback-floodi
221
222 kvm.sh --cpus 448 --configs '56*TREE04
223 --bootargs 'rcutorture.fwd_pro
224
225 Sometimes all that is needed is a full set of
226 what the --buildonly parameter does.
227
228 The --duration parameter can override the defa
229 For example, ``--duration 2d`` would run for t
230 would run for three hours, ``--duration 5m`` w
231 and ``--duration 45s`` would run for 45 second
232 for tracking down rare boot-time failures.
233
234 Finally, the --trust-make parameter allows eac
235 it can from the previous kernel build. Please
236 --trust-make parameter, your tags files may be
237
238 There are additional more arcane arguments tha
239 source code of the kvm.sh script.
240
241 If a run contains failures, the number of buil
242 is listed at the end of the kvm.sh output, whi
243 to a file. The build products and console out
244 tools/testing/selftests/rcutorture/res in time
245 given directory can be supplied to kvm-find-er
246 it cycle you through summaries of errors and f
247
248 tools/testing/selftests/rcutorture/bin
249 tools/testing/selftests/rcutor
250
251 However, it is often more convenient to access
252 Files pertaining to all scenarios in a run res
253 directory (2020.01.20-15.54.23 in the example
254 files reside in a subdirectory named after the
255 "TREE04"). If a given scenario ran more than
256 '56*TREE04'" above), the directories correspon
257 subsequent runs of that scenario include a seq
258 "TREE04.2", "TREE04.3", and so on.
259
260 The most frequently used file in the top-level
261 If the test ran in a git repository, then this
262 that was tested and any uncommitted changes in
263
264 The most frequently used files in each per-sce
265
266 .config:
267 This file contains the Kconfig options
268
269 Make.out:
270 This contains build output for a speci
271
272 console.log:
273 This contains the console output for a
274 This file may be examined once the ker
275 it might not exist if the build failed
276
277 vmlinux:
278 This contains the kernel, which can be
279 objdump and gdb.
280
281 A number of additional files are available, bu
282 Many are intended for debugging of rcutorture
283
284 As of v5.4, a successful run with the default
285 the following summary at the end of the run on
286
287 SRCU-N ------- 804233 GPs (148.932/s) [src
288 SRCU-P ------- 202320 GPs (37.4667/s) [src
289 SRCU-t ------- 1122086 GPs (207.794/s) [sr
290 SRCU-u ------- 1111285 GPs (205.794/s) [sr
291 TASKS01 ------- 19666 GPs (3.64185/s) [tas
292 TASKS02 ------- 20541 GPs (3.80389/s) [tas
293 TASKS03 ------- 19416 GPs (3.59556/s) [tas
294 TINY01 ------- 836134 GPs (154.84/s) [rcu:
295 TINY02 ------- 850371 GPs (157.476/s) [rcu
296 TREE01 ------- 162625 GPs (30.1157/s) [rcu
297 TREE02 ------- 333003 GPs (61.6672/s) [rcu
298 TREE03 ------- 306623 GPs (56.782/s) [rcu:
299 CPU count limited from 16 to 12
300 TREE04 ------- 246149 GPs (45.5831/s) [rcu
301 TREE05 ------- 314603 GPs (58.2598/s) [rcu
302 TREE07 ------- 167347 GPs (30.9902/s) [rcu
303 CPU count limited from 16 to 12
304 TREE09 ------- 752238 GPs (139.303/s) [rcu
305
306
307 Repeated Runs
308 =============
309
310 Suppose that you are chasing down a rare boot-
311 could use kvm.sh, doing so will rebuild the ke
312 need (say) 1,000 runs to have confidence that
313 these pointless rebuilds can become extremely
314
315 This is why kvm-again.sh exists.
316
317 Suppose that a previous kvm.sh run left its ou
318
319 tools/testing/selftests/rcutorture/res
320
321 Then this run can be re-run without rebuilding
322
323 kvm-again.sh tools/testing/selftests/r
324
325 A few of the original run's kvm.sh parameters
326 most notably --duration and --bootargs. For e
327
328 kvm-again.sh tools/testing/selftests/r
329 --duration 45s
330
331 would re-run the previous test, but for only 4
332 tracking down the aforementioned rare boot-tim
333
334
335 Distributed Runs
336 ================
337
338 Although kvm.sh is quite useful, its testing i
339 system. It is not all that hard to use your f
340 (say) 5 instances of kvm.sh to run on your 5 s
341 likely unnecessarily rebuild kernels. In addi
342 the desired rcutorture scenarios across the av
343 painstaking and error-prone.
344
345 And this is why the kvm-remote.sh script exist
346
347 If you the following command works::
348
349 ssh system0 date
350
351 and if it also works for system1, system2, sys
352 and all of these systems have 64 CPUs, you can
353
354 kvm-remote.sh "system0 system1 system2
355 --cpus 64 --duration 8h --conf
356
357 This will build each default scenario's kernel
358 spread each of five instances of each scenario
359 running each scenario for eight hours. At the
360 results will be gathered, recorded, and printe
361 that kvm.sh will accept can be passed to kvm-r
362 systems must come first.
363
364 The kvm.sh ``--dryrun scenarios`` argument is
365 how many scenarios may be run in one batch acr
366
367 You can also re-run a previous remote run in a
368
369 kvm-remote.sh "system0 system1 system2
370 tools/testing/selftests/rcutor
371 --duration 24h
372
373 In this case, most of the kvm-again.sh paramet
374 the pathname of the old run-results directory.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.