1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ============================ 3 ============================
4 Tips For Running KUnit Tests 4 Tips For Running KUnit Tests
5 ============================ 5 ============================
6 6
7 Using ``kunit.py run`` ("kunit tool") 7 Using ``kunit.py run`` ("kunit tool")
8 ===================================== 8 =====================================
9 9
10 Running from any directory 10 Running from any directory
11 -------------------------- 11 --------------------------
12 12
13 It can be handy to create a bash function like 13 It can be handy to create a bash function like:
14 14
15 .. code-block:: bash 15 .. code-block:: bash
16 16
17 function run_kunit() { 17 function run_kunit() {
18 ( cd "$(git rev-parse --show-topleve !! 18 ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run $@ )
19 } 19 }
20 20
21 .. note:: 21 .. note::
22 Early versions of ``kunit.py`` (before 22 Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
23 the kernel root, hence the use of a su 23 the kernel root, hence the use of a subshell and ``cd``.
24 24
25 Running a subset of tests 25 Running a subset of tests
26 ------------------------- 26 -------------------------
27 27
28 ``kunit.py run`` accepts an optional glob argu !! 28 ``kunit.py run`` accepts an optional glob argument to filter tests. Currently
29 is ``"<suite_glob>[.test_glob]"``. !! 29 this only matches against suite names, but this may change in the future.
30 30
31 Say that we wanted to run the sysctl tests, we 31 Say that we wanted to run the sysctl tests, we could do so via:
32 32
33 .. code-block:: bash 33 .. code-block:: bash
34 34
35 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNI 35 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig
36 $ ./tools/testing/kunit/kunit.py run ' 36 $ ./tools/testing/kunit/kunit.py run 'sysctl*'
37 37
38 We can filter down to just the "write" tests v <<
39 <<
40 .. code-block:: bash <<
41 <<
42 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNI <<
43 $ ./tools/testing/kunit/kunit.py run ' <<
44 <<
45 We're paying the cost of building more tests t 38 We're paying the cost of building more tests than we need this way, but it's
46 easier than fiddling with ``.kunitconfig`` fil 39 easier than fiddling with ``.kunitconfig`` files or commenting out
47 ``kunit_suite``'s. 40 ``kunit_suite``'s.
48 41
49 However, if we wanted to define a set of tests 42 However, if we wanted to define a set of tests in a less ad hoc way, the next
50 tip is useful. 43 tip is useful.
51 44
52 Defining a set of tests 45 Defining a set of tests
53 ----------------------- 46 -----------------------
54 47
55 ``kunit.py run`` (along with ``build``, and `` 48 ``kunit.py run`` (along with ``build``, and ``config``) supports a
56 ``--kunitconfig`` flag. So if you have a set o 49 ``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
57 regular basis (especially if they have other d 50 regular basis (especially if they have other dependencies), you can create a
58 specific ``.kunitconfig`` for them. 51 specific ``.kunitconfig`` for them.
59 52
60 E.g. kunit has one for its tests: 53 E.g. kunit has one for its tests:
61 54
62 .. code-block:: bash 55 .. code-block:: bash
63 56
64 $ ./tools/testing/kunit/kunit.py run - 57 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig
65 58
66 Alternatively, if you're following the convent 59 Alternatively, if you're following the convention of naming your
67 file ``.kunitconfig``, you can just pass in th 60 file ``.kunitconfig``, you can just pass in the dir, e.g.
68 61
69 .. code-block:: bash 62 .. code-block:: bash
70 63
71 $ ./tools/testing/kunit/kunit.py run - 64 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
72 65
73 .. note:: 66 .. note::
74 This is a relatively new feature (5.12 67 This is a relatively new feature (5.12+) so we don't have any
75 conventions yet about on what files sh 68 conventions yet about on what files should be checked in versus just
76 kept around locally. It's up to you an 69 kept around locally. It's up to you and your maintainer to decide if a
77 config is useful enough to submit (and 70 config is useful enough to submit (and therefore have to maintain).
78 71
79 .. note:: 72 .. note::
80 Having ``.kunitconfig`` fragments in a 73 Having ``.kunitconfig`` fragments in a parent and child directory is
81 iffy. There's discussion about adding 74 iffy. There's discussion about adding an "import" statement in these
82 files to make it possible to have a to 75 files to make it possible to have a top-level config run tests from all
83 child directories. But that would mean 76 child directories. But that would mean ``.kunitconfig`` files are no
84 longer just simple .config fragments. 77 longer just simple .config fragments.
85 78
86 One alternative would be to have kunit 79 One alternative would be to have kunit tool recursively combine configs
87 automagically, but tests could theoret 80 automagically, but tests could theoretically depend on incompatible
88 options, so handling that would be tri 81 options, so handling that would be tricky.
89 82
90 Setting kernel commandline parameters <<
91 ------------------------------------- <<
92 <<
93 You can use ``--kernel_args`` to pass arbitrar <<
94 <<
95 .. code-block:: bash <<
96 <<
97 $ ./tools/testing/kunit/kunit.py run - <<
98 <<
99 <<
100 Generating code coverage reports under UML 83 Generating code coverage reports under UML
101 ------------------------------------------ 84 ------------------------------------------
102 85
103 .. note:: 86 .. note::
104 TODO(brendanhiggins@google.com): There 87 TODO(brendanhiggins@google.com): There are various issues with UML and
105 versions of gcc 7 and up. You're likel 88 versions of gcc 7 and up. You're likely to run into missing ``.gcda``
106 files or compile errors. 89 files or compile errors.
107 90
108 This is different from the "normal" way of get 91 This is different from the "normal" way of getting coverage information that is
109 documented in Documentation/dev-tools/gcov.rst 92 documented in Documentation/dev-tools/gcov.rst.
110 93
111 Instead of enabling ``CONFIG_GCOV_KERNEL=y``, 94 Instead of enabling ``CONFIG_GCOV_KERNEL=y``, we can set these options:
112 95
113 .. code-block:: none 96 .. code-block:: none
114 97
115 CONFIG_DEBUG_KERNEL=y 98 CONFIG_DEBUG_KERNEL=y
116 CONFIG_DEBUG_INFO=y 99 CONFIG_DEBUG_INFO=y
117 CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFA <<
118 CONFIG_GCOV=y 100 CONFIG_GCOV=y
119 101
120 102
121 Putting it together into a copy-pastable seque 103 Putting it together into a copy-pastable sequence of commands:
122 104
123 .. code-block:: bash 105 .. code-block:: bash
124 106
125 # Append coverage options to the curre 107 # Append coverage options to the current config
126 $ ./tools/testing/kunit/kunit.py run - !! 108 $ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig
>> 109 $ ./tools/testing/kunit/kunit.py run
127 # Extract the coverage information fro 110 # Extract the coverage information from the build dir (.kunit/)
128 $ lcov -t "my_kunit_tests" -o coverage 111 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/
129 112
130 # From here on, it's the same process 113 # From here on, it's the same process as with CONFIG_GCOV_KERNEL=y
131 # E.g. can generate an HTML report in 114 # E.g. can generate an HTML report in a tmp dir like so:
132 $ genhtml -o /tmp/coverage_html covera 115 $ genhtml -o /tmp/coverage_html coverage.info
133 116
134 117
135 If your installed version of gcc doesn't work, 118 If your installed version of gcc doesn't work, you can tweak the steps:
136 119
137 .. code-block:: bash 120 .. code-block:: bash
138 121
139 $ ./tools/testing/kunit/kunit.py run - 122 $ ./tools/testing/kunit/kunit.py run --make_options=CC=/usr/bin/gcc-6
140 $ lcov -t "my_kunit_tests" -o coverage 123 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcov-6
141 124
142 Alternatively, LLVM-based toolchains can also <<
143 <<
144 .. code-block:: bash <<
145 <<
146 # Build with LLVM and append coverage <<
147 $ ./tools/testing/kunit/kunit.py run - <<
148 $ llvm-profdata merge -sparse default. <<
149 $ llvm-cov export --format=lcov .kunit <<
150 # The coverage.info file is in lcov-co <<
151 $ genhtml -o /tmp/coverage_html covera <<
152 <<
153 125
154 Running tests manually 126 Running tests manually
155 ====================== 127 ======================
156 128
157 Running tests without using ``kunit.py run`` i 129 Running tests without using ``kunit.py run`` is also an important use case.
158 Currently it's your only option if you want to 130 Currently it's your only option if you want to test on architectures other than
159 UML. 131 UML.
160 132
161 As running the tests under UML is fairly strai 133 As running the tests under UML is fairly straightforward (configure and compile
162 the kernel, run the ``./linux`` binary), this 134 the kernel, run the ``./linux`` binary), this section will focus on testing
163 non-UML architectures. 135 non-UML architectures.
164 136
165 137
166 Running built-in tests 138 Running built-in tests
167 ---------------------- 139 ----------------------
168 140
169 When setting tests to ``=y``, the tests will r 141 When setting tests to ``=y``, the tests will run as part of boot and print
170 results to dmesg in TAP format. So you just ne 142 results to dmesg in TAP format. So you just need to add your tests to your
171 ``.config``, build and boot your kernel as nor 143 ``.config``, build and boot your kernel as normal.
172 144
173 So if we compiled our kernel with: 145 So if we compiled our kernel with:
174 146
175 .. code-block:: none 147 .. code-block:: none
176 148
177 CONFIG_KUNIT=y 149 CONFIG_KUNIT=y
178 CONFIG_KUNIT_EXAMPLE_TEST=y 150 CONFIG_KUNIT_EXAMPLE_TEST=y
179 151
180 Then we'd see output like this in dmesg signal 152 Then we'd see output like this in dmesg signaling the test ran and passed:
181 153
182 .. code-block:: none 154 .. code-block:: none
183 155
184 TAP version 14 156 TAP version 14
185 1..1 157 1..1
186 # Subtest: example 158 # Subtest: example
187 1..1 159 1..1
188 # example_simple_test: initializin 160 # example_simple_test: initializing
189 ok 1 - example_simple_test 161 ok 1 - example_simple_test
190 ok 1 - example 162 ok 1 - example
191 163
192 Running tests as modules 164 Running tests as modules
193 ------------------------ 165 ------------------------
194 166
195 Depending on the tests, you can build them as 167 Depending on the tests, you can build them as loadable modules.
196 168
197 For example, we'd change the config options fr 169 For example, we'd change the config options from before to
198 170
199 .. code-block:: none 171 .. code-block:: none
200 172
201 CONFIG_KUNIT=y 173 CONFIG_KUNIT=y
202 CONFIG_KUNIT_EXAMPLE_TEST=m 174 CONFIG_KUNIT_EXAMPLE_TEST=m
203 175
204 Then after booting into our kernel, we can run 176 Then after booting into our kernel, we can run the test via
205 177
206 .. code-block:: none 178 .. code-block:: none
207 179
208 $ modprobe kunit-example-test 180 $ modprobe kunit-example-test
209 181
210 This will then cause it to print TAP output to 182 This will then cause it to print TAP output to stdout.
211 183
212 .. note:: 184 .. note::
213 The ``modprobe`` will *not* have a non 185 The ``modprobe`` will *not* have a non-zero exit code if any test
214 failed (as of 5.13). But ``kunit.py pa 186 failed (as of 5.13). But ``kunit.py parse`` would, see below.
215 187
216 .. note:: 188 .. note::
217 You can set ``CONFIG_KUNIT=m`` as well 189 You can set ``CONFIG_KUNIT=m`` as well, however, some features will not
218 work and thus some tests might break. 190 work and thus some tests might break. Ideally tests would specify they
219 depend on ``KUNIT=y`` in their ``Kconf 191 depend on ``KUNIT=y`` in their ``Kconfig``'s, but this is an edge case
220 most test authors won't think about. 192 most test authors won't think about.
221 As of 5.13, the only difference is tha 193 As of 5.13, the only difference is that ``current->kunit_test`` will
222 not exist. 194 not exist.
223 195
224 Pretty-printing results 196 Pretty-printing results
225 ----------------------- 197 -----------------------
226 198
227 You can use ``kunit.py parse`` to parse dmesg 199 You can use ``kunit.py parse`` to parse dmesg for test output and print out
228 results in the same familiar format that ``kun 200 results in the same familiar format that ``kunit.py run`` does.
229 201
230 .. code-block:: bash 202 .. code-block:: bash
231 203
232 $ ./tools/testing/kunit/kunit.py parse 204 $ ./tools/testing/kunit/kunit.py parse /var/log/dmesg
233 205
234 206
235 Retrieving per suite results 207 Retrieving per suite results
236 ---------------------------- 208 ----------------------------
237 209
238 Regardless of how you're running your tests, y 210 Regardless of how you're running your tests, you can enable
239 ``CONFIG_KUNIT_DEBUGFS`` to expose per-suite T 211 ``CONFIG_KUNIT_DEBUGFS`` to expose per-suite TAP-formatted results:
240 212
241 .. code-block:: none 213 .. code-block:: none
242 214
243 CONFIG_KUNIT=y 215 CONFIG_KUNIT=y
244 CONFIG_KUNIT_EXAMPLE_TEST=m 216 CONFIG_KUNIT_EXAMPLE_TEST=m
245 CONFIG_KUNIT_DEBUGFS=y 217 CONFIG_KUNIT_DEBUGFS=y
246 218
247 The results for each suite will be exposed und 219 The results for each suite will be exposed under
248 ``/sys/kernel/debug/kunit/<suite>/results``. 220 ``/sys/kernel/debug/kunit/<suite>/results``.
249 So using our example config: 221 So using our example config:
250 222
251 .. code-block:: bash 223 .. code-block:: bash
252 224
253 $ modprobe kunit-example-test > /dev/n 225 $ modprobe kunit-example-test > /dev/null
254 $ cat /sys/kernel/debug/kunit/example/ 226 $ cat /sys/kernel/debug/kunit/example/results
255 ... <TAP output> ... 227 ... <TAP output> ...
256 228
257 # After removing the module, the corre 229 # After removing the module, the corresponding files will go away
258 $ modprobe -r kunit-example-test 230 $ modprobe -r kunit-example-test
259 $ cat /sys/kernel/debug/kunit/example/ 231 $ cat /sys/kernel/debug/kunit/example/results
260 /sys/kernel/debug/kunit/example/result 232 /sys/kernel/debug/kunit/example/results: No such file or directory
261 233
262 Generating code coverage reports 234 Generating code coverage reports
263 -------------------------------- 235 --------------------------------
264 236
265 See Documentation/dev-tools/gcov.rst for detai 237 See Documentation/dev-tools/gcov.rst for details on how to do this.
266 238
267 The only vaguely KUnit-specific advice here is 239 The only vaguely KUnit-specific advice here is that you probably want to build
268 your tests as modules. That way you can isolat 240 your tests as modules. That way you can isolate the coverage from tests from
269 other code executed during boot, e.g. 241 other code executed during boot, e.g.
270 242
271 .. code-block:: bash 243 .. code-block:: bash
272 244
273 # Reset coverage counters before runni 245 # Reset coverage counters before running the test.
274 $ echo 0 > /sys/kernel/debug/gcov/rese 246 $ echo 0 > /sys/kernel/debug/gcov/reset
275 $ modprobe kunit-example-test 247 $ modprobe kunit-example-test
276 <<
277 <<
278 Test Attributes and Filtering <<
279 ============================= <<
280 <<
281 Test suites and cases can be marked with test <<
282 test. These attributes will later be printed i <<
283 filter test execution. <<
284 <<
285 Marking Test Attributes <<
286 ----------------------- <<
287 <<
288 Tests are marked with an attribute by includin <<
289 in the test definition. <<
290 <<
291 Test cases can be marked using the ``KUNIT_CAS <<
292 macro to define the test case instead of ``KUN <<
293 <<
294 .. code-block:: c <<
295 <<
296 static const struct kunit_attributes e <<
297 .speed = KUNIT_VERY_SLOW, <<
298 }; <<
299 <<
300 static struct kunit_case example_test_ <<
301 KUNIT_CASE_ATTR(example_test, <<
302 }; <<
303 <<
304 .. note:: <<
305 To mark a test case as slow, you can a <<
306 This is a helpful macro as the slow at <<
307 <<
308 Test suites can be marked with an attribute by <<
309 suite definition. <<
310 <<
311 .. code-block:: c <<
312 <<
313 static const struct kunit_attributes e <<
314 .speed = KUNIT_VERY_SLOW, <<
315 }; <<
316 <<
317 static struct kunit_suite example_test <<
318 ..., <<
319 .attr = example_attr, <<
320 }; <<
321 <<
322 .. note:: <<
323 Not all attributes need to be set in a <<
324 attributes will remain uninitialized a <<
325 to 0 or NULL. Thus, if an attribute is <<
326 These unset attributes will not be rep <<
327 for filtering purposes. <<
328 <<
329 Reporting Attributes <<
330 -------------------- <<
331 <<
332 When a user runs tests, attributes will be pre <<
333 KTAP format). Note that attributes will be hid <<
334 for all passing tests but the raw kernel outpu <<
335 ``--raw_output`` flag. This is an example of h <<
336 will be formatted in kernel output: <<
337 <<
338 .. code-block:: none <<
339 <<
340 # example_test.speed: slow <<
341 ok 1 example_test <<
342 <<
343 This is an example of how test attributes for <<
344 kernel output: <<
345 <<
346 .. code-block:: none <<
347 <<
348 KTAP version 2 <<
349 # Subtest: example_suite <<
350 # module: kunit_example_test <<
351 1..3 <<
352 ... <<
353 ok 1 example_suite <<
354 <<
355 Additionally, users can output a full attribut <<
356 attributes, using the command line flag ``--li <<
357 <<
358 .. code-block:: bash <<
359 <<
360 kunit.py run "example" --list_tests_at <<
361 <<
362 .. note:: <<
363 This report can be accessed when runni <<
364 module_param ``kunit.action=list_attr` <<
365 <<
366 Filtering <<
367 --------- <<
368 <<
369 Users can filter tests using the ``--filter`` <<
370 tests. As an example: <<
371 <<
372 .. code-block:: bash <<
373 <<
374 kunit.py run --filter speed=slow <<
375 <<
376 <<
377 You can also use the following operations on f <<
378 "!=", and "=". Example: <<
379 <<
380 .. code-block:: bash <<
381 <<
382 kunit.py run --filter "speed>slow" <<
383 <<
384 This example will run all tests with speeds fa <<
385 characters < and > are often interpreted by th <<
386 quoted or escaped, as above. <<
387 <<
388 Additionally, you can use multiple filters at <<
389 using commas. Example: <<
390 <<
391 .. code-block:: bash <<
392 <<
393 kunit.py run --filter "speed>slow, mod <<
394 <<
395 .. note:: <<
396 You can use this filtering feature whe <<
397 the filter as a module param: ``kunit. <<
398 <<
399 Filtered tests will not run or show up in the <<
400 ``--filter_action=skip`` flag to skip filtered <<
401 shown in the test output in the test but will <<
402 running KUnit manually, use the module param ` <<
403 <<
404 Rules of Filtering Procedure <<
405 ---------------------------- <<
406 <<
407 Since both suites and test cases can have attr <<
408 between attributes during filtering. The proce <<
409 rules: <<
410 <<
411 - Filtering always operates at a per-test leve <<
412 <<
413 - If a test has an attribute set, then the tes <<
414 <<
415 - Otherwise, the value falls back to the suite <<
416 <<
417 - If neither are set, the attribute has a glob <<
418 <<
419 List of Current Attributes <<
420 -------------------------- <<
421 <<
422 ``speed`` <<
423 <<
424 This attribute indicates the speed of a test's <<
425 test is). <<
426 <<
427 This attribute is saved as an enum with the fo <<
428 "slow", or "very_slow". The assumed default sp <<
429 indicates that the test takes a relatively tri <<
430 1 second), regardless of the machine it is run <<
431 this could be marked as "slow" or "very_slow". <<
432 <<
433 The macro ``KUNIT_CASE_SLOW(test_name)`` can b <<
434 of a test case to "slow". <<
435 <<
436 ``module`` <<
437 <<
438 This attribute indicates the name of the modul <<
439 <<
440 This attribute is automatically saved as a str <<
441 Tests can also be filtered using this attribut <<
442 <<
443 ``is_init`` <<
444 <<
445 This attribute indicates whether the test uses <<
446 <<
447 This attribute is automatically saved as a boo <<
448 filtered using this attribute. <<
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.