1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ============================ 3 ============================
4 Tips For Running KUnit Tests 4 Tips For Running KUnit Tests
5 ============================ 5 ============================
6 6
7 Using ``kunit.py run`` ("kunit tool") 7 Using ``kunit.py run`` ("kunit tool")
8 ===================================== 8 =====================================
9 9
10 Running from any directory 10 Running from any directory
11 -------------------------- 11 --------------------------
12 12
13 It can be handy to create a bash function like 13 It can be handy to create a bash function like:
14 14
15 .. code-block:: bash 15 .. code-block:: bash
16 16
17 function run_kunit() { 17 function run_kunit() {
18 ( cd "$(git rev-parse --show-topleve 18 ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run "$@" )
19 } 19 }
20 20
21 .. note:: 21 .. note::
22 Early versions of ``kunit.py`` (before 22 Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
23 the kernel root, hence the use of a su 23 the kernel root, hence the use of a subshell and ``cd``.
24 24
25 Running a subset of tests 25 Running a subset of tests
26 ------------------------- 26 -------------------------
27 27
28 ``kunit.py run`` accepts an optional glob argu 28 ``kunit.py run`` accepts an optional glob argument to filter tests. The format
29 is ``"<suite_glob>[.test_glob]"``. 29 is ``"<suite_glob>[.test_glob]"``.
30 30
31 Say that we wanted to run the sysctl tests, we 31 Say that we wanted to run the sysctl tests, we could do so via:
32 32
33 .. code-block:: bash 33 .. code-block:: bash
34 34
35 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNI 35 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig
36 $ ./tools/testing/kunit/kunit.py run ' 36 $ ./tools/testing/kunit/kunit.py run 'sysctl*'
37 37
38 We can filter down to just the "write" tests v 38 We can filter down to just the "write" tests via:
39 39
40 .. code-block:: bash 40 .. code-block:: bash
41 41
42 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNI 42 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig
43 $ ./tools/testing/kunit/kunit.py run ' 43 $ ./tools/testing/kunit/kunit.py run 'sysctl*.*write*'
44 44
45 We're paying the cost of building more tests t 45 We're paying the cost of building more tests than we need this way, but it's
46 easier than fiddling with ``.kunitconfig`` fil 46 easier than fiddling with ``.kunitconfig`` files or commenting out
47 ``kunit_suite``'s. 47 ``kunit_suite``'s.
48 48
49 However, if we wanted to define a set of tests 49 However, if we wanted to define a set of tests in a less ad hoc way, the next
50 tip is useful. 50 tip is useful.
51 51
52 Defining a set of tests 52 Defining a set of tests
53 ----------------------- 53 -----------------------
54 54
55 ``kunit.py run`` (along with ``build``, and `` 55 ``kunit.py run`` (along with ``build``, and ``config``) supports a
56 ``--kunitconfig`` flag. So if you have a set o 56 ``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
57 regular basis (especially if they have other d 57 regular basis (especially if they have other dependencies), you can create a
58 specific ``.kunitconfig`` for them. 58 specific ``.kunitconfig`` for them.
59 59
60 E.g. kunit has one for its tests: 60 E.g. kunit has one for its tests:
61 61
62 .. code-block:: bash 62 .. code-block:: bash
63 63
64 $ ./tools/testing/kunit/kunit.py run - 64 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig
65 65
66 Alternatively, if you're following the convent 66 Alternatively, if you're following the convention of naming your
67 file ``.kunitconfig``, you can just pass in th 67 file ``.kunitconfig``, you can just pass in the dir, e.g.
68 68
69 .. code-block:: bash 69 .. code-block:: bash
70 70
71 $ ./tools/testing/kunit/kunit.py run - 71 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
72 72
73 .. note:: 73 .. note::
74 This is a relatively new feature (5.12 74 This is a relatively new feature (5.12+) so we don't have any
75 conventions yet about on what files sh 75 conventions yet about on what files should be checked in versus just
76 kept around locally. It's up to you an 76 kept around locally. It's up to you and your maintainer to decide if a
77 config is useful enough to submit (and 77 config is useful enough to submit (and therefore have to maintain).
78 78
79 .. note:: 79 .. note::
80 Having ``.kunitconfig`` fragments in a 80 Having ``.kunitconfig`` fragments in a parent and child directory is
81 iffy. There's discussion about adding 81 iffy. There's discussion about adding an "import" statement in these
82 files to make it possible to have a to 82 files to make it possible to have a top-level config run tests from all
83 child directories. But that would mean 83 child directories. But that would mean ``.kunitconfig`` files are no
84 longer just simple .config fragments. 84 longer just simple .config fragments.
85 85
86 One alternative would be to have kunit 86 One alternative would be to have kunit tool recursively combine configs
87 automagically, but tests could theoret 87 automagically, but tests could theoretically depend on incompatible
88 options, so handling that would be tri 88 options, so handling that would be tricky.
89 89
90 Setting kernel commandline parameters 90 Setting kernel commandline parameters
91 ------------------------------------- 91 -------------------------------------
92 92
93 You can use ``--kernel_args`` to pass arbitrar 93 You can use ``--kernel_args`` to pass arbitrary kernel arguments, e.g.
94 94
95 .. code-block:: bash 95 .. code-block:: bash
96 96
97 $ ./tools/testing/kunit/kunit.py run - 97 $ ./tools/testing/kunit/kunit.py run --kernel_args=param=42 --kernel_args=param2=false
98 98
99 99
100 Generating code coverage reports under UML 100 Generating code coverage reports under UML
101 ------------------------------------------ 101 ------------------------------------------
102 102
103 .. note:: 103 .. note::
104 TODO(brendanhiggins@google.com): There 104 TODO(brendanhiggins@google.com): There are various issues with UML and
105 versions of gcc 7 and up. You're likel 105 versions of gcc 7 and up. You're likely to run into missing ``.gcda``
106 files or compile errors. 106 files or compile errors.
107 107
108 This is different from the "normal" way of get 108 This is different from the "normal" way of getting coverage information that is
109 documented in Documentation/dev-tools/gcov.rst 109 documented in Documentation/dev-tools/gcov.rst.
110 110
111 Instead of enabling ``CONFIG_GCOV_KERNEL=y``, 111 Instead of enabling ``CONFIG_GCOV_KERNEL=y``, we can set these options:
112 112
113 .. code-block:: none 113 .. code-block:: none
114 114
115 CONFIG_DEBUG_KERNEL=y 115 CONFIG_DEBUG_KERNEL=y
116 CONFIG_DEBUG_INFO=y 116 CONFIG_DEBUG_INFO=y
117 CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFA 117 CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
118 CONFIG_GCOV=y 118 CONFIG_GCOV=y
119 119
120 120
121 Putting it together into a copy-pastable seque 121 Putting it together into a copy-pastable sequence of commands:
122 122
123 .. code-block:: bash 123 .. code-block:: bash
124 124
125 # Append coverage options to the curre 125 # Append coverage options to the current config
126 $ ./tools/testing/kunit/kunit.py run - 126 $ ./tools/testing/kunit/kunit.py run --kunitconfig=.kunit/ --kunitconfig=tools/testing/kunit/configs/coverage_uml.config
127 # Extract the coverage information fro 127 # Extract the coverage information from the build dir (.kunit/)
128 $ lcov -t "my_kunit_tests" -o coverage 128 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/
129 129
130 # From here on, it's the same process 130 # From here on, it's the same process as with CONFIG_GCOV_KERNEL=y
131 # E.g. can generate an HTML report in 131 # E.g. can generate an HTML report in a tmp dir like so:
132 $ genhtml -o /tmp/coverage_html covera 132 $ genhtml -o /tmp/coverage_html coverage.info
133 133
134 134
135 If your installed version of gcc doesn't work, 135 If your installed version of gcc doesn't work, you can tweak the steps:
136 136
137 .. code-block:: bash 137 .. code-block:: bash
138 138
139 $ ./tools/testing/kunit/kunit.py run - 139 $ ./tools/testing/kunit/kunit.py run --make_options=CC=/usr/bin/gcc-6
140 $ lcov -t "my_kunit_tests" -o coverage 140 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcov-6
141 141
142 Alternatively, LLVM-based toolchains can also <<
143 <<
144 .. code-block:: bash <<
145 <<
146 # Build with LLVM and append coverage <<
147 $ ./tools/testing/kunit/kunit.py run - <<
148 $ llvm-profdata merge -sparse default. <<
149 $ llvm-cov export --format=lcov .kunit <<
150 # The coverage.info file is in lcov-co <<
151 $ genhtml -o /tmp/coverage_html covera <<
152 <<
153 142
154 Running tests manually 143 Running tests manually
155 ====================== 144 ======================
156 145
157 Running tests without using ``kunit.py run`` i 146 Running tests without using ``kunit.py run`` is also an important use case.
158 Currently it's your only option if you want to 147 Currently it's your only option if you want to test on architectures other than
159 UML. 148 UML.
160 149
161 As running the tests under UML is fairly strai 150 As running the tests under UML is fairly straightforward (configure and compile
162 the kernel, run the ``./linux`` binary), this 151 the kernel, run the ``./linux`` binary), this section will focus on testing
163 non-UML architectures. 152 non-UML architectures.
164 153
165 154
166 Running built-in tests 155 Running built-in tests
167 ---------------------- 156 ----------------------
168 157
169 When setting tests to ``=y``, the tests will r 158 When setting tests to ``=y``, the tests will run as part of boot and print
170 results to dmesg in TAP format. So you just ne 159 results to dmesg in TAP format. So you just need to add your tests to your
171 ``.config``, build and boot your kernel as nor 160 ``.config``, build and boot your kernel as normal.
172 161
173 So if we compiled our kernel with: 162 So if we compiled our kernel with:
174 163
175 .. code-block:: none 164 .. code-block:: none
176 165
177 CONFIG_KUNIT=y 166 CONFIG_KUNIT=y
178 CONFIG_KUNIT_EXAMPLE_TEST=y 167 CONFIG_KUNIT_EXAMPLE_TEST=y
179 168
180 Then we'd see output like this in dmesg signal 169 Then we'd see output like this in dmesg signaling the test ran and passed:
181 170
182 .. code-block:: none 171 .. code-block:: none
183 172
184 TAP version 14 173 TAP version 14
185 1..1 174 1..1
186 # Subtest: example 175 # Subtest: example
187 1..1 176 1..1
188 # example_simple_test: initializin 177 # example_simple_test: initializing
189 ok 1 - example_simple_test 178 ok 1 - example_simple_test
190 ok 1 - example 179 ok 1 - example
191 180
192 Running tests as modules 181 Running tests as modules
193 ------------------------ 182 ------------------------
194 183
195 Depending on the tests, you can build them as 184 Depending on the tests, you can build them as loadable modules.
196 185
197 For example, we'd change the config options fr 186 For example, we'd change the config options from before to
198 187
199 .. code-block:: none 188 .. code-block:: none
200 189
201 CONFIG_KUNIT=y 190 CONFIG_KUNIT=y
202 CONFIG_KUNIT_EXAMPLE_TEST=m 191 CONFIG_KUNIT_EXAMPLE_TEST=m
203 192
204 Then after booting into our kernel, we can run 193 Then after booting into our kernel, we can run the test via
205 194
206 .. code-block:: none 195 .. code-block:: none
207 196
208 $ modprobe kunit-example-test 197 $ modprobe kunit-example-test
209 198
210 This will then cause it to print TAP output to 199 This will then cause it to print TAP output to stdout.
211 200
212 .. note:: 201 .. note::
213 The ``modprobe`` will *not* have a non 202 The ``modprobe`` will *not* have a non-zero exit code if any test
214 failed (as of 5.13). But ``kunit.py pa 203 failed (as of 5.13). But ``kunit.py parse`` would, see below.
215 204
216 .. note:: 205 .. note::
217 You can set ``CONFIG_KUNIT=m`` as well 206 You can set ``CONFIG_KUNIT=m`` as well, however, some features will not
218 work and thus some tests might break. 207 work and thus some tests might break. Ideally tests would specify they
219 depend on ``KUNIT=y`` in their ``Kconf 208 depend on ``KUNIT=y`` in their ``Kconfig``'s, but this is an edge case
220 most test authors won't think about. 209 most test authors won't think about.
221 As of 5.13, the only difference is tha 210 As of 5.13, the only difference is that ``current->kunit_test`` will
222 not exist. 211 not exist.
223 212
224 Pretty-printing results 213 Pretty-printing results
225 ----------------------- 214 -----------------------
226 215
227 You can use ``kunit.py parse`` to parse dmesg 216 You can use ``kunit.py parse`` to parse dmesg for test output and print out
228 results in the same familiar format that ``kun 217 results in the same familiar format that ``kunit.py run`` does.
229 218
230 .. code-block:: bash 219 .. code-block:: bash
231 220
232 $ ./tools/testing/kunit/kunit.py parse 221 $ ./tools/testing/kunit/kunit.py parse /var/log/dmesg
233 222
234 223
235 Retrieving per suite results 224 Retrieving per suite results
236 ---------------------------- 225 ----------------------------
237 226
238 Regardless of how you're running your tests, y 227 Regardless of how you're running your tests, you can enable
239 ``CONFIG_KUNIT_DEBUGFS`` to expose per-suite T 228 ``CONFIG_KUNIT_DEBUGFS`` to expose per-suite TAP-formatted results:
240 229
241 .. code-block:: none 230 .. code-block:: none
242 231
243 CONFIG_KUNIT=y 232 CONFIG_KUNIT=y
244 CONFIG_KUNIT_EXAMPLE_TEST=m 233 CONFIG_KUNIT_EXAMPLE_TEST=m
245 CONFIG_KUNIT_DEBUGFS=y 234 CONFIG_KUNIT_DEBUGFS=y
246 235
247 The results for each suite will be exposed und 236 The results for each suite will be exposed under
248 ``/sys/kernel/debug/kunit/<suite>/results``. 237 ``/sys/kernel/debug/kunit/<suite>/results``.
249 So using our example config: 238 So using our example config:
250 239
251 .. code-block:: bash 240 .. code-block:: bash
252 241
253 $ modprobe kunit-example-test > /dev/n 242 $ modprobe kunit-example-test > /dev/null
254 $ cat /sys/kernel/debug/kunit/example/ 243 $ cat /sys/kernel/debug/kunit/example/results
255 ... <TAP output> ... 244 ... <TAP output> ...
256 245
257 # After removing the module, the corre 246 # After removing the module, the corresponding files will go away
258 $ modprobe -r kunit-example-test 247 $ modprobe -r kunit-example-test
259 $ cat /sys/kernel/debug/kunit/example/ 248 $ cat /sys/kernel/debug/kunit/example/results
260 /sys/kernel/debug/kunit/example/result 249 /sys/kernel/debug/kunit/example/results: No such file or directory
261 250
262 Generating code coverage reports 251 Generating code coverage reports
263 -------------------------------- 252 --------------------------------
264 253
265 See Documentation/dev-tools/gcov.rst for detai 254 See Documentation/dev-tools/gcov.rst for details on how to do this.
266 255
267 The only vaguely KUnit-specific advice here is 256 The only vaguely KUnit-specific advice here is that you probably want to build
268 your tests as modules. That way you can isolat 257 your tests as modules. That way you can isolate the coverage from tests from
269 other code executed during boot, e.g. 258 other code executed during boot, e.g.
270 259
271 .. code-block:: bash 260 .. code-block:: bash
272 261
273 # Reset coverage counters before runni 262 # Reset coverage counters before running the test.
274 $ echo 0 > /sys/kernel/debug/gcov/rese 263 $ echo 0 > /sys/kernel/debug/gcov/reset
275 $ modprobe kunit-example-test 264 $ modprobe kunit-example-test
276 <<
277 <<
278 Test Attributes and Filtering <<
279 ============================= <<
280 <<
281 Test suites and cases can be marked with test <<
282 test. These attributes will later be printed i <<
283 filter test execution. <<
284 <<
285 Marking Test Attributes <<
286 ----------------------- <<
287 <<
288 Tests are marked with an attribute by includin <<
289 in the test definition. <<
290 <<
291 Test cases can be marked using the ``KUNIT_CAS <<
292 macro to define the test case instead of ``KUN <<
293 <<
294 .. code-block:: c <<
295 <<
296 static const struct kunit_attributes e <<
297 .speed = KUNIT_VERY_SLOW, <<
298 }; <<
299 <<
300 static struct kunit_case example_test_ <<
301 KUNIT_CASE_ATTR(example_test, <<
302 }; <<
303 <<
304 .. note:: <<
305 To mark a test case as slow, you can a <<
306 This is a helpful macro as the slow at <<
307 <<
308 Test suites can be marked with an attribute by <<
309 suite definition. <<
310 <<
311 .. code-block:: c <<
312 <<
313 static const struct kunit_attributes e <<
314 .speed = KUNIT_VERY_SLOW, <<
315 }; <<
316 <<
317 static struct kunit_suite example_test <<
318 ..., <<
319 .attr = example_attr, <<
320 }; <<
321 <<
322 .. note:: <<
323 Not all attributes need to be set in a <<
324 attributes will remain uninitialized a <<
325 to 0 or NULL. Thus, if an attribute is <<
326 These unset attributes will not be rep <<
327 for filtering purposes. <<
328 <<
329 Reporting Attributes <<
330 -------------------- <<
331 <<
332 When a user runs tests, attributes will be pre <<
333 KTAP format). Note that attributes will be hid <<
334 for all passing tests but the raw kernel outpu <<
335 ``--raw_output`` flag. This is an example of h <<
336 will be formatted in kernel output: <<
337 <<
338 .. code-block:: none <<
339 <<
340 # example_test.speed: slow <<
341 ok 1 example_test <<
342 <<
343 This is an example of how test attributes for <<
344 kernel output: <<
345 <<
346 .. code-block:: none <<
347 <<
348 KTAP version 2 <<
349 # Subtest: example_suite <<
350 # module: kunit_example_test <<
351 1..3 <<
352 ... <<
353 ok 1 example_suite <<
354 <<
355 Additionally, users can output a full attribut <<
356 attributes, using the command line flag ``--li <<
357 <<
358 .. code-block:: bash <<
359 <<
360 kunit.py run "example" --list_tests_at <<
361 <<
362 .. note:: <<
363 This report can be accessed when runni <<
364 module_param ``kunit.action=list_attr` <<
365 <<
366 Filtering <<
367 --------- <<
368 <<
369 Users can filter tests using the ``--filter`` <<
370 tests. As an example: <<
371 <<
372 .. code-block:: bash <<
373 <<
374 kunit.py run --filter speed=slow <<
375 <<
376 <<
377 You can also use the following operations on f <<
378 "!=", and "=". Example: <<
379 <<
380 .. code-block:: bash <<
381 <<
382 kunit.py run --filter "speed>slow" <<
383 <<
384 This example will run all tests with speeds fa <<
385 characters < and > are often interpreted by th <<
386 quoted or escaped, as above. <<
387 <<
388 Additionally, you can use multiple filters at <<
389 using commas. Example: <<
390 <<
391 .. code-block:: bash <<
392 <<
393 kunit.py run --filter "speed>slow, mod <<
394 <<
395 .. note:: <<
396 You can use this filtering feature whe <<
397 the filter as a module param: ``kunit. <<
398 <<
399 Filtered tests will not run or show up in the <<
400 ``--filter_action=skip`` flag to skip filtered <<
401 shown in the test output in the test but will <<
402 running KUnit manually, use the module param ` <<
403 <<
404 Rules of Filtering Procedure <<
405 ---------------------------- <<
406 <<
407 Since both suites and test cases can have attr <<
408 between attributes during filtering. The proce <<
409 rules: <<
410 <<
411 - Filtering always operates at a per-test leve <<
412 <<
413 - If a test has an attribute set, then the tes <<
414 <<
415 - Otherwise, the value falls back to the suite <<
416 <<
417 - If neither are set, the attribute has a glob <<
418 <<
419 List of Current Attributes <<
420 -------------------------- <<
421 <<
422 ``speed`` <<
423 <<
424 This attribute indicates the speed of a test's <<
425 test is). <<
426 <<
427 This attribute is saved as an enum with the fo <<
428 "slow", or "very_slow". The assumed default sp <<
429 indicates that the test takes a relatively tri <<
430 1 second), regardless of the machine it is run <<
431 this could be marked as "slow" or "very_slow". <<
432 <<
433 The macro ``KUNIT_CASE_SLOW(test_name)`` can b <<
434 of a test case to "slow". <<
435 <<
436 ``module`` <<
437 <<
438 This attribute indicates the name of the modul <<
439 <<
440 This attribute is automatically saved as a str <<
441 Tests can also be filtered using this attribut <<
442 <<
443 ``is_init`` <<
444 <<
445 This attribute indicates whether the test uses <<
446 <<
447 This attribute is automatically saved as a boo <<
448 filtered using this attribute. <<
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.