1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ============================ 3 ============================
4 Tips For Running KUnit Tests 4 Tips For Running KUnit Tests
5 ============================ 5 ============================
6 6
7 Using ``kunit.py run`` ("kunit tool") 7 Using ``kunit.py run`` ("kunit tool")
8 ===================================== 8 =====================================
9 9
10 Running from any directory 10 Running from any directory
11 -------------------------- 11 --------------------------
12 12
13 It can be handy to create a bash function like 13 It can be handy to create a bash function like:
14 14
15 .. code-block:: bash 15 .. code-block:: bash
16 16
17 function run_kunit() { 17 function run_kunit() {
18 ( cd "$(git rev-parse --show-topleve !! 18 ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run $@ )
19 } 19 }
20 20
21 .. note:: 21 .. note::
22 Early versions of ``kunit.py`` (before 22 Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
23 the kernel root, hence the use of a su 23 the kernel root, hence the use of a subshell and ``cd``.
24 24
25 Running a subset of tests 25 Running a subset of tests
26 ------------------------- 26 -------------------------
27 27
28 ``kunit.py run`` accepts an optional glob argu !! 28 ``kunit.py run`` accepts an optional glob argument to filter tests. Currently
29 is ``"<suite_glob>[.test_glob]"``. !! 29 this only matches against suite names, but this may change in the future.
30 30
31 Say that we wanted to run the sysctl tests, we 31 Say that we wanted to run the sysctl tests, we could do so via:
32 32
33 .. code-block:: bash 33 .. code-block:: bash
34 34
35 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNI 35 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig
36 $ ./tools/testing/kunit/kunit.py run ' 36 $ ./tools/testing/kunit/kunit.py run 'sysctl*'
37 37
38 We can filter down to just the "write" tests v <<
39 <<
40 .. code-block:: bash <<
41 <<
42 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNI <<
43 $ ./tools/testing/kunit/kunit.py run ' <<
44 <<
45 We're paying the cost of building more tests t 38 We're paying the cost of building more tests than we need this way, but it's
46 easier than fiddling with ``.kunitconfig`` fil 39 easier than fiddling with ``.kunitconfig`` files or commenting out
47 ``kunit_suite``'s. 40 ``kunit_suite``'s.
48 41
49 However, if we wanted to define a set of tests 42 However, if we wanted to define a set of tests in a less ad hoc way, the next
50 tip is useful. 43 tip is useful.
51 44
52 Defining a set of tests 45 Defining a set of tests
53 ----------------------- 46 -----------------------
54 47
55 ``kunit.py run`` (along with ``build``, and `` 48 ``kunit.py run`` (along with ``build``, and ``config``) supports a
56 ``--kunitconfig`` flag. So if you have a set o 49 ``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
57 regular basis (especially if they have other d 50 regular basis (especially if they have other dependencies), you can create a
58 specific ``.kunitconfig`` for them. 51 specific ``.kunitconfig`` for them.
59 52
60 E.g. kunit has one for its tests: 53 E.g. kunit has one for its tests:
61 54
62 .. code-block:: bash 55 .. code-block:: bash
63 56
64 $ ./tools/testing/kunit/kunit.py run - 57 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig
65 58
66 Alternatively, if you're following the convent 59 Alternatively, if you're following the convention of naming your
67 file ``.kunitconfig``, you can just pass in th 60 file ``.kunitconfig``, you can just pass in the dir, e.g.
68 61
69 .. code-block:: bash 62 .. code-block:: bash
70 63
71 $ ./tools/testing/kunit/kunit.py run - 64 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
72 65
73 .. note:: 66 .. note::
74 This is a relatively new feature (5.12 67 This is a relatively new feature (5.12+) so we don't have any
75 conventions yet about on what files sh 68 conventions yet about on what files should be checked in versus just
76 kept around locally. It's up to you an 69 kept around locally. It's up to you and your maintainer to decide if a
77 config is useful enough to submit (and 70 config is useful enough to submit (and therefore have to maintain).
78 71
79 .. note:: 72 .. note::
80 Having ``.kunitconfig`` fragments in a 73 Having ``.kunitconfig`` fragments in a parent and child directory is
81 iffy. There's discussion about adding 74 iffy. There's discussion about adding an "import" statement in these
82 files to make it possible to have a to 75 files to make it possible to have a top-level config run tests from all
83 child directories. But that would mean 76 child directories. But that would mean ``.kunitconfig`` files are no
84 longer just simple .config fragments. 77 longer just simple .config fragments.
85 78
86 One alternative would be to have kunit 79 One alternative would be to have kunit tool recursively combine configs
87 automagically, but tests could theoret 80 automagically, but tests could theoretically depend on incompatible
88 options, so handling that would be tri 81 options, so handling that would be tricky.
89 82
90 Setting kernel commandline parameters 83 Setting kernel commandline parameters
91 ------------------------------------- 84 -------------------------------------
92 85
93 You can use ``--kernel_args`` to pass arbitrar 86 You can use ``--kernel_args`` to pass arbitrary kernel arguments, e.g.
94 87
95 .. code-block:: bash 88 .. code-block:: bash
96 89
97 $ ./tools/testing/kunit/kunit.py run - 90 $ ./tools/testing/kunit/kunit.py run --kernel_args=param=42 --kernel_args=param2=false
98 91
99 92
100 Generating code coverage reports under UML 93 Generating code coverage reports under UML
101 ------------------------------------------ 94 ------------------------------------------
102 95
103 .. note:: 96 .. note::
104 TODO(brendanhiggins@google.com): There 97 TODO(brendanhiggins@google.com): There are various issues with UML and
105 versions of gcc 7 and up. You're likel 98 versions of gcc 7 and up. You're likely to run into missing ``.gcda``
106 files or compile errors. 99 files or compile errors.
107 100
108 This is different from the "normal" way of get 101 This is different from the "normal" way of getting coverage information that is
109 documented in Documentation/dev-tools/gcov.rst 102 documented in Documentation/dev-tools/gcov.rst.
110 103
111 Instead of enabling ``CONFIG_GCOV_KERNEL=y``, 104 Instead of enabling ``CONFIG_GCOV_KERNEL=y``, we can set these options:
112 105
113 .. code-block:: none 106 .. code-block:: none
114 107
115 CONFIG_DEBUG_KERNEL=y 108 CONFIG_DEBUG_KERNEL=y
116 CONFIG_DEBUG_INFO=y 109 CONFIG_DEBUG_INFO=y
117 CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFA <<
118 CONFIG_GCOV=y 110 CONFIG_GCOV=y
119 111
120 112
121 Putting it together into a copy-pastable seque 113 Putting it together into a copy-pastable sequence of commands:
122 114
123 .. code-block:: bash 115 .. code-block:: bash
124 116
125 # Append coverage options to the curre 117 # Append coverage options to the current config
126 $ ./tools/testing/kunit/kunit.py run - !! 118 $ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig
>> 119 $ ./tools/testing/kunit/kunit.py run
127 # Extract the coverage information fro 120 # Extract the coverage information from the build dir (.kunit/)
128 $ lcov -t "my_kunit_tests" -o coverage 121 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/
129 122
130 # From here on, it's the same process 123 # From here on, it's the same process as with CONFIG_GCOV_KERNEL=y
131 # E.g. can generate an HTML report in 124 # E.g. can generate an HTML report in a tmp dir like so:
132 $ genhtml -o /tmp/coverage_html covera 125 $ genhtml -o /tmp/coverage_html coverage.info
133 126
134 127
135 If your installed version of gcc doesn't work, 128 If your installed version of gcc doesn't work, you can tweak the steps:
136 129
137 .. code-block:: bash 130 .. code-block:: bash
138 131
139 $ ./tools/testing/kunit/kunit.py run - 132 $ ./tools/testing/kunit/kunit.py run --make_options=CC=/usr/bin/gcc-6
140 $ lcov -t "my_kunit_tests" -o coverage 133 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcov-6
141 134
142 Alternatively, LLVM-based toolchains can also <<
143 <<
144 .. code-block:: bash <<
145 <<
146 # Build with LLVM and append coverage <<
147 $ ./tools/testing/kunit/kunit.py run - <<
148 $ llvm-profdata merge -sparse default. <<
149 $ llvm-cov export --format=lcov .kunit <<
150 # The coverage.info file is in lcov-co <<
151 $ genhtml -o /tmp/coverage_html covera <<
152 <<
153 135
154 Running tests manually 136 Running tests manually
155 ====================== 137 ======================
156 138
157 Running tests without using ``kunit.py run`` i 139 Running tests without using ``kunit.py run`` is also an important use case.
158 Currently it's your only option if you want to 140 Currently it's your only option if you want to test on architectures other than
159 UML. 141 UML.
160 142
161 As running the tests under UML is fairly strai 143 As running the tests under UML is fairly straightforward (configure and compile
162 the kernel, run the ``./linux`` binary), this 144 the kernel, run the ``./linux`` binary), this section will focus on testing
163 non-UML architectures. 145 non-UML architectures.
164 146
165 147
166 Running built-in tests 148 Running built-in tests
167 ---------------------- 149 ----------------------
168 150
169 When setting tests to ``=y``, the tests will r 151 When setting tests to ``=y``, the tests will run as part of boot and print
170 results to dmesg in TAP format. So you just ne 152 results to dmesg in TAP format. So you just need to add your tests to your
171 ``.config``, build and boot your kernel as nor 153 ``.config``, build and boot your kernel as normal.
172 154
173 So if we compiled our kernel with: 155 So if we compiled our kernel with:
174 156
175 .. code-block:: none 157 .. code-block:: none
176 158
177 CONFIG_KUNIT=y 159 CONFIG_KUNIT=y
178 CONFIG_KUNIT_EXAMPLE_TEST=y 160 CONFIG_KUNIT_EXAMPLE_TEST=y
179 161
180 Then we'd see output like this in dmesg signal 162 Then we'd see output like this in dmesg signaling the test ran and passed:
181 163
182 .. code-block:: none 164 .. code-block:: none
183 165
184 TAP version 14 166 TAP version 14
185 1..1 167 1..1
186 # Subtest: example 168 # Subtest: example
187 1..1 169 1..1
188 # example_simple_test: initializin 170 # example_simple_test: initializing
189 ok 1 - example_simple_test 171 ok 1 - example_simple_test
190 ok 1 - example 172 ok 1 - example
191 173
192 Running tests as modules 174 Running tests as modules
193 ------------------------ 175 ------------------------
194 176
195 Depending on the tests, you can build them as 177 Depending on the tests, you can build them as loadable modules.
196 178
197 For example, we'd change the config options fr 179 For example, we'd change the config options from before to
198 180
199 .. code-block:: none 181 .. code-block:: none
200 182
201 CONFIG_KUNIT=y 183 CONFIG_KUNIT=y
202 CONFIG_KUNIT_EXAMPLE_TEST=m 184 CONFIG_KUNIT_EXAMPLE_TEST=m
203 185
204 Then after booting into our kernel, we can run 186 Then after booting into our kernel, we can run the test via
205 187
206 .. code-block:: none 188 .. code-block:: none
207 189
208 $ modprobe kunit-example-test 190 $ modprobe kunit-example-test
209 191
210 This will then cause it to print TAP output to 192 This will then cause it to print TAP output to stdout.
211 193
212 .. note:: 194 .. note::
213 The ``modprobe`` will *not* have a non 195 The ``modprobe`` will *not* have a non-zero exit code if any test
214 failed (as of 5.13). But ``kunit.py pa 196 failed (as of 5.13). But ``kunit.py parse`` would, see below.
215 197
216 .. note:: 198 .. note::
217 You can set ``CONFIG_KUNIT=m`` as well 199 You can set ``CONFIG_KUNIT=m`` as well, however, some features will not
218 work and thus some tests might break. 200 work and thus some tests might break. Ideally tests would specify they
219 depend on ``KUNIT=y`` in their ``Kconf 201 depend on ``KUNIT=y`` in their ``Kconfig``'s, but this is an edge case
220 most test authors won't think about. 202 most test authors won't think about.
221 As of 5.13, the only difference is tha 203 As of 5.13, the only difference is that ``current->kunit_test`` will
222 not exist. 204 not exist.
223 205
224 Pretty-printing results 206 Pretty-printing results
225 ----------------------- 207 -----------------------
226 208
227 You can use ``kunit.py parse`` to parse dmesg 209 You can use ``kunit.py parse`` to parse dmesg for test output and print out
228 results in the same familiar format that ``kun 210 results in the same familiar format that ``kunit.py run`` does.
229 211
230 .. code-block:: bash 212 .. code-block:: bash
231 213
232 $ ./tools/testing/kunit/kunit.py parse 214 $ ./tools/testing/kunit/kunit.py parse /var/log/dmesg
233 215
234 216
235 Retrieving per suite results 217 Retrieving per suite results
236 ---------------------------- 218 ----------------------------
237 219
238 Regardless of how you're running your tests, y 220 Regardless of how you're running your tests, you can enable
239 ``CONFIG_KUNIT_DEBUGFS`` to expose per-suite T 221 ``CONFIG_KUNIT_DEBUGFS`` to expose per-suite TAP-formatted results:
240 222
241 .. code-block:: none 223 .. code-block:: none
242 224
243 CONFIG_KUNIT=y 225 CONFIG_KUNIT=y
244 CONFIG_KUNIT_EXAMPLE_TEST=m 226 CONFIG_KUNIT_EXAMPLE_TEST=m
245 CONFIG_KUNIT_DEBUGFS=y 227 CONFIG_KUNIT_DEBUGFS=y
246 228
247 The results for each suite will be exposed und 229 The results for each suite will be exposed under
248 ``/sys/kernel/debug/kunit/<suite>/results``. 230 ``/sys/kernel/debug/kunit/<suite>/results``.
249 So using our example config: 231 So using our example config:
250 232
251 .. code-block:: bash 233 .. code-block:: bash
252 234
253 $ modprobe kunit-example-test > /dev/n 235 $ modprobe kunit-example-test > /dev/null
254 $ cat /sys/kernel/debug/kunit/example/ 236 $ cat /sys/kernel/debug/kunit/example/results
255 ... <TAP output> ... 237 ... <TAP output> ...
256 238
257 # After removing the module, the corre 239 # After removing the module, the corresponding files will go away
258 $ modprobe -r kunit-example-test 240 $ modprobe -r kunit-example-test
259 $ cat /sys/kernel/debug/kunit/example/ 241 $ cat /sys/kernel/debug/kunit/example/results
260 /sys/kernel/debug/kunit/example/result 242 /sys/kernel/debug/kunit/example/results: No such file or directory
261 243
262 Generating code coverage reports 244 Generating code coverage reports
263 -------------------------------- 245 --------------------------------
264 246
265 See Documentation/dev-tools/gcov.rst for detai 247 See Documentation/dev-tools/gcov.rst for details on how to do this.
266 248
267 The only vaguely KUnit-specific advice here is 249 The only vaguely KUnit-specific advice here is that you probably want to build
268 your tests as modules. That way you can isolat 250 your tests as modules. That way you can isolate the coverage from tests from
269 other code executed during boot, e.g. 251 other code executed during boot, e.g.
270 252
271 .. code-block:: bash 253 .. code-block:: bash
272 254
273 # Reset coverage counters before runni 255 # Reset coverage counters before running the test.
274 $ echo 0 > /sys/kernel/debug/gcov/rese 256 $ echo 0 > /sys/kernel/debug/gcov/reset
275 $ modprobe kunit-example-test 257 $ modprobe kunit-example-test
276 <<
277 <<
278 Test Attributes and Filtering <<
279 ============================= <<
280 <<
281 Test suites and cases can be marked with test <<
282 test. These attributes will later be printed i <<
283 filter test execution. <<
284 <<
285 Marking Test Attributes <<
286 ----------------------- <<
287 <<
288 Tests are marked with an attribute by includin <<
289 in the test definition. <<
290 <<
291 Test cases can be marked using the ``KUNIT_CAS <<
292 macro to define the test case instead of ``KUN <<
293 <<
294 .. code-block:: c <<
295 <<
296 static const struct kunit_attributes e <<
297 .speed = KUNIT_VERY_SLOW, <<
298 }; <<
299 <<
300 static struct kunit_case example_test_ <<
301 KUNIT_CASE_ATTR(example_test, <<
302 }; <<
303 <<
304 .. note:: <<
305 To mark a test case as slow, you can a <<
306 This is a helpful macro as the slow at <<
307 <<
308 Test suites can be marked with an attribute by <<
309 suite definition. <<
310 <<
311 .. code-block:: c <<
312 <<
313 static const struct kunit_attributes e <<
314 .speed = KUNIT_VERY_SLOW, <<
315 }; <<
316 <<
317 static struct kunit_suite example_test <<
318 ..., <<
319 .attr = example_attr, <<
320 }; <<
321 <<
322 .. note:: <<
323 Not all attributes need to be set in a <<
324 attributes will remain uninitialized a <<
325 to 0 or NULL. Thus, if an attribute is <<
326 These unset attributes will not be rep <<
327 for filtering purposes. <<
328 <<
329 Reporting Attributes <<
330 -------------------- <<
331 <<
332 When a user runs tests, attributes will be pre <<
333 KTAP format). Note that attributes will be hid <<
334 for all passing tests but the raw kernel outpu <<
335 ``--raw_output`` flag. This is an example of h <<
336 will be formatted in kernel output: <<
337 <<
338 .. code-block:: none <<
339 <<
340 # example_test.speed: slow <<
341 ok 1 example_test <<
342 <<
343 This is an example of how test attributes for <<
344 kernel output: <<
345 <<
346 .. code-block:: none <<
347 <<
348 KTAP version 2 <<
349 # Subtest: example_suite <<
350 # module: kunit_example_test <<
351 1..3 <<
352 ... <<
353 ok 1 example_suite <<
354 <<
355 Additionally, users can output a full attribut <<
356 attributes, using the command line flag ``--li <<
357 <<
358 .. code-block:: bash <<
359 <<
360 kunit.py run "example" --list_tests_at <<
361 <<
362 .. note:: <<
363 This report can be accessed when runni <<
364 module_param ``kunit.action=list_attr` <<
365 <<
366 Filtering <<
367 --------- <<
368 <<
369 Users can filter tests using the ``--filter`` <<
370 tests. As an example: <<
371 <<
372 .. code-block:: bash <<
373 <<
374 kunit.py run --filter speed=slow <<
375 <<
376 <<
377 You can also use the following operations on f <<
378 "!=", and "=". Example: <<
379 <<
380 .. code-block:: bash <<
381 <<
382 kunit.py run --filter "speed>slow" <<
383 <<
384 This example will run all tests with speeds fa <<
385 characters < and > are often interpreted by th <<
386 quoted or escaped, as above. <<
387 <<
388 Additionally, you can use multiple filters at <<
389 using commas. Example: <<
390 <<
391 .. code-block:: bash <<
392 <<
393 kunit.py run --filter "speed>slow, mod <<
394 <<
395 .. note:: <<
396 You can use this filtering feature whe <<
397 the filter as a module param: ``kunit. <<
398 <<
399 Filtered tests will not run or show up in the <<
400 ``--filter_action=skip`` flag to skip filtered <<
401 shown in the test output in the test but will <<
402 running KUnit manually, use the module param ` <<
403 <<
404 Rules of Filtering Procedure <<
405 ---------------------------- <<
406 <<
407 Since both suites and test cases can have attr <<
408 between attributes during filtering. The proce <<
409 rules: <<
410 <<
411 - Filtering always operates at a per-test leve <<
412 <<
413 - If a test has an attribute set, then the tes <<
414 <<
415 - Otherwise, the value falls back to the suite <<
416 <<
417 - If neither are set, the attribute has a glob <<
418 <<
419 List of Current Attributes <<
420 -------------------------- <<
421 <<
422 ``speed`` <<
423 <<
424 This attribute indicates the speed of a test's <<
425 test is). <<
426 <<
427 This attribute is saved as an enum with the fo <<
428 "slow", or "very_slow". The assumed default sp <<
429 indicates that the test takes a relatively tri <<
430 1 second), regardless of the machine it is run <<
431 this could be marked as "slow" or "very_slow". <<
432 <<
433 The macro ``KUNIT_CASE_SLOW(test_name)`` can b <<
434 of a test case to "slow". <<
435 <<
436 ``module`` <<
437 <<
438 This attribute indicates the name of the modul <<
439 <<
440 This attribute is automatically saved as a str <<
441 Tests can also be filtered using this attribut <<
442 <<
443 ``is_init`` <<
444 <<
445 This attribute indicates whether the test uses <<
446 <<
447 This attribute is automatically saved as a boo <<
448 filtered using this attribute. <<
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.