1 ================== 1 ==================
2 BPF Selftest Notes 2 BPF Selftest Notes
3 ================== 3 ==================
4 General instructions on running selftests can 4 General instructions on running selftests can be found in
5 `Documentation/bpf/bpf_devel_QA.rst`__. 5 `Documentation/bpf/bpf_devel_QA.rst`__.
6 6
7 __ /Documentation/bpf/bpf_devel_QA.rst#q-how-t 7 __ /Documentation/bpf/bpf_devel_QA.rst#q-how-to-run-bpf-selftests
8 8
9 ============= <<
10 BPF CI System <<
11 ============= <<
12 <<
13 BPF employs a continuous integration (CI) syst <<
14 automated fashion. The system runs selftests f <<
15 are propagated to patchwork, where failures ar <<
16 violations of other checks (such as additional <<
17 ``scripts/checkpatch.pl`` reported deficiency) <<
18 <<
19 https://patchwork.kernel.org/project/netdevb <<
20 <<
21 The CI system executes tests on multiple archi <<
22 configuration derived from both the generic an <<
23 file fragments below ``tools/testing/selftests <<
24 ``config.x86_64``). <<
25 <<
26 Denylisting Tests <<
27 ================= <<
28 <<
29 It is possible for some architectures to not h <<
30 In such a case tests in CI may fail. An exampl <<
31 trampoline support on IBM's s390x architecture <<
32 deny list file, located at ``tools/testing/sel <<
33 be used to prevent the test from running on su <<
34 <<
35 In addition to that, the generic ``tools/testi <<
36 honored on every architecture running tests. <<
37 <<
38 These files are organized in three columns. Th <<
39 question. This can be the name of a test suite <<
40 remaining two columns provide additional meta <<
41 classify the entry: column two is a copy and p <<
42 when running the test in the setting in questi <<
43 available, summarizes the underlying problem. <<
44 example, indicates that lack of trampoline sup <<
45 This last entry helps identify tests that can <<
46 added. <<
47 <<
48 ========================= 9 =========================
49 Running Selftests in a VM 10 Running Selftests in a VM
50 ========================= 11 =========================
51 12
52 It's now possible to run the selftests using ` 13 It's now possible to run the selftests using ``tools/testing/selftests/bpf/vmtest.sh``.
53 The script tries to ensure that the tests are 14 The script tries to ensure that the tests are run with the same environment as they
54 would be run post-submit in the CI used by the !! 15 would be run post-submit in the CI used by the Maintainers.
55 that deny lists are not automatically honored. <<
56 16
57 This script uses the in-tree kernel configurat !! 17 This script downloads a suitable Kconfig and VM userspace image from the system used by
58 image from the system used by the CI. It build !! 18 the CI. It builds the kernel (without overwriting your existing Kconfig), recompiles the
59 your existing Kconfig), recompiles the bpf sel !! 19 bpf selftests, runs them (by default ``tools/testing/selftests/bpf/test_progs``) and
60 ``tools/testing/selftests/bpf/test_progs``) an !! 20 saves the resulting output (by default in ``~/.bpf_selftests``).
61 default in ``~/.bpf_selftests``). <<
62 21
63 Script dependencies: 22 Script dependencies:
64 - clang (preferably built from sources, https: 23 - clang (preferably built from sources, https://github.com/llvm/llvm-project);
65 - pahole (preferably built from sources, https 24 - pahole (preferably built from sources, https://git.kernel.org/pub/scm/devel/pahole/pahole.git/);
66 - qemu; 25 - qemu;
67 - docutils (for ``rst2man``); 26 - docutils (for ``rst2man``);
68 - libcap-devel. 27 - libcap-devel.
69 28
70 For more information about using the script, r !! 29 For more information on about using the script, run:
71 30
72 .. code-block:: console 31 .. code-block:: console
73 32
74 $ tools/testing/selftests/bpf/vmtest.sh -h 33 $ tools/testing/selftests/bpf/vmtest.sh -h
75 34
76 In case of linker errors when running selftest 35 In case of linker errors when running selftests, try using static linking:
77 36
78 .. code-block:: console 37 .. code-block:: console
79 38
80 $ LDLIBS=-static PKG_CONFIG='pkg-config --st !! 39 $ LDLIBS=-static vmtest.sh
81 40
82 .. note:: Some distros may not support static 41 .. note:: Some distros may not support static linking.
83 42
84 .. note:: The script uses pahole and clang bas 43 .. note:: The script uses pahole and clang based on host environment setting.
85 If you want to change pahole and llv 44 If you want to change pahole and llvm, you can change `PATH` environment
86 variable in the beginning of script. 45 variable in the beginning of script.
87 46
88 Running vmtest on RV64 !! 47 .. note:: The script currently only supports x86_64 and s390x architectures.
89 ====================== <<
90 To speed up testing and avoid various dependen <<
91 run vmtest in a Docker container. Before runni <<
92 Docker container and local rootfs image. The o <<
93 <<
94 1. Create Docker container as shown in link [0 <<
95 <<
96 2. Use mkrootfs_debian.sh script [1] to build <<
97 <<
98 .. code-block:: console <<
99 <<
100 $ sudo ./mkrootfs_debian.sh --arch riscv64 - <<
101 <<
102 3. Start Docker container [0] and run vmtest i <<
103 <<
104 .. code-block:: console <<
105 <<
106 $ PLATFORM=riscv64 CROSS_COMPILE=riscv64-lin <<
107 tools/testing/selftests/bpf/vmtest.sh \ <<
108 -l <path of local rootfs image> -- \ <<
109 ./test_progs -d \ <<
110 \"$(cat tools/testing/selftests/bpf/DE <<
111 | cut -d'#' -f1 \ <<
112 | sed -e 's/^[[:space:]]*//' \ <<
113 -e 's/[[:space:]]*$//' \ <<
114 | tr -s '\n' ',' \ <<
115 )\" <<
116 <<
117 Link: https://github.com/pulehui/riscv-bpf-vmt <<
118 Link: https://github.com/libbpf/ci/blob/main/r <<
119 48
120 Additional information about selftest failures 49 Additional information about selftest failures are
121 documented here. 50 documented here.
122 51
123 profiler[23] test failures with clang/llvm <12 52 profiler[23] test failures with clang/llvm <12.0.0
124 ============================================== 53 ==================================================
125 54
126 With clang/llvm <12.0.0, the profiler[23] test 55 With clang/llvm <12.0.0, the profiler[23] test may fail.
127 The symptom looks like 56 The symptom looks like
128 57
129 .. code-block:: c 58 .. code-block:: c
130 59
131 // r9 is a pointer to map_value 60 // r9 is a pointer to map_value
132 // r7 is a scalar 61 // r7 is a scalar
133 17: bf 96 00 00 00 00 00 00 r6 = r9 62 17: bf 96 00 00 00 00 00 00 r6 = r9
134 18: 0f 76 00 00 00 00 00 00 r6 += r7 63 18: 0f 76 00 00 00 00 00 00 r6 += r7
135 math between map_value pointer and register 64 math between map_value pointer and register with unbounded min value is not allowed
136 65
137 // the instructions below will not be seen i 66 // the instructions below will not be seen in the verifier log
138 19: a5 07 01 00 01 01 00 00 if r7 < 25 67 19: a5 07 01 00 01 01 00 00 if r7 < 257 goto +1
139 20: bf 96 00 00 00 00 00 00 r6 = r9 68 20: bf 96 00 00 00 00 00 00 r6 = r9
140 // r6 is used here 69 // r6 is used here
141 70
142 The verifier will reject such code with above 71 The verifier will reject such code with above error.
143 At insn 18 the r7 is indeed unbounded. The lat 72 At insn 18 the r7 is indeed unbounded. The later insn 19 checks the bounds and
144 the insn 20 undoes map_value addition. It is c 73 the insn 20 undoes map_value addition. It is currently impossible for the
145 verifier to understand such speculative pointe 74 verifier to understand such speculative pointer arithmetic.
146 Hence `this patch`__ addresses it on the compi 75 Hence `this patch`__ addresses it on the compiler side. It was committed on llvm 12.
147 76
148 __ https://github.com/llvm/llvm-project/commit !! 77 __ https://reviews.llvm.org/D85570
149 78
150 The corresponding C code 79 The corresponding C code
151 80
152 .. code-block:: c 81 .. code-block:: c
153 82
154 for (int i = 0; i < MAX_CGROUPS_PATH_DEPTH; 83 for (int i = 0; i < MAX_CGROUPS_PATH_DEPTH; i++) {
155 filepart_length = bpf_probe_read_str 84 filepart_length = bpf_probe_read_str(payload, ...);
156 if (filepart_length <= MAX_PATH) { 85 if (filepart_length <= MAX_PATH) {
157 barrier_var(filepart_length) 86 barrier_var(filepart_length); // workaround
158 payload += filepart_length; 87 payload += filepart_length;
159 } 88 }
160 } 89 }
161 90
162 bpf_iter test failures with clang/llvm 10.0.0 91 bpf_iter test failures with clang/llvm 10.0.0
163 ============================================= 92 =============================================
164 93
165 With clang/llvm 10.0.0, the following two bpf_ 94 With clang/llvm 10.0.0, the following two bpf_iter tests failed:
166 * ``bpf_iter/ipv6_route`` 95 * ``bpf_iter/ipv6_route``
167 * ``bpf_iter/netlink`` 96 * ``bpf_iter/netlink``
168 97
169 The symptom for ``bpf_iter/ipv6_route`` looks 98 The symptom for ``bpf_iter/ipv6_route`` looks like
170 99
171 .. code-block:: c 100 .. code-block:: c
172 101
173 2: (79) r8 = *(u64 *)(r1 +8) 102 2: (79) r8 = *(u64 *)(r1 +8)
174 ... 103 ...
175 14: (bf) r2 = r8 104 14: (bf) r2 = r8
176 15: (0f) r2 += r1 105 15: (0f) r2 += r1
177 ; BPF_SEQ_PRINTF(seq, "%pi6 %02x ", &rt->fib 106 ; BPF_SEQ_PRINTF(seq, "%pi6 %02x ", &rt->fib6_dst.addr, rt->fib6_dst.plen);
178 16: (7b) *(u64 *)(r8 +64) = r2 107 16: (7b) *(u64 *)(r8 +64) = r2
179 only read is supported 108 only read is supported
180 109
181 The symptom for ``bpf_iter/netlink`` looks lik 110 The symptom for ``bpf_iter/netlink`` looks like
182 111
183 .. code-block:: c 112 .. code-block:: c
184 113
185 ; struct netlink_sock *nlk = ctx->sk; 114 ; struct netlink_sock *nlk = ctx->sk;
186 2: (79) r7 = *(u64 *)(r1 +8) 115 2: (79) r7 = *(u64 *)(r1 +8)
187 ... 116 ...
188 15: (bf) r2 = r7 117 15: (bf) r2 = r7
189 16: (0f) r2 += r1 118 16: (0f) r2 += r1
190 ; BPF_SEQ_PRINTF(seq, "%pK %-3d ", s, s->sk_ 119 ; BPF_SEQ_PRINTF(seq, "%pK %-3d ", s, s->sk_protocol);
191 17: (7b) *(u64 *)(r7 +0) = r2 120 17: (7b) *(u64 *)(r7 +0) = r2
192 only read is supported 121 only read is supported
193 122
194 This is due to a llvm BPF backend bug. `The fi 123 This is due to a llvm BPF backend bug. `The fix`__
195 has been pushed to llvm 10.x release branch an 124 has been pushed to llvm 10.x release branch and will be
196 available in 10.0.1. The patch is available in 125 available in 10.0.1. The patch is available in llvm 11.0.0 trunk.
197 126
198 __ https://github.com/llvm/llvm-project/commi !! 127 __ https://reviews.llvm.org/D78466
199 128
200 bpf_verif_scale/loop6.bpf.o test failure with 129 bpf_verif_scale/loop6.bpf.o test failure with Clang 12
201 ============================================== 130 ======================================================
202 131
203 With Clang 12, the following bpf_verif_scale t 132 With Clang 12, the following bpf_verif_scale test failed:
204 * ``bpf_verif_scale/loop6.bpf.o`` 133 * ``bpf_verif_scale/loop6.bpf.o``
205 134
206 The verifier output looks like 135 The verifier output looks like
207 136
208 .. code-block:: c 137 .. code-block:: c
209 138
210 R1 type=ctx expected=fp 139 R1 type=ctx expected=fp
211 The sequence of 8193 jumps is too complex. 140 The sequence of 8193 jumps is too complex.
212 141
213 The reason is compiler generating the followin 142 The reason is compiler generating the following code
214 143
215 .. code-block:: c 144 .. code-block:: c
216 145
217 ; for (i = 0; (i < VIRTIO_MAX_SGS) && 146 ; for (i = 0; (i < VIRTIO_MAX_SGS) && (i < num); i++) {
218 14: 16 05 40 00 00 00 00 00 if w5 147 14: 16 05 40 00 00 00 00 00 if w5 == 0 goto +64 <LBB0_6>
219 15: bc 51 00 00 00 00 00 00 w1 = w 148 15: bc 51 00 00 00 00 00 00 w1 = w5
220 16: 04 01 00 00 ff ff ff ff w1 += 149 16: 04 01 00 00 ff ff ff ff w1 += -1
221 17: 67 05 00 00 20 00 00 00 r5 <<= 150 17: 67 05 00 00 20 00 00 00 r5 <<= 32
222 18: 77 05 00 00 20 00 00 00 r5 >>= 151 18: 77 05 00 00 20 00 00 00 r5 >>= 32
223 19: a6 01 01 00 05 00 00 00 if w1 152 19: a6 01 01 00 05 00 00 00 if w1 < 5 goto +1 <LBB0_4>
224 20: b7 05 00 00 06 00 00 00 r5 = 6 153 20: b7 05 00 00 06 00 00 00 r5 = 6
225 00000000000000a8 <LBB0_4>: 154 00000000000000a8 <LBB0_4>:
226 21: b7 02 00 00 00 00 00 00 r2 = 0 155 21: b7 02 00 00 00 00 00 00 r2 = 0
227 22: b7 01 00 00 00 00 00 00 r1 = 0 156 22: b7 01 00 00 00 00 00 00 r1 = 0
228 ; for (i = 0; (i < VIRTIO_MAX_SGS) && 157 ; for (i = 0; (i < VIRTIO_MAX_SGS) && (i < num); i++) {
229 23: 7b 1a e0 ff 00 00 00 00 *(u64 158 23: 7b 1a e0 ff 00 00 00 00 *(u64 *)(r10 - 32) = r1
230 24: 7b 5a c0 ff 00 00 00 00 *(u64 159 24: 7b 5a c0 ff 00 00 00 00 *(u64 *)(r10 - 64) = r5
231 160
232 Note that insn #15 has w1 = w5 and w1 is refin 161 Note that insn #15 has w1 = w5 and w1 is refined later but
233 r5(w5) is eventually saved on stack at insn #2 162 r5(w5) is eventually saved on stack at insn #24 for later use.
234 This cause later verifier failure. The bug has 163 This cause later verifier failure. The bug has been `fixed`__ in
235 Clang 13. 164 Clang 13.
236 165
237 __ https://github.com/llvm/llvm-project/commi !! 166 __ https://reviews.llvm.org/D97479
238 167
239 BPF CO-RE-based tests and Clang version 168 BPF CO-RE-based tests and Clang version
240 ======================================= 169 =======================================
241 170
242 A set of selftests use BPF target-specific bui 171 A set of selftests use BPF target-specific built-ins, which might require
243 bleeding-edge Clang versions (Clang 12 nightly 172 bleeding-edge Clang versions (Clang 12 nightly at this time).
244 173
245 Few sub-tests of core_reloc test suit (part of 174 Few sub-tests of core_reloc test suit (part of test_progs test runner) require
246 the following built-ins, listed with correspon 175 the following built-ins, listed with corresponding Clang diffs introducing
247 them to Clang/LLVM. These sub-tests are going 176 them to Clang/LLVM. These sub-tests are going to be skipped if Clang is too
248 old to support them, they shouldn't cause buil 177 old to support them, they shouldn't cause build failures or runtime test
249 failures: 178 failures:
250 179
251 - __builtin_btf_type_id() [0_, 1_, 2_]; 180 - __builtin_btf_type_id() [0_, 1_, 2_];
252 - __builtin_preserve_type_info(), __builtin_pr 181 - __builtin_preserve_type_info(), __builtin_preserve_enum_value() [3_, 4_].
253 182
254 .. _0: https://github.com/llvm/llvm-project/co !! 183 .. _0: https://reviews.llvm.org/D74572
255 .. _1: https://github.com/llvm/llvm-project/co !! 184 .. _1: https://reviews.llvm.org/D74668
256 .. _2: https://github.com/llvm/llvm-project/co !! 185 .. _2: https://reviews.llvm.org/D85174
257 .. _3: https://github.com/llvm/llvm-project/co !! 186 .. _3: https://reviews.llvm.org/D83878
258 .. _4: https://github.com/llvm/llvm-project/co !! 187 .. _4: https://reviews.llvm.org/D83242
259 188
260 Floating-point tests and Clang version 189 Floating-point tests and Clang version
261 ====================================== 190 ======================================
262 191
263 Certain selftests, e.g. core_reloc, require su 192 Certain selftests, e.g. core_reloc, require support for the floating-point
264 types, which was introduced in `Clang 13`__. T 193 types, which was introduced in `Clang 13`__. The older Clang versions will
265 either crash when compiling these tests, or ge 194 either crash when compiling these tests, or generate an incorrect BTF.
266 195
267 __ https://github.com/llvm/llvm-project/commi !! 196 __ https://reviews.llvm.org/D83289
268 197
269 Kernel function call test and Clang version 198 Kernel function call test and Clang version
270 =========================================== 199 ===========================================
271 200
272 Some selftests (e.g. kfunc_call and bpf_tcp_ca 201 Some selftests (e.g. kfunc_call and bpf_tcp_ca) require a LLVM support
273 to generate extern function in BTF. It was in 202 to generate extern function in BTF. It was introduced in `Clang 13`__.
274 203
275 Without it, the error from compiling bpf selft 204 Without it, the error from compiling bpf selftests looks like:
276 205
277 .. code-block:: console 206 .. code-block:: console
278 207
279 libbpf: failed to find BTF for extern 'tcp_s 208 libbpf: failed to find BTF for extern 'tcp_slow_start' [25] section: -2
280 209
281 __ https://github.com/llvm/llvm-project/commit !! 210 __ https://reviews.llvm.org/D93563
282 211
283 btf_tag test and Clang version 212 btf_tag test and Clang version
284 ============================== 213 ==============================
285 214
286 The btf_tag selftest requires LLVM support to 215 The btf_tag selftest requires LLVM support to recognize the btf_decl_tag and
287 btf_type_tag attributes. They are introduced i 216 btf_type_tag attributes. They are introduced in `Clang 14` [0_, 1_].
288 The subtests ``btf_type_tag_user_{mod1, mod2, 217 The subtests ``btf_type_tag_user_{mod1, mod2, vmlinux}`` also requires
289 pahole version ``1.23``. 218 pahole version ``1.23``.
290 219
291 Without them, the btf_tag selftest will be ski 220 Without them, the btf_tag selftest will be skipped and you will observe:
292 221
293 .. code-block:: console 222 .. code-block:: console
294 223
295 #<test_num> btf_tag:SKIP 224 #<test_num> btf_tag:SKIP
296 225
297 .. _0: https://github.com/llvm/llvm-project/co !! 226 .. _0: https://reviews.llvm.org/D111588
298 .. _1: https://github.com/llvm/llvm-project/co !! 227 .. _1: https://reviews.llvm.org/D111199
299 228
300 Clang dependencies for static linking tests 229 Clang dependencies for static linking tests
301 =========================================== 230 ===========================================
302 231
303 linked_vars, linked_maps, and linked_funcs tes 232 linked_vars, linked_maps, and linked_funcs tests depend on `Clang fix`__ to
304 generate valid BTF information for weak variab 233 generate valid BTF information for weak variables. Please make sure you use
305 Clang that contains the fix. 234 Clang that contains the fix.
306 235
307 __ https://github.com/llvm/llvm-project/commit !! 236 __ https://reviews.llvm.org/D100362
308 237
309 Clang relocation changes 238 Clang relocation changes
310 ======================== 239 ========================
311 240
312 Clang 13 patch `clang reloc patch`_ made some 241 Clang 13 patch `clang reloc patch`_ made some changes on relocations such
313 that existing relocation types are broken into 242 that existing relocation types are broken into more types and
314 each new type corresponds to only one way to r 243 each new type corresponds to only one way to resolve relocation.
315 See `kernel llvm reloc`_ for more explanation 244 See `kernel llvm reloc`_ for more explanation and some examples.
316 Using clang 13 to compile old libbpf which has 245 Using clang 13 to compile old libbpf which has static linker support,
317 there will be a compilation failure:: 246 there will be a compilation failure::
318 247
319 libbpf: ELF relo #0 in section #6 has unexpe 248 libbpf: ELF relo #0 in section #6 has unexpected type 2 in .../bpf_tcp_nogpl.bpf.o
320 249
321 Here, ``type 2`` refers to new relocation type 250 Here, ``type 2`` refers to new relocation type ``R_BPF_64_ABS64``.
322 To fix this issue, user newer libbpf. 251 To fix this issue, user newer libbpf.
323 252
324 .. Links 253 .. Links
325 .. _clang reloc patch: https://github.com/llvm !! 254 .. _clang reloc patch: https://reviews.llvm.org/D102712
326 .. _kernel llvm reloc: /Documentation/bpf/llvm 255 .. _kernel llvm reloc: /Documentation/bpf/llvm_reloc.rst
327 256
328 Clang dependencies for the u32 spill test (xdp 257 Clang dependencies for the u32 spill test (xdpwall)
329 ============================================== 258 ===================================================
330 The xdpwall selftest requires a change in `Cla 259 The xdpwall selftest requires a change in `Clang 14`__.
331 260
332 Without it, the xdpwall selftest will fail and 261 Without it, the xdpwall selftest will fail and the error message
333 from running test_progs will look like: 262 from running test_progs will look like:
334 263
335 .. code-block:: console 264 .. code-block:: console
336 265
337 test_xdpwall:FAIL:Does LLVM have https://git !! 266 test_xdpwall:FAIL:Does LLVM have https://reviews.llvm.org/D109073? unexpected error: -4007
338 267
339 __ https://github.com/llvm/llvm-project/commit !! 268 __ https://reviews.llvm.org/D109073
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.