1 ================== 1 ==================
2 BPF Selftest Notes 2 BPF Selftest Notes
3 ================== 3 ==================
4 General instructions on running selftests can 4 General instructions on running selftests can be found in
5 `Documentation/bpf/bpf_devel_QA.rst`__. 5 `Documentation/bpf/bpf_devel_QA.rst`__.
6 6
7 __ /Documentation/bpf/bpf_devel_QA.rst#q-how-t 7 __ /Documentation/bpf/bpf_devel_QA.rst#q-how-to-run-bpf-selftests
8 8
9 ============= <<
10 BPF CI System <<
11 ============= <<
12 <<
13 BPF employs a continuous integration (CI) syst <<
14 automated fashion. The system runs selftests f <<
15 are propagated to patchwork, where failures ar <<
16 violations of other checks (such as additional <<
17 ``scripts/checkpatch.pl`` reported deficiency) <<
18 <<
19 https://patchwork.kernel.org/project/netdevb <<
20 <<
21 The CI system executes tests on multiple archi <<
22 configuration derived from both the generic an <<
23 file fragments below ``tools/testing/selftests <<
24 ``config.x86_64``). <<
25 <<
26 Denylisting Tests <<
27 ================= <<
28 <<
29 It is possible for some architectures to not h <<
30 In such a case tests in CI may fail. An exampl <<
31 trampoline support on IBM's s390x architecture <<
32 deny list file, located at ``tools/testing/sel <<
33 be used to prevent the test from running on su <<
34 <<
35 In addition to that, the generic ``tools/testi <<
36 honored on every architecture running tests. <<
37 <<
38 These files are organized in three columns. Th <<
39 question. This can be the name of a test suite <<
40 remaining two columns provide additional meta <<
41 classify the entry: column two is a copy and p <<
42 when running the test in the setting in questi <<
43 available, summarizes the underlying problem. <<
44 example, indicates that lack of trampoline sup <<
45 This last entry helps identify tests that can <<
46 added. <<
47 <<
48 ========================= 9 =========================
49 Running Selftests in a VM 10 Running Selftests in a VM
50 ========================= 11 =========================
51 12
52 It's now possible to run the selftests using ` 13 It's now possible to run the selftests using ``tools/testing/selftests/bpf/vmtest.sh``.
53 The script tries to ensure that the tests are 14 The script tries to ensure that the tests are run with the same environment as they
54 would be run post-submit in the CI used by the !! 15 would be run post-submit in the CI used by the Maintainers.
55 that deny lists are not automatically honored. <<
56 16
57 This script uses the in-tree kernel configurat !! 17 This script downloads a suitable Kconfig and VM userspace image from the system used by
58 image from the system used by the CI. It build !! 18 the CI. It builds the kernel (without overwriting your existing Kconfig), recompiles the
59 your existing Kconfig), recompiles the bpf sel !! 19 bpf selftests, runs them (by default ``tools/testing/selftests/bpf/test_progs``) and
60 ``tools/testing/selftests/bpf/test_progs``) an !! 20 saves the resulting output (by default in ``~/.bpf_selftests``).
61 default in ``~/.bpf_selftests``). <<
62 21
63 Script dependencies: 22 Script dependencies:
64 - clang (preferably built from sources, https: 23 - clang (preferably built from sources, https://github.com/llvm/llvm-project);
65 - pahole (preferably built from sources, https 24 - pahole (preferably built from sources, https://git.kernel.org/pub/scm/devel/pahole/pahole.git/);
66 - qemu; 25 - qemu;
67 - docutils (for ``rst2man``); 26 - docutils (for ``rst2man``);
68 - libcap-devel. 27 - libcap-devel.
69 28
70 For more information about using the script, r !! 29 For more information on about using the script, run:
71 30
72 .. code-block:: console 31 .. code-block:: console
73 32
74 $ tools/testing/selftests/bpf/vmtest.sh -h 33 $ tools/testing/selftests/bpf/vmtest.sh -h
75 34
76 In case of linker errors when running selftest <<
77 <<
78 .. code-block:: console <<
79 <<
80 $ LDLIBS=-static PKG_CONFIG='pkg-config --st <<
81 <<
82 .. note:: Some distros may not support static <<
83 <<
84 .. note:: The script uses pahole and clang bas 35 .. note:: The script uses pahole and clang based on host environment setting.
85 If you want to change pahole and llv 36 If you want to change pahole and llvm, you can change `PATH` environment
86 variable in the beginning of script. 37 variable in the beginning of script.
87 38
88 Running vmtest on RV64 !! 39 .. note:: The script currently only supports x86_64.
89 ====================== <<
90 To speed up testing and avoid various dependen <<
91 run vmtest in a Docker container. Before runni <<
92 Docker container and local rootfs image. The o <<
93 <<
94 1. Create Docker container as shown in link [0 <<
95 <<
96 2. Use mkrootfs_debian.sh script [1] to build <<
97 <<
98 .. code-block:: console <<
99 <<
100 $ sudo ./mkrootfs_debian.sh --arch riscv64 - <<
101 <<
102 3. Start Docker container [0] and run vmtest i <<
103 <<
104 .. code-block:: console <<
105 <<
106 $ PLATFORM=riscv64 CROSS_COMPILE=riscv64-lin <<
107 tools/testing/selftests/bpf/vmtest.sh \ <<
108 -l <path of local rootfs image> -- \ <<
109 ./test_progs -d \ <<
110 \"$(cat tools/testing/selftests/bpf/DE <<
111 | cut -d'#' -f1 \ <<
112 | sed -e 's/^[[:space:]]*//' \ <<
113 -e 's/[[:space:]]*$//' \ <<
114 | tr -s '\n' ',' \ <<
115 )\" <<
116 <<
117 Link: https://github.com/pulehui/riscv-bpf-vmt <<
118 Link: https://github.com/libbpf/ci/blob/main/r <<
119 40
120 Additional information about selftest failures 41 Additional information about selftest failures are
121 documented here. 42 documented here.
122 43
123 profiler[23] test failures with clang/llvm <12 44 profiler[23] test failures with clang/llvm <12.0.0
124 ============================================== 45 ==================================================
125 46
126 With clang/llvm <12.0.0, the profiler[23] test 47 With clang/llvm <12.0.0, the profiler[23] test may fail.
127 The symptom looks like 48 The symptom looks like
128 49
129 .. code-block:: c 50 .. code-block:: c
130 51
131 // r9 is a pointer to map_value 52 // r9 is a pointer to map_value
132 // r7 is a scalar 53 // r7 is a scalar
133 17: bf 96 00 00 00 00 00 00 r6 = r9 54 17: bf 96 00 00 00 00 00 00 r6 = r9
134 18: 0f 76 00 00 00 00 00 00 r6 += r7 55 18: 0f 76 00 00 00 00 00 00 r6 += r7
135 math between map_value pointer and register 56 math between map_value pointer and register with unbounded min value is not allowed
136 57
137 // the instructions below will not be seen i 58 // the instructions below will not be seen in the verifier log
138 19: a5 07 01 00 01 01 00 00 if r7 < 25 59 19: a5 07 01 00 01 01 00 00 if r7 < 257 goto +1
139 20: bf 96 00 00 00 00 00 00 r6 = r9 60 20: bf 96 00 00 00 00 00 00 r6 = r9
140 // r6 is used here 61 // r6 is used here
141 62
142 The verifier will reject such code with above 63 The verifier will reject such code with above error.
143 At insn 18 the r7 is indeed unbounded. The lat 64 At insn 18 the r7 is indeed unbounded. The later insn 19 checks the bounds and
144 the insn 20 undoes map_value addition. It is c 65 the insn 20 undoes map_value addition. It is currently impossible for the
145 verifier to understand such speculative pointe 66 verifier to understand such speculative pointer arithmetic.
146 Hence `this patch`__ addresses it on the compi 67 Hence `this patch`__ addresses it on the compiler side. It was committed on llvm 12.
147 68
148 __ https://github.com/llvm/llvm-project/commit !! 69 __ https://reviews.llvm.org/D85570
149 70
150 The corresponding C code 71 The corresponding C code
151 72
152 .. code-block:: c 73 .. code-block:: c
153 74
154 for (int i = 0; i < MAX_CGROUPS_PATH_DEPTH; 75 for (int i = 0; i < MAX_CGROUPS_PATH_DEPTH; i++) {
155 filepart_length = bpf_probe_read_str 76 filepart_length = bpf_probe_read_str(payload, ...);
156 if (filepart_length <= MAX_PATH) { 77 if (filepart_length <= MAX_PATH) {
157 barrier_var(filepart_length) 78 barrier_var(filepart_length); // workaround
158 payload += filepart_length; 79 payload += filepart_length;
159 } 80 }
160 } 81 }
161 82
162 bpf_iter test failures with clang/llvm 10.0.0 83 bpf_iter test failures with clang/llvm 10.0.0
163 ============================================= 84 =============================================
164 85
165 With clang/llvm 10.0.0, the following two bpf_ 86 With clang/llvm 10.0.0, the following two bpf_iter tests failed:
166 * ``bpf_iter/ipv6_route`` 87 * ``bpf_iter/ipv6_route``
167 * ``bpf_iter/netlink`` 88 * ``bpf_iter/netlink``
168 89
169 The symptom for ``bpf_iter/ipv6_route`` looks 90 The symptom for ``bpf_iter/ipv6_route`` looks like
170 91
171 .. code-block:: c 92 .. code-block:: c
172 93
173 2: (79) r8 = *(u64 *)(r1 +8) 94 2: (79) r8 = *(u64 *)(r1 +8)
174 ... 95 ...
175 14: (bf) r2 = r8 96 14: (bf) r2 = r8
176 15: (0f) r2 += r1 97 15: (0f) r2 += r1
177 ; BPF_SEQ_PRINTF(seq, "%pi6 %02x ", &rt->fib 98 ; BPF_SEQ_PRINTF(seq, "%pi6 %02x ", &rt->fib6_dst.addr, rt->fib6_dst.plen);
178 16: (7b) *(u64 *)(r8 +64) = r2 99 16: (7b) *(u64 *)(r8 +64) = r2
179 only read is supported 100 only read is supported
180 101
181 The symptom for ``bpf_iter/netlink`` looks lik 102 The symptom for ``bpf_iter/netlink`` looks like
182 103
183 .. code-block:: c 104 .. code-block:: c
184 105
185 ; struct netlink_sock *nlk = ctx->sk; 106 ; struct netlink_sock *nlk = ctx->sk;
186 2: (79) r7 = *(u64 *)(r1 +8) 107 2: (79) r7 = *(u64 *)(r1 +8)
187 ... 108 ...
188 15: (bf) r2 = r7 109 15: (bf) r2 = r7
189 16: (0f) r2 += r1 110 16: (0f) r2 += r1
190 ; BPF_SEQ_PRINTF(seq, "%pK %-3d ", s, s->sk_ 111 ; BPF_SEQ_PRINTF(seq, "%pK %-3d ", s, s->sk_protocol);
191 17: (7b) *(u64 *)(r7 +0) = r2 112 17: (7b) *(u64 *)(r7 +0) = r2
192 only read is supported 113 only read is supported
193 114
194 This is due to a llvm BPF backend bug. `The fi 115 This is due to a llvm BPF backend bug. `The fix`__
195 has been pushed to llvm 10.x release branch an 116 has been pushed to llvm 10.x release branch and will be
196 available in 10.0.1. The patch is available in 117 available in 10.0.1. The patch is available in llvm 11.0.0 trunk.
197 118
198 __ https://github.com/llvm/llvm-project/commi !! 119 __ https://reviews.llvm.org/D78466
199 120
200 bpf_verif_scale/loop6.bpf.o test failure with !! 121 bpf_verif_scale/loop6.o test failure with Clang 12
201 ============================================== !! 122 ==================================================
202 123
203 With Clang 12, the following bpf_verif_scale t 124 With Clang 12, the following bpf_verif_scale test failed:
204 * ``bpf_verif_scale/loop6.bpf.o`` !! 125 * ``bpf_verif_scale/loop6.o``
205 126
206 The verifier output looks like 127 The verifier output looks like
207 128
208 .. code-block:: c 129 .. code-block:: c
209 130
210 R1 type=ctx expected=fp 131 R1 type=ctx expected=fp
211 The sequence of 8193 jumps is too complex. 132 The sequence of 8193 jumps is too complex.
212 133
213 The reason is compiler generating the followin 134 The reason is compiler generating the following code
214 135
215 .. code-block:: c 136 .. code-block:: c
216 137
217 ; for (i = 0; (i < VIRTIO_MAX_SGS) && 138 ; for (i = 0; (i < VIRTIO_MAX_SGS) && (i < num); i++) {
218 14: 16 05 40 00 00 00 00 00 if w5 139 14: 16 05 40 00 00 00 00 00 if w5 == 0 goto +64 <LBB0_6>
219 15: bc 51 00 00 00 00 00 00 w1 = w 140 15: bc 51 00 00 00 00 00 00 w1 = w5
220 16: 04 01 00 00 ff ff ff ff w1 += 141 16: 04 01 00 00 ff ff ff ff w1 += -1
221 17: 67 05 00 00 20 00 00 00 r5 <<= 142 17: 67 05 00 00 20 00 00 00 r5 <<= 32
222 18: 77 05 00 00 20 00 00 00 r5 >>= 143 18: 77 05 00 00 20 00 00 00 r5 >>= 32
223 19: a6 01 01 00 05 00 00 00 if w1 144 19: a6 01 01 00 05 00 00 00 if w1 < 5 goto +1 <LBB0_4>
224 20: b7 05 00 00 06 00 00 00 r5 = 6 145 20: b7 05 00 00 06 00 00 00 r5 = 6
225 00000000000000a8 <LBB0_4>: 146 00000000000000a8 <LBB0_4>:
226 21: b7 02 00 00 00 00 00 00 r2 = 0 147 21: b7 02 00 00 00 00 00 00 r2 = 0
227 22: b7 01 00 00 00 00 00 00 r1 = 0 148 22: b7 01 00 00 00 00 00 00 r1 = 0
228 ; for (i = 0; (i < VIRTIO_MAX_SGS) && 149 ; for (i = 0; (i < VIRTIO_MAX_SGS) && (i < num); i++) {
229 23: 7b 1a e0 ff 00 00 00 00 *(u64 150 23: 7b 1a e0 ff 00 00 00 00 *(u64 *)(r10 - 32) = r1
230 24: 7b 5a c0 ff 00 00 00 00 *(u64 151 24: 7b 5a c0 ff 00 00 00 00 *(u64 *)(r10 - 64) = r5
231 152
232 Note that insn #15 has w1 = w5 and w1 is refin 153 Note that insn #15 has w1 = w5 and w1 is refined later but
233 r5(w5) is eventually saved on stack at insn #2 154 r5(w5) is eventually saved on stack at insn #24 for later use.
234 This cause later verifier failure. The bug has 155 This cause later verifier failure. The bug has been `fixed`__ in
235 Clang 13. 156 Clang 13.
236 157
237 __ https://github.com/llvm/llvm-project/commi !! 158 __ https://reviews.llvm.org/D97479
238 159
239 BPF CO-RE-based tests and Clang version 160 BPF CO-RE-based tests and Clang version
240 ======================================= 161 =======================================
241 162
242 A set of selftests use BPF target-specific bui 163 A set of selftests use BPF target-specific built-ins, which might require
243 bleeding-edge Clang versions (Clang 12 nightly 164 bleeding-edge Clang versions (Clang 12 nightly at this time).
244 165
245 Few sub-tests of core_reloc test suit (part of 166 Few sub-tests of core_reloc test suit (part of test_progs test runner) require
246 the following built-ins, listed with correspon 167 the following built-ins, listed with corresponding Clang diffs introducing
247 them to Clang/LLVM. These sub-tests are going 168 them to Clang/LLVM. These sub-tests are going to be skipped if Clang is too
248 old to support them, they shouldn't cause buil 169 old to support them, they shouldn't cause build failures or runtime test
249 failures: 170 failures:
250 171
251 - __builtin_btf_type_id() [0_, 1_, 2_]; 172 - __builtin_btf_type_id() [0_, 1_, 2_];
252 - __builtin_preserve_type_info(), __builtin_pr 173 - __builtin_preserve_type_info(), __builtin_preserve_enum_value() [3_, 4_].
253 174
254 .. _0: https://github.com/llvm/llvm-project/co !! 175 .. _0: https://reviews.llvm.org/D74572
255 .. _1: https://github.com/llvm/llvm-project/co !! 176 .. _1: https://reviews.llvm.org/D74668
256 .. _2: https://github.com/llvm/llvm-project/co !! 177 .. _2: https://reviews.llvm.org/D85174
257 .. _3: https://github.com/llvm/llvm-project/co !! 178 .. _3: https://reviews.llvm.org/D83878
258 .. _4: https://github.com/llvm/llvm-project/co !! 179 .. _4: https://reviews.llvm.org/D83242
259 180
260 Floating-point tests and Clang version 181 Floating-point tests and Clang version
261 ====================================== 182 ======================================
262 183
263 Certain selftests, e.g. core_reloc, require su 184 Certain selftests, e.g. core_reloc, require support for the floating-point
264 types, which was introduced in `Clang 13`__. T 185 types, which was introduced in `Clang 13`__. The older Clang versions will
265 either crash when compiling these tests, or ge 186 either crash when compiling these tests, or generate an incorrect BTF.
266 187
267 __ https://github.com/llvm/llvm-project/commi !! 188 __ https://reviews.llvm.org/D83289
268 189
269 Kernel function call test and Clang version 190 Kernel function call test and Clang version
270 =========================================== 191 ===========================================
271 192
272 Some selftests (e.g. kfunc_call and bpf_tcp_ca 193 Some selftests (e.g. kfunc_call and bpf_tcp_ca) require a LLVM support
273 to generate extern function in BTF. It was in 194 to generate extern function in BTF. It was introduced in `Clang 13`__.
274 195
275 Without it, the error from compiling bpf selft 196 Without it, the error from compiling bpf selftests looks like:
276 197
277 .. code-block:: console 198 .. code-block:: console
278 199
279 libbpf: failed to find BTF for extern 'tcp_s 200 libbpf: failed to find BTF for extern 'tcp_slow_start' [25] section: -2
280 201
281 __ https://github.com/llvm/llvm-project/commit !! 202 __ https://reviews.llvm.org/D93563
282 203
283 btf_tag test and Clang version 204 btf_tag test and Clang version
284 ============================== 205 ==============================
285 206
286 The btf_tag selftest requires LLVM support to 207 The btf_tag selftest requires LLVM support to recognize the btf_decl_tag and
287 btf_type_tag attributes. They are introduced i 208 btf_type_tag attributes. They are introduced in `Clang 14` [0_, 1_].
288 The subtests ``btf_type_tag_user_{mod1, mod2, <<
289 pahole version ``1.23``. <<
290 209
291 Without them, the btf_tag selftest will be ski 210 Without them, the btf_tag selftest will be skipped and you will observe:
292 211
293 .. code-block:: console 212 .. code-block:: console
294 213
295 #<test_num> btf_tag:SKIP 214 #<test_num> btf_tag:SKIP
296 215
297 .. _0: https://github.com/llvm/llvm-project/co !! 216 .. _0: https://reviews.llvm.org/D111588
298 .. _1: https://github.com/llvm/llvm-project/co !! 217 .. _1: https://reviews.llvm.org/D111199
299 218
300 Clang dependencies for static linking tests 219 Clang dependencies for static linking tests
301 =========================================== 220 ===========================================
302 221
303 linked_vars, linked_maps, and linked_funcs tes 222 linked_vars, linked_maps, and linked_funcs tests depend on `Clang fix`__ to
304 generate valid BTF information for weak variab 223 generate valid BTF information for weak variables. Please make sure you use
305 Clang that contains the fix. 224 Clang that contains the fix.
306 225
307 __ https://github.com/llvm/llvm-project/commit !! 226 __ https://reviews.llvm.org/D100362
308 227
309 Clang relocation changes 228 Clang relocation changes
310 ======================== 229 ========================
311 230
312 Clang 13 patch `clang reloc patch`_ made some 231 Clang 13 patch `clang reloc patch`_ made some changes on relocations such
313 that existing relocation types are broken into 232 that existing relocation types are broken into more types and
314 each new type corresponds to only one way to r 233 each new type corresponds to only one way to resolve relocation.
315 See `kernel llvm reloc`_ for more explanation 234 See `kernel llvm reloc`_ for more explanation and some examples.
316 Using clang 13 to compile old libbpf which has 235 Using clang 13 to compile old libbpf which has static linker support,
317 there will be a compilation failure:: 236 there will be a compilation failure::
318 237
319 libbpf: ELF relo #0 in section #6 has unexpe !! 238 libbpf: ELF relo #0 in section #6 has unexpected type 2 in .../bpf_tcp_nogpl.o
320 239
321 Here, ``type 2`` refers to new relocation type 240 Here, ``type 2`` refers to new relocation type ``R_BPF_64_ABS64``.
322 To fix this issue, user newer libbpf. 241 To fix this issue, user newer libbpf.
323 242
324 .. Links 243 .. Links
325 .. _clang reloc patch: https://github.com/llvm !! 244 .. _clang reloc patch: https://reviews.llvm.org/D102712
326 .. _kernel llvm reloc: /Documentation/bpf/llvm 245 .. _kernel llvm reloc: /Documentation/bpf/llvm_reloc.rst
327 246
328 Clang dependencies for the u32 spill test (xdp 247 Clang dependencies for the u32 spill test (xdpwall)
329 ============================================== 248 ===================================================
330 The xdpwall selftest requires a change in `Cla 249 The xdpwall selftest requires a change in `Clang 14`__.
331 250
332 Without it, the xdpwall selftest will fail and 251 Without it, the xdpwall selftest will fail and the error message
333 from running test_progs will look like: 252 from running test_progs will look like:
334 253
335 .. code-block:: console 254 .. code-block:: console
336 255
337 test_xdpwall:FAIL:Does LLVM have https://git !! 256 test_xdpwall:FAIL:Does LLVM have https://reviews.llvm.org/D109073? unexpected error: -4007
338 257
339 __ https://github.com/llvm/llvm-project/commit !! 258 __ https://reviews.llvm.org/D109073
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.