1 ============== 2 BPF Design Q&A 3 ============== 4 5 BPF extensibility and applicability to networking, tracing, security 6 in the linux kernel and several user space implementations of BPF 7 virtual machine led to a number of misunderstanding on what BPF actually is. 8 This short QA is an attempt to address that and outline a direction 9 of where BPF is heading long term. 10 11 .. contents:: 12 :local: 13 :depth: 3 14 15 Questions and Answers 16 ===================== 17 18 Q: Is BPF a generic instruction set similar to x64 and arm64? 19 ------------------------------------------------------------- 20 A: NO. 21 22 Q: Is BPF a generic virtual machine ? 23 ------------------------------------- 24 A: NO. 25 26 BPF is generic instruction set *with* C calling convention. 27 ----------------------------------------------------------- 28 29 Q: Why C calling convention was chosen? 30 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 31 32 A: Because BPF programs are designed to run in the linux kernel 33 which is written in C, hence BPF defines instruction set compatible 34 with two most used architectures x64 and arm64 (and takes into 35 consideration important quirks of other architectures) and 36 defines calling convention that is compatible with C calling 37 convention of the linux kernel on those architectures. 38 39 Q: Can multiple return values be supported in the future? 40 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 41 A: NO. BPF allows only register R0 to be used as return value. 42 43 Q: Can more than 5 function arguments be supported in the future? 44 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 45 A: NO. BPF calling convention only allows registers R1-R5 to be used 46 as arguments. BPF is not a standalone instruction set. 47 (unlike x64 ISA that allows msft, cdecl and other conventions) 48 49 Q: Can BPF programs access instruction pointer or return address? 50 ----------------------------------------------------------------- 51 A: NO. 52 53 Q: Can BPF programs access stack pointer ? 54 ------------------------------------------ 55 A: NO. 56 57 Only frame pointer (register R10) is accessible. 58 From compiler point of view it's necessary to have stack pointer. 59 For example, LLVM defines register R11 as stack pointer in its 60 BPF backend, but it makes sure that generated code never uses it. 61 62 Q: Does C-calling convention diminishes possible use cases? 63 ----------------------------------------------------------- 64 A: YES. 65 66 BPF design forces addition of major functionality in the form 67 of kernel helper functions and kernel objects like BPF maps with 68 seamless interoperability between them. It lets kernel call into 69 BPF programs and programs call kernel helpers with zero overhead, 70 as all of them were native C code. That is particularly the case 71 for JITed BPF programs that are indistinguishable from 72 native kernel C code. 73 74 Q: Does it mean that 'innovative' extensions to BPF code are disallowed? 75 ------------------------------------------------------------------------ 76 A: Soft yes. 77 78 At least for now, until BPF core has support for 79 bpf-to-bpf calls, indirect calls, loops, global variables, 80 jump tables, read-only sections, and all other normal constructs 81 that C code can produce. 82 83 Q: Can loops be supported in a safe way? 84 ---------------------------------------- 85 A: It's not clear yet. 86 87 BPF developers are trying to find a way to 88 support bounded loops. 89 90 Q: What are the verifier limits? 91 -------------------------------- 92 A: The only limit known to the user space is BPF_MAXINSNS (4096). 93 It's the maximum number of instructions that the unprivileged bpf 94 program can have. The verifier has various internal limits. 95 Like the maximum number of instructions that can be explored during 96 program analysis. Currently, that limit is set to 1 million. 97 Which essentially means that the largest program can consist 98 of 1 million NOP instructions. There is a limit to the maximum number 99 of subsequent branches, a limit to the number of nested bpf-to-bpf 100 calls, a limit to the number of the verifier states per instruction, 101 a limit to the number of maps used by the program. 102 All these limits can be hit with a sufficiently complex program. 103 There are also non-numerical limits that can cause the program 104 to be rejected. The verifier used to recognize only pointer + constant 105 expressions. Now it can recognize pointer + bounded_register. 106 bpf_lookup_map_elem(key) had a requirement that 'key' must be 107 a pointer to the stack. Now, 'key' can be a pointer to map value. 108 The verifier is steadily getting 'smarter'. The limits are 109 being removed. The only way to know that the program is going to 110 be accepted by the verifier is to try to load it. 111 The bpf development process guarantees that the future kernel 112 versions will accept all bpf programs that were accepted by 113 the earlier versions. 114 115 116 Instruction level questions 117 --------------------------- 118 119 Q: LD_ABS and LD_IND instructions vs C code 120 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 121 122 Q: How come LD_ABS and LD_IND instruction are present in BPF whereas 123 C code cannot express them and has to use builtin intrinsics? 124 125 A: This is artifact of compatibility with classic BPF. Modern 126 networking code in BPF performs better without them. 127 See 'direct packet access'. 128 129 Q: BPF instructions mapping not one-to-one to native CPU 130 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 131 Q: It seems not all BPF instructions are one-to-one to native CPU. 132 For example why BPF_JNE and other compare and jumps are not cpu-like? 133 134 A: This was necessary to avoid introducing flags into ISA which are 135 impossible to make generic and efficient across CPU architectures. 136 137 Q: Why BPF_DIV instruction doesn't map to x64 div? 138 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 139 A: Because if we picked one-to-one relationship to x64 it would have made 140 it more complicated to support on arm64 and other archs. Also it 141 needs div-by-zero runtime check. 142 143 Q: Why BPF has implicit prologue and epilogue? 144 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 145 A: Because architectures like sparc have register windows and in general 146 there are enough subtle differences between architectures, so naive 147 store return address into stack won't work. Another reason is BPF has 148 to be safe from division by zero (and legacy exception path 149 of LD_ABS insn). Those instructions need to invoke epilogue and 150 return implicitly. 151 152 Q: Why BPF_JLT and BPF_JLE instructions were not introduced in the beginning? 153 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 154 A: Because classic BPF didn't have them and BPF authors felt that compiler 155 workaround would be acceptable. Turned out that programs lose performance 156 due to lack of these compare instructions and they were added. 157 These two instructions is a perfect example what kind of new BPF 158 instructions are acceptable and can be added in the future. 159 These two already had equivalent instructions in native CPUs. 160 New instructions that don't have one-to-one mapping to HW instructions 161 will not be accepted. 162 163 Q: BPF 32-bit subregister requirements 164 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 165 Q: BPF 32-bit subregisters have a requirement to zero upper 32-bits of BPF 166 registers which makes BPF inefficient virtual machine for 32-bit 167 CPU architectures and 32-bit HW accelerators. Can true 32-bit registers 168 be added to BPF in the future? 169 170 A: NO. 171 172 But some optimizations on zero-ing the upper 32 bits for BPF registers are 173 available, and can be leveraged to improve the performance of JITed BPF 174 programs for 32-bit architectures. 175 176 Starting with version 7, LLVM is able to generate instructions that operate 177 on 32-bit subregisters, provided the option -mattr=+alu32 is passed for 178 compiling a program. Furthermore, the verifier can now mark the 179 instructions for which zero-ing the upper bits of the destination register 180 is required, and insert an explicit zero-extension (zext) instruction 181 (a mov32 variant). This means that for architectures without zext hardware 182 support, the JIT back-ends do not need to clear the upper bits for 183 subregisters written by alu32 instructions or narrow loads. Instead, the 184 back-ends simply need to support code generation for that mov32 variant, 185 and to overwrite bpf_jit_needs_zext() to make it return "true" (in order to 186 enable zext insertion in the verifier). 187 188 Note that it is possible for a JIT back-end to have partial hardware 189 support for zext. In that case, if verifier zext insertion is enabled, 190 it could lead to the insertion of unnecessary zext instructions. Such 191 instructions could be removed by creating a simple peephole inside the JIT 192 back-end: if one instruction has hardware support for zext and if the next 193 instruction is an explicit zext, then the latter can be skipped when doing 194 the code generation. 195 196 Q: Does BPF have a stable ABI? 197 ------------------------------ 198 A: YES. BPF instructions, arguments to BPF programs, set of helper 199 functions and their arguments, recognized return codes are all part 200 of ABI. However there is one specific exception to tracing programs 201 which are using helpers like bpf_probe_read() to walk kernel internal 202 data structures and compile with kernel internal headers. Both of these 203 kernel internals are subject to change and can break with newer kernels 204 such that the program needs to be adapted accordingly. 205 206 New BPF functionality is generally added through the use of kfuncs instead of 207 new helpers. Kfuncs are not considered part of the stable API, and have their own 208 lifecycle expectations as described in :ref:`BPF_kfunc_lifecycle_expectations`. 209 210 Q: Are tracepoints part of the stable ABI? 211 ------------------------------------------ 212 A: NO. Tracepoints are tied to internal implementation details hence they are 213 subject to change and can break with newer kernels. BPF programs need to change 214 accordingly when this happens. 215 216 Q: Are places where kprobes can attach part of the stable ABI? 217 -------------------------------------------------------------- 218 A: NO. The places to which kprobes can attach are internal implementation 219 details, which means that they are subject to change and can break with 220 newer kernels. BPF programs need to change accordingly when this happens. 221 222 Q: How much stack space a BPF program uses? 223 ------------------------------------------- 224 A: Currently all program types are limited to 512 bytes of stack 225 space, but the verifier computes the actual amount of stack used 226 and both interpreter and most JITed code consume necessary amount. 227 228 Q: Can BPF be offloaded to HW? 229 ------------------------------ 230 A: YES. BPF HW offload is supported by NFP driver. 231 232 Q: Does classic BPF interpreter still exist? 233 -------------------------------------------- 234 A: NO. Classic BPF programs are converted into extend BPF instructions. 235 236 Q: Can BPF call arbitrary kernel functions? 237 ------------------------------------------- 238 A: NO. BPF programs can only call specific functions exposed as BPF helpers or 239 kfuncs. The set of available functions is defined for every program type. 240 241 Q: Can BPF overwrite arbitrary kernel memory? 242 --------------------------------------------- 243 A: NO. 244 245 Tracing bpf programs can *read* arbitrary memory with bpf_probe_read() 246 and bpf_probe_read_str() helpers. Networking programs cannot read 247 arbitrary memory, since they don't have access to these helpers. 248 Programs can never read or write arbitrary memory directly. 249 250 Q: Can BPF overwrite arbitrary user memory? 251 ------------------------------------------- 252 A: Sort-of. 253 254 Tracing BPF programs can overwrite the user memory 255 of the current task with bpf_probe_write_user(). Every time such 256 program is loaded the kernel will print warning message, so 257 this helper is only useful for experiments and prototypes. 258 Tracing BPF programs are root only. 259 260 Q: New functionality via kernel modules? 261 ---------------------------------------- 262 Q: Can BPF functionality such as new program or map types, new 263 helpers, etc be added out of kernel module code? 264 265 A: Yes, through kfuncs and kptrs 266 267 The core BPF functionality such as program types, maps and helpers cannot be 268 added to by modules. However, modules can expose functionality to BPF programs 269 by exporting kfuncs (which may return pointers to module-internal data 270 structures as kptrs). 271 272 Q: Directly calling kernel function is an ABI? 273 ---------------------------------------------- 274 Q: Some kernel functions (e.g. tcp_slow_start) can be called 275 by BPF programs. Do these kernel functions become an ABI? 276 277 A: NO. 278 279 The kernel function protos will change and the bpf programs will be 280 rejected by the verifier. Also, for example, some of the bpf-callable 281 kernel functions have already been used by other kernel tcp 282 cc (congestion-control) implementations. If any of these kernel 283 functions has changed, both the in-tree and out-of-tree kernel tcp cc 284 implementations have to be changed. The same goes for the bpf 285 programs and they have to be adjusted accordingly. See 286 :ref:`BPF_kfunc_lifecycle_expectations` for details. 287 288 Q: Attaching to arbitrary kernel functions is an ABI? 289 ----------------------------------------------------- 290 Q: BPF programs can be attached to many kernel functions. Do these 291 kernel functions become part of the ABI? 292 293 A: NO. 294 295 The kernel function prototypes will change, and BPF programs attaching to 296 them will need to change. The BPF compile-once-run-everywhere (CO-RE) 297 should be used in order to make it easier to adapt your BPF programs to 298 different versions of the kernel. 299 300 Q: Marking a function with BTF_ID makes that function an ABI? 301 ------------------------------------------------------------- 302 A: NO. 303 304 The BTF_ID macro does not cause a function to become part of the ABI 305 any more than does the EXPORT_SYMBOL_GPL macro. 306 307 Q: What is the compatibility story for special BPF types in map values? 308 ----------------------------------------------------------------------- 309 Q: Users are allowed to embed bpf_spin_lock, bpf_timer fields in their BPF map 310 values (when using BTF support for BPF maps). This allows to use helpers for 311 such objects on these fields inside map values. Users are also allowed to embed 312 pointers to some kernel types (with __kptr_untrusted and __kptr BTF tags). Will the 313 kernel preserve backwards compatibility for these features? 314 315 A: It depends. For bpf_spin_lock, bpf_timer: YES, for kptr and everything else: 316 NO, but see below. 317 318 For struct types that have been added already, like bpf_spin_lock and bpf_timer, 319 the kernel will preserve backwards compatibility, as they are part of UAPI. 320 321 For kptrs, they are also part of UAPI, but only with respect to the kptr 322 mechanism. The types that you can use with a __kptr_untrusted and __kptr tagged 323 pointer in your struct are NOT part of the UAPI contract. The supported types can 324 and will change across kernel releases. However, operations like accessing kptr 325 fields and bpf_kptr_xchg() helper will continue to be supported across kernel 326 releases for the supported types. 327 328 For any other supported struct type, unless explicitly stated in this document 329 and added to bpf.h UAPI header, such types can and will arbitrarily change their 330 size, type, and alignment, or any other user visible API or ABI detail across 331 kernel releases. The users must adapt their BPF programs to the new changes and 332 update them to make sure their programs continue to work correctly. 333 334 NOTE: BPF subsystem specially reserves the 'bpf\_' prefix for type names, in 335 order to introduce more special fields in the future. Hence, user programs must 336 avoid defining types with 'bpf\_' prefix to not be broken in future releases. 337 In other words, no backwards compatibility is guaranteed if one using a type 338 in BTF with 'bpf\_' prefix. 339 340 Q: What is the compatibility story for special BPF types in allocated objects? 341 ------------------------------------------------------------------------------ 342 Q: Same as above, but for allocated objects (i.e. objects allocated using 343 bpf_obj_new for user defined types). Will the kernel preserve backwards 344 compatibility for these features? 345 346 A: NO. 347 348 Unlike map value types, the API to work with allocated objects and any support 349 for special fields inside them is exposed through kfuncs, and thus has the same 350 lifecycle expectations as the kfuncs themselves. See 351 :ref:`BPF_kfunc_lifecycle_expectations` for details.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.