1 .. SPDX-License-Identifier: GPL-2.0
2
3 ======
4 AF_XDP
5 ======
6
7 Overview
8 ========
9
10 AF_XDP is an address family that is optimized for high performance
11 packet processing.
12
13 This document assumes that the reader is familiar with BPF and XDP. If
14 not, the Cilium project has an excellent reference guide at
15 http://cilium.readthedocs.io/en/latest/bpf/.
16
17 Using the XDP_REDIRECT action from an XDP program, the program can
18 redirect ingress frames to other XDP enabled netdevs, using the
19 bpf_redirect_map() function. AF_XDP sockets enable the possibility for
20 XDP programs to redirect frames to a memory buffer in a user-space
21 application.
22
23 An AF_XDP socket (XSK) is created with the normal socket()
24 syscall. Associated with each XSK are two rings: the RX ring and the
25 TX ring. A socket can receive packets on the RX ring and it can send
26 packets on the TX ring. These rings are registered and sized with the
27 setsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatory
28 to have at least one of these rings for each socket. An RX or TX
29 descriptor ring points to a data buffer in a memory area called a
30 UMEM. RX and TX can share the same UMEM so that a packet does not have
31 to be copied between RX and TX. Moreover, if a packet needs to be kept
32 for a while due to a possible retransmit, the descriptor that points
33 to that packet can be changed to point to another and reused right
34 away. This again avoids copying data.
35
36 The UMEM consists of a number of equally sized chunks. A descriptor in
37 one of the rings references a frame by referencing its addr. The addr
38 is simply an offset within the entire UMEM region. The user space
39 allocates memory for this UMEM using whatever means it feels is most
40 appropriate (malloc, mmap, huge pages, etc). This memory area is then
41 registered with the kernel using the new setsockopt XDP_UMEM_REG. The
42 UMEM also has two rings: the FILL ring and the COMPLETION ring. The
43 FILL ring is used by the application to send down addr for the kernel
44 to fill in with RX packet data. References to these frames will then
45 appear in the RX ring once each packet has been received. The
46 COMPLETION ring, on the other hand, contains frame addr that the
47 kernel has transmitted completely and can now be used again by user
48 space, for either TX or RX. Thus, the frame addrs appearing in the
49 COMPLETION ring are addrs that were previously transmitted using the
50 TX ring. In summary, the RX and FILL rings are used for the RX path
51 and the TX and COMPLETION rings are used for the TX path.
52
53 The socket is then finally bound with a bind() call to a device and a
54 specific queue id on that device, and it is not until bind is
55 completed that traffic starts to flow.
56
57 The UMEM can be shared between processes, if desired. If a process
58 wants to do this, it simply skips the registration of the UMEM and its
59 corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind
60 call and submits the XSK of the process it would like to share UMEM
61 with as well as its own newly created XSK socket. The new process will
62 then receive frame addr references in its own RX ring that point to
63 this shared UMEM. Note that since the ring structures are
64 single-consumer / single-producer (for performance reasons), the new
65 process has to create its own socket with associated RX and TX rings,
66 since it cannot share this with the other process. This is also the
67 reason that there is only one set of FILL and COMPLETION rings per
68 UMEM. It is the responsibility of a single process to handle the UMEM.
69
70 How is then packets distributed from an XDP program to the XSKs? There
71 is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The
72 user-space application can place an XSK at an arbitrary place in this
73 map. The XDP program can then redirect a packet to a specific index in
74 this map and at this point XDP validates that the XSK in that map was
75 indeed bound to that device and ring number. If not, the packet is
76 dropped. If the map is empty at that index, the packet is also
77 dropped. This also means that it is currently mandatory to have an XDP
78 program loaded (and one XSK in the XSKMAP) to be able to get any
79 traffic to user space through the XSK.
80
81 AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the
82 driver does not have support for XDP, or XDP_SKB is explicitly chosen
83 when loading the XDP program, XDP_SKB mode is employed that uses SKBs
84 together with the generic XDP support and copies out the data to user
85 space. A fallback mode that works for any network device. On the other
86 hand, if the driver has support for XDP, it will be used by the AF_XDP
87 code to provide better performance, but there is still a copy of the
88 data into user space.
89
90 Concepts
91 ========
92
93 In order to use an AF_XDP socket, a number of associated objects need
94 to be setup. These objects and their options are explained in the
95 following sections.
96
97 For an overview on how AF_XDP works, you can also take a look at the
98 Linux Plumbers paper from 2018 on the subject:
99 http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. Do
100 NOT consult the paper from 2017 on "AF_PACKET v4", the first attempt
101 at AF_XDP. Nearly everything changed since then. Jonathan Corbet has
102 also written an excellent article on LWN, "Accelerating networking
103 with AF_XDP". It can be found at https://lwn.net/Articles/750845/.
104
105 UMEM
106 ----
107
108 UMEM is a region of virtual contiguous memory, divided into
109 equal-sized frames. An UMEM is associated to a netdev and a specific
110 queue id of that netdev. It is created and configured (chunk size,
111 headroom, start address and size) by using the XDP_UMEM_REG setsockopt
112 system call. A UMEM is bound to a netdev and queue id, via the bind()
113 system call.
114
115 An AF_XDP is socket linked to a single UMEM, but one UMEM can have
116 multiple AF_XDP sockets. To share an UMEM created via one socket A,
117 the next socket B can do this by setting the XDP_SHARED_UMEM flag in
118 struct sockaddr_xdp member sxdp_flags, and passing the file descriptor
119 of A to struct sockaddr_xdp member sxdp_shared_umem_fd.
120
121 The UMEM has two single-producer/single-consumer rings that are used
122 to transfer ownership of UMEM frames between the kernel and the
123 user-space application.
124
125 Rings
126 -----
127
128 There are a four different kind of rings: FILL, COMPLETION, RX and
129 TX. All rings are single-producer/single-consumer, so the user-space
130 application need explicit synchronization of multiple
131 processes/threads are reading/writing to them.
132
133 The UMEM uses two rings: FILL and COMPLETION. Each socket associated
134 with the UMEM must have an RX queue, TX queue or both. Say, that there
135 is a setup with four sockets (all doing TX and RX). Then there will be
136 one FILL ring, one COMPLETION ring, four TX rings and four RX rings.
137
138 The rings are head(producer)/tail(consumer) based rings. A producer
139 writes the data ring at the index pointed out by struct xdp_ring
140 producer member, and increasing the producer index. A consumer reads
141 the data ring at the index pointed out by struct xdp_ring consumer
142 member, and increasing the consumer index.
143
144 The rings are configured and created via the _RING setsockopt system
145 calls and mmapped to user-space using the appropriate offset to mmap()
146 (XDP_PGOFF_RX_RING, XDP_PGOFF_TX_RING, XDP_UMEM_PGOFF_FILL_RING and
147 XDP_UMEM_PGOFF_COMPLETION_RING).
148
149 The size of the rings need to be of size power of two.
150
151 UMEM Fill Ring
152 ~~~~~~~~~~~~~~
153
154 The FILL ring is used to transfer ownership of UMEM frames from
155 user-space to kernel-space. The UMEM addrs are passed in the ring. As
156 an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has
157 16 chunks and can pass addrs between 0 and 64k.
158
159 Frames passed to the kernel are used for the ingress path (RX rings).
160
161 The user application produces UMEM addrs to this ring. Note that, if
162 running the application with aligned chunk mode, the kernel will mask
163 the incoming addr. E.g. for a chunk size of 2k, the log2(2048) LSB of
164 the addr will be masked off, meaning that 2048, 2050 and 3000 refers
165 to the same chunk. If the user application is run in the unaligned
166 chunks mode, then the incoming addr will be left untouched.
167
168
169 UMEM Completion Ring
170 ~~~~~~~~~~~~~~~~~~~~
171
172 The COMPLETION Ring is used transfer ownership of UMEM frames from
173 kernel-space to user-space. Just like the FILL ring, UMEM indices are
174 used.
175
176 Frames passed from the kernel to user-space are frames that has been
177 sent (TX ring) and can be used by user-space again.
178
179 The user application consumes UMEM addrs from this ring.
180
181
182 RX Ring
183 ~~~~~~~
184
185 The RX ring is the receiving side of a socket. Each entry in the ring
186 is a struct xdp_desc descriptor. The descriptor contains UMEM offset
187 (addr) and the length of the data (len).
188
189 If no frames have been passed to kernel via the FILL ring, no
190 descriptors will (or can) appear on the RX ring.
191
192 The user application consumes struct xdp_desc descriptors from this
193 ring.
194
195 TX Ring
196 ~~~~~~~
197
198 The TX ring is used to send frames. The struct xdp_desc descriptor is
199 filled (index, length and offset) and passed into the ring.
200
201 To start the transfer a sendmsg() system call is required. This might
202 be relaxed in the future.
203
204 The user application produces struct xdp_desc descriptors to this
205 ring.
206
207 Libbpf
208 ======
209
210 Libbpf is a helper library for eBPF and XDP that makes using these
211 technologies a lot simpler. It also contains specific helper functions
212 in tools/lib/bpf/xsk.h for facilitating the use of AF_XDP. It
213 contains two types of functions: those that can be used to make the
214 setup of AF_XDP socket easier and ones that can be used in the data
215 plane to access the rings safely and quickly. To see an example on how
216 to use this API, please take a look at the sample application in
217 samples/bpf/xdpsock_usr.c which uses libbpf for both setup and data
218 plane operations.
219
220 We recommend that you use this library unless you have become a power
221 user. It will make your program a lot simpler.
222
223 XSKMAP / BPF_MAP_TYPE_XSKMAP
224 ============================
225
226 On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) that
227 is used in conjunction with bpf_redirect_map() to pass the ingress
228 frame to a socket.
229
230 The user application inserts the socket into the map, via the bpf()
231 system call.
232
233 Note that if an XDP program tries to redirect to a socket that does
234 not match the queue configuration and netdev, the frame will be
235 dropped. E.g. an AF_XDP socket is bound to netdev eth0 and
236 queue 17. Only the XDP program executing for eth0 and queue 17 will
237 successfully pass data to the socket. Please refer to the sample
238 application (samples/bpf/) in for an example.
239
240 Configuration Flags and Socket Options
241 ======================================
242
243 These are the various configuration flags that can be used to control
244 and monitor the behavior of AF_XDP sockets.
245
246 XDP_COPY and XDP_ZEROCOPY bind flags
247 ------------------------------------
248
249 When you bind to a socket, the kernel will first try to use zero-copy
250 copy. If zero-copy is not supported, it will fall back on using copy
251 mode, i.e. copying all packets out to user space. But if you would
252 like to force a certain mode, you can use the following flags. If you
253 pass the XDP_COPY flag to the bind call, the kernel will force the
254 socket into copy mode. If it cannot use copy mode, the bind call will
255 fail with an error. Conversely, the XDP_ZEROCOPY flag will force the
256 socket into zero-copy mode or fail.
257
258 XDP_SHARED_UMEM bind flag
259 -------------------------
260
261 This flag enables you to bind multiple sockets to the same UMEM. It
262 works on the same queue id, between queue ids and between
263 netdevs/devices. In this mode, each socket has their own RX and TX
264 rings as usual, but you are going to have one or more FILL and
265 COMPLETION ring pairs. You have to create one of these pairs per
266 unique netdev and queue id tuple that you bind to.
267
268 Starting with the case were we would like to share a UMEM between
269 sockets bound to the same netdev and queue id. The UMEM (tied to the
270 fist socket created) will only have a single FILL ring and a single
271 COMPLETION ring as there is only on unique netdev,queue_id tuple that
272 we have bound to. To use this mode, create the first socket and bind
273 it in the normal way. Create a second socket and create an RX and a TX
274 ring, or at least one of them, but no FILL or COMPLETION rings as the
275 ones from the first socket will be used. In the bind call, set he
276 XDP_SHARED_UMEM option and provide the initial socket's fd in the
277 sxdp_shared_umem_fd field. You can attach an arbitrary number of extra
278 sockets this way.
279
280 What socket will then a packet arrive on? This is decided by the XDP
281 program. Put all the sockets in the XSK_MAP and just indicate which
282 index in the array you would like to send each packet to. A simple
283 round-robin example of distributing packets is shown below:
284
285 .. code-block:: c
286
287 #include <linux/bpf.h>
288 #include "bpf_helpers.h"
289
290 #define MAX_SOCKS 16
291
292 struct {
293 __uint(type, BPF_MAP_TYPE_XSKMAP);
294 __uint(max_entries, MAX_SOCKS);
295 __uint(key_size, sizeof(int));
296 __uint(value_size, sizeof(int));
297 } xsks_map SEC(".maps");
298
299 static unsigned int rr;
300
301 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
302 {
303 rr = (rr + 1) & (MAX_SOCKS - 1);
304
305 return bpf_redirect_map(&xsks_map, rr, XDP_DROP);
306 }
307
308 Note, that since there is only a single set of FILL and COMPLETION
309 rings, and they are single producer, single consumer rings, you need
310 to make sure that multiple processes or threads do not use these rings
311 concurrently. There are no synchronization primitives in the
312 libbpf code that protects multiple users at this point in time.
313
314 Libbpf uses this mode if you create more than one socket tied to the
315 same UMEM. However, note that you need to supply the
316 XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf_flag with the
317 xsk_socket__create calls and load your own XDP program as there is no
318 built in one in libbpf that will route the traffic for you.
319
320 The second case is when you share a UMEM between sockets that are
321 bound to different queue ids and/or netdevs. In this case you have to
322 create one FILL ring and one COMPLETION ring for each unique
323 netdev,queue_id pair. Let us say you want to create two sockets bound
324 to two different queue ids on the same netdev. Create the first socket
325 and bind it in the normal way. Create a second socket and create an RX
326 and a TX ring, or at least one of them, and then one FILL and
327 COMPLETION ring for this socket. Then in the bind call, set he
328 XDP_SHARED_UMEM option and provide the initial socket's fd in the
329 sxdp_shared_umem_fd field as you registered the UMEM on that
330 socket. These two sockets will now share one and the same UMEM.
331
332 There is no need to supply an XDP program like the one in the previous
333 case where sockets were bound to the same queue id and
334 device. Instead, use the NIC's packet steering capabilities to steer
335 the packets to the right queue. In the previous example, there is only
336 one queue shared among sockets, so the NIC cannot do this steering. It
337 can only steer between queues.
338
339 In libbpf, you need to use the xsk_socket__create_shared() API as it
340 takes a reference to a FILL ring and a COMPLETION ring that will be
341 created for you and bound to the shared UMEM. You can use this
342 function for all the sockets you create, or you can use it for the
343 second and following ones and use xsk_socket__create() for the first
344 one. Both methods yield the same result.
345
346 Note that a UMEM can be shared between sockets on the same queue id
347 and device, as well as between queues on the same device and between
348 devices at the same time.
349
350 XDP_USE_NEED_WAKEUP bind flag
351 -----------------------------
352
353 This option adds support for a new flag called need_wakeup that is
354 present in the FILL ring and the TX ring, the rings for which user
355 space is a producer. When this option is set in the bind call, the
356 need_wakeup flag will be set if the kernel needs to be explicitly
357 woken up by a syscall to continue processing packets. If the flag is
358 zero, no syscall is needed.
359
360 If the flag is set on the FILL ring, the application needs to call
361 poll() to be able to continue to receive packets on the RX ring. This
362 can happen, for example, when the kernel has detected that there are no
363 more buffers on the FILL ring and no buffers left on the RX HW ring of
364 the NIC. In this case, interrupts are turned off as the NIC cannot
365 receive any packets (as there are no buffers to put them in), and the
366 need_wakeup flag is set so that user space can put buffers on the
367 FILL ring and then call poll() so that the kernel driver can put these
368 buffers on the HW ring and start to receive packets.
369
370 If the flag is set for the TX ring, it means that the application
371 needs to explicitly notify the kernel to send any packets put on the
372 TX ring. This can be accomplished either by a poll() call, as in the
373 RX path, or by calling sendto().
374
375 An example of how to use this flag can be found in
376 samples/bpf/xdpsock_user.c. An example with the use of libbpf helpers
377 would look like this for the TX path:
378
379 .. code-block:: c
380
381 if (xsk_ring_prod__needs_wakeup(&my_tx_ring))
382 sendto(xsk_socket__fd(xsk_handle), NULL, 0, MSG_DONTWAIT, NULL, 0);
383
384 I.e., only use the syscall if the flag is set.
385
386 We recommend that you always enable this mode as it usually leads to
387 better performance especially if you run the application and the
388 driver on the same core, but also if you use different cores for the
389 application and the kernel driver, as it reduces the number of
390 syscalls needed for the TX path.
391
392 XDP_{RX|TX|UMEM_FILL|UMEM_COMPLETION}_RING setsockopts
393 ------------------------------------------------------
394
395 These setsockopts sets the number of descriptors that the RX, TX,
396 FILL, and COMPLETION rings respectively should have. It is mandatory
397 to set the size of at least one of the RX and TX rings. If you set
398 both, you will be able to both receive and send traffic from your
399 application, but if you only want to do one of them, you can save
400 resources by only setting up one of them. Both the FILL ring and the
401 COMPLETION ring are mandatory as you need to have a UMEM tied to your
402 socket. But if the XDP_SHARED_UMEM flag is used, any socket after the
403 first one does not have a UMEM and should in that case not have any
404 FILL or COMPLETION rings created as the ones from the shared UMEM will
405 be used. Note, that the rings are single-producer single-consumer, so
406 do not try to access them from multiple processes at the same
407 time. See the XDP_SHARED_UMEM section.
408
409 In libbpf, you can create Rx-only and Tx-only sockets by supplying
410 NULL to the rx and tx arguments, respectively, to the
411 xsk_socket__create function.
412
413 If you create a Tx-only socket, we recommend that you do not put any
414 packets on the fill ring. If you do this, drivers might think you are
415 going to receive something when you in fact will not, and this can
416 negatively impact performance.
417
418 XDP_UMEM_REG setsockopt
419 -----------------------
420
421 This setsockopt registers a UMEM to a socket. This is the area that
422 contain all the buffers that packet can reside in. The call takes a
423 pointer to the beginning of this area and the size of it. Moreover, it
424 also has parameter called chunk_size that is the size that the UMEM is
425 divided into. It can only be 2K or 4K at the moment. If you have an
426 UMEM area that is 128K and a chunk size of 2K, this means that you
427 will be able to hold a maximum of 128K / 2K = 64 packets in your UMEM
428 area and that your largest packet size can be 2K.
429
430 There is also an option to set the headroom of each single buffer in
431 the UMEM. If you set this to N bytes, it means that the packet will
432 start N bytes into the buffer leaving the first N bytes for the
433 application to use. The final option is the flags field, but it will
434 be dealt with in separate sections for each UMEM flag.
435
436 SO_BINDTODEVICE setsockopt
437 --------------------------
438
439 This is a generic SOL_SOCKET option that can be used to tie AF_XDP
440 socket to a particular network interface. It is useful when a socket
441 is created by a privileged process and passed to a non-privileged one.
442 Once the option is set, kernel will refuse attempts to bind that socket
443 to a different interface. Updating the value requires CAP_NET_RAW.
444
445 XDP_STATISTICS getsockopt
446 -------------------------
447
448 Gets drop statistics of a socket that can be useful for debug
449 purposes. The supported statistics are shown below:
450
451 .. code-block:: c
452
453 struct xdp_statistics {
454 __u64 rx_dropped; /* Dropped for reasons other than invalid desc */
455 __u64 rx_invalid_descs; /* Dropped due to invalid descriptor */
456 __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
457 };
458
459 XDP_OPTIONS getsockopt
460 ----------------------
461
462 Gets options from an XDP socket. The only one supported so far is
463 XDP_OPTIONS_ZEROCOPY which tells you if zero-copy is on or not.
464
465 Multi-Buffer Support
466 ====================
467
468 With multi-buffer support, programs using AF_XDP sockets can receive
469 and transmit packets consisting of multiple buffers both in copy and
470 zero-copy mode. For example, a packet can consist of two
471 frames/buffers, one with the header and the other one with the data,
472 or a 9K Ethernet jumbo frame can be constructed by chaining together
473 three 4K frames.
474
475 Some definitions:
476
477 * A packet consists of one or more frames
478
479 * A descriptor in one of the AF_XDP rings always refers to a single
480 frame. In the case the packet consists of a single frame, the
481 descriptor refers to the whole packet.
482
483 To enable multi-buffer support for an AF_XDP socket, use the new bind
484 flag XDP_USE_SG. If this is not provided, all multi-buffer packets
485 will be dropped just as before. Note that the XDP program loaded also
486 needs to be in multi-buffer mode. This can be accomplished by using
487 "xdp.frags" as the section name of the XDP program used.
488
489 To represent a packet consisting of multiple frames, a new flag called
490 XDP_PKT_CONTD is introduced in the options field of the Rx and Tx
491 descriptors. If it is true (1) the packet continues with the next
492 descriptor and if it is false (0) it means this is the last descriptor
493 of the packet. Why the reverse logic of end-of-packet (eop) flag found
494 in many NICs? Just to preserve compatibility with non-multi-buffer
495 applications that have this bit set to false for all packets on Rx,
496 and the apps set the options field to zero for Tx, as anything else
497 will be treated as an invalid descriptor.
498
499 These are the semantics for producing packets onto AF_XDP Tx ring
500 consisting of multiple frames:
501
502 * When an invalid descriptor is found, all the other
503 descriptors/frames of this packet are marked as invalid and not
504 completed. The next descriptor is treated as the start of a new
505 packet, even if this was not the intent (because we cannot guess
506 the intent). As before, if your program is producing invalid
507 descriptors you have a bug that must be fixed.
508
509 * Zero length descriptors are treated as invalid descriptors.
510
511 * For copy mode, the maximum supported number of frames in a packet is
512 equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all
513 descriptors accumulated so far are dropped and treated as
514 invalid. To produce an application that will work on any system
515 regardless of this config setting, limit the number of frags to 18,
516 as the minimum value of the config is 17.
517
518 * For zero-copy mode, the limit is up to what the NIC HW
519 supports. Usually at least five on the NICs we have checked. We
520 consciously chose to not enforce a rigid limit (such as
521 CONFIG_MAX_SKB_FRAGS + 1) for zero-copy mode, as it would have
522 resulted in copy actions under the hood to fit into what limit the
523 NIC supports. Kind of defeats the purpose of zero-copy mode. How to
524 probe for this limit is explained in the "probe for multi-buffer
525 support" section.
526
527 On the Rx path in copy-mode, the xsk core copies the XDP data into
528 multiple descriptors, if needed, and sets the XDP_PKT_CONTD flag as
529 detailed before. Zero-copy mode works the same, though the data is not
530 copied. When the application gets a descriptor with the XDP_PKT_CONTD
531 flag set to one, it means that the packet consists of multiple buffers
532 and it continues with the next buffer in the following
533 descriptor. When a descriptor with XDP_PKT_CONTD == 0 is received, it
534 means that this is the last buffer of the packet. AF_XDP guarantees
535 that only a complete packet (all frames in the packet) is sent to the
536 application. If there is not enough space in the AF_XDP Rx ring, all
537 frames of the packet will be dropped.
538
539 If application reads a batch of descriptors, using for example the libxdp
540 interfaces, it is not guaranteed that the batch will end with a full
541 packet. It might end in the middle of a packet and the rest of the
542 buffers of that packet will arrive at the beginning of the next batch,
543 since the libxdp interface does not read the whole ring (unless you
544 have an enormous batch size or a very small ring size).
545
546 An example program each for Rx and Tx multi-buffer support can be found
547 later in this document.
548
549 Usage
550 -----
551
552 In order to use AF_XDP sockets two parts are needed. The
553 user-space application and the XDP program. For a complete setup and
554 usage example, please refer to the sample application. The user-space
555 side is xdpsock_user.c and the XDP side is part of libbpf.
556
557 The XDP code sample included in tools/lib/bpf/xsk.c is the following:
558
559 .. code-block:: c
560
561 SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
562 {
563 int index = ctx->rx_queue_index;
564
565 // A set entry here means that the corresponding queue_id
566 // has an active AF_XDP socket bound to it.
567 if (bpf_map_lookup_elem(&xsks_map, &index))
568 return bpf_redirect_map(&xsks_map, index, 0);
569
570 return XDP_PASS;
571 }
572
573 A simple but not so performance ring dequeue and enqueue could look
574 like this:
575
576 .. code-block:: c
577
578 // struct xdp_rxtx_ring {
579 // __u32 *producer;
580 // __u32 *consumer;
581 // struct xdp_desc *desc;
582 // };
583
584 // struct xdp_umem_ring {
585 // __u32 *producer;
586 // __u32 *consumer;
587 // __u64 *desc;
588 // };
589
590 // typedef struct xdp_rxtx_ring RING;
591 // typedef struct xdp_umem_ring RING;
592
593 // typedef struct xdp_desc RING_TYPE;
594 // typedef __u64 RING_TYPE;
595
596 int dequeue_one(RING *ring, RING_TYPE *item)
597 {
598 __u32 entries = *ring->producer - *ring->consumer;
599
600 if (entries == 0)
601 return -1;
602
603 // read-barrier!
604
605 *item = ring->desc[*ring->consumer & (RING_SIZE - 1)];
606 (*ring->consumer)++;
607 return 0;
608 }
609
610 int enqueue_one(RING *ring, const RING_TYPE *item)
611 {
612 u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer);
613
614 if (free_entries == 0)
615 return -1;
616
617 ring->desc[*ring->producer & (RING_SIZE - 1)] = *item;
618
619 // write-barrier!
620
621 (*ring->producer)++;
622 return 0;
623 }
624
625 But please use the libbpf functions as they are optimized and ready to
626 use. Will make your life easier.
627
628 Usage Multi-Buffer Rx
629 ---------------------
630
631 Here is a simple Rx path pseudo-code example (using libxdp interfaces
632 for simplicity). Error paths have been excluded to keep it short:
633
634 .. code-block:: c
635
636 void rx_packets(struct xsk_socket_info *xsk)
637 {
638 static bool new_packet = true;
639 u32 idx_rx = 0, idx_fq = 0;
640 static char *pkt;
641
642 int rcvd = xsk_ring_cons__peek(&xsk->rx, opt_batch_size, &idx_rx);
643
644 xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq);
645
646 for (int i = 0; i < rcvd; i++) {
647 struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++);
648 char *frag = xsk_umem__get_data(xsk->umem->buffer, desc->addr);
649 bool eop = !(desc->options & XDP_PKT_CONTD);
650
651 if (new_packet)
652 pkt = frag;
653 else
654 add_frag_to_pkt(pkt, frag);
655
656 if (eop)
657 process_pkt(pkt);
658
659 new_packet = eop;
660
661 *xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = desc->addr;
662 }
663
664 xsk_ring_prod__submit(&xsk->umem->fq, rcvd);
665 xsk_ring_cons__release(&xsk->rx, rcvd);
666 }
667
668 Usage Multi-Buffer Tx
669 ---------------------
670
671 Here is an example Tx path pseudo-code (using libxdp interfaces for
672 simplicity) ignoring that the umem is finite in size, and that we
673 eventually will run out of packets to send. Also assumes pkts.addr
674 points to a valid location in the umem.
675
676 .. code-block:: c
677
678 void tx_packets(struct xsk_socket_info *xsk, struct pkt *pkts,
679 int batch_size)
680 {
681 u32 idx, i, pkt_nb = 0;
682
683 xsk_ring_prod__reserve(&xsk->tx, batch_size, &idx);
684
685 for (i = 0; i < batch_size;) {
686 u64 addr = pkts[pkt_nb].addr;
687 u32 len = pkts[pkt_nb].size;
688
689 do {
690 struct xdp_desc *tx_desc;
691
692 tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx + i++);
693 tx_desc->addr = addr;
694
695 if (len > xsk_frame_size) {
696 tx_desc->len = xsk_frame_size;
697 tx_desc->options = XDP_PKT_CONTD;
698 } else {
699 tx_desc->len = len;
700 tx_desc->options = 0;
701 pkt_nb++;
702 }
703 len -= tx_desc->len;
704 addr += xsk_frame_size;
705
706 if (i == batch_size) {
707 /* Remember len, addr, pkt_nb for next iteration.
708 * Skipped for simplicity.
709 */
710 break;
711 }
712 } while (len);
713 }
714
715 xsk_ring_prod__submit(&xsk->tx, i);
716 }
717
718 Probing for Multi-Buffer Support
719 --------------------------------
720
721 To discover if a driver supports multi-buffer AF_XDP in SKB or DRV
722 mode, use the XDP_FEATURES feature of netlink in linux/netdev.h to
723 query for NETDEV_XDP_ACT_RX_SG support. This is the same flag as for
724 querying for XDP multi-buffer support. If XDP supports multi-buffer in
725 a driver, then AF_XDP will also support that in SKB and DRV mode.
726
727 To discover if a driver supports multi-buffer AF_XDP in zero-copy
728 mode, use XDP_FEATURES and first check the NETDEV_XDP_ACT_XSK_ZEROCOPY
729 flag. If it is set, it means that at least zero-copy is supported and
730 you should go and check the netlink attribute
731 NETDEV_A_DEV_XDP_ZC_MAX_SEGS in linux/netdev.h. An unsigned integer
732 value will be returned stating the max number of frags that are
733 supported by this device in zero-copy mode. These are the possible
734 return values:
735
736 1: Multi-buffer for zero-copy is not supported by this device, as max
737 one fragment supported means that multi-buffer is not possible.
738
739 >=2: Multi-buffer is supported in zero-copy mode for this device. The
740 returned number signifies the max number of frags supported.
741
742 For an example on how these are used through libbpf, please take a
743 look at tools/testing/selftests/bpf/xskxceiver.c.
744
745 Multi-Buffer Support for Zero-Copy Drivers
746 ------------------------------------------
747
748 Zero-copy drivers usually use the batched APIs for Rx and Tx
749 processing. Note that the Tx batch API guarantees that it will provide
750 a batch of Tx descriptors that ends with full packet at the end. This
751 to facilitate extending a zero-copy driver with multi-buffer support.
752
753 Sample application
754 ==================
755
756 There is a xdpsock benchmarking/test application included that
757 demonstrates how to use AF_XDP sockets with private UMEMs. Say that
758 you would like your UDP traffic from port 4242 to end up in queue 16,
759 that we will enable AF_XDP on. Here, we use ethtool for this::
760
761 ethtool -N p3p2 rx-flow-hash udp4 fn
762 ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \
763 action 16
764
765 Running the rxdrop benchmark in XDP_DRV mode can then be done
766 using::
767
768 samples/bpf/xdpsock -i p3p2 -q 16 -r -N
769
770 For XDP_SKB mode, use the switch "-S" instead of "-N" and all options
771 can be displayed with "-h", as usual.
772
773 This sample application uses libbpf to make the setup and usage of
774 AF_XDP simpler. If you want to know how the raw uapi of AF_XDP is
775 really used to make something more advanced, take a look at the libbpf
776 code in tools/lib/bpf/xsk.[ch].
777
778 FAQ
779 =======
780
781 Q: I am not seeing any traffic on the socket. What am I doing wrong?
782
783 A: When a netdev of a physical NIC is initialized, Linux usually
784 allocates one RX and TX queue pair per core. So on a 8 core system,
785 queue ids 0 to 7 will be allocated, one per core. In the AF_XDP
786 bind call or the xsk_socket__create libbpf function call, you
787 specify a specific queue id to bind to and it is only the traffic
788 towards that queue you are going to get on you socket. So in the
789 example above, if you bind to queue 0, you are NOT going to get any
790 traffic that is distributed to queues 1 through 7. If you are
791 lucky, you will see the traffic, but usually it will end up on one
792 of the queues you have not bound to.
793
794 There are a number of ways to solve the problem of getting the
795 traffic you want to the queue id you bound to. If you want to see
796 all the traffic, you can force the netdev to only have 1 queue, queue
797 id 0, and then bind to queue 0. You can use ethtool to do this::
798
799 sudo ethtool -L <interface> combined 1
800
801 If you want to only see part of the traffic, you can program the
802 NIC through ethtool to filter out your traffic to a single queue id
803 that you can bind your XDP socket to. Here is one example in which
804 UDP traffic to and from port 4242 are sent to queue 2::
805
806 sudo ethtool -N <interface> rx-flow-hash udp4 fn
807 sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \
808 4242 action 2
809
810 A number of other ways are possible all up to the capabilities of
811 the NIC you have.
812
813 Q: Can I use the XSKMAP to implement a switch between different umems
814 in copy mode?
815
816 A: The short answer is no, that is not supported at the moment. The
817 XSKMAP can only be used to switch traffic coming in on queue id X
818 to sockets bound to the same queue id X. The XSKMAP can contain
819 sockets bound to different queue ids, for example X and Y, but only
820 traffic goming in from queue id Y can be directed to sockets bound
821 to the same queue id Y. In zero-copy mode, you should use the
822 switch, or other distribution mechanism, in your NIC to direct
823 traffic to the correct queue id and socket.
824
825 Q: My packets are sometimes corrupted. What is wrong?
826
827 A: Care has to be taken not to feed the same buffer in the UMEM into
828 more than one ring at the same time. If you for example feed the
829 same buffer into the FILL ring and the TX ring at the same time, the
830 NIC might receive data into the buffer at the same time it is
831 sending it. This will cause some packets to become corrupted. Same
832 thing goes for feeding the same buffer into the FILL rings
833 belonging to different queue ids or netdevs bound with the
834 XDP_SHARED_UMEM flag.
835
836 Credits
837 =======
838
839 - Björn Töpel (AF_XDP core)
840 - Magnus Karlsson (AF_XDP core)
841 - Alexander Duyck
842 - Alexei Starovoitov
843 - Daniel Borkmann
844 - Jesper Dangaard Brouer
845 - John Fastabend
846 - Jonathan Corbet (LWN coverage)
847 - Michael S. Tsirkin
848 - Qi Z Zhang
849 - Willem de Bruijn
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.