1 .. SPDX-License-Identifier: GPL-2.0
2
3 =============
4 Page Pool API
5 =============
6
7 .. kernel-doc:: include/net/page_pool/helpers.h
8 :doc: page_pool allocator
9
10 Architecture overview
11 =====================
12
13 .. code-block:: none
14
15 +------------------+
16 | Driver |
17 +------------------+
18 ^
19 |
20 |
21 |
22 v
23 +--------------------------------------------+
24 | request memory |
25 +--------------------------------------------+
26 ^ ^
27 | |
28 | Pool empty | Pool has entries
29 | |
30 v v
31 +-----------------------+ +------------------------+
32 | alloc (and map) pages | | get page from cache |
33 +-----------------------+ +------------------------+
34 ^ ^
35 | |
36 | cache available | No entries, refill
37 | | from ptr-ring
38 | |
39 v v
40 +-----------------+ +------------------+
41 | Fast cache | | ptr-ring cache |
42 +-----------------+ +------------------+
43
44 Monitoring
45 ==========
46 Information about page pools on the system can be accessed via the netdev
47 genetlink family (see Documentation/netlink/specs/netdev.yaml).
48
49 API interface
50 =============
51 The number of pools created **must** match the number of hardware queues
52 unless hardware restrictions make that impossible. This would otherwise beat the
53 purpose of page pool, which is allocate pages fast from cache without locking.
54 This lockless guarantee naturally comes from running under a NAPI softirq.
55 The protection doesn't strictly have to be NAPI, any guarantee that allocating
56 a page will cause no race conditions is enough.
57
58 .. kernel-doc:: net/core/page_pool.c
59 :identifiers: page_pool_create
60
61 .. kernel-doc:: include/net/page_pool/types.h
62 :identifiers: struct page_pool_params
63
64 .. kernel-doc:: include/net/page_pool/helpers.h
65 :identifiers: page_pool_put_page page_pool_put_full_page
66 page_pool_recycle_direct page_pool_free_va
67 page_pool_dev_alloc_pages page_pool_dev_alloc_frag
68 page_pool_dev_alloc page_pool_dev_alloc_va
69 page_pool_get_dma_addr page_pool_get_dma_dir
70
71 .. kernel-doc:: net/core/page_pool.c
72 :identifiers: page_pool_put_page_bulk page_pool_get_stats
73
74 DMA sync
75 --------
76 Driver is always responsible for syncing the pages for the CPU.
77 Drivers may choose to take care of syncing for the device as well
78 or set the ``PP_FLAG_DMA_SYNC_DEV`` flag to request that pages
79 allocated from the page pool are already synced for the device.
80
81 If ``PP_FLAG_DMA_SYNC_DEV`` is set, the driver must inform the core what portion
82 of the buffer has to be synced. This allows the core to avoid syncing the entire
83 page when the drivers knows that the device only accessed a portion of the page.
84
85 Most drivers will reserve headroom in front of the frame. This part
86 of the buffer is not touched by the device, so to avoid syncing
87 it drivers can set the ``offset`` field in struct page_pool_params
88 appropriately.
89
90 For pages recycled on the XDP xmit and skb paths the page pool will
91 use the ``max_len`` member of struct page_pool_params to decide how
92 much of the page needs to be synced (starting at ``offset``).
93 When directly freeing pages in the driver (page_pool_put_page())
94 the ``dma_sync_size`` argument specifies how much of the buffer needs
95 to be synced.
96
97 If in doubt set ``offset`` to 0, ``max_len`` to ``PAGE_SIZE`` and
98 pass -1 as ``dma_sync_size``. That combination of arguments is always
99 correct.
100
101 Note that the syncing parameters are for the entire page.
102 This is important to remember when using fragments (``PP_FLAG_PAGE_FRAG``),
103 where allocated buffers may be smaller than a full page.
104 Unless the driver author really understands page pool internals
105 it's recommended to always use ``offset = 0``, ``max_len = PAGE_SIZE``
106 with fragmented page pools.
107
108 Stats API and structures
109 ------------------------
110 If the kernel is configured with ``CONFIG_PAGE_POOL_STATS=y``, the API
111 page_pool_get_stats() and structures described below are available.
112 It takes a pointer to a ``struct page_pool`` and a pointer to a struct
113 page_pool_stats allocated by the caller.
114
115 Older drivers expose page pool statistics via ethtool or debugfs.
116 The same statistics are accessible via the netlink netdev family
117 in a driver-independent fashion.
118
119 .. kernel-doc:: include/net/page_pool/types.h
120 :identifiers: struct page_pool_recycle_stats
121 struct page_pool_alloc_stats
122 struct page_pool_stats
123
124 Coding examples
125 ===============
126
127 Registration
128 ------------
129
130 .. code-block:: c
131
132 /* Page pool registration */
133 struct page_pool_params pp_params = { 0 };
134 struct xdp_rxq_info xdp_rxq;
135 int err;
136
137 pp_params.order = 0;
138 /* internal DMA mapping in page_pool */
139 pp_params.flags = PP_FLAG_DMA_MAP;
140 pp_params.pool_size = DESC_NUM;
141 pp_params.nid = NUMA_NO_NODE;
142 pp_params.dev = priv->dev;
143 pp_params.napi = napi; /* only if locking is tied to NAPI */
144 pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
145 page_pool = page_pool_create(&pp_params);
146
147 err = xdp_rxq_info_reg(&xdp_rxq, ndev, 0);
148 if (err)
149 goto err_out;
150
151 err = xdp_rxq_info_reg_mem_model(&xdp_rxq, MEM_TYPE_PAGE_POOL, page_pool);
152 if (err)
153 goto err_out;
154
155 NAPI poller
156 -----------
157
158
159 .. code-block:: c
160
161 /* NAPI Rx poller */
162 enum dma_data_direction dma_dir;
163
164 dma_dir = page_pool_get_dma_dir(dring->page_pool);
165 while (done < budget) {
166 if (some error)
167 page_pool_recycle_direct(page_pool, page);
168 if (packet_is_xdp) {
169 if XDP_DROP:
170 page_pool_recycle_direct(page_pool, page);
171 } else (packet_is_skb) {
172 skb_mark_for_recycle(skb);
173 new_page = page_pool_dev_alloc_pages(page_pool);
174 }
175 }
176
177 Stats
178 -----
179
180 .. code-block:: c
181
182 #ifdef CONFIG_PAGE_POOL_STATS
183 /* retrieve stats */
184 struct page_pool_stats stats = { 0 };
185 if (page_pool_get_stats(page_pool, &stats)) {
186 /* perhaps the driver reports statistics with ethool */
187 ethtool_print_allocation_stats(&stats.alloc_stats);
188 ethtool_print_recycle_stats(&stats.recycle_stats);
189 }
190 #endif
191
192 Driver unload
193 -------------
194
195 .. code-block:: c
196
197 /* Driver unload */
198 page_pool_put_full_page(page_pool, page, false);
199 xdp_rxq_info_unreg(&xdp_rxq);
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.