1 .. SPDX-License-Identifier: GPL-2.0
2
3 ============================
4 PCI Peer-to-Peer DMA Support
5 ============================
6
7 The PCI bus has pretty decent support for perf
8 between two devices on the bus. This type of t
9 called Peer-to-Peer (or P2P). However, there a
10 make P2P transactions tricky to do in a perfec
11
12 One of the biggest issues is that PCI doesn't
13 transactions between hierarchy domains, and in
14 defines a separate hierarchy domain. To make t
15 simple way to determine if a given Root Comple
16 (See PCIe r4.0, sec 1.3.1). Therefore, as of t
17 only supports doing P2P when the endpoints inv
18 same PCI bridge, as such devices are all in th
19 domain, and the spec guarantees that all trans
20 hierarchy will be routable, but it does not re
21 between hierarchies.
22
23 The second issue is that to make use of existi
24 memory that is used for P2P transactions needs
25 pages. However, PCI BARs are not typically cac
26 a few corner case gotchas with these pages so
27 be careful about what they do with them.
28
29
30 Driver Writer's Guide
31 =====================
32
33 In a given P2P implementation there may be thr
34 types of kernel drivers in play:
35
36 * Provider - A driver which provides or publis
37 memory or doorbell registers to other driver
38 * Client - A driver which makes use of a resou
39 DMA transaction to or from it.
40 * Orchestrator - A driver which orchestrates t
41 clients and providers.
42
43 In many cases there could be overlap between t
44 it may be typical for a driver to be both a pr
45
46 For example, in the NVMe Target Copy Offload i
47
48 * The NVMe PCI driver is both a client, provid
49 in that it exposes any CMB (Controller Memor
50 resource (provider), it accepts P2P memory p
51 to be used directly (client) and it can also
52 submission queue entries (orchestrator).
53 * The RDMA driver is a client in this arrangem
54 can DMA directly to the memory exposed by th
55 * The NVMe Target driver (nvmet) can orchestra
56 to the P2P memory (CMB) and then to the NVMe
57
58 This is currently the only arrangement support
59 one could imagine slight tweaks to this that w
60 functionality. For example, if a specific RNIC
61 memory behind it, its driver could add support
62 then the NVMe Target could use the RNIC's memo
63 in cases where the NVMe cards in use do not ha
64
65
66 Provider Drivers
67 ----------------
68
69 A provider simply needs to register a BAR (or
70 as a P2P DMA resource using :c:func:`pci_p2pdm
71 This will register struct pages for all the sp
72
73 After that it may optionally publish all of it
74 P2P memory using :c:func:`pci_p2pmem_publish()
75 any orchestrator drivers to find and use the m
76 this way, the resource must be regular memory
77
78 For the time being this is fairly rudimentary
79 are typically going to be P2P memory. Future w
80 this to include other types of resources like
81
82
83 Client Drivers
84 --------------
85
86 A client driver only has to use the mapping AP
87 and :c:func:`dma_unmap_sg()` functions as usua
88 will do the right thing for the P2P capable me
89
90
91 Orchestrator Drivers
92 --------------------
93
94 The first task an orchestrator driver must do
95 all client devices that will be involved in a
96 example, the NVMe Target driver creates a list
97 block device and the RNIC in use. If the orche
98 a specific P2P provider to use it may check co
99 :c:func:`pci_p2pdma_distance()` otherwise it m
100 that's compatible with all clients using :c:f
101 If more than one provider is supported, the on
102 be chosen first. If more than one provider is
103 one returned will be chosen at random (it is n
104 truly random). This function returns the PCI d
105 with a reference taken and therefore when it's
106 returned with pci_dev_put().
107
108 Once a provider is selected, the orchestrator
109 :c:func:`pci_alloc_p2pmem()` and :c:func:`pci_
110 allocate P2P memory from the provider. :c:func
111 and :c:func:`pci_p2pmem_free_sgl()` are conven
112 allocating scatter-gather lists with P2P memor
113
114 Struct Page Caveats
115 -------------------
116
117 Driver writers should be very careful about no
118 struct pages to code that isn't prepared for i
119 interfaces do not have any checks for ensuring
120 precludes passing these pages to userspace.
121
122 P2P memory is also technically IO memory but s
123 effects behind it. Thus, the order of loads an
124 and ioreadX(), iowriteX() and friends should n
125
126
127 P2P DMA Support Library
128 =======================
129
130 .. kernel-doc:: drivers/pci/p2pdma.c
131 :export:
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.