1 .. SPDX-License-Identifier: GPL-2.0
2
3 =================
4 PCI NTB Function
5 =================
6
7 :Author: Kishon Vijay Abraham I <kishon@ti.com>
8
9 PCI Non-Transparent Bridges (NTB) allow two ho
10 with each other by exposing each host as a dev
11 NTBs typically support the ability to generate
12 machine, expose memory ranges as BARs, and per
13 scratchpads, which are areas of memory within
14 from both machines.
15
16 PCI NTB Function allows two different systems
17 with each other by configuring the endpoint in
18 transactions from one system are routed to the
19
20 In the below diagram, PCI NTB function configu
21 PCI Endpoint (EP) instances in such a way that
22 controller are routed to the other EP controll
23 configures the SoC with multiple EP instances,
24 communicate with each other using SoC as a bri
25
26 .. code-block:: text
27
28 +-------------+
29 | |
30 | HOST1 |
31 | |
32 +------^------+
33 |
34 |
35 +---------|----------------------------------
36 | +------v------+
37 | | |
38 | | EP |
39 | | CONTROLLER1 |
40 | | <---------------------------
41 | | |
42 | | |
43 | | | SoC With Multiple EP Inst
44 | | | (Configured using NTB Fun
45 | +-------------+
46 +--------------------------------------------
47
48 Constructs used for Implementing NTB
49 ====================================
50
51 1) Config Region
52 2) Self Scratchpad Registers
53 3) Peer Scratchpad Registers
54 4) Doorbell (DB) Registers
55 5) Memory Window (MW)
56
57
58 Config Region:
59 --------------
60
61 Config Region is a construct that is specific
62 Endpoint Function Driver. The host and endpoin
63 exchange information with each other using thi
64 Control/Status Registers for configuring the E
65 write into this region for configuring the out
66 (ATU) and to indicate the link status. Endpoin
67 commands issued by host in this region. Endpoi
68 scratchpad offset and number of memory windows
69
70 The format of Config Region is given below. Al
71
72 .. code-block:: text
73
74 +------------------------+
75 | COMMAND |
76 +------------------------+
77 | ARGUMENT |
78 +------------------------+
79 | STATUS |
80 +------------------------+
81 | TOPOLOGY |
82 +------------------------+
83 | ADDRESS (LOWER 32) |
84 +------------------------+
85 | ADDRESS (UPPER 32) |
86 +------------------------+
87 | SIZE |
88 +------------------------+
89 | NO OF MEMORY WINDOW |
90 +------------------------+
91 | MEMORY WINDOW1 OFFSET |
92 +------------------------+
93 | SPAD OFFSET |
94 +------------------------+
95 | SPAD COUNT |
96 +------------------------+
97 | DB ENTRY SIZE |
98 +------------------------+
99 | DB DATA |
100 +------------------------+
101 | : |
102 +------------------------+
103 | : |
104 +------------------------+
105 | DB DATA |
106 +------------------------+
107
108
109 COMMAND:
110
111 NTB function supports three commands:
112
113 CMD_CONFIGURE_DOORBELL (0x1): Comman
114 invoking this command, the host should
115 MSI/MSI-X vectors (i.e., initialize th
116 Endpoint). The endpoint on receiving t
117 the outbound ATU such that transaction
118 to the MSI/MSI-X address programmed by
119 register should be populated with numb
120 lower 16 bits) and if MSI or MSI-X sho
121
122 CMD_CONFIGURE_MW (0x2): Command to c
123 host invokes this command after alloca
124 accessed by remote host. The allocated
125 in the ADDRESS register (64 bit), the
126 the SIZE register and the memory windo
127 in the ARGUMENT register. The endpoint
128 will configure the outbound ATU such t
129 are routed to the address provided by
130
131 CMD_LINK_UP (0x3): Command to indica
132 bound to the EP device on the host sid
133 receives this command from both the ho
134 raise a LINK_UP event to both the host
135 NTB applications can start communicati
136
137 ARGUMENT:
138
139 The value of this register is based on
140 command register. See COMMAND section
141
142 TOPOLOGY:
143
144 Set to NTB_TOPO_B2B_USD for Primary in
145 Set to NTB_TOPO_B2B_DSD for Secondary
146
147 ADDRESS/SIZE:
148
149 Address and Size to be used while conf
150 See "CMD_CONFIGURE_MW" for more info.
151
152 MEMORY WINDOW1 OFFSET:
153
154 Memory Window 1 and Doorbell registers
155 same BAR. The initial portion of the r
156 registers and the latter portion of th
157 This register will specify the offset
158
159 NO OF MEMORY WINDOW:
160
161 Specifies the number of memory windows
162
163 SPAD OFFSET:
164
165 Self scratchpad region and config regi
166 same BAR. The initial portion of the r
167 and the latter portion of the region i
168 register will specify the offset of th
169
170 SPAD COUNT:
171
172 Specifies the number of scratchpad reg
173 device.
174
175 DB ENTRY SIZE:
176
177 Used to determine the offset within th
178 in order to raise doorbell. EPF NTB ca
179 ring doorbell (MSI-X support will be a
180 address for all the interrupts and MSI
181 addresses for different interrupts. Th
182 by the host and the address it gives i
183 implementation supported by the host.
184 using GIC ITS will have the same MSI-X
185 In order to support all the combinatio
186 for both MSI and MSI-X, EPF NTB alloca
187 Outbound Address Space for each of the
188 be mapped to the MSI/MSI-X address pro
189 provides the same address for all the
190 will be translated to the same address
191 addresses, the regions will be transla
192 will ensure there is no difference whi
193
194 DB DATA:
195
196 EPF NTB supports 32 interrupts, so the
197 This holds the MSI/MSI-X data that has
198 for raising doorbell interrupt. This w
199 while invoking CMD_CONFIGURE_DOORBELL.
200
201 Scratchpad Registers:
202 ---------------------
203
204 Each host has its own register space allocat
205 controller. They are both readable and writa
206 They are used by applications built over NTB
207 and status information between both sides of
208
209 Scratchpad registers has 2 parts
210 1) Self Scratchpad: Host's own registe
211 2) Peer Scratchpad: Remote host's regi
212
213 Doorbell Registers:
214 -------------------
215
216 Doorbell Registers are used by the hosts to
217
218 Memory Window:
219 --------------
220
221 Actual transfer of data between the two host
222 memory window.
223
224 Modeling Constructs:
225 ====================
226
227 There are 5 or more distinct regions (config,
228 scratchpad, doorbell, one or more memory windo
229 NTB functionality. At least one memory window
230 one is permitted. All these regions should be
231 access these regions.
232
233 If one 32-bit BAR is allocated for each of the
234 look like this:
235
236 ====== ===============
237 BAR NO CONSTRUCTS USED
238 ====== ===============
239 BAR0 Config Region
240 BAR1 Self Scratchpad
241 BAR2 Peer Scratchpad
242 BAR3 Doorbell
243 BAR4 Memory Window 1
244 BAR5 Memory Window 2
245 ====== ===============
246
247 However if we allocate a separate BAR for each
248 be enough BARs for all the regions in a platfo
249 BARs.
250
251 In order to be supported by most of the platfo
252 packed and mapped to BARs in a way that provid
253 also makes sure the host doesn't access any re
254 to.
255
256 The following scheme is used in EPF NTB Functi
257
258 ====== ===============================
259 BAR NO CONSTRUCTS USED
260 ====== ===============================
261 BAR0 Config Region + Self Scratchpad
262 BAR1 Peer Scratchpad
263 BAR2 Doorbell + Memory Window 1
264 BAR3 Memory Window 2
265 BAR4 Memory Window 3
266 BAR5 Memory Window 4
267 ====== ===============================
268
269 With this scheme, for the basic NTB functional
270
271 Modeling Config/Scratchpad Region:
272 ----------------------------------
273
274 .. code-block:: text
275
276 +-----------------+------->+-----------------
277 | BAR0 | | CONFIG REGION
278 +-----------------+----+ +-----------------
279 | BAR1 | | |SCRATCHPAD REGION
280 +-----------------+ +-->+-----------------
281 | BAR2 | Local Memory
282 +-----------------+
283 | BAR3 |
284 +-----------------+
285 | BAR4 |
286 +-----------------+
287 | BAR5 |
288 +-----------------+
289 EP CONTROLLER 1
290
291 Above diagram shows Config region + Scratchpad
292 EP controller 1) allocated in local memory. Th
293 region and scratchpad region (self scratchpad)
294 The peer host (HOST2 connected to EP controlle
295 scratchpad region (peer scratchpad) using BAR1
296 diagram shows the case where Config region and
297 for HOST1, however the same is applicable for
298
299 Modeling Doorbell/Memory Window 1:
300 ----------------------------------
301
302 .. code-block:: text
303
304 +-----------------+ +----->+--------------
305 | BAR0 | | | Doorbell 1
306 +-----------------+ | +--------------
307 | BAR1 | | | Doorbell 2
308 +-----------------+----+ +--------------
309 | BAR2 | | Doorbell 3
310 +-----------------+----+ +--------------
311 | BAR3 | | | Doorbell 4
312 +-----------------+ | |--------------
313 | BAR4 | | |
314 +-----------------+ | | MW1
315 | BAR5 | | |
316 +-----------------+ +----->---------------
317 EP CONTROLLER 1 |
318 |
319 +--------------
320 EP CONTROLLER
321 (OB SPACE)
322
323
324
325
326
327
328
329
330
331
332
333
334
335 Above diagram shows how the doorbell and memor
336 HOST1 can raise doorbell interrupt on HOST2 an
337 buffers exposed by HOST2 using memory window1
338 memory window 1 regions are allocated in EP co
339 space. Allocating and configuring BARs for doo
340 is done during the initialization phase of NTB
341 Mapping from EP controller 2 OB space to PCI a
342 sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL.
343
344 Modeling Optional Memory Windows:
345 ---------------------------------
346
347 This is modeled the same was as MW1 but each o
348 is mapped to separate BARs.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.