1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ================= 3 =================
4 PCI NTB Function 4 PCI NTB Function
5 ================= 5 =================
6 6
7 :Author: Kishon Vijay Abraham I <kishon@ti.com> 7 :Author: Kishon Vijay Abraham I <kishon@ti.com>
8 8
9 PCI Non-Transparent Bridges (NTB) allow two ho 9 PCI Non-Transparent Bridges (NTB) allow two host systems to communicate
10 with each other by exposing each host as a dev 10 with each other by exposing each host as a device to the other host.
11 NTBs typically support the ability to generate 11 NTBs typically support the ability to generate interrupts on the remote
12 machine, expose memory ranges as BARs, and per 12 machine, expose memory ranges as BARs, and perform DMA. They also support
13 scratchpads, which are areas of memory within 13 scratchpads, which are areas of memory within the NTB that are accessible
14 from both machines. 14 from both machines.
15 15
16 PCI NTB Function allows two different systems 16 PCI NTB Function allows two different systems (or hosts) to communicate
17 with each other by configuring the endpoint in 17 with each other by configuring the endpoint instances in such a way that
18 transactions from one system are routed to the 18 transactions from one system are routed to the other system.
19 19
20 In the below diagram, PCI NTB function configu 20 In the below diagram, PCI NTB function configures the SoC with multiple
21 PCI Endpoint (EP) instances in such a way that 21 PCI Endpoint (EP) instances in such a way that transactions from one EP
22 controller are routed to the other EP controll 22 controller are routed to the other EP controller. Once PCI NTB function
23 configures the SoC with multiple EP instances, 23 configures the SoC with multiple EP instances, HOST1 and HOST2 can
24 communicate with each other using SoC as a bri 24 communicate with each other using SoC as a bridge.
25 25
26 .. code-block:: text 26 .. code-block:: text
27 27
28 +-------------+ 28 +-------------+ +-------------+
29 | | 29 | | | |
30 | HOST1 | 30 | HOST1 | | HOST2 |
31 | | 31 | | | |
32 +------^------+ 32 +------^------+ +------^------+
33 | 33 | |
34 | 34 | |
35 +---------|---------------------------------- 35 +---------|-------------------------------------------------|---------+
36 | +------v------+ 36 | +------v------+ +------v------+ |
37 | | | 37 | | | | | |
38 | | EP | 38 | | EP | | EP | |
39 | | CONTROLLER1 | 39 | | CONTROLLER1 | | CONTROLLER2 | |
40 | | <--------------------------- 40 | | <-----------------------------------> | |
41 | | | 41 | | | | | |
42 | | | 42 | | | | | |
43 | | | SoC With Multiple EP Inst 43 | | | SoC With Multiple EP Instances | | |
44 | | | (Configured using NTB Fun 44 | | | (Configured using NTB Function) | | |
45 | +-------------+ 45 | +-------------+ +-------------+ |
46 +-------------------------------------------- 46 +---------------------------------------------------------------------+
47 47
48 Constructs used for Implementing NTB 48 Constructs used for Implementing NTB
49 ==================================== 49 ====================================
50 50
51 1) Config Region 51 1) Config Region
52 2) Self Scratchpad Registers 52 2) Self Scratchpad Registers
53 3) Peer Scratchpad Registers 53 3) Peer Scratchpad Registers
54 4) Doorbell (DB) Registers 54 4) Doorbell (DB) Registers
55 5) Memory Window (MW) 55 5) Memory Window (MW)
56 56
57 57
58 Config Region: 58 Config Region:
59 -------------- 59 --------------
60 60
61 Config Region is a construct that is specific 61 Config Region is a construct that is specific to NTB implemented using NTB
62 Endpoint Function Driver. The host and endpoin 62 Endpoint Function Driver. The host and endpoint side NTB function driver will
63 exchange information with each other using thi 63 exchange information with each other using this region. Config Region has
64 Control/Status Registers for configuring the E 64 Control/Status Registers for configuring the Endpoint Controller. Host can
65 write into this region for configuring the out 65 write into this region for configuring the outbound Address Translation Unit
66 (ATU) and to indicate the link status. Endpoin 66 (ATU) and to indicate the link status. Endpoint can indicate the status of
67 commands issued by host in this region. Endpoi 67 commands issued by host in this region. Endpoint can also indicate the
68 scratchpad offset and number of memory windows 68 scratchpad offset and number of memory windows to the host using this region.
69 69
70 The format of Config Region is given below. Al 70 The format of Config Region is given below. All the fields here are 32 bits.
71 71
72 .. code-block:: text 72 .. code-block:: text
73 73
74 +------------------------+ 74 +------------------------+
75 | COMMAND | 75 | COMMAND |
76 +------------------------+ 76 +------------------------+
77 | ARGUMENT | 77 | ARGUMENT |
78 +------------------------+ 78 +------------------------+
79 | STATUS | 79 | STATUS |
80 +------------------------+ 80 +------------------------+
81 | TOPOLOGY | 81 | TOPOLOGY |
82 +------------------------+ 82 +------------------------+
83 | ADDRESS (LOWER 32) | 83 | ADDRESS (LOWER 32) |
84 +------------------------+ 84 +------------------------+
85 | ADDRESS (UPPER 32) | 85 | ADDRESS (UPPER 32) |
86 +------------------------+ 86 +------------------------+
87 | SIZE | 87 | SIZE |
88 +------------------------+ 88 +------------------------+
89 | NO OF MEMORY WINDOW | 89 | NO OF MEMORY WINDOW |
90 +------------------------+ 90 +------------------------+
91 | MEMORY WINDOW1 OFFSET | 91 | MEMORY WINDOW1 OFFSET |
92 +------------------------+ 92 +------------------------+
93 | SPAD OFFSET | 93 | SPAD OFFSET |
94 +------------------------+ 94 +------------------------+
95 | SPAD COUNT | 95 | SPAD COUNT |
96 +------------------------+ 96 +------------------------+
97 | DB ENTRY SIZE | 97 | DB ENTRY SIZE |
98 +------------------------+ 98 +------------------------+
99 | DB DATA | 99 | DB DATA |
100 +------------------------+ 100 +------------------------+
101 | : | 101 | : |
102 +------------------------+ 102 +------------------------+
103 | : | 103 | : |
104 +------------------------+ 104 +------------------------+
105 | DB DATA | 105 | DB DATA |
106 +------------------------+ 106 +------------------------+
107 107
108 108
109 COMMAND: 109 COMMAND:
110 110
111 NTB function supports three commands: 111 NTB function supports three commands:
112 112
113 CMD_CONFIGURE_DOORBELL (0x1): Comman 113 CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
114 invoking this command, the host should 114 invoking this command, the host should allocate and initialize
115 MSI/MSI-X vectors (i.e., initialize th 115 MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the
116 Endpoint). The endpoint on receiving t 116 Endpoint). The endpoint on receiving this command will configure
117 the outbound ATU such that transaction 117 the outbound ATU such that transactions to Doorbell BAR will be routed
118 to the MSI/MSI-X address programmed by 118 to the MSI/MSI-X address programmed by the host. The ARGUMENT
119 register should be populated with numb 119 register should be populated with number of DBs to configure (in the
120 lower 16 bits) and if MSI or MSI-X sho 120 lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
121 121
122 CMD_CONFIGURE_MW (0x2): Command to c 122 CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The
123 host invokes this command after alloca 123 host invokes this command after allocating a buffer that can be
124 accessed by remote host. The allocated 124 accessed by remote host. The allocated address should be programmed
125 in the ADDRESS register (64 bit), the 125 in the ADDRESS register (64 bit), the size should be programmed in
126 the SIZE register and the memory windo 126 the SIZE register and the memory window index should be programmed
127 in the ARGUMENT register. The endpoint 127 in the ARGUMENT register. The endpoint on receiving this command
128 will configure the outbound ATU such t 128 will configure the outbound ATU such that transactions to MW BAR
129 are routed to the address provided by 129 are routed to the address provided by the host.
130 130
131 CMD_LINK_UP (0x3): Command to indica 131 CMD_LINK_UP (0x3): Command to indicate an NTB application is
132 bound to the EP device on the host sid 132 bound to the EP device on the host side. Once the endpoint
133 receives this command from both the ho 133 receives this command from both the hosts, the endpoint will
134 raise a LINK_UP event to both the host 134 raise a LINK_UP event to both the hosts to indicate the host
135 NTB applications can start communicati 135 NTB applications can start communicating with each other.
136 136
137 ARGUMENT: 137 ARGUMENT:
138 138
139 The value of this register is based on 139 The value of this register is based on the commands issued in
140 command register. See COMMAND section 140 command register. See COMMAND section for more information.
141 141
142 TOPOLOGY: 142 TOPOLOGY:
143 143
144 Set to NTB_TOPO_B2B_USD for Primary in 144 Set to NTB_TOPO_B2B_USD for Primary interface
145 Set to NTB_TOPO_B2B_DSD for Secondary 145 Set to NTB_TOPO_B2B_DSD for Secondary interface
146 146
147 ADDRESS/SIZE: 147 ADDRESS/SIZE:
148 148
149 Address and Size to be used while conf 149 Address and Size to be used while configuring the memory window.
150 See "CMD_CONFIGURE_MW" for more info. 150 See "CMD_CONFIGURE_MW" for more info.
151 151
152 MEMORY WINDOW1 OFFSET: 152 MEMORY WINDOW1 OFFSET:
153 153
154 Memory Window 1 and Doorbell registers 154 Memory Window 1 and Doorbell registers are packed together in the
155 same BAR. The initial portion of the r 155 same BAR. The initial portion of the region will have doorbell
156 registers and the latter portion of th 156 registers and the latter portion of the region is for memory window 1.
157 This register will specify the offset 157 This register will specify the offset of the memory window 1.
158 158
159 NO OF MEMORY WINDOW: 159 NO OF MEMORY WINDOW:
160 160
161 Specifies the number of memory windows 161 Specifies the number of memory windows supported by the NTB device.
162 162
163 SPAD OFFSET: 163 SPAD OFFSET:
164 164
165 Self scratchpad region and config regi 165 Self scratchpad region and config region are packed together in the
166 same BAR. The initial portion of the r 166 same BAR. The initial portion of the region will have config region
167 and the latter portion of the region i 167 and the latter portion of the region is for self scratchpad. This
168 register will specify the offset of th 168 register will specify the offset of the self scratchpad registers.
169 169
170 SPAD COUNT: 170 SPAD COUNT:
171 171
172 Specifies the number of scratchpad reg 172 Specifies the number of scratchpad registers supported by the NTB
173 device. 173 device.
174 174
175 DB ENTRY SIZE: 175 DB ENTRY SIZE:
176 176
177 Used to determine the offset within th 177 Used to determine the offset within the DB BAR that should be written
178 in order to raise doorbell. EPF NTB ca 178 in order to raise doorbell. EPF NTB can use either MSI or MSI-X to
179 ring doorbell (MSI-X support will be a 179 ring doorbell (MSI-X support will be added later). MSI uses same
180 address for all the interrupts and MSI 180 address for all the interrupts and MSI-X can provide different
181 addresses for different interrupts. Th 181 addresses for different interrupts. The MSI/MSI-X address is provided
182 by the host and the address it gives i 182 by the host and the address it gives is based on the MSI/MSI-X
183 implementation supported by the host. 183 implementation supported by the host. For instance, ARM platform
184 using GIC ITS will have the same MSI-X 184 using GIC ITS will have the same MSI-X address for all the interrupts.
185 In order to support all the combinatio 185 In order to support all the combinations and use the same mechanism
186 for both MSI and MSI-X, EPF NTB alloca 186 for both MSI and MSI-X, EPF NTB allocates a separate region in the
187 Outbound Address Space for each of the 187 Outbound Address Space for each of the interrupts. This region will
188 be mapped to the MSI/MSI-X address pro 188 be mapped to the MSI/MSI-X address provided by the host. If a host
189 provides the same address for all the 189 provides the same address for all the interrupts, all the regions
190 will be translated to the same address 190 will be translated to the same address. If a host provides different
191 addresses, the regions will be transla 191 addresses, the regions will be translated to different addresses. This
192 will ensure there is no difference whi 192 will ensure there is no difference while raising the doorbell.
193 193
194 DB DATA: 194 DB DATA:
195 195
196 EPF NTB supports 32 interrupts, so the 196 EPF NTB supports 32 interrupts, so there are 32 DB DATA registers.
197 This holds the MSI/MSI-X data that has 197 This holds the MSI/MSI-X data that has to be written to MSI address
198 for raising doorbell interrupt. This w 198 for raising doorbell interrupt. This will be populated by EPF NTB
199 while invoking CMD_CONFIGURE_DOORBELL. 199 while invoking CMD_CONFIGURE_DOORBELL.
200 200
201 Scratchpad Registers: 201 Scratchpad Registers:
202 --------------------- 202 ---------------------
203 203
204 Each host has its own register space allocat 204 Each host has its own register space allocated in the memory of NTB endpoint
205 controller. They are both readable and writa 205 controller. They are both readable and writable from both sides of the bridge.
206 They are used by applications built over NTB 206 They are used by applications built over NTB and can be used to pass control
207 and status information between both sides of 207 and status information between both sides of a device.
208 208
209 Scratchpad registers has 2 parts 209 Scratchpad registers has 2 parts
210 1) Self Scratchpad: Host's own registe 210 1) Self Scratchpad: Host's own register space
211 2) Peer Scratchpad: Remote host's regi 211 2) Peer Scratchpad: Remote host's register space.
212 212
213 Doorbell Registers: 213 Doorbell Registers:
214 ------------------- 214 -------------------
215 215
216 Doorbell Registers are used by the hosts to 216 Doorbell Registers are used by the hosts to interrupt each other.
217 217
218 Memory Window: 218 Memory Window:
219 -------------- 219 --------------
220 220
221 Actual transfer of data between the two host 221 Actual transfer of data between the two hosts will happen using the
222 memory window. 222 memory window.
223 223
224 Modeling Constructs: 224 Modeling Constructs:
225 ==================== 225 ====================
226 226
227 There are 5 or more distinct regions (config, 227 There are 5 or more distinct regions (config, self scratchpad, peer
228 scratchpad, doorbell, one or more memory windo 228 scratchpad, doorbell, one or more memory windows) to be modeled to achieve
229 NTB functionality. At least one memory window 229 NTB functionality. At least one memory window is required while more than
230 one is permitted. All these regions should be 230 one is permitted. All these regions should be mapped to BARs for hosts to
231 access these regions. 231 access these regions.
232 232
233 If one 32-bit BAR is allocated for each of the 233 If one 32-bit BAR is allocated for each of these regions, the scheme would
234 look like this: 234 look like this:
235 235
236 ====== =============== 236 ====== ===============
237 BAR NO CONSTRUCTS USED 237 BAR NO CONSTRUCTS USED
238 ====== =============== 238 ====== ===============
239 BAR0 Config Region 239 BAR0 Config Region
240 BAR1 Self Scratchpad 240 BAR1 Self Scratchpad
241 BAR2 Peer Scratchpad 241 BAR2 Peer Scratchpad
242 BAR3 Doorbell 242 BAR3 Doorbell
243 BAR4 Memory Window 1 243 BAR4 Memory Window 1
244 BAR5 Memory Window 2 244 BAR5 Memory Window 2
245 ====== =============== 245 ====== ===============
246 246
247 However if we allocate a separate BAR for each 247 However if we allocate a separate BAR for each of the regions, there would not
248 be enough BARs for all the regions in a platfo 248 be enough BARs for all the regions in a platform that supports only 64-bit
249 BARs. 249 BARs.
250 250
251 In order to be supported by most of the platfo 251 In order to be supported by most of the platforms, the regions should be
252 packed and mapped to BARs in a way that provid 252 packed and mapped to BARs in a way that provides NTB functionality and
253 also makes sure the host doesn't access any re 253 also makes sure the host doesn't access any region that it is not supposed
254 to. 254 to.
255 255
256 The following scheme is used in EPF NTB Functi 256 The following scheme is used in EPF NTB Function:
257 257
258 ====== =============================== 258 ====== ===============================
259 BAR NO CONSTRUCTS USED 259 BAR NO CONSTRUCTS USED
260 ====== =============================== 260 ====== ===============================
261 BAR0 Config Region + Self Scratchpad 261 BAR0 Config Region + Self Scratchpad
262 BAR1 Peer Scratchpad 262 BAR1 Peer Scratchpad
263 BAR2 Doorbell + Memory Window 1 263 BAR2 Doorbell + Memory Window 1
264 BAR3 Memory Window 2 264 BAR3 Memory Window 2
265 BAR4 Memory Window 3 265 BAR4 Memory Window 3
266 BAR5 Memory Window 4 266 BAR5 Memory Window 4
267 ====== =============================== 267 ====== ===============================
268 268
269 With this scheme, for the basic NTB functional 269 With this scheme, for the basic NTB functionality 3 BARs should be sufficient.
270 270
271 Modeling Config/Scratchpad Region: 271 Modeling Config/Scratchpad Region:
272 ---------------------------------- 272 ----------------------------------
273 273
274 .. code-block:: text 274 .. code-block:: text
275 275
276 +-----------------+------->+----------------- 276 +-----------------+------->+------------------+ +-----------------+
277 | BAR0 | | CONFIG REGION 277 | BAR0 | | CONFIG REGION | | BAR0 |
278 +-----------------+----+ +----------------- 278 +-----------------+----+ +------------------+<-------+-----------------+
279 | BAR1 | | |SCRATCHPAD REGION 279 | BAR1 | | |SCRATCHPAD REGION | | BAR1 |
280 +-----------------+ +-->+----------------- 280 +-----------------+ +-->+------------------+<-------+-----------------+
281 | BAR2 | Local Memory 281 | BAR2 | Local Memory | BAR2 |
282 +-----------------+ 282 +-----------------+ +-----------------+
283 | BAR3 | 283 | BAR3 | | BAR3 |
284 +-----------------+ 284 +-----------------+ +-----------------+
285 | BAR4 | 285 | BAR4 | | BAR4 |
286 +-----------------+ 286 +-----------------+ +-----------------+
287 | BAR5 | 287 | BAR5 | | BAR5 |
288 +-----------------+ 288 +-----------------+ +-----------------+
289 EP CONTROLLER 1 289 EP CONTROLLER 1 EP CONTROLLER 2
290 290
291 Above diagram shows Config region + Scratchpad 291 Above diagram shows Config region + Scratchpad region for HOST1 (connected to
292 EP controller 1) allocated in local memory. Th 292 EP controller 1) allocated in local memory. The HOST1 can access the config
293 region and scratchpad region (self scratchpad) 293 region and scratchpad region (self scratchpad) using BAR0 of EP controller 1.
294 The peer host (HOST2 connected to EP controlle 294 The peer host (HOST2 connected to EP controller 2) can also access this
295 scratchpad region (peer scratchpad) using BAR1 295 scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This
296 diagram shows the case where Config region and 296 diagram shows the case where Config region and Scratchpad regions are allocated
297 for HOST1, however the same is applicable for 297 for HOST1, however the same is applicable for HOST2.
298 298
299 Modeling Doorbell/Memory Window 1: 299 Modeling Doorbell/Memory Window 1:
300 ---------------------------------- 300 ----------------------------------
301 301
302 .. code-block:: text 302 .. code-block:: text
303 303
304 +-----------------+ +----->+-------------- 304 +-----------------+ +----->+----------------+-----------+-----------------+
305 | BAR0 | | | Doorbell 1 305 | BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 |
306 +-----------------+ | +-------------- 306 +-----------------+ | +----------------+ +-----------------+
307 | BAR1 | | | Doorbell 2 307 | BAR1 | | | Doorbell 2 +---------+ | |
308 +-----------------+----+ +-------------- 308 +-----------------+----+ +----------------+ | | |
309 | BAR2 | | Doorbell 3 309 | BAR2 | | Doorbell 3 +-------+ | +-----------------+
310 +-----------------+----+ +-------------- 310 +-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 |
311 | BAR3 | | | Doorbell 4 311 | BAR3 | | | Doorbell 4 +-----+ | +-----------------+
312 +-----------------+ | |-------------- 312 +-----------------+ | |----------------+ | | | |
313 | BAR4 | | | 313 | BAR4 | | | | | | +-----------------+
314 +-----------------+ | | MW1 314 +-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3||
315 | BAR5 | | | 315 | BAR5 | | | | | | +-----------------+
316 +-----------------+ +----->--------------- 316 +-----------------+ +----->-----------------+ | | | |
317 EP CONTROLLER 1 | 317 EP CONTROLLER 1 | | | | +-----------------+
318 | 318 | | | +---->+ MSI-X ADDRESS 4 |
319 +-------------- 319 +----------------+ | +-----------------+
320 EP CONTROLLER 320 EP CONTROLLER 2 | | |
321 (OB SPACE) 321 (OB SPACE) | | |
322 322 +-------> MW1 |
323 323 | |
324 324 | |
325 325 +-----------------+
326 326 | |
327 327 | |
328 328 | |
329 329 | |
330 330 | |
331 331 +-----------------+
332 332 PCI Address Space
333 333 (Managed by HOST2)
334 334
335 Above diagram shows how the doorbell and memor 335 Above diagram shows how the doorbell and memory window 1 is mapped so that
336 HOST1 can raise doorbell interrupt on HOST2 an 336 HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access
337 buffers exposed by HOST2 using memory window1 337 buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and
338 memory window 1 regions are allocated in EP co 338 memory window 1 regions are allocated in EP controller 2 outbound (OB) address
339 space. Allocating and configuring BARs for doo 339 space. Allocating and configuring BARs for doorbell and memory window1
340 is done during the initialization phase of NTB 340 is done during the initialization phase of NTB endpoint function driver.
341 Mapping from EP controller 2 OB space to PCI a 341 Mapping from EP controller 2 OB space to PCI address space is done when HOST2
342 sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL. 342 sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL.
343 343
344 Modeling Optional Memory Windows: 344 Modeling Optional Memory Windows:
345 --------------------------------- 345 ---------------------------------
346 346
347 This is modeled the same was as MW1 but each o 347 This is modeled the same was as MW1 but each of the additional memory windows
348 is mapped to separate BARs. 348 is mapped to separate BARs.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.