1 .. include:: <isonum.txt> 2 3 ========================================================= 4 DPAA2 (Data Path Acceleration Architecture Gen2) Overview 5 ========================================================= 6 7 :Copyright: |copy| 2015 Freescale Semiconductor Inc. 8 :Copyright: |copy| 2018 NXP 9 10 This document provides an overview of the Freescale DPAA2 architecture 11 and how it is integrated into the Linux kernel. 12 13 Introduction 14 ============ 15 16 DPAA2 is a hardware architecture designed for high-speeed network 17 packet processing. DPAA2 consists of sophisticated mechanisms for 18 processing Ethernet packets, queue management, buffer management, 19 autonomous L2 switching, virtual Ethernet bridging, and accelerator 20 (e.g. crypto) sharing. 21 22 A DPAA2 hardware component called the Management Complex (or MC) manages the 23 DPAA2 hardware resources. The MC provides an object-based abstraction for 24 software drivers to use the DPAA2 hardware. 25 The MC uses DPAA2 hardware resources such as queues, buffer pools, and 26 network ports to create functional objects/devices such as network 27 interfaces, an L2 switch, or accelerator instances. 28 The MC provides memory-mapped I/O command interfaces (MC portals) 29 which DPAA2 software drivers use to operate on DPAA2 objects. 30 31 The diagram below shows an overview of the DPAA2 resource management 32 architecture:: 33 34 +--------------------------------------+ 35 | OS | 36 | DPAA2 drivers | 37 | | | 38 +-----------------------------|--------+ 39 | 40 | (create,discover,connect 41 | config,use,destroy) 42 | 43 DPAA2 | 44 +------------------------| mc portal |-+ 45 | | | 46 | +- - - - - - - - - - - - -V- - -+ | 47 | | | | 48 | | Management Complex (MC) | | 49 | | | | 50 | +- - - - - - - - - - - - - - - -+ | 51 | | 52 | Hardware Hardware | 53 | Resources Objects | 54 | --------- ------- | 55 | -queues -DPRC | 56 | -buffer pools -DPMCP | 57 | -Eth MACs/ports -DPIO | 58 | -network interface -DPNI | 59 | profiles -DPMAC | 60 | -queue portals -DPBP | 61 | -MC portals ... | 62 | ... | 63 | | 64 +--------------------------------------+ 65 66 67 The MC mediates operations such as create, discover, 68 connect, configuration, and destroy. Fast-path operations 69 on data, such as packet transmit/receive, are not mediated by 70 the MC and are done directly using memory mapped regions in 71 DPIO objects. 72 73 Overview of DPAA2 Objects 74 ========================= 75 76 The section provides a brief overview of some key DPAA2 objects. 77 A simple scenario is described illustrating the objects involved 78 in creating a network interfaces. 79 80 DPRC (Datapath Resource Container) 81 ---------------------------------- 82 83 A DPRC is a container object that holds all the other 84 types of DPAA2 objects. In the example diagram below there 85 are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) 86 in the container. 87 88 :: 89 90 +---------------------------------------------------------+ 91 | DPRC | 92 | | 93 | +-------+ +-------+ +-------+ +-------+ +-------+ | 94 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | 95 | +-------+ +-------+ +-------+ +---+---+ +---+---+ | 96 | | DPMCP | | DPIO | | 97 | +-------+ +-------+ | 98 | | DPMCP | | 99 | +-------+ | 100 | | 101 +---------------------------------------------------------+ 102 103 From the point of view of an OS, a DPRC behaves similar to a plug and 104 play bus, like PCI. DPRC commands can be used to enumerate the contents 105 of the DPRC, discover the hardware objects present (including mappable 106 regions and interrupts). 107 108 :: 109 110 DPRC.1 (bus) 111 | 112 +--+--------+-------+-------+-------+ 113 | | | | | 114 DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1 115 DPMCP.2 DPIO.2 116 DPMCP.3 117 118 Hardware objects can be created and destroyed dynamically, providing 119 the ability to hot plug/unplug objects in and out of the DPRC. 120 121 A DPRC has a mappable MMIO region (an MC portal) that can be used 122 to send MC commands. It has an interrupt for status events (like 123 hotplug). 124 All objects in a container share the same hardware "isolation context". 125 This means that with respect to an IOMMU the isolation granularity 126 is at the DPRC (container) level, not at the individual object 127 level. 128 129 DPRCs can be defined statically and populated with objects 130 via a config file passed to the MC when firmware starts it. 131 132 DPAA2 Objects for an Ethernet Network Interface 133 ----------------------------------------------- 134 135 A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX 136 queuing mechanisms, configuration mechanisms, buffer management, 137 physical ports, and interrupts. DPAA2 uses a more granular approach 138 utilizing multiple hardware objects. Each object provides specialized 139 functions. Groups of these objects are used by software to provide 140 Ethernet network interface functionality. This approach provides 141 efficient use of finite hardware resources, flexibility, and 142 performance advantages. 143 144 The diagram below shows the objects needed for a simple 145 network interface configuration on a system with 2 CPUs. 146 147 :: 148 149 +---+---+ +---+---+ 150 CPU0 CPU1 151 +---+---+ +---+---+ 152 | | 153 +---+---+ +---+---+ 154 DPIO DPIO 155 +---+---+ +---+---+ 156 \ / 157 \ / 158 \ / 159 +---+---+ 160 DPNI --- DPBP,DPMCP 161 +---+---+ 162 | 163 | 164 +---+---+ 165 DPMAC 166 +---+---+ 167 | 168 port/PHY 169 170 Below the objects are described. For each object a brief description 171 is provided along with a summary of the kinds of operations the object 172 supports and a summary of key resources of the object (MMIO regions 173 and IRQs). 174 175 DPMAC (Datapath Ethernet MAC) 176 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 177 Represents an Ethernet MAC, a hardware device that connects to an Ethernet 178 PHY and allows physical transmission and reception of Ethernet frames. 179 180 - MMIO regions: none 181 - IRQs: DPNI link change 182 - commands: set link up/down, link config, get stats, 183 IRQ config, enable, reset 184 185 DPNI (Datapath Network Interface) 186 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 187 Contains TX/RX queues, network interface configuration, and RX buffer pool 188 configuration mechanisms. The TX/RX queues are in memory and are identified 189 by queue number. 190 191 - MMIO regions: none 192 - IRQs: link state 193 - commands: port config, offload config, queue config, 194 parse/classify config, IRQ config, enable, reset 195 196 DPIO (Datapath I/O) 197 ~~~~~~~~~~~~~~~~~~~ 198 Provides interfaces to enqueue and dequeue 199 packets and do hardware buffer pool management operations. The DPAA2 200 architecture separates the mechanism to access queues (the DPIO object) 201 from the queues themselves. The DPIO provides an MMIO interface to 202 enqueue/dequeue packets. To enqueue something a descriptor is written 203 to the DPIO MMIO region, which includes the target queue number. 204 There will typically be one DPIO assigned to each CPU. This allows all 205 CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are 206 expected to be shared by different DPAA2 drivers. 207 208 - MMIO regions: queue operations, buffer management 209 - IRQs: data availability, congestion notification, buffer 210 pool depletion 211 - commands: IRQ config, enable, reset 212 213 DPBP (Datapath Buffer Pool) 214 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 215 Represents a hardware buffer pool. 216 217 - MMIO regions: none 218 - IRQs: none 219 - commands: enable, reset 220 221 DPMCP (Datapath MC Portal) 222 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 223 Provides an MC command portal. 224 Used by drivers to send commands to the MC to manage 225 objects. 226 227 - MMIO regions: MC command portal 228 - IRQs: command completion 229 - commands: IRQ config, enable, reset 230 231 Object Connections 232 ================== 233 Some objects have explicit relationships that must 234 be configured: 235 236 - DPNI <--> DPMAC 237 - DPNI <--> DPNI 238 - DPNI <--> L2-switch-port 239 240 A DPNI must be connected to something such as a DPMAC, 241 another DPNI, or L2 switch port. The DPNI connection 242 is made via a DPRC command. 243 244 :: 245 246 +-------+ +-------+ 247 | DPNI | | DPMAC | 248 +---+---+ +---+---+ 249 | | 250 +==========+ 251 252 - DPNI <--> DPBP 253 254 A network interface requires a 'buffer pool' (DPBP 255 object) which provides a list of pointers to memory 256 where received Ethernet data is to be copied. The 257 Ethernet driver configures the DPBPs associated with 258 the network interface. 259 260 Interrupts 261 ========== 262 All interrupts generated by DPAA2 objects are message 263 interrupts. At the hardware level message interrupts 264 generated by devices will normally have 3 components-- 265 1) a non-spoofable 'device-id' expressed on the hardware 266 bus, 2) an address, 3) a data value. 267 268 In the case of DPAA2 devices/objects, all objects in the 269 same container/DPRC share the same 'device-id'. 270 For ARM-based SoC this is the same as the stream ID. 271 272 273 DPAA2 Linux Drivers Overview 274 ============================ 275 276 This section provides an overview of the Linux kernel drivers for 277 DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" 278 drivers and 2) functional object drivers (such as Ethernet). 279 280 As described previously, a DPRC is a container that holds the other 281 types of DPAA2 objects. It is functionally similar to a plug-and-play 282 bus controller. 283 Each object in the DPRC is a Linux "device" and is bound to a driver. 284 The diagram below shows the Linux drivers involved in a networking 285 scenario and the objects bound to each driver. A brief description 286 of each driver follows. 287 288 :: 289 290 +------------+ 291 | OS Network | 292 | Stack | 293 +------------+ +------------+ 294 | Allocator |. . . . . . . | Ethernet | 295 |(DPMCP,DPBP)| | (DPNI) | 296 +-.----------+ +---+---+----+ 297 . . ^ | 298 . . <data avail, | | <enqueue, 299 . . tx confirm> | | dequeue> 300 +-------------+ . | | 301 | DPRC driver | . +---+---V----+ +---------+ 302 | (DPRC) | . . . . . .| DPIO driver| | MAC | 303 +----------+--+ | (DPIO) | | (DPMAC) | 304 | +------+-----+ +-----+---+ 305 |<dev add/remove> | | 306 | | | 307 +--------+----------+ | +--+---+ 308 | MC-bus driver | | | PHY | 309 | | | |driver| 310 | /bus/fsl-mc | | +--+---+ 311 +-------------------+ | | 312 | | 313 ========================= HARDWARE =========|=================|====== 314 DPIO | 315 | | 316 DPNI---DPBP | 317 | | 318 DPMAC | 319 | | 320 PHY ---------------+ 321 ============================================|======================== 322 323 A brief description of each driver is provided below. 324 325 MC-bus driver 326 ------------- 327 The MC-bus driver is a platform driver and is probed from a 328 node in the device tree (compatible "fsl,qoriq-mc") passed in by boot 329 firmware. It is responsible for bootstrapping the DPAA2 kernel 330 infrastructure. 331 Key functions include: 332 333 - registering a new bus type named "fsl-mc" with the kernel, 334 and implementing bus call-backs (e.g. match/uevent/dev_groups) 335 - implementing APIs for DPAA2 driver registration and for device 336 add/remove 337 - creates an MSI IRQ domain 338 - doing a 'device add' to expose the 'root' DPRC, in turn triggering 339 a bind of the root DPRC to the DPRC driver 340 341 The binding for the MC-bus device-tree node can be consulted at 342 *Documentation/devicetree/bindings/misc/fsl,qoriq-mc.yaml*. 343 The sysfs bind/unbind interfaces for the MC-bus can be consulted at 344 *Documentation/ABI/testing/sysfs-bus-fsl-mc*. 345 346 DPRC driver 347 ----------- 348 The DPRC driver is bound to DPRC objects and does runtime management 349 of a bus instance. It performs the initial bus scan of the DPRC 350 and handles interrupts for container events such as hot plug by 351 re-scanning the DPRC. 352 353 Allocator 354 --------- 355 Certain objects such as DPMCP and DPBP are generic and fungible, 356 and are intended to be used by other drivers. For example, 357 the DPAA2 Ethernet driver needs: 358 359 - DPMCPs to send MC commands, to configure network interfaces 360 - DPBPs for network buffer pools 361 362 The allocator driver registers for these allocatable object types 363 and those objects are bound to the allocator when the bus is probed. 364 The allocator maintains a pool of objects that are available for 365 allocation by other DPAA2 drivers. 366 367 DPIO driver 368 ----------- 369 The DPIO driver is bound to DPIO objects and provides services that allow 370 other drivers such as the Ethernet driver to enqueue and dequeue data for 371 their respective objects. 372 Key services include: 373 374 - data availability notifications 375 - hardware queuing operations (enqueue and dequeue of data) 376 - hardware buffer pool management 377 378 To transmit a packet the Ethernet driver puts data on a queue and 379 invokes a DPIO API. For receive, the Ethernet driver registers 380 a data availability notification callback. To dequeue a packet 381 a DPIO API is used. 382 There is typically one DPIO object per physical CPU for optimum 383 performance, allowing different CPUs to simultaneously enqueue 384 and dequeue data. 385 386 The DPIO driver operates on behalf of all DPAA2 drivers 387 active in the kernel-- Ethernet, crypto, compression, 388 etc. 389 390 Ethernet driver 391 --------------- 392 The Ethernet driver is bound to a DPNI and implements the kernel 393 interfaces needed to connect the DPAA2 network interface to 394 the network stack. 395 Each DPNI corresponds to a Linux network interface. 396 397 MAC driver 398 ---------- 399 An Ethernet PHY is an off-chip, board specific component and is managed 400 by the appropriate PHY driver via an mdio bus. The MAC driver 401 plays a role of being a proxy between the PHY driver and the 402 MC. It does this proxy via the MC commands to a DPMAC object. 403 If the PHY driver signals a link change, the MAC driver notifies 404 the MC via a DPMAC command. If a network interface is brought 405 up or down, the MC notifies the DPMAC driver via an interrupt and 406 the driver can take appropriate action.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.