1 ================================================= 2 FPGA Device Feature List (DFL) Framework Overview 3 ================================================= 4 5 Authors: 6 7 - Enno Luebbers <enno.luebbers@intel.com> 8 - Xiao Guangrong <guangrong.xiao@linux.intel.com> 9 - Wu Hao <hao.wu@intel.com> 10 - Xu Yilun <yilun.xu@intel.com> 11 12 The Device Feature List (DFL) FPGA framework (and drivers according to 13 this framework) hides the very details of low layer hardware and provides 14 unified interfaces to userspace. Applications could use these interfaces to 15 configure, enumerate, open and access FPGA accelerators on platforms which 16 implement the DFL in the device memory. Besides this, the DFL framework 17 enables system level management functions such as FPGA reconfiguration. 18 19 20 Device Feature List (DFL) Overview 21 ================================== 22 Device Feature List (DFL) defines a linked list of feature headers within the 23 device MMIO space to provide an extensible way of adding features. Software can 24 walk through these predefined data structures to enumerate FPGA features: 25 FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features, 26 as illustrated below:: 27 28 Header Header Header Header 29 +----------+ +-->+----------+ +-->+----------+ +-->+----------+ 30 | Type | | | Type | | | Type | | | Type | 31 | FIU | | | Private | | | Private | | | Private | 32 +----------+ | | Feature | | | Feature | | | Feature | 33 | Next_DFH |--+ +----------+ | +----------+ | +----------+ 34 +----------+ | Next_DFH |--+ | Next_DFH |--+ | Next_DFH |--> NULL 35 | ID | +----------+ +----------+ +----------+ 36 +----------+ | ID | | ID | | ID | 37 | Next_AFU |--+ +----------+ +----------+ +----------+ 38 +----------+ | | Feature | | Feature | | Feature | 39 | Header | | | Register | | Register | | Register | 40 | Register | | | Set | | Set | | Set | 41 | Set | | +----------+ +----------+ +----------+ 42 +----------+ | Header 43 +-->+----------+ 44 | Type | 45 | AFU | 46 +----------+ 47 | Next_DFH |--> NULL 48 +----------+ 49 | GUID | 50 +----------+ 51 | Header | 52 | Register | 53 | Set | 54 +----------+ 55 56 FPGA Interface Unit (FIU) represents a standalone functional unit for the 57 interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more 58 descriptions on FME and Port in later sections). 59 60 Accelerated Function Unit (AFU) represents an FPGA programmable region and 61 always connects to a FIU (e.g. a Port) as its child as illustrated above. 62 63 Private Features represent sub features of the FIU and AFU. They could be 64 various function blocks with different IDs, but all private features which 65 belong to the same FIU or AFU, must be linked to one list via the Next Device 66 Feature Header (Next_DFH) pointer. 67 68 Each FIU, AFU and Private Feature could implement its own functional registers. 69 The functional register set for FIU and AFU, is named as Header Register Set, 70 e.g. FME Header Register Set, and the one for Private Feature, is named as 71 Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set. 72 73 This Device Feature List provides a way of linking features together, it's 74 convenient for software to locate each feature by walking through this list, 75 and can be implemented in register regions of any FPGA device. 76 77 78 Device Feature Header - Version 0 79 ================================= 80 Version 0 (DFHv0) is the original version of the Device Feature Header. 81 All multi-byte quantities in DFHv0 are little-endian. 82 The format of DFHv0 is shown below:: 83 84 +-----------------------------------------------------------------------+ 85 |63 Type 60|59 DFH VER 52|51 Rsvd 41|40 EOL|39 Next 16|15 REV 12|11 ID 0| 0x00 86 +-----------------------------------------------------------------------+ 87 |63 GUID_L 0| 0x08 88 +-----------------------------------------------------------------------+ 89 |63 GUID_H 0| 0x10 90 +-----------------------------------------------------------------------+ 91 92 - Offset 0x00 93 94 * Type - The type of DFH (e.g. FME, AFU, or private feature). 95 * DFH VER - The version of the DFH. 96 * Rsvd - Currently unused. 97 * EOL - Set if the DFH is the end of the Device Feature List (DFL). 98 * Next - The offset in bytes of the next DFH in the DFL from the DFH start, 99 and the start of a DFH must be aligned to an 8 byte boundary. 100 If EOL is set, Next is the size of MMIO of the last feature in the list. 101 * REV - The revision of the feature associated with this header. 102 * ID - The feature ID if Type is private feature. 103 104 - Offset 0x08 105 106 * GUID_L - Least significant 64 bits of a 128-bit Globally Unique Identifier 107 (present only if Type is FME or AFU). 108 109 - Offset 0x10 110 111 * GUID_H - Most significant 64 bits of a 128-bit Globally Unique Identifier 112 (present only if Type is FME or AFU). 113 114 115 Device Feature Header - Version 1 116 ================================= 117 Version 1 (DFHv1) of the Device Feature Header adds the following functionality: 118 119 * Provides a standardized mechanism for features to describe 120 parameters/capabilities to software. 121 * Standardize the use of a GUID for all DFHv1 types. 122 * Decouples the DFH location from the register space of the feature itself. 123 124 All multi-byte quantities in DFHv1 are little-endian. 125 The format of Version 1 of the Device Feature Header (DFH) is shown below:: 126 127 +-----------------------------------------------------------------------+ 128 |63 Type 60|59 DFH VER 52|51 Rsvd 41|40 EOL|39 Next 16|15 REV 12|11 ID 0| 0x00 129 +-----------------------------------------------------------------------+ 130 |63 GUID_L 0| 0x08 131 +-----------------------------------------------------------------------+ 132 |63 GUID_H 0| 0x10 133 +-----------------------------------------------------------------------+ 134 |63 Reg Address/Offset 1| Rel 0| 0x18 135 +-----------------------------------------------------------------------+ 136 |63 Reg Size 32|Params 31|30 Group 16|15 Instance 0| 0x20 137 +-----------------------------------------------------------------------+ 138 |63 Next 35|34RSV33|EOP32|31 Param Version 16|15 Param ID 0| 0x28 139 +-----------------------------------------------------------------------+ 140 |63 Parameter Data 0| 0x30 141 +-----------------------------------------------------------------------+ 142 143 ... 144 145 +-----------------------------------------------------------------------+ 146 |63 Next 35|34RSV33|EOP32|31 Param Version 16|15 Param ID 0| 147 +-----------------------------------------------------------------------+ 148 |63 Parameter Data 0| 149 +-----------------------------------------------------------------------+ 150 151 - Offset 0x00 152 153 * Type - The type of DFH (e.g. FME, AFU, or private feature). 154 * DFH VER - The version of the DFH. 155 * Rsvd - Currently unused. 156 * EOL - Set if the DFH is the end of the Device Feature List (DFL). 157 * Next - The offset in bytes of the next DFH in the DFL from the DFH start, 158 and the start of a DFH must be aligned to an 8 byte boundary. 159 If EOL is set, Next is the size of MMIO of the last feature in the list. 160 * REV - The revision of the feature associated with this header. 161 * ID - The feature ID if Type is private feature. 162 163 - Offset 0x08 164 165 * GUID_L - Least significant 64 bits of a 128-bit Globally Unique Identifier. 166 167 - Offset 0x10 168 169 * GUID_H - Most significant 64 bits of a 128-bit Globally Unique Identifier. 170 171 - Offset 0x18 172 173 * Reg Address/Offset - If Rel bit is set, then the value is the high 63 bits 174 of a 16-bit aligned absolute address of the feature's registers. Otherwise 175 the value is the offset from the start of the DFH of the feature's registers. 176 177 - Offset 0x20 178 179 * Reg Size - Size of feature's register set in bytes. 180 * Params - Set if DFH has a list of parameter blocks. 181 * Group - Id of group if feature is part of a group. 182 * Instance - Id of feature instance within a group. 183 184 - Offset 0x28 if feature has parameters 185 186 * Next - Offset to the next parameter block in 8 byte words. If EOP set, 187 size in 8 byte words of last parameter. 188 * Param Version - Version of Param ID. 189 * Param ID - ID of parameter. 190 191 - Offset 0x30 192 193 * Parameter Data - Parameter data whose size and format is defined by 194 version and ID of the parameter. 195 196 197 FIU - FME (FPGA Management Engine) 198 ================================== 199 The FPGA Management Engine performs reconfiguration and other infrastructure 200 functions. Each FPGA device only has one FME. 201 202 User-space applications can acquire exclusive access to the FME using open(), 203 and release it using close(). 204 205 The following functions are exposed through ioctls: 206 207 - Get driver API version (DFL_FPGA_GET_API_VERSION) 208 - Check for extensions (DFL_FPGA_CHECK_EXTENSION) 209 - Program bitstream (DFL_FPGA_FME_PORT_PR) 210 - Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN) 211 - Release port from PF (DFL_FPGA_FME_PORT_RELEASE) 212 - Get number of irqs of FME global error (DFL_FPGA_FME_ERR_GET_IRQ_NUM) 213 - Set interrupt trigger for FME error (DFL_FPGA_FME_ERR_SET_IRQ) 214 215 More functions are exposed through sysfs 216 (/sys/class/fpga_region/regionX/dfl-fme.n/): 217 218 Read bitstream ID (bitstream_id) 219 bitstream_id indicates version of the static FPGA region. 220 221 Read bitstream metadata (bitstream_metadata) 222 bitstream_metadata includes detailed information of static FPGA region, 223 e.g. synthesis date and seed. 224 225 Read number of ports (ports_num) 226 one FPGA device may have more than one port, this sysfs interface indicates 227 how many ports the FPGA device has. 228 229 Global error reporting management (errors/) 230 error reporting sysfs interfaces allow user to read errors detected by the 231 hardware, and clear the logged errors. 232 233 Power management (dfl_fme_power hwmon) 234 power management hwmon sysfs interfaces allow user to read power management 235 information (power consumption, thresholds, threshold status, limits, etc.) 236 and configure power thresholds for different throttling levels. 237 238 Thermal management (dfl_fme_thermal hwmon) 239 thermal management hwmon sysfs interfaces allow user to read thermal 240 management information (current temperature, thresholds, threshold status, 241 etc.). 242 243 Performance reporting 244 performance counters are exposed through perf PMU APIs. Standard perf tool 245 can be used to monitor all available perf events. Please see performance 246 counter section below for more detailed information. 247 248 249 FIU - PORT 250 ========== 251 A port represents the interface between the static FPGA fabric and a partially 252 reconfigurable region containing an AFU. It controls the communication from SW 253 to the accelerator and exposes features such as reset and debug. Each FPGA 254 device may have more than one port, but always one AFU per port. 255 256 257 AFU 258 === 259 An AFU is attached to a port FIU and exposes a fixed length MMIO region to be 260 used for accelerator-specific control registers. 261 262 User-space applications can acquire exclusive access to an AFU attached to a 263 port by using open() on the port device node and release it using close(). 264 265 The following functions are exposed through ioctls: 266 267 - Get driver API version (DFL_FPGA_GET_API_VERSION) 268 - Check for extensions (DFL_FPGA_CHECK_EXTENSION) 269 - Get port info (DFL_FPGA_PORT_GET_INFO) 270 - Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO) 271 - Map DMA buffer (DFL_FPGA_PORT_DMA_MAP) 272 - Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP) 273 - Reset AFU (DFL_FPGA_PORT_RESET) 274 - Get number of irqs of port error (DFL_FPGA_PORT_ERR_GET_IRQ_NUM) 275 - Set interrupt trigger for port error (DFL_FPGA_PORT_ERR_SET_IRQ) 276 - Get number of irqs of UINT (DFL_FPGA_PORT_UINT_GET_IRQ_NUM) 277 - Set interrupt trigger for UINT (DFL_FPGA_PORT_UINT_SET_IRQ) 278 279 DFL_FPGA_PORT_RESET: 280 reset the FPGA Port and its AFU. Userspace can do Port 281 reset at any time, e.g. during DMA or Partial Reconfiguration. But it should 282 never cause any system level issue, only functional failure (e.g. DMA or PR 283 operation failure) and be recoverable from the failure. 284 285 User-space applications can also mmap() accelerator MMIO regions. 286 287 More functions are exposed through sysfs: 288 (/sys/class/fpga_region/<regionX>/<dfl-port.m>/): 289 290 Read Accelerator GUID (afu_id) 291 afu_id indicates which PR bitstream is programmed to this AFU. 292 293 Error reporting (errors/) 294 error reporting sysfs interfaces allow user to read port/afu errors 295 detected by the hardware, and clear the logged errors. 296 297 298 DFL Framework Overview 299 ====================== 300 301 :: 302 303 +----------+ +--------+ +--------+ +--------+ 304 | FME | | AFU | | AFU | | AFU | 305 | Module | | Module | | Module | | Module | 306 +----------+ +--------+ +--------+ +--------+ 307 +-----------------------+ 308 | FPGA Container Device | Device Feature List 309 | (FPGA Base Region) | Framework 310 +-----------------------+ 311 ------------------------------------------------------------------ 312 +----------------------------+ 313 | FPGA DFL Device Module | 314 | (e.g. PCIE/Platform Device)| 315 +----------------------------+ 316 +------------------------+ 317 | FPGA Hardware Device | 318 +------------------------+ 319 320 DFL framework in kernel provides common interfaces to create container device 321 (FPGA base region), discover feature devices and their private features from the 322 given Device Feature Lists and create platform devices for feature devices 323 (e.g. FME, Port and AFU) with related resources under the container device. It 324 also abstracts operations for the private features and exposes common ops to 325 feature device drivers. 326 327 The FPGA DFL Device could be different hardware, e.g. PCIe device, platform 328 device and etc. Its driver module is always loaded first once the device is 329 created by the system. This driver plays an infrastructural role in the 330 driver architecture. It locates the DFLs in the device memory, handles them 331 and related resources to common interfaces from DFL framework for enumeration. 332 (Please refer to drivers/fpga/dfl.c for detailed enumeration APIs). 333 334 The FPGA Management Engine (FME) driver is a platform driver which is loaded 335 automatically after FME platform device creation from the DFL device module. It 336 provides the key features for FPGA management, including: 337 338 a) Expose static FPGA region information, e.g. version and metadata. 339 Users can read related information via sysfs interfaces exposed 340 by FME driver. 341 342 b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA 343 bridges and FPGA regions during PR sub feature initialization. Once 344 it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the 345 common interface function from FPGA Region to complete the partial 346 reconfiguration of the PR bitstream to the given port. 347 348 Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is 349 probed once the AFU platform device is created. The main function of this module 350 is to provide an interface for userspace applications to access the individual 351 accelerators, including basic reset control on port, AFU MMIO region export, dma 352 buffer mapping service functions. 353 354 After feature platform devices creation, matched platform drivers will be loaded 355 automatically to handle different functionalities. Please refer to next sections 356 for detailed information on functional units which have been already implemented 357 under this DFL framework. 358 359 360 Partial Reconfiguration 361 ======================= 362 As mentioned above, accelerators can be reconfigured through partial 363 reconfiguration of a PR bitstream file. The PR bitstream file must have been 364 generated for the exact static FPGA region and targeted reconfigurable region 365 (port) of the FPGA, otherwise, the reconfiguration operation will fail and 366 possibly cause system instability. This compatibility can be checked by 367 comparing the compatibility ID noted in the header of PR bitstream file against 368 the compat_id exposed by the target FPGA region. This check is usually done by 369 userspace before calling the reconfiguration IOCTL. 370 371 372 FPGA virtualization - PCIe SRIOV 373 ================================ 374 This section describes the virtualization support on DFL based FPGA device to 375 enable accessing an accelerator from applications running in a virtual machine 376 (VM). This section only describes the PCIe based FPGA device with SRIOV support. 377 378 Features supported by the particular FPGA device are exposed through Device 379 Feature Lists, as illustrated below: 380 381 :: 382 383 +-------------------------------+ +-------------+ 384 | PF | | VF | 385 +-------------------------------+ +-------------+ 386 ^ ^ ^ ^ 387 | | | | 388 +-----|------------|---------|--------------|-------+ 389 | | | | | | 390 | +-----+ +-------+ +-------+ +-------+ | 391 | | FME | | Port0 | | Port1 | | Port2 | | 392 | +-----+ +-------+ +-------+ +-------+ | 393 | ^ ^ ^ | 394 | | | | | 395 | +-------+ +------+ +-------+ | 396 | | AFU | | AFU | | AFU | | 397 | +-------+ +------+ +-------+ | 398 | | 399 | DFL based FPGA PCIe Device | 400 +---------------------------------------------------+ 401 402 FME is always accessed through the physical function (PF). 403 404 Ports (and related AFUs) are accessed via PF by default, but could be exposed 405 through virtual function (VF) devices via PCIe SRIOV. Each VF only contains 406 1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators) 407 created via PCIe SRIOV interface, to virtual machines. 408 409 The driver organization in virtualization case is illustrated below: 410 :: 411 412 +-------++------++------+ | 413 | FME || FME || FME | | 414 | FPGA || FPGA || FPGA | | 415 |Manager||Bridge||Region| | 416 +-------++------++------+ | 417 +-----------------------+ +--------+ | +--------+ 418 | FME | | AFU | | | AFU | 419 | Module | | Module | | | Module | 420 +-----------------------+ +--------+ | +--------+ 421 +-----------------------+ | +-----------------------+ 422 | FPGA Container Device | | | FPGA Container Device | 423 | (FPGA Base Region) | | | (FPGA Base Region) | 424 +-----------------------+ | +-----------------------+ 425 +------------------+ | +------------------+ 426 | FPGA PCIE Module | | Virtual | FPGA PCIE Module | 427 +------------------+ Host | Machine +------------------+ 428 -------------------------------------- | ------------------------------ 429 +---------------+ | +---------------+ 430 | PCI PF Device | | | PCI VF Device | 431 +---------------+ | +---------------+ 432 433 FPGA PCIe device driver is always loaded first once an FPGA PCIe PF or VF device 434 is detected. It: 435 436 * Finishes enumeration on both FPGA PCIe PF and VF device using common 437 interfaces from DFL framework. 438 * Supports SRIOV. 439 440 The FME device driver plays a management role in this driver architecture, it 441 provides ioctls to release Port from PF and assign Port to PF. After release 442 a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV 443 sysfs interface. 444 445 To enable accessing an accelerator from applications running in a VM, the 446 respective AFU's port needs to be assigned to a VF using the following steps: 447 448 #. The PF owns all AFU ports by default. Any port that needs to be 449 reassigned to a VF must first be released through the 450 DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device. 451 452 #. Once N ports are released from PF, then user can use command below 453 to enable SRIOV and VFs. Each VF owns only one Port with AFU. 454 455 :: 456 457 echo N > $PCI_DEVICE_PATH/sriov_numvfs 458 459 #. Pass through the VFs to VMs 460 461 #. The AFU under VF is accessible from applications in VM (using the 462 same driver inside the VF). 463 464 Note that an FME can't be assigned to a VF, thus PR and other management 465 functions are only available via the PF. 466 467 Device enumeration 468 ================== 469 This section introduces how applications enumerate the fpga device from 470 the sysfs hierarchy under /sys/class/fpga_region. 471 472 In the example below, two DFL based FPGA devices are installed in the host. Each 473 fpga device has one FME and two ports (AFUs). 474 475 FPGA regions are created under /sys/class/fpga_region/:: 476 477 /sys/class/fpga_region/region0 478 /sys/class/fpga_region/region1 479 /sys/class/fpga_region/region2 480 ... 481 482 Application needs to search each regionX folder, if feature device is found, 483 (e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base 484 fpga region which represents the FPGA device. 485 486 Each base region has one FME and two ports (AFUs) as child devices:: 487 488 /sys/class/fpga_region/region0/dfl-fme.0 489 /sys/class/fpga_region/region0/dfl-port.0 490 /sys/class/fpga_region/region0/dfl-port.1 491 ... 492 493 /sys/class/fpga_region/region3/dfl-fme.1 494 /sys/class/fpga_region/region3/dfl-port.2 495 /sys/class/fpga_region/region3/dfl-port.3 496 ... 497 498 In general, the FME/AFU sysfs interfaces are named as follows:: 499 500 /sys/class/fpga_region/<regionX>/<dfl-fme.n>/ 501 /sys/class/fpga_region/<regionX>/<dfl-port.m>/ 502 503 with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all 504 ports. 505 506 The device nodes used for ioctl() or mmap() can be referenced through:: 507 508 /sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev 509 /sys/class/fpga_region/<regionX>/<dfl-port.n>/dev 510 511 512 Performance Counters 513 ==================== 514 Performance reporting is one private feature implemented in FME. It could 515 supports several independent, system-wide, device counter sets in hardware to 516 monitor and count for performance events, including "basic", "cache", "fabric", 517 "vtd" and "vtd_sip" counters. Users could use standard perf tool to monitor 518 FPGA cache hit/miss rate, transaction number, interface clock counter of AFU 519 and other FPGA performance events. 520 521 Different FPGA devices may have different counter sets, depending on hardware 522 implementation. E.g., some discrete FPGA cards don't have any cache. User could 523 use "perf list" to check which perf events are supported by target hardware. 524 525 In order to allow user to use standard perf API to access these performance 526 counters, driver creates a perf PMU, and related sysfs interfaces in 527 /sys/bus/event_source/devices/dfl_fme* to describe available perf events and 528 configuration options. 529 530 The "format" directory describes the format of the config field of struct 531 perf_event_attr. There are 3 bitfields for config: "evtype" defines which type 532 the perf event belongs to; "event" is the identity of the event within its 533 category; "portid" is introduced to decide counters set to monitor on FPGA 534 overall data or a specific port. 535 536 The "events" directory describes the configuration templates for all available 537 events which can be used with perf tool directly. For example, fab_mmio_read 538 has the configuration "event=0x06,evtype=0x02,portid=0xff", which shows this 539 event belongs to fabric type (0x02), the local event id is 0x06 and it is for 540 overall monitoring (portid=0xff). 541 542 Example usage of perf:: 543 544 $# perf list |grep dfl_fme 545 546 dfl_fme0/fab_mmio_read/ [Kernel PMU event] 547 <...> 548 dfl_fme0/fab_port_mmio_read,portid=?/ [Kernel PMU event] 549 <...> 550 551 $# perf stat -a -e dfl_fme0/fab_mmio_read/ <command> 552 or 553 $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0xff/ <command> 554 or 555 $# perf stat -a -e dfl_fme0/config=0xff2006/ <command> 556 557 Another example, fab_port_mmio_read monitors mmio read of a specific port. So 558 its configuration template is "event=0x06,evtype=0x01,portid=?". The portid 559 should be explicitly set. 560 561 Its usage of perf:: 562 563 $# perf stat -a -e dfl_fme0/fab_port_mmio_read,portid=0x0/ <command> 564 or 565 $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0x0/ <command> 566 or 567 $# perf stat -a -e dfl_fme0/config=0x2006/ <command> 568 569 Please note for fabric counters, overall perf events (fab_*) and port perf 570 events (fab_port_*) actually share one set of counters in hardware, so it can't 571 monitor both at the same time. If this set of counters is configured to monitor 572 overall data, then per port perf data is not supported. See below example:: 573 574 $# perf stat -e dfl_fme0/fab_mmio_read/,dfl_fme0/fab_port_mmio_write,\ 575 portid=0/ sleep 1 576 577 Performance counter stats for 'system wide': 578 579 3 dfl_fme0/fab_mmio_read/ 580 <not supported> dfl_fme0/fab_port_mmio_write,portid=0x0/ 581 582 1.001750904 seconds time elapsed 583 584 The driver also provides a "cpumask" sysfs attribute, which contains only one 585 CPU id used to access these perf events. Counting on multiple CPU is not allowed 586 since they are system-wide counters on FPGA device. 587 588 The current driver does not support sampling. So "perf record" is unsupported. 589 590 591 Interrupt support 592 ================= 593 Some FME and AFU private features are able to generate interrupts. As mentioned 594 above, users could call ioctl (DFL_FPGA_*_GET_IRQ_NUM) to know whether or how 595 many interrupts are supported for this private feature. Drivers also implement 596 an eventfd based interrupt handling mechanism for users to get notified when 597 interrupt happens. Users could set eventfds to driver via 598 ioctl (DFL_FPGA_*_SET_IRQ), and then poll/select on these eventfds waiting for 599 notification. 600 In Current DFL, 3 sub features (Port error, FME global error and AFU interrupt) 601 support interrupts. 602 603 604 Add new FIUs support 605 ==================== 606 It's possible that developers made some new function blocks (FIUs) under this 607 DFL framework, then new platform device driver needs to be developed for the 608 new feature dev (FIU) following the same way as existing feature dev drivers 609 (e.g. FME and Port/AFU platform device driver). Besides that, it requires 610 modification on DFL framework enumeration code too, for new FIU type detection 611 and related platform devices creation. 612 613 614 Add new private features support 615 ================================ 616 In some cases, we may need to add some new private features to existing FIUs 617 (e.g. FME or Port). Developers don't need to touch enumeration code in DFL 618 framework, as each private feature will be parsed automatically and related 619 mmio resources can be found under FIU platform device created by DFL framework. 620 Developer only needs to provide a sub feature driver with matched feature id. 621 FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c) 622 could be a reference. 623 624 Please refer to below link to existing feature id table and guide for new feature 625 ids application. 626 https://github.com/OPAE/dfl-feature-id 627 628 629 Location of DFLs on a PCI Device 630 ================================ 631 The original method for finding a DFL on a PCI device assumed the start of the 632 first DFL to offset 0 of bar 0. If the first node of the DFL is an FME, 633 then further DFLs in the port(s) are specified in FME header registers. 634 Alternatively, a PCIe vendor specific capability structure can be used to 635 specify the location of all the DFLs on the device, providing flexibility 636 for the type of starting node in the DFL. Intel has reserved the 637 VSEC ID of 0x43 for this purpose. The vendor specific 638 data begins with a 4 byte vendor specific register for the number of DFLs followed 4 byte 639 Offset/BIR vendor specific registers for each DFL. Bits 2:0 of Offset/BIR register 640 indicates the BAR, and bits 31:3 form the 8 byte aligned offset where bits 2:0 are 641 zero. 642 :: 643 644 +----------------------------+ 645 |31 Number of DFLS 0| 646 +----------------------------+ 647 |31 Offset 3|2 BIR 0| 648 +----------------------------+ 649 . . . 650 +----------------------------+ 651 |31 Offset 3|2 BIR 0| 652 +----------------------------+ 653 654 Being able to specify more than one DFL per BAR has been considered, but it 655 was determined the use case did not provide value. Specifying a single DFL 656 per BAR simplifies the implementation and allows for extra error checking. 657 658 659 Userspace driver support for DFL devices 660 ======================================== 661 The purpose of an FPGA is to be reprogrammed with newly developed hardware 662 components. New hardware can instantiate a new private feature in the DFL, and 663 then present a DFL device in the system. In some cases users may need a 664 userspace driver for the DFL device: 665 666 * Users may need to run some diagnostic test for their hardware. 667 * Users may prototype the kernel driver in user space. 668 * Some hardware is designed for specific purposes and does not fit into one of 669 the standard kernel subsystems. 670 671 This requires direct access to MMIO space and interrupt handling from 672 userspace. The uio_dfl module exposes the UIO device interfaces for this 673 purpose. 674 675 Currently the uio_dfl driver only supports the Ether Group sub feature, which 676 has no irq in hardware. So the interrupt handling is not added in this driver. 677 678 UIO_DFL should be selected to enable the uio_dfl module driver. To support a 679 new DFL feature via UIO direct access, its feature id should be added to the 680 driver's id_table. 681 682 683 Open discussion 684 =============== 685 FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration 686 to user now. In the future, if unified user interfaces for reconfiguration are 687 added, FME driver should switch to them from ioctl interface.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.