1 ======================================================== 2 OpenCAPI (Open Coherent Accelerator Processor Interface) 3 ======================================================== 4 5 OpenCAPI is an interface between processors and accelerators. It aims 6 at being low-latency and high-bandwidth. The specification is 7 developed by the `OpenCAPI Consortium <http://opencapi.org/>`_. 8 9 It allows an accelerator (which could be an FPGA, ASICs, ...) to access 10 the host memory coherently, using virtual addresses. An OpenCAPI 11 device can also host its own memory, that can be accessed from the 12 host. 13 14 OpenCAPI is known in linux as 'ocxl', as the open, processor-agnostic 15 evolution of 'cxl' (the driver for the IBM CAPI interface for 16 powerpc), which was named that way to avoid confusion with the ISDN 17 CAPI subsystem. 18 19 20 High-level view 21 =============== 22 23 OpenCAPI defines a Data Link Layer (DL) and Transaction Layer (TL), to 24 be implemented on top of a physical link. Any processor or device 25 implementing the DL and TL can start sharing memory. 26 27 :: 28 29 +-----------+ +-------------+ 30 | | | | 31 | | | Accelerated | 32 | Processor | | Function | 33 | | +--------+ | Unit | +--------+ 34 | |--| Memory | | (AFU) |--| Memory | 35 | | +--------+ | | +--------+ 36 +-----------+ +-------------+ 37 | | 38 +-----------+ +-------------+ 39 | TL | | TLX | 40 +-----------+ +-------------+ 41 | | 42 +-----------+ +-------------+ 43 | DL | | DLX | 44 +-----------+ +-------------+ 45 | | 46 | PHY | 47 +---------------------------------------+ 48 49 50 51 Device discovery 52 ================ 53 54 OpenCAPI relies on a PCI-like configuration space, implemented on the 55 device. So the host can discover AFUs by querying the config space. 56 57 OpenCAPI devices in Linux are treated like PCI devices (with a few 58 caveats). The firmware is expected to abstract the hardware as if it 59 was a PCI link. A lot of the existing PCI infrastructure is reused: 60 devices are scanned and BARs are assigned during the standard PCI 61 enumeration. Commands like 'lspci' can therefore be used to see what 62 devices are available. 63 64 The configuration space defines the AFU(s) that can be found on the 65 physical adapter, such as its name, how many memory contexts it can 66 work with, the size of its MMIO areas, ... 67 68 69 70 MMIO 71 ==== 72 73 OpenCAPI defines two MMIO areas for each AFU: 74 75 * the global MMIO area, with registers pertinent to the whole AFU. 76 * a per-process MMIO area, which has a fixed size for each context. 77 78 79 80 AFU interrupts 81 ============== 82 83 OpenCAPI includes the possibility for an AFU to send an interrupt to a 84 host process. It is done through a 'intrp_req' defined in the 85 Transaction Layer, specifying a 64-bit object handle which defines the 86 interrupt. 87 88 The driver allows a process to allocate an interrupt and obtain its 89 64-bit object handle, that can be passed to the AFU. 90 91 92 93 char devices 94 ============ 95 96 The driver creates one char device per AFU found on the physical 97 device. A physical device may have multiple functions and each 98 function can have multiple AFUs. At the time of this writing though, 99 it has only been tested with devices exporting only one AFU. 100 101 Char devices can be found in /dev/ocxl/ and are named as: 102 /dev/ocxl/<AFU name>.<location>.<index> 103 104 where <AFU name> is a max 20-character long name, as found in the 105 config space of the AFU. 106 <location> is added by the driver and can help distinguish devices 107 when a system has more than one instance of the same OpenCAPI device. 108 <index> is also to help distinguish AFUs in the unlikely case where a 109 device carries multiple copies of the same AFU. 110 111 112 113 Sysfs class 114 =========== 115 116 An ocxl class is added for the devices representing the AFUs. See 117 /sys/class/ocxl. The layout is described in 118 Documentation/ABI/testing/sysfs-class-ocxl 119 120 121 122 User API 123 ======== 124 125 open 126 ---- 127 128 Based on the AFU definition found in the config space, an AFU may 129 support working with more than one memory context, in which case the 130 associated char device may be opened multiple times by different 131 processes. 132 133 134 ioctl 135 ----- 136 137 OCXL_IOCTL_ATTACH: 138 139 Attach the memory context of the calling process to the AFU so that 140 the AFU can access its memory. 141 142 OCXL_IOCTL_IRQ_ALLOC: 143 144 Allocate an AFU interrupt and return an identifier. 145 146 OCXL_IOCTL_IRQ_FREE: 147 148 Free a previously allocated AFU interrupt. 149 150 OCXL_IOCTL_IRQ_SET_FD: 151 152 Associate an event fd to an AFU interrupt so that the user process 153 can be notified when the AFU sends an interrupt. 154 155 OCXL_IOCTL_GET_METADATA: 156 157 Obtains configuration information from the card, such at the size of 158 MMIO areas, the AFU version, and the PASID for the current context. 159 160 OCXL_IOCTL_ENABLE_P9_WAIT: 161 162 Allows the AFU to wake a userspace thread executing 'wait'. Returns 163 information to userspace to allow it to configure the AFU. Note that 164 this is only available on POWER9. 165 166 OCXL_IOCTL_GET_FEATURES: 167 168 Reports on which CPU features that affect OpenCAPI are usable from 169 userspace. 170 171 172 mmap 173 ---- 174 175 A process can mmap the per-process MMIO area for interactions with the 176 AFU.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.