1 ============================================== 1 ========================================================
2 OpenCAPI (Open Coherent Accelerator Processor 2 OpenCAPI (Open Coherent Accelerator Processor Interface)
3 ============================================== 3 ========================================================
4 4
5 OpenCAPI is an interface between processors an 5 OpenCAPI is an interface between processors and accelerators. It aims
6 at being low-latency and high-bandwidth. The s 6 at being low-latency and high-bandwidth. The specification is
7 developed by the `OpenCAPI Consortium <http:// 7 developed by the `OpenCAPI Consortium <http://opencapi.org/>`_.
8 8
9 It allows an accelerator (which could be an FP 9 It allows an accelerator (which could be an FPGA, ASICs, ...) to access
10 the host memory coherently, using virtual addr 10 the host memory coherently, using virtual addresses. An OpenCAPI
11 device can also host its own memory, that can 11 device can also host its own memory, that can be accessed from the
12 host. 12 host.
13 13
14 OpenCAPI is known in linux as 'ocxl', as the o 14 OpenCAPI is known in linux as 'ocxl', as the open, processor-agnostic
15 evolution of 'cxl' (the driver for the IBM CAP 15 evolution of 'cxl' (the driver for the IBM CAPI interface for
16 powerpc), which was named that way to avoid co 16 powerpc), which was named that way to avoid confusion with the ISDN
17 CAPI subsystem. 17 CAPI subsystem.
18 18
19 19
20 High-level view 20 High-level view
21 =============== 21 ===============
22 22
23 OpenCAPI defines a Data Link Layer (DL) and Tr 23 OpenCAPI defines a Data Link Layer (DL) and Transaction Layer (TL), to
24 be implemented on top of a physical link. Any 24 be implemented on top of a physical link. Any processor or device
25 implementing the DL and TL can start sharing m 25 implementing the DL and TL can start sharing memory.
26 26
27 :: 27 ::
28 28
29 +-----------+ +----- 29 +-----------+ +-------------+
30 | | | 30 | | | |
31 | | | Acce 31 | | | Accelerated |
32 | Processor | | Fun 32 | Processor | | Function |
33 | | +--------+ | U 33 | | +--------+ | Unit | +--------+
34 | |--| Memory | | ( 34 | |--| Memory | | (AFU) |--| Memory |
35 | | +--------+ | 35 | | +--------+ | | +--------+
36 +-----------+ +----- 36 +-----------+ +-------------+
37 | 37 | |
38 +-----------+ +----- 38 +-----------+ +-------------+
39 | TL | | T 39 | TL | | TLX |
40 +-----------+ +----- 40 +-----------+ +-------------+
41 | 41 | |
42 +-----------+ +----- 42 +-----------+ +-------------+
43 | DL | | D 43 | DL | | DLX |
44 +-----------+ +----- 44 +-----------+ +-------------+
45 | 45 | |
46 | PHY 46 | PHY |
47 +-------------------------------------- 47 +---------------------------------------+
48 48
49 49
50 50
51 Device discovery 51 Device discovery
52 ================ 52 ================
53 53
54 OpenCAPI relies on a PCI-like configuration sp 54 OpenCAPI relies on a PCI-like configuration space, implemented on the
55 device. So the host can discover AFUs by query 55 device. So the host can discover AFUs by querying the config space.
56 56
57 OpenCAPI devices in Linux are treated like PCI 57 OpenCAPI devices in Linux are treated like PCI devices (with a few
58 caveats). The firmware is expected to abstract 58 caveats). The firmware is expected to abstract the hardware as if it
59 was a PCI link. A lot of the existing PCI infr 59 was a PCI link. A lot of the existing PCI infrastructure is reused:
60 devices are scanned and BARs are assigned duri 60 devices are scanned and BARs are assigned during the standard PCI
61 enumeration. Commands like 'lspci' can therefo 61 enumeration. Commands like 'lspci' can therefore be used to see what
62 devices are available. 62 devices are available.
63 63
64 The configuration space defines the AFU(s) tha 64 The configuration space defines the AFU(s) that can be found on the
65 physical adapter, such as its name, how many m 65 physical adapter, such as its name, how many memory contexts it can
66 work with, the size of its MMIO areas, ... 66 work with, the size of its MMIO areas, ...
67 67
68 68
69 69
70 MMIO 70 MMIO
71 ==== 71 ====
72 72
73 OpenCAPI defines two MMIO areas for each AFU: 73 OpenCAPI defines two MMIO areas for each AFU:
74 74
75 * the global MMIO area, with registers pertine 75 * the global MMIO area, with registers pertinent to the whole AFU.
76 * a per-process MMIO area, which has a fixed s 76 * a per-process MMIO area, which has a fixed size for each context.
77 77
78 78
79 79
80 AFU interrupts 80 AFU interrupts
81 ============== 81 ==============
82 82
83 OpenCAPI includes the possibility for an AFU t 83 OpenCAPI includes the possibility for an AFU to send an interrupt to a
84 host process. It is done through a 'intrp_req' 84 host process. It is done through a 'intrp_req' defined in the
85 Transaction Layer, specifying a 64-bit object 85 Transaction Layer, specifying a 64-bit object handle which defines the
86 interrupt. 86 interrupt.
87 87
88 The driver allows a process to allocate an int 88 The driver allows a process to allocate an interrupt and obtain its
89 64-bit object handle, that can be passed to th 89 64-bit object handle, that can be passed to the AFU.
90 90
91 91
92 92
93 char devices 93 char devices
94 ============ 94 ============
95 95
96 The driver creates one char device per AFU fou 96 The driver creates one char device per AFU found on the physical
97 device. A physical device may have multiple fu 97 device. A physical device may have multiple functions and each
98 function can have multiple AFUs. At the time o 98 function can have multiple AFUs. At the time of this writing though,
99 it has only been tested with devices exporting 99 it has only been tested with devices exporting only one AFU.
100 100
101 Char devices can be found in /dev/ocxl/ and ar 101 Char devices can be found in /dev/ocxl/ and are named as:
102 /dev/ocxl/<AFU name>.<location>.<index> 102 /dev/ocxl/<AFU name>.<location>.<index>
103 103
104 where <AFU name> is a max 20-character long na 104 where <AFU name> is a max 20-character long name, as found in the
105 config space of the AFU. 105 config space of the AFU.
106 <location> is added by the driver and can help 106 <location> is added by the driver and can help distinguish devices
107 when a system has more than one instance of th 107 when a system has more than one instance of the same OpenCAPI device.
108 <index> is also to help distinguish AFUs in th 108 <index> is also to help distinguish AFUs in the unlikely case where a
109 device carries multiple copies of the same AFU 109 device carries multiple copies of the same AFU.
110 110
111 111
112 112
113 Sysfs class 113 Sysfs class
114 =========== 114 ===========
115 115
116 An ocxl class is added for the devices represe 116 An ocxl class is added for the devices representing the AFUs. See
117 /sys/class/ocxl. The layout is described in 117 /sys/class/ocxl. The layout is described in
118 Documentation/ABI/testing/sysfs-class-ocxl 118 Documentation/ABI/testing/sysfs-class-ocxl
119 119
120 120
121 121
122 User API 122 User API
123 ======== 123 ========
124 124
125 open 125 open
126 ---- 126 ----
127 127
128 Based on the AFU definition found in the confi 128 Based on the AFU definition found in the config space, an AFU may
129 support working with more than one memory cont 129 support working with more than one memory context, in which case the
130 associated char device may be opened multiple 130 associated char device may be opened multiple times by different
131 processes. 131 processes.
132 132
133 133
134 ioctl 134 ioctl
135 ----- 135 -----
136 136
137 OCXL_IOCTL_ATTACH: 137 OCXL_IOCTL_ATTACH:
138 138
139 Attach the memory context of the calling pro 139 Attach the memory context of the calling process to the AFU so that
140 the AFU can access its memory. 140 the AFU can access its memory.
141 141
142 OCXL_IOCTL_IRQ_ALLOC: 142 OCXL_IOCTL_IRQ_ALLOC:
143 143
144 Allocate an AFU interrupt and return an iden 144 Allocate an AFU interrupt and return an identifier.
145 145
146 OCXL_IOCTL_IRQ_FREE: 146 OCXL_IOCTL_IRQ_FREE:
147 147
148 Free a previously allocated AFU interrupt. 148 Free a previously allocated AFU interrupt.
149 149
150 OCXL_IOCTL_IRQ_SET_FD: 150 OCXL_IOCTL_IRQ_SET_FD:
151 151
152 Associate an event fd to an AFU interrupt so 152 Associate an event fd to an AFU interrupt so that the user process
153 can be notified when the AFU sends an interr 153 can be notified when the AFU sends an interrupt.
154 154
155 OCXL_IOCTL_GET_METADATA: 155 OCXL_IOCTL_GET_METADATA:
156 156
157 Obtains configuration information from the c 157 Obtains configuration information from the card, such at the size of
158 MMIO areas, the AFU version, and the PASID f 158 MMIO areas, the AFU version, and the PASID for the current context.
159 159
160 OCXL_IOCTL_ENABLE_P9_WAIT: 160 OCXL_IOCTL_ENABLE_P9_WAIT:
161 161
162 Allows the AFU to wake a userspace thread ex 162 Allows the AFU to wake a userspace thread executing 'wait'. Returns
163 information to userspace to allow it to conf 163 information to userspace to allow it to configure the AFU. Note that
164 this is only available on POWER9. 164 this is only available on POWER9.
165 165
166 OCXL_IOCTL_GET_FEATURES: 166 OCXL_IOCTL_GET_FEATURES:
167 167
168 Reports on which CPU features that affect Op 168 Reports on which CPU features that affect OpenCAPI are usable from
169 userspace. 169 userspace.
170 170
171 171
172 mmap 172 mmap
173 ---- 173 ----
174 174
175 A process can mmap the per-process MMIO area f 175 A process can mmap the per-process MMIO area for interactions with the
176 AFU. 176 AFU.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.