1 ================================
2 Coherent Accelerator (CXL) Flash
3 ================================
4
5 Introduction
6 ============
7
8 The IBM Power architecture provides suppor
9 Accelerator Power Interface), which is ava
10 on Power 8 systems. CAPI can be thought of
11 protocol through PCIe that allow PCIe adap
12 purpose co-processors which can read or wr
13 memory and generate page faults. As a resu
14 an adapter running in CAPI mode does not r
15 be mapped to the device's memory (IOMMU by
16 memory to be pinned.
17
18 On Linux, Coherent Accelerator (CXL) kerne
19 devices as a PCI device by implementing a
20 This abstraction simplifies the infrastruc
21 model, allowing for drivers to look simila
22 device drivers.
23
24 CXL provides a mechanism by which user spa
25 directly talk to a device (network or stor
26 kernel/device driver stack. The CXL Flash
27 user space application direct access to Fl
28
29 The CXL Flash Adapter Driver is a kernel m
30 SCSI stack as a low level device driver (b
31 protocol drivers) for the IBM CXL Flash Ad
32 responsible for the initialization of the
33 special path for user space access, and pe
34 communicates directly the Flash Accelerato
35 as described in Documentation/arch/powerpc
36
37 The cxlflash driver supports two, mutually
38 operation at the device (LUN) level:
39
40 - Any flash device (LUN) can be config
41 regular disk device (i.e.: /dev/sdc)
42
43 - Any flash device (LUN) can be config
44 user space with a special block libr
45 specifies the means of accessing the
46 either raw access to the entire LUN
47 or physical LUN access) or access to
48 partition of the LUN (referred to as
49 segmentation of a disk device into v
50 by special translation services prov
51
52 Overview
53 ========
54
55 The Coherent Accelerator Interface Archite
56 concept of a master context. A master typi
57 granted to it by the kernel or hypervisor
58 wide management and control. The master ma
59 directly in each user I/O, but at the mini
60 initial setup before the user application
61 directly to the AFU.
62
63 The CXL Flash Adapter Driver establishes a
64 AFU. It uses memory mapped I/O (MMIO) for
65 Adapter Problem Space Memory Map looks lik
66
67 +------------------------
68 | 512 * 64 KB User MMI
69 | (per context)
70 | User Accessible
71 +------------------------
72 | 512 * 128 B per cont
73 | Provisioning and Con
74 | Trusted Process acces
75 +------------------------
76 | 64 KB Global
77 | Trusted Process acces
78 +------------------------
79
80 This driver configures itself into the SCS
81 adapter driver. The driver is the only ent
82 Trusted Process to program the Provisionin
83 areas in the MMIO Space shown above. The
84 discovers all LUNs attached to the CXL Fla
85 scsi block devices (/dev/sdb, /dev/sdc etc
86 seen from each path.
87
88 Once these scsi block devices are instanti
89 written to a specification provided by the
90 access to the Flash from user space (witho
91
92 This master context driver also provides a
93 block library to enable this user space ac
94 two modes for accessing the block device.
95
96 The first mode is called a virtual mode. I
97 block device (/dev/sdb) may be carved up i
98 virtual LUNs. The virtual LUNs may be resi
99 the sizes of all the virtual LUNs, along w
100 with it does not exceed the physical capac
101
102 The second mode is called the physical mod
103 block device (/dev/sdb) may be opened dire
104 and the entire space for the LUN is availa
105
106 Only the physical mode provides persistenc
107 data written to the block device will surv
108 restart and also reboot. The virtual LUNs
109 not survive after the application terminat
110
111
112 Block library API
113 =================
114
115 Applications intending to get access to th
116 space should use the block library, as it
117 interfacing directly with the cxlflash dri
118 performing administrative actions (i.e.: s
119 The block library can be thought of as a '
120 implemented as IOCTLs, that are provided b
121 specifically for devices (LUNs) operating
122 mode. While it is not a requirement that a
123 the interface between the block library an
124 a high-level overview of each supported se
125 below.
126
127 The block library can be found on GitHub:
128 http://github.com/open-power/capiflash
129
130
131 CXL Flash Driver LUN IOCTLs
132 ===========================
133
134 Users, such as the block library, that wis
135 device (LUN) via user space access need to
136 by the cxlflash driver. As these services
137 a file descriptor handle must first be obt
138 the communication channel between a user a
139 descriptor is obtained by opening the devi
140 with the scsi disk device (/dev/sdb) that
141 discovery. As per the location of the cxlf
142 SCSI protocol stack, this open is actually
143 driver. Upon successful open, the user rec
144 (herein referred to as fd1) that should be
145 subsequent ioctls listed below.
146
147 The structure definitions for these IOCTLs
148 uapi/scsi/cxlflash_ioctl.h
149
150 DK_CXLFLASH_ATTACH
151 ------------------
152
153 This ioctl obtains, initializes, and start
154 kernel services. These services specify a
155 to uniquely identify the context and its a
156 services additionally provide a second fil
157 referred to as fd2) that is used by the bl
158 memory mapped I/O (via mmap()) to the CXL
159 completion events. This file descriptor is
160 this driver and not the CXL kernel service
161 notification and access in the event of a
162 such as a killed process. This design poin
163 detail in the description for the DK_CXLFL
164
165 There are a few important aspects regardin
166 and fd2) that are provided back to the use
167
168 - These tokens are only valid for the
169 were created. The child of a forked
170 to use the context id or file descri
171 (see DK_CXLFLASH_VLUN_CLONE for furt
172
173 - These tokens are only valid for the
174 the process under which they were cr
175 destroyed, the tokens are to be cons
176 usage will result in errors.
177
178 - A valid adapter file descriptor (fd2
179 the initial attach for a context. Su
180 existing context (DK_CXLFLASH_ATTACH
181 do not provide the adapter file desc
182 made known to the application.
183
184 - When a context is no longer needed,
185 the context via the DK_CXLFLASH_DETA
186 returns with a valid adapter file de
187 DK_CXLFLASH_APP_CLOSE_ADAP_FD is pre
188 close the adapter file descriptor fo
189
190 - When this ioctl returns with a valid
191 DK_CXLFLASH_APP_CLOSE_ADAP_FD is pre
192 close fd2 in the following circumsta
193
194 + Following a successful detac
195 + Following a successful recov
196 + In the child process of a fo
197 on the fd2 associated with t
198
199 - At any time, a close on fd2 will inv
200 should exercise caution to only clos
201 in the previous bullet) to avoid pre
202
203 DK_CXLFLASH_USER_DIRECT
204 -----------------------
205 This ioctl is responsible for transitionin
206 (physical) mode access and configuring the
207 user space on a per-context basis. Additio
208 last logical block address (LBA) are retur
209
210 As mentioned previously, when operating in
211 LUNs may be accessed in whole or in part.
212 at a time and if one mode is active (outst
213 requests to use the LUN in a different mod
214
215 The AFU is configured for direct access fr
216 entry to the AFU's resource handle table.
217 treated as a resource handle that is retur
218 is then able to use the handle to referenc
219
220 DK_CXLFLASH_USER_VIRTUAL
221 ------------------------
222 This ioctl is responsible for transitionin
223 of access and configuring the AFU for virt
224 on a per-context basis. Additionally, the
225 block address (LBA) are returned to the us
226
227 As mentioned previously, when operating in
228 LUNs may be accessed in whole or in part.
229 at a time and if one mode is active (outst
230 requests to use the LUN in a different mod
231
232 The AFU is configured for virtual access f
233 an entry to the AFU's resource handle tabl
234 is treated as a resource handle that is re
235 user is then able to use the handle to ref
236
237 By default, the virtual LUN is created wit
238 would need to use the DK_CXLFLASH_VLUN_RES
239 the virtual LUN to a desired size. To avoi
240 resize for the initial creation of the vir
241 option of specifying a size as part of the
242 ioctl, such that when success is returned
243 resource handle that is provided is alread
244 storage. This is reflected by the last LBA
245
246 When a LUN is accessible from more than on
247 return with the DK_CXLFLASH_ALL_PORTS_ACTI
248 provides the user with a hint that I/O can
249 of an I/O error as the LUN can be reached
250
251 DK_CXLFLASH_VLUN_RESIZE
252 -----------------------
253 This ioctl is responsible for resizing a p
254 LUN and will fail if invoked upon a LUN th
255 mode. Upon success, an updated last LBA is
256 indicating the new size of the virtual LUN
257 resource handle.
258
259 The partitioning of virtual LUNs is jointl
260 driver and the AFU. An allocation table is
261 operating in the virtual mode and used to
262 table that the AFU references when provide
263
264 This ioctl can return -EAGAIN if an AFU sy
265 In addition to returning a failure to user
266 an asynchronous AFU reset. Should the user
267 it is expected to succeed. If this ioctl f
268 can either retry the operation or treat it
269
270 DK_CXLFLASH_RELEASE
271 -------------------
272 This ioctl is responsible for releasing a
273 reference to either a physical or virtual
274 thought of as the inverse of the DK_CXLFLA
275 DK_CXLFLASH_USER_VIRTUAL ioctls. Upon succ
276 is no longer valid and the entry in the re
277 made available to be used again.
278
279 As part of the release process for virtual
280 is first resized to 0 to clear out and fre
281 associated with the virtual LUN reference.
282
283 DK_CXLFLASH_DETACH
284 ------------------
285 This ioctl is responsible for unregisterin
286 cxlflash driver and release outstanding re
287 not explicitly released via the DK_CXLFLAS
288 success, all "tokens" which had been provi
289 DK_CXLFLASH_ATTACH onward are no longer va
290
291 When the DK_CXLFLASH_APP_CLOSE_ADAP_FD fla
292 attach, the application _must_ close the f
293 following the detach of the final user of
294
295 DK_CXLFLASH_VLUN_CLONE
296 ----------------------
297 This ioctl is responsible for cloning a pr
298 context to a more recently created context
299 support maintaining user space access to s
300 forks. Upon success, the child process (wh
301 will have access to the same LUNs via the
302 as the parent, but under a different conte
303
304 Context sharing across processes is not su
305 therefore each fork must be met with estab
306 for the child process. This ioctl simplifi
307 and playback required by a user in such a
308 forks, child process can clone the parents
309 a context (via DK_CXLFLASH_ATTACH) and the
310 perform the clone from the parent to the c
311
312 The clone itself is fairly simple. The res
313 translation tables are copied from the par
314 and then synced with the AFU.
315
316 When the DK_CXLFLASH_APP_CLOSE_ADAP_FD fla
317 attach, the application _must_ close the f
318 context (still resident/accessible in the
319 clone. This is to avoid a stale entry in t
320 child process.
321
322 This ioctl can return -EAGAIN if an AFU sy
323 In addition to returning a failure to user
324 an asynchronous AFU reset. Should the user
325 it is expected to succeed. If this ioctl f
326 can either retry the operation or treat it
327
328 DK_CXLFLASH_VERIFY
329 ------------------
330 This ioctl is used to detect various chang
331 the disk changing, the number of LUNs visi
332 where the changes affect the application (
333 cxlflash driver will report the changed st
334
335 The user calls in when they want to valida
336 changed in response to a check condition.
337 of band from the kernel, they will see the
338 the kernel's knowledge. When encountered,
339 behavior is to call in to this ioctl, indi
340 verify and passing along any appropriate i
341 verifying a LUN change (ie: size different
342 supported.
343
344 DK_CXLFLASH_RECOVER_AFU
345 -----------------------
346 This ioctl is used to drive recovery (if s
347 of a specified user context. Any state ass
348 is re-established upon successful recovery
349
350 User contexts are put into an error condit
351 be reset or is terminating. Users are noti
352 by seeing all 0xF's on an MMIO read. Upon
353 architected behavior for a user is to call
354 their context. A user may also call into t
355 check if the device is operating normally.
356 from this ioctl, the user is expected to g
357 context via release/detach ioctls. Until t
358 hold is not relinquished. The user may als
359 at which time the context/resources they h
360 the release fop.
361
362 When the DK_CXLFLASH_APP_CLOSE_ADAP_FD fla
363 attach, the application _must_ unmap and c
364 original context following this ioctl retu
365 the context was recovered (DK_CXLFLASH_REC
366
367 DK_CXLFLASH_MANAGE_LUN
368 ----------------------
369 This ioctl is used to switch a LUN from a
370 for file-system access (legacy), to a mode
371 exclusive user space access (superpipe). I
372 across multiple ports and adapters, this i
373 identify each LUN by its World Wide Node N
374
375
376 CXL Flash Driver Host IOCTLs
377 ============================
378
379 Each host adapter instance that is support
380 has a special character device associated
381 host management function. These character
382 class dedicated for cxlflash and can be ac
383
384 Applications can be written to perform var
385 host ioctl APIs below.
386
387 The structure definitions for these IOCTLs
388 uapi/scsi/cxlflash_ioctl.h
389
390 HT_CXLFLASH_LUN_PROVISION
391 -------------------------
392 This ioctl is used to create and delete pe
393 devices that lack an external LUN manageme
394 valid when used with AFUs that support the
395
396 When sufficient space is available, LUNs c
397 the target port to host the LUN and a desi
398 success, the LUN ID and WWID of the create
399 the SCSI bus can be scanned to detect the
400 that partial allocations are not supported
401 to a space issue, the target port can be q
402 geometry.
403
404 To remove a LUN, the device must first be
405 SCSI subsystem. The LUN deletion can then
406 target port and LUN ID. Upon success, the
407 the port will be updated to reflect new nu
408 available capacity.
409
410 To query the LUN geometry of a port, the t
411 upon success, the following information is
412
413 - Maximum number of provisioned LUNs a
414 - Current number of provisioned LUNs f
415 - Maximum total capacity of provisione
416 - Current total capacity of provisione
417
418 With this information, the number of avail
419 can be calculated.
420
421 HT_CXLFLASH_AFU_DEBUG
422 ---------------------
423 This ioctl is used to debug AFUs by suppor
424 interface. It is only valid when used with
425 debug capability.
426
427 With exception of buffer management, AFU d
428 cxlflash and treated as pass-through. For
429 data transfer, the user supplies an adequa
430 specify the data transfer direction with r
431 maximum transfer size of 256K imposed. Not
432 are not supported - when errors are experi
433 transfer, the data buffer is not copied ba
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.