1 ===============================
2 Adjunct Processor (AP) facility
3 ===============================
4
5
6 Introduction
7 ============
8 The Adjunct Processor (AP) facility is an IBM
9 of three AP instructions and from 1 up to 256
10 The AP devices provide cryptographic functions
11 linux system running in an IBM Z system LPAR.
12
13 The AP adapter cards are exposed via the AP bu
14 is to make AP cards available to KVM guests us
15 framework. This implementation relies consider
16 facilities which do most of the hard work of p
17 devices.
18
19 AP Architectural Overview
20 =========================
21 To facilitate the comprehension of the design,
22 definitions:
23
24 * AP adapter
25
26 An AP adapter is an IBM Z adapter card that
27 functions. There can be from 0 to 256 adapte
28 assigned to the LPAR in which a linux host i
29 the linux host. Each adapter is identified b
30 the maximum adapter number is determined by
31 When installed, an AP adapter is accessed by
32 CPU.
33
34 The AP adapter cards are assigned to a given
35 Profile which can be edited via the HMC. Whe
36 in the LPAR, the AP bus detects the AP adapt
37 creates a sysfs device for each assigned ada
38 4 and 10 (0x0a) are assigned to the LPAR, th
39 sysfs device entries::
40
41 /sys/devices/ap/card04
42 /sys/devices/ap/card0a
43
44 Symbolic links to these devices will also be
45 sub-directory::
46
47 /sys/bus/ap/devices/[card04]
48 /sys/bus/ap/devices/[card04]
49
50 * AP domain
51
52 An adapter is partitioned into domains. An a
53 depending upon the adapter type and hardware
54 identified by a number from 0 to 255; howeve
55 determined by machine model and/or adapter t
56 as a set of hardware registers and memory us
57 domain can be configured with a secure priva
58 encryption. A domain is classified in one of
59 may be accessed:
60
61 * Usage domains are domains that are targe
62 process an AP command.
63
64 * Control domains are domains that are cha
65 usage domain; for example, to set the se
66 domain.
67
68 The AP usage and control domains are assigne
69 Activation Profile which can be edited via t
70 is IPL'd in the LPAR, the AP bus module dete
71 domains assigned to the LPAR. The domain num
72 adapter number of each AP adapter are combin
73 (see AP Queue section below). The domain num
74 represented in a bitmask and stored in a sys
75 /sys/bus/ap/ap_control_domain_mask. The bits
76 significant bit, correspond to domains 0-255
77
78 * AP Queue
79
80 An AP queue is the means by which an AP comm
81 inside a specific adapter. An AP queue is id
82 comprised of an AP adapter ID (APID) and an
83 APQI corresponds to a given usage domain num
84 forms an AP Queue Number (APQN) uniquely ide
85 instructions include a field containing the
86 which the AP command is to be sent for proce
87
88 The AP bus will create a sysfs device for ea
89 the cross product of the AP adapter and usag
90 AP bus module is loaded. For example, if ada
91 domains 6 and 71 (0x47) are assigned to the
92 following sysfs entries::
93
94 /sys/devices/ap/card04/04.0006
95 /sys/devices/ap/card04/04.0047
96 /sys/devices/ap/card0a/0a.0006
97 /sys/devices/ap/card0a/0a.0047
98
99 The following symbolic links to these device
100 devices subdirectory::
101
102 /sys/bus/ap/devices/[04.0006]
103 /sys/bus/ap/devices/[04.0047]
104 /sys/bus/ap/devices/[0a.0006]
105 /sys/bus/ap/devices/[0a.0047]
106
107 * AP Instructions:
108
109 There are three AP instructions:
110
111 * NQAP: to enqueue an AP command-request mes
112 * DQAP: to dequeue an AP command-reply messa
113 * PQAP: to administer the queues
114
115 AP instructions identify the domain that is
116 command; this must be one of the usage domai
117 domain that is not one of the usage domains,
118 must be one of the control domains.
119
120 AP and SIE
121 ==========
122 Let's now take a look at how AP instructions e
123 by the hardware.
124
125 A satellite control block called the Crypto Co
126 our main hardware virtualization control block
127 Block (APCB) that has three fields to identify
128 control domains assigned to the KVM guest:
129
130 * The AP Mask (APM) field is a bit mask that i
131 to the KVM guest. Each bit in the mask, from
132 an APID from 0-255. If a bit is set, the cor
133 use by the KVM guest.
134
135 * The AP Queue Mask (AQM) field is a bit mask
136 assigned to the KVM guest. Each bit in the m
137 corresponds to an AP queue index (APQI) from
138 corresponding queue is valid for use by the
139
140 * The AP Domain Mask field is a bit mask that
141 assigned to the KVM guest. The ADM bit mask
142 changed by an AP command-request message sen
143 guest. Each bit in the mask, from left to ri
144 0-255. If a bit is set, the corresponding do
145 command-request message sent to a usage doma
146
147 If you recall from the description of an AP Qu
148 an APQN to identify the AP queue to which an A
149 sent (NQAP and PQAP instructions), or from whi
150 be received (DQAP instruction). The validity o
151 calculated from the APM and AQM; it is the Car
152 adapter numbers (APM) with all assigned queue
153 adapters 1 and 2 and usage domains 5 and 6 are
154 (1,5), (1,6), (2,5) and (2,6) will be valid fo
155
156 The APQNs can provide secure key functionality
157 on the adapter card for each of its domains -
158 at most one guest or to the linux host::
159
160 Example 1: Valid configuration:
161 ------------------------------
162 Guest1: adapters 1,2 domains 5,6
163 Guest2: adapter 1,2 domain 7
164
165 This is valid because both guests have a un
166 Guest1 has APQNs (1,5), (1,6), (2,5), (2
167 Guest2 has APQNs (1,7), (2,7)
168
169 Example 2: Valid configuration:
170 ------------------------------
171 Guest1: adapters 1,2 domains 5,6
172 Guest2: adapters 3,4 domains 5,6
173
174 This is also valid because both guests have
175 Guest1 has APQNs (1,5), (1,6), (2,5), (2
176 Guest2 has APQNs (3,5), (3,6), (4,5), (4
177
178 Example 3: Invalid configuration:
179 --------------------------------
180 Guest1: adapters 1,2 domains 5,6
181 Guest2: adapter 1 domains 6,7
182
183 This is an invalid configuration because bo
184 APQN (1,6).
185
186 The Design
187 ==========
188 The design introduces three new objects:
189
190 1. AP matrix device
191 2. VFIO AP device driver (vfio_ap.ko)
192 3. VFIO AP mediated pass-through device
193
194 The VFIO AP device driver
195 -------------------------
196 The VFIO AP (vfio_ap) device driver serves the
197
198 1. Provides the interfaces to secure APQNs for
199
200 2. Sets up the VFIO mediated device interfaces
201 device and creates the sysfs interfaces for
202 domains, and control domains comprising the
203
204 3. Configures the APM, AQM and ADM in the APCB
205 by a KVM guest's SIE state description to g
206 of AP devices
207
208 Reserve APQNs for exclusive use of KVM guests
209 ---------------------------------------------
210 The following block diagram illustrates the me
211 reserved::
212
213 +-------------
214 7 remove |
215 +--------------------> cex4queue dr
216 | |
217 | +-------------
218 |
219 |
220 | +-------------
221 | 5 register driver |
222 | +----------------> Device cor
223 | | |
224 | | +--------^----
225 | | |
226 | | +----
227 | | +----------------------------
228 | | | 4 register AP driver
229 | | |
230 +--------+---+-v---+ +-
231 | | |
232 | ap_bus +--------------------- >
233 | | 8 probe |
234 +--------^---------+ +-
235 6 edit |
236 apmask | +----------------------------
237 aqmask | | 1 modprobe
238 +--------+-----+---+ +------------
239 | | |
240 | admin | | VFIO device
241 | + |
242 +------+-+---------+ +--------^---
243 | | |
244 | | 9 create vfio_ap-passthrough |
245 | +------------------------------+
246 +------------------------------------
247 12 assign adapter/domain
248
249 The process for reserving an AP queue for use
250
251 1. The administrator loads the vfio_ap device
252 2. The vfio-ap driver during its initializatio
253 device with the device core. This will serv
254 all vfio_ap mediated devices used to config
255 3. The /sys/devices/vfio_ap/matrix device is c
256 4. The vfio_ap device driver will register wit
257 of type 10 and higher (CEX4 and newer). The
258 driver's probe and remove callback interfac
259 are not supported to simplify the implement
260 complicating the design by supporting older
261 service in the relatively near future, and
262 systems around on which to test.
263 5. The AP bus registers the vfio_ap device dri
264 6. The administrator edits the AP adapter and
265 for use by the vfio_ap device driver.
266 7. The AP bus removes the AP queues reserved f
267 default zcrypt cex4queue driver.
268 8. The AP bus probes the vfio_ap device driver
269 it.
270 9. The administrator creates a passthrough typ
271 used by a guest
272 10. The administrator assigns the adapters, us
273 to be exclusively used by a guest.
274
275 Set up the VFIO mediated device interfaces
276 ------------------------------------------
277 The VFIO AP device driver utilizes the common
278 device core driver to:
279
280 * Register an AP mediated bus driver to add a
281 remove it from a VFIO group.
282 * Create and destroy a vfio_ap mediated device
283 * Add a vfio_ap mediated device to and remove
284 * Add a vfio_ap mediated device to and remove
285
286 The following high-level block diagram shows t
287 of the VFIO AP mediated device driver::
288
289 +-------------+
290 | |
291 | +---------+ | mdev_register_driver() +---
292 | | Mdev | +<-----------------------+
293 | | bus | | | vf
294 | | driver | +----------------------->+
295 | +---------+ | probe()/remove() +---
296 | |
297 | MDEV CORE |
298 | MODULE |
299 | mdev.ko |
300 | +---------+ | mdev_register_parent() +---
301 | |Physical | +<-----------------------+
302 | | device | | | v
303 | |interface| +----------------------->+
304 | +---------+ | callback +---
305 +-------------+
306
307 During initialization of the vfio_ap module, t
308 with an 'mdev_parent_ops' structure that provi
309 structures, mdev functions and callback interf
310 matrix device.
311
312 * sysfs attribute structures:
313
314 supported_type_groups
315 The VFIO mediated device framework support
316 mediated device types. These mediated devi
317 via the 'supported_type_groups' structure
318 with the mediated device framework. The re
319 sysfs structures for each mediated device
320 'mdev_supported_types' sub-directory of th
321 with the device type, the sysfs attributes
322 provided.
323
324 The VFIO AP device driver will register on
325 passthrough devices:
326
327 /sys/devices/vfio_ap/matrix/mdev_support
328
329 Only the read-only attributes required by
330 be provided::
331
332 ... name
333 ... device_api
334 ... available_instances
335 ... device_api
336
337 Where:
338
339 * name:
340 specifies the name of the mediated
341 * device_api:
342 the mediated device type's API
343 * available_instances:
344 the number of vfio_ap mediated pas
345 that can be created
346 * device_api:
347 specifies the VFIO API
348 mdev_attr_groups
349 This attribute group identifies the user-d
350 mediated device. When a device is register
351 framework, the sysfs attribute files ident
352 structure will be created in the vfio_ap m
353 sysfs attributes for a vfio_ap mediated de
354
355 assign_adapter / unassign_adapter:
356 Write-only attributes for assigning/unas
357 vfio_ap mediated device. To assign/unass
358 adapter is echoed into the respective at
359 assign_domain / unassign_domain:
360 Write-only attributes for assigning/unas
361 the vfio_ap mediated device. To assign/u
362 number of the usage domain is echoed int
363 file.
364 matrix:
365 A read-only file for displaying the APQN
366 product of the adapter and domain number
367 device.
368 guest_matrix:
369 A read-only file for displaying the APQN
370 product of the adapter and domain number
371 fields respectively of the KVM guest's C
372 the APQNs assigned to the vfio_ap mediat
373 reference a queue device bound to the vf
374 queue is not in the host's AP configurat
375 assign_control_domain / unassign_control_d
376 Write-only attributes for assigning/unas
377 to/from the vfio_ap mediated device. To
378 the ID of the domain to be assigned/unas
379 respective attribute file.
380 control_domains:
381 A read-only file for displaying the cont
382 vfio_ap mediated device.
383 ap_config:
384 A read/write file that, when written to,
385 vfio_ap mediated device's ap matrix mask
386 Three masks are given, one for adapters,
387 control domains. If the given state cann
388 made to the vfio-ap mediated device.
389
390 The format of the data written to ap_con
391 {amask},{dmask},{cmask}\n
392
393 \n is a newline character.
394
395 amask, dmask, and cmask are masks identi
396 and control domains should be assigned t
397
398 The format of a mask is as follows:
399 0xNN..NN
400
401 Where NN..NN is 64 hexadecimal character
402 The leftmost (highest order) bit represe
403
404 For an example set of masks that represe
405 configuration, simply cat ap_config.
406
407 Setting an adapter or domain number grea
408 the system will result in an error.
409
410 This attribute is intended to be used by
411 better served using the respective assig
412 adapters, domains, and control domains.
413
414 * functions:
415
416 create:
417 allocates the ap_matrix_mdev structure use
418
419 * Store the reference to the KVM structure
420 * Store the AP matrix configuration for th
421 domains assigned via the corresponding s
422 * Store the AP matrix configuration for th
423 domains available to a guest. A guest ma
424 referencing queue devices that do not ex
425 vfio_ap device driver.
426
427 remove:
428 deallocates the vfio_ap mediated device's
429 This will be allowed only if a running gue
430
431 * callback interfaces
432
433 open_device:
434 The vfio_ap driver uses this callback to r
435 VFIO_GROUP_NOTIFY_SET_KVM notifier callbac
436 devices. The open_device callback is invok
437 VFIO iommu group for the matrix mdev devic
438 KVM structure used to configure the KVM gu
439 The KVM structure, is used to configure th
440 defined via the vfio_ap mediated device's
441
442 close_device:
443 unregisters the VFIO_GROUP_NOTIFY_SET_KVM
444 matrix mdev device and deconfigures the gu
445
446 ioctl:
447 this callback handles the VFIO_DEVICE_GET_
448 defined by the vfio framework.
449
450 Configure the guest's AP resources
451 ----------------------------------
452 Configuring the AP resources for a KVM guest w
453 VFIO_GROUP_NOTIFY_SET_KVM notifier callback is
454 function is called when userspace connects to
455 configured via its APCB by:
456
457 * Setting the bits in the APM corresponding to
458 vfio_ap mediated device via its 'assign_adap
459 * Setting the bits in the AQM corresponding to
460 vfio_ap mediated device via its 'assign_doma
461 * Setting the bits in the ADM corresponding to
462 vfio_ap mediated device via its 'assign_cont
463
464 The linux device model precludes passing a dev
465 is not bound to the device driver facilitating
466 an APQN that does not reference a queue device
467 driver will not be assigned to a KVM guest's m
468 however, does not provide a means to filter in
469 matrix, so the adapters, domains and control d
470 mediated device via its sysfs 'assign_adapter'
471 'assign_control_domain' interfaces will be fil
472 configuration to a guest:
473
474 * The APIDs of the adapters, the APQIs of the
475 the control domains assigned to the matrix m
476 the host's AP configuration will be filtered
477
478 * Each APQN derived from the Cartesian product
479 to the vfio_ap mdev is examined and if any o
480 queue device bound to the vfio_ap device dri
481 plugged into the guest (i.e., the bit corres
482 set in the APM of the guest's APCB).
483
484 The CPU model features for AP
485 -----------------------------
486 The AP stack relies on the presence of the AP
487 facilities: The AP Facilities Test (APFT) faci
488 Configuration Information (QCI) facility; and
489 facility. These features/facilities are made a
490 following CPU model features:
491
492 1. ap: Indicates whether the AP instructions a
493 feature will be enabled by KVM only if the
494 on the host.
495
496 2. apft: Indicates the APFT facility is availa
497 can be made available to the guest only if
498 facility bit 15 is set).
499
500 3. apqci: Indicates the AP QCI facility is ava
501 can be made available to the guest only if
502 facility bit 12 is set).
503
504 4. apqi: Indicates AP Queue Interruption Contr
505 guest. This facility can be made available
506 available on the host (i.e., facility bit 6
507
508 Note: If the user chooses to specify a CPU mod
509 model to QEMU, the CPU model features and faci
510 explicitly; for example::
511
512 /usr/bin/qemu-system-s390x ... -cpu z13,a
513
514 A guest can be precluded from using AP feature
515 explicitly; for example::
516
517 /usr/bin/qemu-system-s390x ... -cpu host,
518
519 Note: If the APFT facility is turned off (apft
520 will not see any AP devices. The zcrypt device
521 register for type 10 and newer AP devices - i.
522 device drivers - need the APFT facility to asc
523 a given AP device. If the APFT facility is not
524 adapter or domain devices will get created by
525 guest because only type 10 and newer devices c
526
527 Example
528 =======
529 Let's now provide an example to illustrate how
530 access to AP facilities. For this example, we
531 three guests such that executing the lszcrypt
532 look like this:
533
534 Guest1
535 ------
536 =========== ===== ============
537 CARD.DOMAIN TYPE MODE
538 =========== ===== ============
539 05 CEX5C CCA-Coproc
540 05.0004 CEX5C CCA-Coproc
541 05.00ab CEX5C CCA-Coproc
542 06 CEX5A Accelerator
543 06.0004 CEX5A Accelerator
544 06.00ab CEX5A Accelerator
545 =========== ===== ============
546
547 Guest2
548 ------
549 =========== ===== ============
550 CARD.DOMAIN TYPE MODE
551 =========== ===== ============
552 05 CEX5C CCA-Coproc
553 05.0047 CEX5C CCA-Coproc
554 05.00ff CEX5C CCA-Coproc
555 =========== ===== ============
556
557 Guest3
558 ------
559 =========== ===== ============
560 CARD.DOMAIN TYPE MODE
561 =========== ===== ============
562 06 CEX5A Accelerator
563 06.0047 CEX5A Accelerator
564 06.00ff CEX5A Accelerator
565 =========== ===== ============
566
567 These are the steps:
568
569 1. Install the vfio_ap module on the linux hos
570 vfio_ap module is:
571 * iommu
572 * s390
573 * zcrypt
574 * vfio
575 * vfio_mdev
576 * vfio_mdev_device
577 * KVM
578
579 To build the vfio_ap module, the kernel bui
580 following Kconfig elements selected:
581 * IOMMU_SUPPORT
582 * S390
583 * AP
584 * VFIO
585 * KVM
586
587 If using make menuconfig select the followi
588
589 -> Device Drivers
590 -> IOMMU Hardware Support
591 select S390 AP IOMMU Support
592 -> VFIO Non-Privileged userspace drive
593 -> Mediated device driver frramewor
594 -> VFIO driver for Mediated devi
595 -> I/O subsystem
596 -> VFIO support for AP devices
597
598 2. Secure the AP queues to be used by the thre
599 access them. To secure them, there are two
600 bitmasks marking a subset of the APQN range
601 queue device drivers. All remaining APQNs a
602 any other device driver. The vfio_ap device
603 non-default device driver. The location of
604 masks are::
605
606 /sys/bus/ap/apmask
607 /sys/bus/ap/aqmask
608
609 The 'apmask' is a 256-bit mask that identif
610 (APID). Each bit in the mask, from left to
611 0-255. If a bit is set, the APID belongs to
612 available only to the default AP queue devi
613
614 The 'aqmask' is a 256-bit mask that identif
615 (APQI). Each bit in the mask, from left to
616 0-255. If a bit is set, the APQI belongs to
617 available only to the default AP queue devi
618
619 The Cartesian product of the APIDs correspo
620 apmask and the APQIs corresponding to the b
621 the subset of APQNs that can be used only b
622 All other APQNs are available to the non-de
623 vfio_ap driver.
624
625 Take, for example, the following masks::
626
627 apmask:
628 0x7d000000000000000000000000000000000000
629
630 aqmask:
631 0x80000000000000000000000000000000000000
632
633 The masks indicate:
634
635 * Adapters 1, 2, 3, 4, 5, and 7 are availab
636 device drivers.
637
638 * Domain 0 is available for use by the host
639
640 * The subset of APQNs available for use onl
641 drivers are:
642
643 (1,0), (2,0), (3,0), (4.0), (5,0) and (7,
644
645 * All other APQNs are available for use by
646
647 The APQN of each AP queue device assigned t
648 AP bus against the set of APQNs derived fro
649 and APQIs marked as available to the defaul
650 match is detected, only the default AP que
651 otherwise, the vfio_ap device driver will b
652
653 By default, the two masks are set to reserv
654 AP queue device drivers. There are two ways
655
656 1. The sysfs mask files can be edited by ec
657 respective sysfs mask file in one of two
658
659 * An absolute hex string starting with 0
660 the mask. If the given string is short
661 with 0s on the right; for example, spe
662 the same as specifying::
663
664 0x410000000000000000000000000000000
665
666 Keep in mind that the mask reads from
667 above identifies device numbers 1 and
668
669 If the string is longer than the mask,
670 an error (EINVAL).
671
672 * Individual bits in the mask can be swi
673 each bit number to be switched in a co
674 number string must be prepended with a
675 the corresponding bit is to be switche
676 valid values are:
677
678 - "+0" switches bit 0 on
679 - "-13" switches bit 13 off
680 - "+0x41" switches bit 65 on
681 - "-0xff" switches bit 255 off
682
683 The following example:
684
685 +0,-6,+0x47,-0xf0
686
687 Switches bits 0 and 71 (0x47) on
688
689 Switches bits 6 and 240 (0xf0) off
690
691 Note that the bits not specified in th
692 the operation.
693
694 2. The masks can also be changed at boot ti
695 command line like this:
696
697 ap.apmask=0xffff ap.aqmask=0x40
698
699 This would create the following masks
700
701 apmask:
702 0xffff0000000000000000000000000000
703
704 aqmask:
705 0x40000000000000000000000000000000
706
707 Resulting in these two pools::
708
709 default drivers pool: adapter 0
710 alternate drivers pool: adapter 1
711
712 **Note:**
713 Changing a mask such that one or more APQNs
714 mediated device (see below) will fail with
715 is logged to the kernel ring buffer which c
716 command. The output identifies each APQN fl
717 the vfio_ap mediated device to which it is
718
719 Userspace may not re-assign queue 05.0054 a
720 Userspace may not re-assign queue 04.0054 a
721
722 Securing the APQNs for our example
723 ----------------------------------
724 To secure the AP queues 05.0004, 05.0047, 0
725 06.00ab, and 06.00ff for use by the vfio_ap
726 APQNs can be removed from the default masks
727 commands::
728
729 echo -5,-6 > /sys/bus/ap/apmask
730
731 echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/
732
733 Or the masks can be set as follows::
734
735 echo 0xf9fffffffffffffffffffffffffffffff
736 > apmask
737
738 echo 0xf7fffffffffffffffefffffffffffffff
739 > aqmask
740
741 This will result in AP queues 05.0004, 05.0
742 06.0047, 06.00ab, and 06.00ff getting bound
743 sysfs directory for the vfio_ap device driv
744 to the AP queue devices bound to it::
745
746 /sys/bus/ap
747 ... [drivers]
748 ...... [vfio_ap]
749 ......... [05.0004]
750 ......... [05.0047]
751 ......... [05.00ab]
752 ......... [05.00ff]
753 ......... [06.0004]
754 ......... [06.0047]
755 ......... [06.00ab]
756 ......... [06.00ff]
757
758 Keep in mind that only type 10 and newer ad
759 can be bound to the vfio_ap device driver.
760 simplify the implementation by not needless
761 supporting older devices that will go out o
762 future and for which there are few older sy
763
764 The administrator, therefore, must take car
765 can be bound to the vfio_ap device driver.
766 queue device can be read from the parent ca
767 to see the hardware type of the queue 05.00
768
769 cat /sys/bus/ap/devices/card05/hwtype
770
771 The hwtype must be 10 or higher (CEX4 or ne
772 vfio_ap device driver.
773
774 3. Create the mediated devices needed to confi
775 three guests and to provide an interface to
776 use by the guests::
777
778 /sys/devices/vfio_ap/matrix/
779 --- [mdev_supported_types]
780 ------ [vfio_ap-passthrough] (passthrough
781 --------- create
782 --------- [devices]
783
784 To create the mediated devices for the thre
785
786 uuidgen > create
787 uuidgen > create
788 uuidgen > create
789
790 or
791
792 echo $uuid1 > create
793 echo $uuid2 > create
794 echo $uuid3 > create
795
796 This will create three mediated devices in
797 after the UUID written to the create attrib
798 $uuid2 and $uuid3 and this is the sysfs dir
799
800 /sys/devices/vfio_ap/matrix/
801 --- [mdev_supported_types]
802 ------ [vfio_ap-passthrough]
803 --------- [devices]
804 ------------ [$uuid1]
805 --------------- assign_adapter
806 --------------- assign_control_domain
807 --------------- assign_domain
808 --------------- matrix
809 --------------- unassign_adapter
810 --------------- unassign_control_domain
811 --------------- unassign_domain
812
813 ------------ [$uuid2]
814 --------------- assign_adapter
815 --------------- assign_control_domain
816 --------------- assign_domain
817 --------------- matrix
818 --------------- unassign_adapter
819 ----------------unassign_control_domain
820 ----------------unassign_domain
821
822 ------------ [$uuid3]
823 --------------- assign_adapter
824 --------------- assign_control_domain
825 --------------- assign_domain
826 --------------- matrix
827 --------------- unassign_adapter
828 ----------------unassign_control_domain
829 ----------------unassign_domain
830
831 Note *****: The vfio_ap mdevs do not persis
832 mdevctl tool is used to create
833
834 4. The administrator now needs to configure th
835 devices $uuid1 (for Guest1), $uuid2 (for Gu
836
837 This is how the matrix is configured for Gu
838
839 echo 5 > assign_adapter
840 echo 6 > assign_adapter
841 echo 4 > assign_domain
842 echo 0xab > assign_domain
843
844 Control domains can similarly be assigned u
845 sysfs file.
846
847 If a mistake is made configuring an adapter
848 you can use the unassign_xxx files to unass
849 control domain.
850
851 To display the matrix configuration for Gue
852
853 cat matrix
854
855 To display the matrix that is or will be as
856
857 cat guest_matrix
858
859 This is how the matrix is configured for Gu
860
861 echo 5 > assign_adapter
862 echo 0x47 > assign_domain
863 echo 0xff > assign_domain
864
865 This is how the matrix is configured for Gu
866
867 echo 6 > assign_adapter
868 echo 0x47 > assign_domain
869 echo 0xff > assign_domain
870
871 In order to successfully assign an adapter:
872
873 * The adapter number specified must represe
874 maximum adapter number configured for the
875 higher than the maximum is specified, the
876 an error (ENODEV).
877
878 Note: The maximum adapter number can be o
879 /sys/bus/ap/ap_max_adapter_id attri
880
881 * Each APQN derived from the Cartesian prod
882 being assigned and the APQIs of the domai
883
884 - Must only be available to the vfio_ap d
885 sysfs /sys/bus/ap/apmask and /sys/bus/a
886 one APQN is reserved for use by the hos
887 will terminate with an error (EADDRNOTA
888
889 - Must NOT be assigned to another vfio_ap
890 is assigned to another vfio_ap mediated
891 terminate with an error (EBUSY).
892
893 - Must NOT be assigned while the sysfs /s
894 sys/bus/ap/aqmask attribute files are b
895 terminate with an error (EBUSY).
896
897 In order to successfully assign a domain:
898
899 * The domain number specified must represen
900 maximum domain number configured for the
901 higher than the maximum is specified, the
902 an error (ENODEV).
903
904 Note: The maximum domain number can be ob
905 /sys/bus/ap/ap_max_domain_id attrib
906
907 * Each APQN derived from the Cartesian pro
908 being assigned and the APIDs of the adap
909
910 - Must only be available to the vfio_ap d
911 sysfs /sys/bus/ap/apmask and /sys/bus/a
912 one APQN is reserved for use by the hos
913 will terminate with an error (EADDRNOTA
914
915 - Must NOT be assigned to another vfio_ap
916 is assigned to another vfio_ap mediated
917 terminate with an error (EBUSY).
918
919 - Must NOT be assigned while the sysfs /s
920 sys/bus/ap/aqmask attribute files are b
921 terminate with an error (EBUSY).
922
923 In order to successfully assign a control d
924
925 * The domain number specified must represen
926 domain number configured for the system.
927 than the maximum is specified, the operat
928 error (ENODEV).
929
930 5. Start Guest1::
931
932 /usr/bin/qemu-system-s390x ... -cpu host,
933 -device vfio-ap,sysfsdev=/sys/devices/
934
935 7. Start Guest2::
936
937 /usr/bin/qemu-system-s390x ... -cpu host,
938 -device vfio-ap,sysfsdev=/sys/devices/
939
940 7. Start Guest3::
941
942 /usr/bin/qemu-system-s390x ... -cpu host,
943 -device vfio-ap,sysfsdev=/sys/devices/
944
945 When the guest is shut down, the vfio_ap media
946
947 Using our example again, to remove the vfio_ap
948
949 /sys/devices/vfio_ap/matrix/
950 --- [mdev_supported_types]
951 ------ [vfio_ap-passthrough]
952 --------- [devices]
953 ------------ [$uuid1]
954 --------------- remove
955
956 ::
957
958 echo 1 > remove
959
960 This will remove all of the matrix mdev device
961 the mdev device itself. To recreate and reconf
962 all of the steps starting with step 3 will hav
963 that the remove will fail if a guest using the
964
965 It is not necessary to remove a vfio_ap mdev,
966 remove it if no guest will use it during the r
967 host. If the vfio_ap mdev is removed, one may
968 the pool of adapters and queues reserved for u
969
970 Hot plug/unplug support:
971 ========================
972 An adapter, domain or control domain may be ho
973 guest by assigning it to the vfio_ap mediated
974 the following conditions are met:
975
976 * The adapter, domain or control domain must a
977 AP configuration.
978
979 * Each APQN derived from the Cartesian product
980 adapter being assigned and the APQIs of the
981 queue device bound to the vfio_ap device dri
982
983 * To hot plug a domain, each APQN derived from
984 comprised of the APQI of the domain being as
985 adapters assigned must reference a queue dev
986 driver.
987
988 An adapter, domain or control domain may be ho
989 guest by unassigning it from the vfio_ap media
990 guest.
991
992 Over-provisioning of AP queues for a KVM guest
993 ==============================================
994 Over-provisioning is defined herein as the ass
995 a vfio_ap mediated device that do not referenc
996 configuration. The idea here is that when the
997 available, it will be automatically hot-plugge
998 the vfio_ap mediated device to which it is ass
999 resulting from plugging it in references a que
1000 device driver.
1001
1002 Driver Features
1003 ===============
1004 The vfio_ap driver exposes a sysfs file conta
1005 This exists so third party tools (like Libvir
1006 availability of specific features.
1007
1008 The features list can be found here: /sys/bus
1009
1010 Entries are space delimited. Each entry consi
1011 alphanumeric and underscore characters.
1012
1013 Example:
1014 cat /sys/bus/matrix/devices/matrix/features
1015 guest_matrix dyn ap_config
1016
1017 the following features are advertised:
1018
1019 ---------------+-----------------------------
1020 | Flag | Description
1021 +==============+=============================
1022 | guest_matrix | guest_matrix attribute exist
1023 | | adapters and domains that ar
1024 | | guest when the mdev is attac
1025 +--------------+-----------------------------
1026 | dyn | Indicates hot plug/unplug of
1027 | | domains for a guest to which
1028 +------------+-------------------------------
1029 | ap_config | ap_config interface for one-
1030 +--------------+-----------------------------
1031
1032 Limitations
1033 ===========
1034 Live guest migration is not supported for gue
1035 intervention by a system administrator. Befor
1036 the vfio_ap mediated device must be removed.
1037 removed manually (i.e., echo 1 > /sys/devices
1038 the mdev is in use by a KVM guest. If the gue
1039 its mdev can be hot unplugged from the guest
1040
1041 1. If the KVM guest was started with libvirt,
1042 the following commands:
1043
1044 virsh detach-device <guestname> <path-t
1045
1046 For example, to hot unplug mdev 6217788
1047 the guest named 'my-guest':
1048
1049 virsh detach-device my-guest ~/confi
1050
1051 The contents of my-guest-hostdev.
1052
1053 .. code-block:: xml
1054
1055 <hostdev mode='subsystem' type='m
1056 <source>
1057 <address uuid='62177883-f1bb-
1058 </source>
1059 </hostdev>
1060
1061
1062 virsh qemu-monitor-command <guest-name>
1063
1064 For example, to hot unplug the vfio_ap
1065 qemu command line with 'id=hostdev0' fr
1066
1067 .. code-block:: sh
1068
1069 virsh qemu-monitor-command my-guest
1070
1071 2. A vfio_ap mediated device can be hot unplu
1072 to the guest and using the following qemu
1073
1074 (QEMU) device-del id=<device-id>
1075
1076 For example, to hot unplug the vfio_ap
1077 on the qemu command line with 'id=hostd
1078
1079 (QEMU) device-del id=hostdev0
1080
1081 After live migration of the KVM guest complet
1082 restored to the KVM guest by hot plugging a v
1083 system into the guest in one of two ways:
1084
1085 1. If the KVM guest was started with libvirt,
1086 device into the guest via the following vi
1087
1088 virsh attach-device <guestname> <path-to-d
1089
1090 For example, to hot plug mdev 62177883-
1091 the guest named 'my-guest':
1092
1093 virsh attach-device my-guest ~/confi
1094
1095 The contents of my-guest-hostdev.
1096
1097 .. code-block:: xml
1098
1099 <hostdev mode='subsystem' type='m
1100 <source>
1101 <address uuid='62177883-f1bb-
1102 </source>
1103 </hostdev>
1104
1105
1106 virsh qemu-monitor-command <guest-name> --
1107 "device_add vfio-ap,sysfsdev=<path-to-mdev
1108
1109 For example, to hot plug the vfio_ap me
1110 62177883-f1bb-47f0-914d-32a22e3a8804 in
1111 device-id hostdev0:
1112
1113 virsh qemu-monitor-command my-guest --h
1114 "device_add vfio-ap,\
1115 sysfsdev=/sys/devices/vfio_ap/matrix/62
1116 id=hostdev0"
1117
1118 2. A vfio_ap mediated device can be hot plugg
1119 to the guest and using the following qemu
1120
1121 (qemu) device_add "vfio-ap,sysfsdev=<pa
1122
1123 For example, to plug the vfio_ap mediat
1124 62177883-f1bb-47f0-914d-32a22e3a8804 in
1125 hostdev0:
1126
1127 (QEMU) device-add "vfio-ap,\
1128 sysfsdev=/sys/devices/vfio_ap/matrix
1129 id=hostdev0"
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.