1 .. SPDX-License-Identifier: GPL-2.0
2
3 ====================================
4 Nested KVM on POWER
5 ====================================
6
7 Introduction
8 ============
9
10 This document explains how a guest operating s
11 hypervisor and run nested guests through the u
12 hypervisor has implemented them. The terms L0,
13 refer to different software entities. L0 is th
14 that would normally be called the "host" or "h
15 guest virtual machine that is directly run und
16 and controlled by L0. L2 is a guest virtual ma
17 and controlled by L1 acting as a hypervisor.
18
19 Existing API
20 ============
21
22 Linux/KVM has had support for Nesting as an L0
23
24 The L0 code was added::
25
26 commit 8e3f5fc1045dc49fd175b978c5457f5f51e7
27 Author: Paul Mackerras <paulus@ozlabs.org>
28 Date: Mon Oct 8 16:31:03 2018 +1100
29 KVM: PPC: Book3S HV: Framework and hcall st
30
31 The L1 code was added::
32
33 commit 360cae313702cdd0b90f82c261a8302fecef
34 Author: Paul Mackerras <paulus@ozlabs.org>
35 Date: Mon Oct 8 16:31:04 2018 +1100
36 KVM: PPC: Book3S HV: Nested guest entry via
37
38 This API works primarily using a single hcall
39 call made by the L1 to tell the L0 to start an
40 state. The L0 then starts this L2 and runs unt
41 is reached. Once the L2 exits, the state of th
42 the L1 by the L0. The full L2 vCPU state is al
43 and to L1 when the L2 is run. The L0 doesn't k
44 vCPU (except in the short sequence in the L0 o
45 -> L1 exit).
46
47 The only state kept by the L0 is the partition
48 it's partition table using the h_set_partition
49 other state held by the L0 about the L2s is ca
50 shadow page tables).
51
52 The L1 may run any L2 or vCPU without first in
53 simply starts the vCPU using h_enter_nested().
54 vCPUs is done implicitly whenever h_enter_nest
55
56 In this document, we call this existing API th
57
58 New PAPR API
59 ===============
60
61 The new PAPR API changes from the v1 API such
62 associated vCPUs is explicit. In this document
63 API.
64
65 h_enter_nested() is replaced with H_GUEST_VCPU
66 be called the L1 must explicitly create the L2
67 and any associated vCPUs() created with h_gues
68 and setting vCPU state can also be performed u
69 hcall.
70
71 The basic execution flow is for an L1 to creat
72 delete it is:
73
74 - L1 and L0 negotiate capabilities with H_GUES
75 (normally at L1 boot time).
76
77 - L1 requests the L0 create an L2 with H_GUEST
78
79 - L1 requests the L0 create an L2 vCPU with H_
80
81 - L1 and L0 communicate the vCPU state using t
82
83 - L1 requests the L0 runs the vCPU running H_G
84
85 - L1 deletes L2 with H_GUEST_DELETE()
86
87 More details of the individual hcalls follows:
88
89 HCALL Details
90 =============
91
92 This documentation is provided to give an over
93 API. It doesn't aim to provide all the details
94 an L1 or L0. Latest version of PAPR can be ref
95
96 All these HCALLs are made by the L1 to the L0.
97
98 H_GUEST_GET_CAPABILITIES()
99 --------------------------
100
101 This is called to get the capabilities of the
102 hypervisor. This includes capabilities such th
103 POWER9, POWER10) that are supported as L2s::
104
105 H_GUEST_GET_CAPABILITIES(uint64 flags)
106
107 Parameters:
108 Input:
109 flags: Reserved
110 Output:
111 R3: Return code
112 R4: Hypervisor Supported Capabilities bi
113
114 H_GUEST_SET_CAPABILITIES()
115 --------------------------
116
117 This is called to inform the L0 of the capabil
118 hypervisor. The set of flags passed here are t
119 H_GUEST_GET_CAPABILITIES()
120
121 Typically, GET will be called first and then S
122 subset of the flags returned from GET. This pr
123 L1 to negotiate an agreed set of capabilities:
124
125 H_GUEST_SET_CAPABILITIES(uint64 flags,
126 uint64 capabilities
127 Parameters:
128 Input:
129 flags: Reserved
130 capabilitiesBitmap1: Only capabilities a
131 H_GUEST_GET_CAPABIL
132 Output:
133 R3: Return code
134 R4: If R3 = H_P2: The number of invalid
135 R5: If R3 = H_P2: The index of first inv
136
137 H_GUEST_CREATE()
138 ----------------
139
140 This is called to create an L2. A unique ID of
141 (similar to an LPID) is returned, which can be
142 identify the L2::
143
144 H_GUEST_CREATE(uint64 flags,
145 uint64 continueToken);
146 Parameters:
147 Input:
148 flags: Reserved
149 continueToken: Initial call set to -1. S
150 after H_Busy or H_LongBus
151 returned, value that was
152 Output:
153 R3: Return code. Notable:
154 H_Not_Enough_Resources: Unable to crea
155 enough Hypervisor memory. See H_GUEST_
156 takeOwnershipOfVcpuState)
157 R4: If R3 = H_Busy or_H_LongBusyOrder ->
158
159 H_GUEST_CREATE_VCPU()
160 ---------------------
161
162 This is called to create a vCPU associated wit
163 (returned from H_GUEST_CREATE()) should be pas
164 is a unique (for this L2) vCPUid. This vCPUid
165 L1::
166
167 H_GUEST_CREATE_VCPU(uint64 flags,
168 uint64 guestId,
169 uint64 vcpuId);
170 Parameters:
171 Input:
172 flags: Reserved
173 guestId: ID obtained from H_GUEST_CREATE
174 vcpuId: ID of the vCPU to be created. Th
175 range of 0 to 2047
176 Output:
177 R3: Return code. Notable:
178 H_Not_Enough_Resources: Unable to crea
179 enough Hypervisor memory. See H_GUEST_
180 takeOwnershipOfVcpuState)
181
182 H_GUEST_GET_STATE()
183 -------------------
184
185 This is called to get state associated with an
186 This info is passed via the Guest State Buffer
187 explained later in this doc, necessary details
188
189 This can get either L2 wide or vcpu specific i
190 L2 wide is the timebase offset or process scop
191 info. Examples of vCPU specific are GPRs or VS
192 parameter specifies if this call is L2 wide or
193 IDs in the GSB must match this.
194
195 The L1 provides a pointer to the GSB as a para
196 provided is the L2 and vCPU IDs associated wit
197
198 The L1 writes only the IDs and sizes in the GS
199 associated values for each ID in the GSB::
200
201 H_GUEST_GET_STATE(uint64 flags,
202 uint64 guestId,
203 uint64 vcpuId,
204 uint64 dataBuffer,
205 uint64 dataBufferSi
206 Parameters:
207 Input:
208 flags:
209 Bit 0: getGuestWideState: Request sta
210 of an individual VCPU.
211 Bit 1: takeOwnershipOfVcpuState Indic
212 over ownership of the VCPU state an
213 the storage holding the state. The
214 be returned to the Hypervisor via H
215 to H_GUEST_RUN_VCPU being called fo
216 returned in the dataBuffer is in a
217 format.
218 Bits 2-63: Reserved
219 guestId: ID obtained from H_GUEST_CREATE
220 vcpuId: ID of the vCPU pass to H_GUEST_C
221 dataBuffer: A L1 real address of the GSB
222 If takeOwnershipOfVcpuState, size must
223 returned by ID=0x0001
224 dataBufferSizeInBytes: Size of dataBuffe
225 Output:
226 R3: Return code
227 R4: If R3 = H_Invalid_Element_Id: The ar
228 element ID.
229 If R3 = H_Invalid_Element_Size: The
230 element size.
231 If R3 = H_Invalid_Element_Value: The
232 element value.
233
234 H_GUEST_SET_STATE()
235 -------------------
236
237 This is called to set L2 wide or vCPU specific
238 passed via the Guest State Buffer (GSB), neces
239
240 This can set either L2 wide or vcpu specific i
241 L2 wide is the timebase offset or process scop
242 info. Examples of vCPU specific are GPRs or VS
243 parameter specifies if this call is L2 wide or
244 IDs in the GSB must match this.
245
246 The L1 provides a pointer to the GSB as a para
247 provided is the L2 and vCPU IDs associated wit
248
249 The L1 writes all values in the GSB and the L0
250 this call::
251
252 H_GUEST_SET_STATE(uint64 flags,
253 uint64 guestId,
254 uint64 vcpuId,
255 uint64 dataBuffer,
256 uint64 dataBufferSizeInByt
257 Parameters:
258 Input:
259 flags:
260 Bit 0: getGuestWideState: Request sta
261 of an individual VCPU.
262 Bit 1: returnOwnershipOfVcpuState Ret
263 GET_STATE takeOwnershipOfVcpuState
264 Bits 2-63: Reserved
265 guestId: ID obtained from H_GUEST_CREATE
266 vcpuId: ID of the vCPU pass to H_GUEST_C
267 dataBuffer: A L1 real address of the GSB
268 If takeOwnershipOfVcpuState, size must
269 returned by ID=0x0001
270 dataBufferSizeInBytes: Size of dataBuffe
271 Output:
272 R3: Return code
273 R4: If R3 = H_Invalid_Element_Id: The ar
274 element ID.
275 If R3 = H_Invalid_Element_Size: The
276 element size.
277 If R3 = H_Invalid_Element_Value: The
278 element value.
279
280 H_GUEST_RUN_VCPU()
281 ------------------
282
283 This is called to run an L2 vCPU. The L2 and v
284 parameters. The vCPU runs with the state set p
285 H_GUEST_SET_STATE(). When the L2 exits, the L1
286 hcall.
287
288 This hcall also has associated input and outpu
289 H_GUEST_{S,G}ET_STATE(), these GSB pointers ar
290 parameters to the hcall (This was done in the
291 performance). The locations of these GSBs must
292 the H_GUEST_SET_STATE() call with ID 0x0c00 an
293 below).
294
295 The input GSB may contain only VCPU specific e
296 GSB may also contain zero elements (ie 0 in th
297 GSB) if nothing needs to be set.
298
299 On exit from the hcall, the output buffer is f
300 determined by the L0. The reason for the exit
301 NIP is put in GPR4). The elements returned de
302 type. For example, if the exit reason is the L
303 0xc00), then GPR3-12 are provided in the outpu
304 state likely needed to service the hcall. If a
305 needed, H_GUEST_GET_STATE() may be called by t
306
307 To synthesize interrupts in the L2, when calli
308 the L1 may set a flag (as a hcall parameter) a
309 synthesize the interrupt in the L2. Alternativ
310 synthesize the interrupt itself using H_GUEST_
311 H_GUEST_RUN_VCPU() input GSB to set the state
312
313 H_GUEST_RUN_VCPU(uint64 flags,
314 uint64 guestId,
315 uint64 vcpuId,
316 uint64 dataBuffer,
317 uint64 dataBufferSizeInByte
318 Parameters:
319 Input:
320 flags:
321 Bit 0: generateExternalInterrupt: Gen
322 Bit 1: generatePrivilegedDoorbell: Ge
323 Bit 2: sendToSystemResetâ: Generate
324 Bits 3-63: Reserved
325 guestId: ID obtained from H_GUEST_CREATE
326 vcpuId: ID of the vCPU pass to H_GUEST_C
327 Output:
328 R3: Return code
329 R4: If R3 = H_Success: The reason L1 VCP
330 0x000: The VCPU stopped running fo
331 example of this is the Hyperviso
332 due to an outstanding interrupt
333 0x980: HDEC
334 0xC00: HCALL
335 0xE00: HDSI
336 0xE20: HISI
337 0xE40: HEA
338 0xF80: HV Fac Unavail
339 If R3 = H_Invalid_Element_Id, H_Inva
340 H_Invalid_Element_Value: R4 is off
341 in the input buffer.
342
343 H_GUEST_DELETE()
344 ----------------
345
346 This is called to delete an L2. All associated
347 deleted. No specific vCPU delete call is provi
348
349 A flag may be provided to delete all guests. T
350 L0 in the case of kdump/kexec::
351
352 H_GUEST_DELETE(uint64 flags,
353 uint64 guestId)
354 Parameters:
355 Input:
356 flags:
357 Bit 0: deleteAllGuests: deletes all g
358 Bits 1-63: Reserved
359 guestId: ID obtained from H_GUEST_CREATE
360 Output:
361 R3: Return code
362
363 Guest State Buffer
364 ==================
365
366 The Guest State Buffer (GSB) is the main metho
367 about the L2 between the L1 and L0 via H_GUEST
368 H_GUEST_VCPU_RUN() calls.
369
370 State may be associated with a whole L2 (eg ti
371 specific L2 vCPU (eg. GPR state). Only L2 VCPU
372 H_GUEST_VCPU_RUN().
373
374 All data in the GSB is big endian (as is stand
375
376 The Guest state buffer has a header which give
377 elements, followed by the GSB elements themsel
378
379 GSB header:
380
381 +----------+----------+-----------------------
382 | Offset | Size | Purpose
383 | Bytes | Bytes |
384 +==========+==========+=======================
385 | 0 | 4 | Number of elements
386 +----------+----------+-----------------------
387 | 4 | | Guest state buffer el
388 +----------+----------+-----------------------
389
390 GSB element:
391
392 +----------+----------+-----------------------
393 | Offset | Size | Purpose
394 | Bytes | Bytes |
395 +==========+==========+=======================
396 | 0 | 2 | ID
397 +----------+----------+-----------------------
398 | 2 | 2 | Size of Value
399 +----------+----------+-----------------------
400 | 4 | As above | Value
401 +----------+----------+-----------------------
402
403 The ID in the GSB element specifies what is to
404 archtected state like GPRs, VSRs, SPRs, plus a
405 the partition like the timebase offset and par
406 table information.
407
408 +--------+-------+----+--------+--------------
409 | ID | Size | RW | Thread | Details
410 | | Bytes | | Guest |
411 | | | | Scope |
412 +========+=======+====+========+==============
413 | 0x0000 | | RW | TG | NOP element
414 +--------+-------+----+--------+--------------
415 | 0x0001 | 0x08 | R | G | Size of L0 vC
416 | | | | | H_GUEST_GET_S
417 | | | | | flags = takeO
418 +--------+-------+----+--------+--------------
419 | 0x0002 | 0x08 | R | G | Size Run vCPU
420 +--------+-------+----+--------+--------------
421 | 0x0003 | 0x04 | RW | G | Logical PVR
422 +--------+-------+----+--------+--------------
423 | 0x0004 | 0x08 | RW | G | TB Offset (L1
424 +--------+-------+----+--------+--------------
425 | 0x0005 | 0x18 | RW | G |Partition scop
426 | | | | |
427 | | | | |- 0x00 Addr pa
428 | | | | |- 0x08 Num add
429 | | | | |- 0x10 Size ro
430 +--------+-------+----+--------+--------------
431 | 0x0006 | 0x10 | RW | G |Process Table
432 | | | | |
433 | | | | |- 0x0 Addr pro
434 | | | | |- 0x8 Table si
435 +--------+-------+----+--------+--------------
436 | 0x0007-| | | | Reserved
437 | 0x0BFF | | | |
438 +--------+-------+----+--------+--------------
439 | 0x0C00 | 0x10 | RW | T |Run vCPU Input
440 | | | | |
441 | | | | |- 0x0 Addr of
442 | | | | |- 0x8 Buffer S
443 +--------+-------+----+--------+--------------
444 | 0x0C01 | 0x10 | RW | T |Run vCPU Outpu
445 | | | | |
446 | | | | |- 0x0 Addr of
447 | | | | |- 0x8 Buffer S
448 +--------+-------+----+--------+--------------
449 | 0x0C02 | 0x08 | RW | T | vCPU VPA Addr
450 +--------+-------+----+--------+--------------
451 | 0x0C03-| | | | Reserved
452 | 0x0FFF | | | |
453 +--------+-------+----+--------+--------------
454 | 0x1000-| 0x08 | RW | T | GPR 0-31
455 | 0x101F | | | |
456 +--------+-------+----+--------+--------------
457 | 0x1020 | 0x08 | T | T | HDEC expiry T
458 +--------+-------+----+--------+--------------
459 | 0x1021 | 0x08 | RW | T | NIA
460 +--------+-------+----+--------+--------------
461 | 0x1022 | 0x08 | RW | T | MSR
462 +--------+-------+----+--------+--------------
463 | 0x1023 | 0x08 | RW | T | LR
464 +--------+-------+----+--------+--------------
465 | 0x1024 | 0x08 | RW | T | XER
466 +--------+-------+----+--------+--------------
467 | 0x1025 | 0x08 | RW | T | CTR
468 +--------+-------+----+--------+--------------
469 | 0x1026 | 0x08 | RW | T | CFAR
470 +--------+-------+----+--------+--------------
471 | 0x1027 | 0x08 | RW | T | SRR0
472 +--------+-------+----+--------+--------------
473 | 0x1028 | 0x08 | RW | T | SRR1
474 +--------+-------+----+--------+--------------
475 | 0x1029 | 0x08 | RW | T | DAR
476 +--------+-------+----+--------+--------------
477 | 0x102A | 0x08 | RW | T | DEC expiry TB
478 +--------+-------+----+--------+--------------
479 | 0x102B | 0x08 | RW | T | VTB
480 +--------+-------+----+--------+--------------
481 | 0x102C | 0x08 | RW | T | LPCR
482 +--------+-------+----+--------+--------------
483 | 0x102D | 0x08 | RW | T | HFSCR
484 +--------+-------+----+--------+--------------
485 | 0x102E | 0x08 | RW | T | FSCR
486 +--------+-------+----+--------+--------------
487 | 0x102F | 0x08 | RW | T | FPSCR
488 +--------+-------+----+--------+--------------
489 | 0x1030 | 0x08 | RW | T | DAWR0
490 +--------+-------+----+--------+--------------
491 | 0x1031 | 0x08 | RW | T | DAWR1
492 +--------+-------+----+--------+--------------
493 | 0x1032 | 0x08 | RW | T | CIABR
494 +--------+-------+----+--------+--------------
495 | 0x1033 | 0x08 | RW | T | PURR
496 +--------+-------+----+--------+--------------
497 | 0x1034 | 0x08 | RW | T | SPURR
498 +--------+-------+----+--------+--------------
499 | 0x1035 | 0x08 | RW | T | IC
500 +--------+-------+----+--------+--------------
501 | 0x1036-| 0x08 | RW | T | SPRG 0-3
502 | 0x1039 | | | |
503 +--------+-------+----+--------+--------------
504 | 0x103A | 0x08 | W | T | PPR
505 +--------+-------+----+--------+--------------
506 | 0x103B | 0x08 | RW | T | MMCR 0-3
507 | 0x103E | | | |
508 +--------+-------+----+--------+--------------
509 | 0x103F | 0x08 | RW | T | MMCRA
510 +--------+-------+----+--------+--------------
511 | 0x1040 | 0x08 | RW | T | SIER
512 +--------+-------+----+--------+--------------
513 | 0x1041 | 0x08 | RW | T | SIER 2
514 +--------+-------+----+--------+--------------
515 | 0x1042 | 0x08 | RW | T | SIER 3
516 +--------+-------+----+--------+--------------
517 | 0x1043 | 0x08 | RW | T | BESCR
518 +--------+-------+----+--------+--------------
519 | 0x1044 | 0x08 | RW | T | EBBHR
520 +--------+-------+----+--------+--------------
521 | 0x1045 | 0x08 | RW | T | EBBRR
522 +--------+-------+----+--------+--------------
523 | 0x1046 | 0x08 | RW | T | AMR
524 +--------+-------+----+--------+--------------
525 | 0x1047 | 0x08 | RW | T | IAMR
526 +--------+-------+----+--------+--------------
527 | 0x1048 | 0x08 | RW | T | AMOR
528 +--------+-------+----+--------+--------------
529 | 0x1049 | 0x08 | RW | T | UAMOR
530 +--------+-------+----+--------+--------------
531 | 0x104A | 0x08 | RW | T | SDAR
532 +--------+-------+----+--------+--------------
533 | 0x104B | 0x08 | RW | T | SIAR
534 +--------+-------+----+--------+--------------
535 | 0x104C | 0x08 | RW | T | DSCR
536 +--------+-------+----+--------+--------------
537 | 0x104D | 0x08 | RW | T | TAR
538 +--------+-------+----+--------+--------------
539 | 0x104E | 0x08 | RW | T | DEXCR
540 +--------+-------+----+--------+--------------
541 | 0x104F | 0x08 | RW | T | HDEXCR
542 +--------+-------+----+--------+--------------
543 | 0x1050 | 0x08 | RW | T | HASHKEYR
544 +--------+-------+----+--------+--------------
545 | 0x1051 | 0x08 | RW | T | HASHPKEYR
546 +--------+-------+----+--------+--------------
547 | 0x1052 | 0x08 | RW | T | CTRL
548 +--------+-------+----+--------+--------------
549 | 0x1053 | 0x08 | RW | T | DPDES
550 +--------+-------+----+--------+--------------
551 | 0x1054-| | | | Reserved
552 | 0x1FFF | | | |
553 +--------+-------+----+--------+--------------
554 | 0x2000 | 0x04 | RW | T | CR
555 +--------+-------+----+--------+--------------
556 | 0x2001 | 0x04 | RW | T | PIDR
557 +--------+-------+----+--------+--------------
558 | 0x2002 | 0x04 | RW | T | DSISR
559 +--------+-------+----+--------+--------------
560 | 0x2003 | 0x04 | RW | T | VSCR
561 +--------+-------+----+--------+--------------
562 | 0x2004 | 0x04 | RW | T | VRSAVE
563 +--------+-------+----+--------+--------------
564 | 0x2005 | 0x04 | RW | T | DAWRX0
565 +--------+-------+----+--------+--------------
566 | 0x2006 | 0x04 | RW | T | DAWRX1
567 +--------+-------+----+--------+--------------
568 | 0x2007-| 0x04 | RW | T | PMC 1-6
569 | 0x200c | | | |
570 +--------+-------+----+--------+--------------
571 | 0x200D | 0x04 | RW | T | WORT
572 +--------+-------+----+--------+--------------
573 | 0x200E | 0x04 | RW | T | PSPB
574 +--------+-------+----+--------+--------------
575 | 0x200F-| | | | Reserved
576 | 0x2FFF | | | |
577 +--------+-------+----+--------+--------------
578 | 0x3000-| 0x10 | RW | T | VSR 0-63
579 | 0x303F | | | |
580 +--------+-------+----+--------+--------------
581 | 0x3040-| | | | Reserved
582 | 0xEFFF | | | |
583 +--------+-------+----+--------+--------------
584 | 0xF000 | 0x08 | R | T | HDAR
585 +--------+-------+----+--------+--------------
586 | 0xF001 | 0x04 | R | T | HDSISR
587 +--------+-------+----+--------+--------------
588 | 0xF002 | 0x04 | R | T | HEIR
589 +--------+-------+----+--------+--------------
590 | 0xF003 | 0x08 | R | T | ASDR
591 +--------+-------+----+--------+--------------
592
593
594 Miscellaneous info
595 ==================
596
597 State not in ptregs/hvregs
598 --------------------------
599
600 In the v1 API, some state is not in the ptregs
601 the vector register and some SPRs. For the L1
602 the L2, the L1 loads up these hardware registe
603 h_enter_nested() call and the L0 ensures they
604 (by not touching them).
605
606 The v2 API removes this and explicitly sets th
607
608 L1 Implementation details: Caching state
609 ----------------------------------------
610
611 In the v1 API, all state is sent from the L1 t
612 on every h_enter_nested() hcall. If the L0 is
613 any L2s, the L0 has no state information about
614 exception to this is the location of the parti
615 via h_set_partition_table().
616
617 The v2 API changes this so that the L0 retains
618 it's vCPUs are no longer running. This means t
619 communicate with the L0 about L2 state when it
620 state, or when it's value is out of date. This
621 for performance optimisation.
622
623 When a vCPU exits from a H_GUEST_RUN_VCPU() ca
624 marks all L2 state as invalid. This means that
625 the L2 state (say via a kvm_get_one_reg() call
626 H_GUEST_GET_STATE() to get that state. Once it
627 valid in L1 until the L2 is run again.
628
629 Also, when an L1 modifies L2 vcpu state, it do
630 to the L0 until that L2 vcpu runs again. Hence
631 state (say via a kvm_set_one_reg() call), it w
632 copy and only flushes this copy to the L0 when
633 the H_GUEST_VCPU_RUN() input buffer.
634
635 This lazy updating of state by the L1 avoids u
636 H_GUEST_{G|S}ET_STATE() calls.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.