1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 Overview 3 Overview
4 ======== 4 ========
5 The Linux kernel contains a variety of code fo 5 The Linux kernel contains a variety of code for running as a fully
6 enlightened guest on Microsoft's Hyper-V hyper 6 enlightened guest on Microsoft's Hyper-V hypervisor. Hyper-V
7 consists primarily of a bare-metal hypervisor 7 consists primarily of a bare-metal hypervisor plus a virtual machine
8 management service running in the parent parti 8 management service running in the parent partition (roughly
9 equivalent to KVM and QEMU, for example). Gue 9 equivalent to KVM and QEMU, for example). Guest VMs run in child
10 partitions. In this documentation, references 10 partitions. In this documentation, references to Hyper-V usually
11 encompass both the hypervisor and the VMM serv 11 encompass both the hypervisor and the VMM service without making a
12 distinction about which functionality is provi 12 distinction about which functionality is provided by which
13 component. 13 component.
14 14
15 Hyper-V runs on x86/x64 and arm64 architecture 15 Hyper-V runs on x86/x64 and arm64 architectures, and Linux guests
16 are supported on both. The functionality and 16 are supported on both. The functionality and behavior of Hyper-V is
17 generally the same on both architectures unles 17 generally the same on both architectures unless noted otherwise.
18 18
19 Linux Guest Communication with Hyper-V 19 Linux Guest Communication with Hyper-V
20 -------------------------------------- 20 --------------------------------------
21 Linux guests communicate with Hyper-V in four 21 Linux guests communicate with Hyper-V in four different ways:
22 22
23 * Implicit traps: As defined by the x86/x64 or 23 * Implicit traps: As defined by the x86/x64 or arm64 architecture,
24 some guest actions trap to Hyper-V. Hyper-V 24 some guest actions trap to Hyper-V. Hyper-V emulates the action and
25 returns control to the guest. This behavior 25 returns control to the guest. This behavior is generally invisible
26 to the Linux kernel. 26 to the Linux kernel.
27 27
28 * Explicit hypercalls: Linux makes an explicit 28 * Explicit hypercalls: Linux makes an explicit function call to
29 Hyper-V, passing parameters. Hyper-V perfor 29 Hyper-V, passing parameters. Hyper-V performs the requested action
30 and returns control to the caller. Paramete 30 and returns control to the caller. Parameters are passed in
31 processor registers or in memory shared betw 31 processor registers or in memory shared between the Linux guest and
32 Hyper-V. On x86/x64, hypercalls use a Hype 32 Hyper-V. On x86/x64, hypercalls use a Hyper-V specific calling
33 sequence. On arm64, hypercalls use the ARM 33 sequence. On arm64, hypercalls use the ARM standard SMCCC calling
34 sequence. 34 sequence.
35 35
36 * Synthetic register access: Hyper-V implement 36 * Synthetic register access: Hyper-V implements a variety of
37 synthetic registers. On x86/x64 these regis 37 synthetic registers. On x86/x64 these registers appear as MSRs in
38 the guest, and the Linux kernel can read or 38 the guest, and the Linux kernel can read or write these MSRs using
39 the normal mechanisms defined by the x86/x64 39 the normal mechanisms defined by the x86/x64 architecture. On
40 arm64, these synthetic registers must be acc 40 arm64, these synthetic registers must be accessed using explicit
41 hypercalls. 41 hypercalls.
42 42
43 * VMBus: VMBus is a higher-level software cons !! 43 * VMbus: VMbus is a higher-level software construct that is built on
44 the other 3 mechanisms. It is a message pas 44 the other 3 mechanisms. It is a message passing interface between
45 the Hyper-V host and the Linux guest. It us 45 the Hyper-V host and the Linux guest. It uses memory that is shared
46 between Hyper-V and the guest, along with va 46 between Hyper-V and the guest, along with various signaling
47 mechanisms. 47 mechanisms.
48 48
49 The first three communication mechanisms are d 49 The first three communication mechanisms are documented in the
50 `Hyper-V Top Level Functional Spec (TLFS)`_. 50 `Hyper-V Top Level Functional Spec (TLFS)`_. The TLFS describes
51 general Hyper-V functionality and provides det 51 general Hyper-V functionality and provides details on the hypercalls
52 and synthetic registers. The TLFS is currentl 52 and synthetic registers. The TLFS is currently written for the
53 x86/x64 architecture only. 53 x86/x64 architecture only.
54 54
55 .. _Hyper-V Top Level Functional Spec (TLFS): 55 .. _Hyper-V Top Level Functional Spec (TLFS): https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/tlfs
56 56
57 VMBus is not documented. This documentation p !! 57 VMbus is not documented. This documentation provides a high-level
58 overview of VMBus and how it works, but the de !! 58 overview of VMbus and how it works, but the details can be discerned
59 only from the code. 59 only from the code.
60 60
61 Sharing Memory 61 Sharing Memory
62 -------------- 62 --------------
63 Many aspects are communication between Hyper-V 63 Many aspects are communication between Hyper-V and Linux are based
64 on sharing memory. Such sharing is generally 64 on sharing memory. Such sharing is generally accomplished as
65 follows: 65 follows:
66 66
67 * Linux allocates memory from its physical add 67 * Linux allocates memory from its physical address space using
68 standard Linux mechanisms. 68 standard Linux mechanisms.
69 69
70 * Linux tells Hyper-V the guest physical addre 70 * Linux tells Hyper-V the guest physical address (GPA) of the
71 allocated memory. Many shared areas are kep 71 allocated memory. Many shared areas are kept to 1 page so that a
72 single GPA is sufficient. Larger shared ar 72 single GPA is sufficient. Larger shared areas require a list of
73 GPAs, which usually do not need to be contig 73 GPAs, which usually do not need to be contiguous in the guest
74 physical address space. How Hyper-V is told 74 physical address space. How Hyper-V is told about the GPA or list
75 of GPAs varies. In some cases, a single GPA 75 of GPAs varies. In some cases, a single GPA is written to a
76 synthetic register. In other cases, a GPA o 76 synthetic register. In other cases, a GPA or list of GPAs is sent
77 in a VMBus message. !! 77 in a VMbus message.
78 78
79 * Hyper-V translates the GPAs into "real" phys 79 * Hyper-V translates the GPAs into "real" physical memory addresses,
80 and creates a virtual mapping that it can us 80 and creates a virtual mapping that it can use to access the memory.
81 81
82 * Linux can later revoke sharing it has previo 82 * Linux can later revoke sharing it has previously established by
83 telling Hyper-V to set the shared GPA to zer 83 telling Hyper-V to set the shared GPA to zero.
84 84
85 Hyper-V operates with a page size of 4 Kbytes. 85 Hyper-V operates with a page size of 4 Kbytes. GPAs communicated to
86 Hyper-V may be in the form of page numbers, an 86 Hyper-V may be in the form of page numbers, and always describe a
87 range of 4 Kbytes. Since the Linux guest page 87 range of 4 Kbytes. Since the Linux guest page size on x86/x64 is
88 also 4 Kbytes, the mapping from guest page to 88 also 4 Kbytes, the mapping from guest page to Hyper-V page is 1-to-1.
89 On arm64, Hyper-V supports guests with 4/16/64 89 On arm64, Hyper-V supports guests with 4/16/64 Kbyte pages as
90 defined by the arm64 architecture. If Linux 90 defined by the arm64 architecture. If Linux is using 16 or 64
91 Kbyte pages, Linux code must be careful to com 91 Kbyte pages, Linux code must be careful to communicate with Hyper-V
92 only in terms of 4 Kbyte pages. HV_HYP_PAGE_S 92 only in terms of 4 Kbyte pages. HV_HYP_PAGE_SIZE and related macros
93 are used in code that communicates with Hyper- 93 are used in code that communicates with Hyper-V so that it works
94 correctly in all configurations. 94 correctly in all configurations.
95 95
96 As described in the TLFS, a few memory pages s 96 As described in the TLFS, a few memory pages shared between Hyper-V
97 and the Linux guest are "overlay" pages. With 97 and the Linux guest are "overlay" pages. With overlay pages, Linux
98 uses the usual approach of allocating guest me 98 uses the usual approach of allocating guest memory and telling
99 Hyper-V the GPA of the allocated memory. But 99 Hyper-V the GPA of the allocated memory. But Hyper-V then replaces
100 that physical memory page with a page it has a 100 that physical memory page with a page it has allocated, and the
101 original physical memory page is no longer acc 101 original physical memory page is no longer accessible in the guest
102 VM. Linux may access the memory normally as i 102 VM. Linux may access the memory normally as if it were the memory
103 that it originally allocated. The "overlay" b 103 that it originally allocated. The "overlay" behavior is visible
104 only because the contents of the page (as seen 104 only because the contents of the page (as seen by Linux) change at
105 the time that Linux originally establishes the 105 the time that Linux originally establishes the sharing and the
106 overlay page is inserted. Similarly, the cont 106 overlay page is inserted. Similarly, the contents change if Linux
107 revokes the sharing, in which case Hyper-V rem 107 revokes the sharing, in which case Hyper-V removes the overlay page,
108 and the guest page originally allocated by Lin 108 and the guest page originally allocated by Linux becomes visible
109 again. 109 again.
110 110
111 Before Linux does a kexec to a kdump kernel or 111 Before Linux does a kexec to a kdump kernel or any other kernel,
112 memory shared with Hyper-V should be revoked. 112 memory shared with Hyper-V should be revoked. Hyper-V could modify
113 a shared page or remove an overlay page after 113 a shared page or remove an overlay page after the new kernel is
114 using the page for a different purpose, corrup 114 using the page for a different purpose, corrupting the new kernel.
115 Hyper-V does not provide a single "set everyth 115 Hyper-V does not provide a single "set everything" operation to
116 guest VMs, so Linux code must individually rev 116 guest VMs, so Linux code must individually revoke all sharing before
117 doing kexec. See hv_kexec_handler() and hv_c 117 doing kexec. See hv_kexec_handler() and hv_crash_handler(). But
118 the crash/panic path still has holes in cleanu 118 the crash/panic path still has holes in cleanup because some shared
119 pages are set using per-CPU synthetic register 119 pages are set using per-CPU synthetic registers and there's no
120 mechanism to revoke the shared pages for CPUs 120 mechanism to revoke the shared pages for CPUs other than the CPU
121 running the panic path. 121 running the panic path.
122 122
123 CPU Management 123 CPU Management
124 -------------- 124 --------------
125 Hyper-V does not have a ability to hot-add or 125 Hyper-V does not have a ability to hot-add or hot-remove a CPU
126 from a running VM. However, Windows Server 20 126 from a running VM. However, Windows Server 2019 Hyper-V and
127 earlier versions may provide guests with ACPI 127 earlier versions may provide guests with ACPI tables that indicate
128 more CPUs than are actually present in the VM. 128 more CPUs than are actually present in the VM. As is normal, Linux
129 treats these additional CPUs as potential hot- 129 treats these additional CPUs as potential hot-add CPUs, and reports
130 them as such even though Hyper-V will never ac 130 them as such even though Hyper-V will never actually hot-add them.
131 Starting in Windows Server 2022 Hyper-V, the A 131 Starting in Windows Server 2022 Hyper-V, the ACPI tables reflect
132 only the CPUs actually present in the VM, so L 132 only the CPUs actually present in the VM, so Linux does not report
133 any hot-add CPUs. 133 any hot-add CPUs.
134 134
135 A Linux guest CPU may be taken offline using t 135 A Linux guest CPU may be taken offline using the normal Linux
136 mechanisms, provided no VMBus channel interrup !! 136 mechanisms, provided no VMbus channel interrupts are assigned to
137 the CPU. See the section on VMBus Interrupts !! 137 the CPU. See the section on VMbus Interrupts for more details
138 on how VMBus channel interrupts can be re-assi !! 138 on how VMbus channel interrupts can be re-assigned to permit
139 taking a CPU offline. 139 taking a CPU offline.
140 140
141 32-bit and 64-bit 141 32-bit and 64-bit
142 ----------------- 142 -----------------
143 On x86/x64, Hyper-V supports 32-bit and 64-bit 143 On x86/x64, Hyper-V supports 32-bit and 64-bit guests, and Linux
144 will build and run in either version. While th 144 will build and run in either version. While the 32-bit version is
145 expected to work, it is used rarely and may su 145 expected to work, it is used rarely and may suffer from undetected
146 regressions. 146 regressions.
147 147
148 On arm64, Hyper-V supports only 64-bit guests. 148 On arm64, Hyper-V supports only 64-bit guests.
149 149
150 Endian-ness 150 Endian-ness
151 ----------- 151 -----------
152 All communication between Hyper-V and guest VM 152 All communication between Hyper-V and guest VMs uses Little-Endian
153 format on both x86/x64 and arm64. Big-endian 153 format on both x86/x64 and arm64. Big-endian format on arm64 is not
154 supported by Hyper-V, and Linux code does not 154 supported by Hyper-V, and Linux code does not use endian-ness macros
155 when accessing data shared with Hyper-V. 155 when accessing data shared with Hyper-V.
156 156
157 Versioning 157 Versioning
158 ---------- 158 ----------
159 Current Linux kernels operate correctly with o 159 Current Linux kernels operate correctly with older versions of
160 Hyper-V back to Windows Server 2012 Hyper-V. S 160 Hyper-V back to Windows Server 2012 Hyper-V. Support for running
161 on the original Hyper-V release in Windows Ser 161 on the original Hyper-V release in Windows Server 2008/2008 R2
162 has been removed. 162 has been removed.
163 163
164 A Linux guest on Hyper-V outputs in dmesg the 164 A Linux guest on Hyper-V outputs in dmesg the version of Hyper-V
165 it is running on. This version is in the form 165 it is running on. This version is in the form of a Windows build
166 number and is for display purposes only. Linux 166 number and is for display purposes only. Linux code does not
167 test this version number at runtime to determi 167 test this version number at runtime to determine available features
168 and functionality. Hyper-V indicates feature/f 168 and functionality. Hyper-V indicates feature/function availability
169 via flags in synthetic MSRs that Hyper-V provi 169 via flags in synthetic MSRs that Hyper-V provides to the guest,
170 and the guest code tests these flags. 170 and the guest code tests these flags.
171 171
172 VMBus has its own protocol version that is neg !! 172 VMbus has its own protocol version that is negotiated during the
173 initial VMBus connection from the guest to Hyp !! 173 initial VMbus connection from the guest to Hyper-V. This version
174 number is also output to dmesg during boot. T 174 number is also output to dmesg during boot. This version number
175 is checked in a few places in the code to dete 175 is checked in a few places in the code to determine if specific
176 functionality is present. 176 functionality is present.
177 177
178 Furthermore, each synthetic device on VMBus al !! 178 Furthermore, each synthetic device on VMbus also has a protocol
179 version that is separate from the VMBus protoc !! 179 version that is separate from the VMbus protocol version. Device
180 drivers for these synthetic devices typically 180 drivers for these synthetic devices typically negotiate the device
181 protocol version, and may test that protocol v 181 protocol version, and may test that protocol version to determine
182 if specific device functionality is present. 182 if specific device functionality is present.
183 183
184 Code Packaging 184 Code Packaging
185 -------------- 185 --------------
186 Hyper-V related code appears in the Linux kern 186 Hyper-V related code appears in the Linux kernel code tree in three
187 main areas: 187 main areas:
188 188
189 1. drivers/hv 189 1. drivers/hv
190 190
191 2. arch/x86/hyperv and arch/arm64/hyperv 191 2. arch/x86/hyperv and arch/arm64/hyperv
192 192
193 3. individual device driver areas such as driv 193 3. individual device driver areas such as drivers/scsi, drivers/net,
194 drivers/clocksource, etc. 194 drivers/clocksource, etc.
195 195
196 A few miscellaneous files appear elsewhere. Se 196 A few miscellaneous files appear elsewhere. See the full list under
197 "Hyper-V/Azure CORE AND DRIVERS" and "DRM DRIV 197 "Hyper-V/Azure CORE AND DRIVERS" and "DRM DRIVER FOR HYPERV
198 SYNTHETIC VIDEO DEVICE" in the MAINTAINERS fil 198 SYNTHETIC VIDEO DEVICE" in the MAINTAINERS file.
199 199
200 The code in #1 and #2 is built only when CONFI 200 The code in #1 and #2 is built only when CONFIG_HYPERV is set.
201 Similarly, the code for most Hyper-V related d 201 Similarly, the code for most Hyper-V related drivers is built only
202 when CONFIG_HYPERV is set. 202 when CONFIG_HYPERV is set.
203 203
204 Most Hyper-V related code in #1 and #3 can be 204 Most Hyper-V related code in #1 and #3 can be built as a module.
205 The architecture specific code in #2 must be b 205 The architecture specific code in #2 must be built-in. Also,
206 drivers/hv/hv_common.c is low-level code that 206 drivers/hv/hv_common.c is low-level code that is common across
207 architectures and must be built-in. 207 architectures and must be built-in.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.