1 .. SPDX-License-Identifier: GPL-2.0+ 1 .. SPDX-License-Identifier: GPL-2.0+
2 2
3 ============================================== 3 ======================================================
4 IBM Virtual Management Channel Kernel Driver ( 4 IBM Virtual Management Channel Kernel Driver (IBMVMC)
5 ============================================== 5 ======================================================
6 6
7 :Authors: 7 :Authors:
8 Dave Engebretsen <engebret@us.ibm.com>, 8 Dave Engebretsen <engebret@us.ibm.com>,
9 Adam Reznechek <adreznec@linux.vnet.ibm 9 Adam Reznechek <adreznec@linux.vnet.ibm.com>,
10 Steven Royer <seroyer@linux.vnet.ibm.co 10 Steven Royer <seroyer@linux.vnet.ibm.com>,
11 Bryant G. Ly <bryantly@linux.vnet.ibm.c 11 Bryant G. Ly <bryantly@linux.vnet.ibm.com>,
12 12
13 Introduction 13 Introduction
14 ============ 14 ============
15 15
16 Note: Knowledge of virtualization technology i 16 Note: Knowledge of virtualization technology is required to understand
17 this document. 17 this document.
18 18
19 A good reference document would be: 19 A good reference document would be:
20 20
21 https://openpowerfoundation.org/wp-content/upl 21 https://openpowerfoundation.org/wp-content/uploads/2016/05/LoPAPR_DRAFT_v11_24March2016_cmt1.pdf
22 22
23 The Virtual Management Channel (VMC) is a logi 23 The Virtual Management Channel (VMC) is a logical device which provides an
24 interface between the hypervisor and a managem 24 interface between the hypervisor and a management partition. This interface
25 is like a message passing interface. This mana 25 is like a message passing interface. This management partition is intended
26 to provide an alternative to systems that use 26 to provide an alternative to systems that use a Hardware Management
27 Console (HMC) - based system management. 27 Console (HMC) - based system management.
28 28
29 The primary hardware management solution that 29 The primary hardware management solution that is developed by IBM relies
30 on an appliance server named the Hardware Mana 30 on an appliance server named the Hardware Management Console (HMC),
31 packaged as an external tower or rack-mounted 31 packaged as an external tower or rack-mounted personal computer. In a
32 Power Systems environment, a single HMC can ma 32 Power Systems environment, a single HMC can manage multiple POWER
33 processor-based systems. 33 processor-based systems.
34 34
35 Management Application 35 Management Application
36 ---------------------- 36 ----------------------
37 37
38 In the management partition, a management appl 38 In the management partition, a management application exists which enables
39 a system administrator to configure the system 39 a system administrator to configure the system’s partitioning
40 characteristics via a command line interface ( 40 characteristics via a command line interface (CLI) or Representational
41 State Transfer Application (REST API's). 41 State Transfer Application (REST API's).
42 42
43 The management application runs on a Linux log 43 The management application runs on a Linux logical partition on a
44 POWER8 or newer processor-based server that is 44 POWER8 or newer processor-based server that is virtualized by PowerVM.
45 System configuration, maintenance, and control 45 System configuration, maintenance, and control functions which
46 traditionally require an HMC can be implemente 46 traditionally require an HMC can be implemented in the management
47 application using a combination of HMC to hype 47 application using a combination of HMC to hypervisor interfaces and
48 existing operating system methods. This tool p 48 existing operating system methods. This tool provides a subset of the
49 functions implemented by the HMC and enables b 49 functions implemented by the HMC and enables basic partition configuration.
50 The set of HMC to hypervisor messages supporte 50 The set of HMC to hypervisor messages supported by the management
51 application component are passed to the hyperv 51 application component are passed to the hypervisor over a VMC interface,
52 which is defined below. 52 which is defined below.
53 53
54 The VMC enables the management partition to pr 54 The VMC enables the management partition to provide basic partitioning
55 functions: 55 functions:
56 56
57 - Logical Partitioning Configuration 57 - Logical Partitioning Configuration
58 - Start, and stop actions for individual parti 58 - Start, and stop actions for individual partitions
59 - Display of partition status 59 - Display of partition status
60 - Management of virtual Ethernet 60 - Management of virtual Ethernet
61 - Management of virtual Storage 61 - Management of virtual Storage
62 - Basic system management 62 - Basic system management
63 63
64 Virtual Management Channel (VMC) 64 Virtual Management Channel (VMC)
65 -------------------------------- 65 --------------------------------
66 66
67 A logical device, called the Virtual Managemen 67 A logical device, called the Virtual Management Channel (VMC), is defined
68 for communicating between the management appli 68 for communicating between the management application and the hypervisor. It
69 basically creates the pipes that enable virtua 69 basically creates the pipes that enable virtualization management
70 software. This device is presented to a design 70 software. This device is presented to a designated management partition as
71 a virtual device. 71 a virtual device.
72 72
73 This communication device uses Command/Respons 73 This communication device uses Command/Response Queue (CRQ) and the
74 Remote Direct Memory Access (RDMA) interfaces. 74 Remote Direct Memory Access (RDMA) interfaces. A three-way handshake is
75 defined that must take place to establish that 75 defined that must take place to establish that both the hypervisor and
76 management partition sides of the channel are 76 management partition sides of the channel are running prior to
77 sending/receiving any of the protocol messages 77 sending/receiving any of the protocol messages.
78 78
79 This driver also utilizes Transport Event CRQs 79 This driver also utilizes Transport Event CRQs. CRQ messages are sent
80 when the hypervisor detects one of the peer pa 80 when the hypervisor detects one of the peer partitions has abnormally
81 terminated, or one side has called H_FREE_CRQ 81 terminated, or one side has called H_FREE_CRQ to close their CRQ.
82 Two new classes of CRQ messages are introduced 82 Two new classes of CRQ messages are introduced for the VMC device. VMC
83 Administrative messages are used for each part 83 Administrative messages are used for each partition using the VMC to
84 communicate capabilities to their partner. HMC 84 communicate capabilities to their partner. HMC Interface messages are used
85 for the actual flow of HMC messages between th 85 for the actual flow of HMC messages between the management partition and
86 the hypervisor. As most HMC messages far excee 86 the hypervisor. As most HMC messages far exceed the size of a CRQ buffer,
87 a virtual DMA (RMDA) of the HMC message data i 87 a virtual DMA (RMDA) of the HMC message data is done prior to each HMC
88 Interface CRQ message. Only the management par 88 Interface CRQ message. Only the management partition drives RDMA
89 operations; hypervisors never directly cause t 89 operations; hypervisors never directly cause the movement of message data.
90 90
91 91
92 Terminology 92 Terminology
93 ----------- 93 -----------
94 RDMA 94 RDMA
95 Remote Direct Memory Access is DMA tra 95 Remote Direct Memory Access is DMA transfer from the server to its
96 client or from the server to its partn 96 client or from the server to its partner partition. DMA refers
97 to both physical I/O to and from memor 97 to both physical I/O to and from memory operations and to memory
98 to memory move operations. 98 to memory move operations.
99 CRQ 99 CRQ
100 Command/Response Queue a facility whic 100 Command/Response Queue a facility which is used to communicate
101 between partner partitions. Transport 101 between partner partitions. Transport events which are signaled
102 from the hypervisor to partition are a 102 from the hypervisor to partition are also reported in this queue.
103 103
104 Example Management Partition VMC Driver Interf 104 Example Management Partition VMC Driver Interface
105 ============================================== 105 =================================================
106 106
107 This section provides an example for the manag 107 This section provides an example for the management application
108 implementation where a device driver is used t 108 implementation where a device driver is used to interface to the VMC
109 device. This driver consists of a new device, 109 device. This driver consists of a new device, for example /dev/ibmvmc,
110 which provides interfaces to open, close, read 110 which provides interfaces to open, close, read, write, and perform
111 ioctl’s against the VMC device. 111 ioctl’s against the VMC device.
112 112
113 VMC Interface Initialization 113 VMC Interface Initialization
114 ---------------------------- 114 ----------------------------
115 115
116 The device driver is responsible for initializ 116 The device driver is responsible for initializing the VMC when the driver
117 is loaded. It first creates and initializes th 117 is loaded. It first creates and initializes the CRQ. Next, an exchange of
118 VMC capabilities is performed to indicate the 118 VMC capabilities is performed to indicate the code version and number of
119 resources available in both the management par 119 resources available in both the management partition and the hypervisor.
120 Finally, the hypervisor requests that the mana 120 Finally, the hypervisor requests that the management partition create an
121 initial pool of VMC buffers, one buffer for ea 121 initial pool of VMC buffers, one buffer for each possible HMC connection,
122 which will be used for management application 122 which will be used for management application session initialization.
123 Prior to completion of this initialization seq 123 Prior to completion of this initialization sequence, the device returns
124 EBUSY to open() calls. EIO is returned for all 124 EBUSY to open() calls. EIO is returned for all open() failures.
125 125
126 :: 126 ::
127 127
128 Management Partition Hyperv 128 Management Partition Hypervisor
129 CRQ INIT 129 CRQ INIT
130 -------------------------------------- 130 ---------------------------------------->
131 CRQ INIT COMPLETE 131 CRQ INIT COMPLETE
132 <------------------------------------- 132 <----------------------------------------
133 CAPABILITIES 133 CAPABILITIES
134 -------------------------------------- 134 ---------------------------------------->
135 CAPABILITIES RESPONSE 135 CAPABILITIES RESPONSE
136 <------------------------------------- 136 <----------------------------------------
137 ADD BUFFER (HMC IDX=0,1,..) 137 ADD BUFFER (HMC IDX=0,1,..) _
138 <------------------------------------- 138 <---------------------------------------- |
139 ADD BUFFER RESPONSE 139 ADD BUFFER RESPONSE | - Perform # HMCs Iterations
140 -------------------------------------- 140 ----------------------------------------> -
141 141
142 VMC Interface Open 142 VMC Interface Open
143 ------------------ 143 ------------------
144 144
145 After the basic VMC channel has been initializ 145 After the basic VMC channel has been initialized, an HMC session level
146 connection can be established. The application 146 connection can be established. The application layer performs an open() to
147 the VMC device and executes an ioctl() against 147 the VMC device and executes an ioctl() against it, indicating the HMC ID
148 (32 bytes of data) for this session. If the VM 148 (32 bytes of data) for this session. If the VMC device is in an invalid
149 state, EIO will be returned for the ioctl(). T 149 state, EIO will be returned for the ioctl(). The device driver creates a
150 new HMC session value (ranging from 1 to 255) 150 new HMC session value (ranging from 1 to 255) and HMC index value (starting
151 at index 0 and ranging to 254) for this HMC ID 151 at index 0 and ranging to 254) for this HMC ID. The driver then does an
152 RDMA of the HMC ID to the hypervisor, and then 152 RDMA of the HMC ID to the hypervisor, and then sends an Interface Open
153 message to the hypervisor to establish the ses 153 message to the hypervisor to establish the session over the VMC. After the
154 hypervisor receives this information, it sends 154 hypervisor receives this information, it sends Add Buffer messages to the
155 management partition to seed an initial pool o 155 management partition to seed an initial pool of buffers for the new HMC
156 connection. Finally, the hypervisor sends an I 156 connection. Finally, the hypervisor sends an Interface Open Response
157 message, to indicate that it is ready for norm 157 message, to indicate that it is ready for normal runtime messaging. The
158 following illustrates this VMC flow: 158 following illustrates this VMC flow:
159 159
160 :: 160 ::
161 161
162 Management Partition Hyper 162 Management Partition Hypervisor
163 RDMA HMC ID 163 RDMA HMC ID
164 -------------------------------------- 164 ---------------------------------------->
165 Interface Open 165 Interface Open
166 -------------------------------------- 166 ---------------------------------------->
167 Add Buffer 167 Add Buffer _
168 <------------------------------------- 168 <---------------------------------------- |
169 Add Buffer Response 169 Add Buffer Response | - Perform N Iterations
170 -------------------------------------- 170 ----------------------------------------> -
171 Interface Open Response 171 Interface Open Response
172 <------------------------------------- 172 <----------------------------------------
173 173
174 VMC Interface Runtime 174 VMC Interface Runtime
175 --------------------- 175 ---------------------
176 176
177 During normal runtime, the management applicat 177 During normal runtime, the management application and the hypervisor
178 exchange HMC messages via the Signal VMC messa 178 exchange HMC messages via the Signal VMC message and RDMA operations. When
179 sending data to the hypervisor, the management 179 sending data to the hypervisor, the management application performs a
180 write() to the VMC device, and the driver RDMA 180 write() to the VMC device, and the driver RDMA’s the data to the hypervisor
181 and then sends a Signal Message. If a write() 181 and then sends a Signal Message. If a write() is attempted before VMC
182 device buffers have been made available by the 182 device buffers have been made available by the hypervisor, or no buffers
183 are currently available, EBUSY is returned in 183 are currently available, EBUSY is returned in response to the write(). A
184 write() will return EIO for all other errors, 184 write() will return EIO for all other errors, such as an invalid device
185 state. When the hypervisor sends a message to 185 state. When the hypervisor sends a message to the management, the data is
186 put into a VMC buffer and an Signal Message is 186 put into a VMC buffer and an Signal Message is sent to the VMC driver in
187 the management partition. The driver RDMA’s 187 the management partition. The driver RDMA’s the buffer into the partition
188 and passes the data up to the appropriate mana 188 and passes the data up to the appropriate management application via a
189 read() to the VMC device. The read() request b 189 read() to the VMC device. The read() request blocks if there is no buffer
190 available to read. The management application 190 available to read. The management application may use select() to wait for
191 the VMC device to become ready with data to re 191 the VMC device to become ready with data to read.
192 192
193 :: 193 ::
194 194
195 Management Partition Hyper 195 Management Partition Hypervisor
196 MSG RDMA 196 MSG RDMA
197 -------------------------------------- 197 ---------------------------------------->
198 SIGNAL MSG 198 SIGNAL MSG
199 -------------------------------------- 199 ---------------------------------------->
200 SIGNAL MSG 200 SIGNAL MSG
201 <------------------------------------- 201 <----------------------------------------
202 MSG RDMA 202 MSG RDMA
203 <------------------------------------- 203 <----------------------------------------
204 204
205 VMC Interface Close 205 VMC Interface Close
206 ------------------- 206 -------------------
207 207
208 HMC session level connections are closed by th 208 HMC session level connections are closed by the management partition when
209 the application layer performs a close() again 209 the application layer performs a close() against the device. This action
210 results in an Interface Close message flowing 210 results in an Interface Close message flowing to the hypervisor, which
211 causes the session to be terminated. The devic 211 causes the session to be terminated. The device driver must free any
212 storage allocated for buffers for this HMC con 212 storage allocated for buffers for this HMC connection.
213 213
214 :: 214 ::
215 215
216 Management Partition Hyper 216 Management Partition Hypervisor
217 INTERFACE CLOSE 217 INTERFACE CLOSE
218 -------------------------------------- 218 ---------------------------------------->
219 INTERFACE CLOSE RESPONSE 219 INTERFACE CLOSE RESPONSE
220 <------------------------------------- 220 <----------------------------------------
221 221
222 Additional Information 222 Additional Information
223 ====================== 223 ======================
224 224
225 For more information on the documentation for 225 For more information on the documentation for CRQ Messages, VMC Messages,
226 HMC interface Buffers, and signal messages ple 226 HMC interface Buffers, and signal messages please refer to the Linux on
227 Power Architecture Platform Reference. Section 227 Power Architecture Platform Reference. Section F.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.