1 .. SPDX-License-Identifier: GPL-2.0+ 2 3 ====================================================== 4 IBM Virtual Management Channel Kernel Driver (IBMVMC) 5 ====================================================== 6 7 :Authors: 8 Dave Engebretsen <engebret@us.ibm.com>, 9 Adam Reznechek <adreznec@linux.vnet.ibm.com>, 10 Steven Royer <seroyer@linux.vnet.ibm.com>, 11 Bryant G. Ly <bryantly@linux.vnet.ibm.com>, 12 13 Introduction 14 ============ 15 16 Note: Knowledge of virtualization technology is required to understand 17 this document. 18 19 A good reference document would be: 20 21 https://openpowerfoundation.org/wp-content/uploads/2016/05/LoPAPR_DRAFT_v11_24March2016_cmt1.pdf 22 23 The Virtual Management Channel (VMC) is a logical device which provides an 24 interface between the hypervisor and a management partition. This interface 25 is like a message passing interface. This management partition is intended 26 to provide an alternative to systems that use a Hardware Management 27 Console (HMC) - based system management. 28 29 The primary hardware management solution that is developed by IBM relies 30 on an appliance server named the Hardware Management Console (HMC), 31 packaged as an external tower or rack-mounted personal computer. In a 32 Power Systems environment, a single HMC can manage multiple POWER 33 processor-based systems. 34 35 Management Application 36 ---------------------- 37 38 In the management partition, a management application exists which enables 39 a system administrator to configure the system’s partitioning 40 characteristics via a command line interface (CLI) or Representational 41 State Transfer Application (REST API's). 42 43 The management application runs on a Linux logical partition on a 44 POWER8 or newer processor-based server that is virtualized by PowerVM. 45 System configuration, maintenance, and control functions which 46 traditionally require an HMC can be implemented in the management 47 application using a combination of HMC to hypervisor interfaces and 48 existing operating system methods. This tool provides a subset of the 49 functions implemented by the HMC and enables basic partition configuration. 50 The set of HMC to hypervisor messages supported by the management 51 application component are passed to the hypervisor over a VMC interface, 52 which is defined below. 53 54 The VMC enables the management partition to provide basic partitioning 55 functions: 56 57 - Logical Partitioning Configuration 58 - Start, and stop actions for individual partitions 59 - Display of partition status 60 - Management of virtual Ethernet 61 - Management of virtual Storage 62 - Basic system management 63 64 Virtual Management Channel (VMC) 65 -------------------------------- 66 67 A logical device, called the Virtual Management Channel (VMC), is defined 68 for communicating between the management application and the hypervisor. It 69 basically creates the pipes that enable virtualization management 70 software. This device is presented to a designated management partition as 71 a virtual device. 72 73 This communication device uses Command/Response Queue (CRQ) and the 74 Remote Direct Memory Access (RDMA) interfaces. A three-way handshake is 75 defined that must take place to establish that both the hypervisor and 76 management partition sides of the channel are running prior to 77 sending/receiving any of the protocol messages. 78 79 This driver also utilizes Transport Event CRQs. CRQ messages are sent 80 when the hypervisor detects one of the peer partitions has abnormally 81 terminated, or one side has called H_FREE_CRQ to close their CRQ. 82 Two new classes of CRQ messages are introduced for the VMC device. VMC 83 Administrative messages are used for each partition using the VMC to 84 communicate capabilities to their partner. HMC Interface messages are used 85 for the actual flow of HMC messages between the management partition and 86 the hypervisor. As most HMC messages far exceed the size of a CRQ buffer, 87 a virtual DMA (RMDA) of the HMC message data is done prior to each HMC 88 Interface CRQ message. Only the management partition drives RDMA 89 operations; hypervisors never directly cause the movement of message data. 90 91 92 Terminology 93 ----------- 94 RDMA 95 Remote Direct Memory Access is DMA transfer from the server to its 96 client or from the server to its partner partition. DMA refers 97 to both physical I/O to and from memory operations and to memory 98 to memory move operations. 99 CRQ 100 Command/Response Queue a facility which is used to communicate 101 between partner partitions. Transport events which are signaled 102 from the hypervisor to partition are also reported in this queue. 103 104 Example Management Partition VMC Driver Interface 105 ================================================= 106 107 This section provides an example for the management application 108 implementation where a device driver is used to interface to the VMC 109 device. This driver consists of a new device, for example /dev/ibmvmc, 110 which provides interfaces to open, close, read, write, and perform 111 ioctl’s against the VMC device. 112 113 VMC Interface Initialization 114 ---------------------------- 115 116 The device driver is responsible for initializing the VMC when the driver 117 is loaded. It first creates and initializes the CRQ. Next, an exchange of 118 VMC capabilities is performed to indicate the code version and number of 119 resources available in both the management partition and the hypervisor. 120 Finally, the hypervisor requests that the management partition create an 121 initial pool of VMC buffers, one buffer for each possible HMC connection, 122 which will be used for management application session initialization. 123 Prior to completion of this initialization sequence, the device returns 124 EBUSY to open() calls. EIO is returned for all open() failures. 125 126 :: 127 128 Management Partition Hypervisor 129 CRQ INIT 130 ----------------------------------------> 131 CRQ INIT COMPLETE 132 <---------------------------------------- 133 CAPABILITIES 134 ----------------------------------------> 135 CAPABILITIES RESPONSE 136 <---------------------------------------- 137 ADD BUFFER (HMC IDX=0,1,..) _ 138 <---------------------------------------- | 139 ADD BUFFER RESPONSE | - Perform # HMCs Iterations 140 ----------------------------------------> - 141 142 VMC Interface Open 143 ------------------ 144 145 After the basic VMC channel has been initialized, an HMC session level 146 connection can be established. The application layer performs an open() to 147 the VMC device and executes an ioctl() against it, indicating the HMC ID 148 (32 bytes of data) for this session. If the VMC device is in an invalid 149 state, EIO will be returned for the ioctl(). The device driver creates a 150 new HMC session value (ranging from 1 to 255) and HMC index value (starting 151 at index 0 and ranging to 254) for this HMC ID. The driver then does an 152 RDMA of the HMC ID to the hypervisor, and then sends an Interface Open 153 message to the hypervisor to establish the session over the VMC. After the 154 hypervisor receives this information, it sends Add Buffer messages to the 155 management partition to seed an initial pool of buffers for the new HMC 156 connection. Finally, the hypervisor sends an Interface Open Response 157 message, to indicate that it is ready for normal runtime messaging. The 158 following illustrates this VMC flow: 159 160 :: 161 162 Management Partition Hypervisor 163 RDMA HMC ID 164 ----------------------------------------> 165 Interface Open 166 ----------------------------------------> 167 Add Buffer _ 168 <---------------------------------------- | 169 Add Buffer Response | - Perform N Iterations 170 ----------------------------------------> - 171 Interface Open Response 172 <---------------------------------------- 173 174 VMC Interface Runtime 175 --------------------- 176 177 During normal runtime, the management application and the hypervisor 178 exchange HMC messages via the Signal VMC message and RDMA operations. When 179 sending data to the hypervisor, the management application performs a 180 write() to the VMC device, and the driver RDMA’s the data to the hypervisor 181 and then sends a Signal Message. If a write() is attempted before VMC 182 device buffers have been made available by the hypervisor, or no buffers 183 are currently available, EBUSY is returned in response to the write(). A 184 write() will return EIO for all other errors, such as an invalid device 185 state. When the hypervisor sends a message to the management, the data is 186 put into a VMC buffer and an Signal Message is sent to the VMC driver in 187 the management partition. The driver RDMA’s the buffer into the partition 188 and passes the data up to the appropriate management application via a 189 read() to the VMC device. The read() request blocks if there is no buffer 190 available to read. The management application may use select() to wait for 191 the VMC device to become ready with data to read. 192 193 :: 194 195 Management Partition Hypervisor 196 MSG RDMA 197 ----------------------------------------> 198 SIGNAL MSG 199 ----------------------------------------> 200 SIGNAL MSG 201 <---------------------------------------- 202 MSG RDMA 203 <---------------------------------------- 204 205 VMC Interface Close 206 ------------------- 207 208 HMC session level connections are closed by the management partition when 209 the application layer performs a close() against the device. This action 210 results in an Interface Close message flowing to the hypervisor, which 211 causes the session to be terminated. The device driver must free any 212 storage allocated for buffers for this HMC connection. 213 214 :: 215 216 Management Partition Hypervisor 217 INTERFACE CLOSE 218 ----------------------------------------> 219 INTERFACE CLOSE RESPONSE 220 <---------------------------------------- 221 222 Additional Information 223 ====================== 224 225 For more information on the documentation for CRQ Messages, VMC Messages, 226 HMC interface Buffers, and signal messages please refer to the Linux on 227 Power Architecture Platform Reference. Section F.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.