1 .. SPDX-License-Identifier: GPL-2.0 2 3 ============================================================ 4 Linux kernel driver for Elastic Network Adapter (ENA) family 5 ============================================================ 6 7 Overview 8 ======== 9 10 ENA is a networking interface designed to make good use of modern CPU 11 features and system architectures. 12 13 The ENA device exposes a lightweight management interface with a 14 minimal set of memory mapped registers and extendible command set 15 through an Admin Queue. 16 17 The driver supports a range of ENA devices, is link-speed independent 18 (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc), and has 19 a negotiated and extendible feature set. 20 21 Some ENA devices support SR-IOV. This driver is used for both the 22 SR-IOV Physical Function (PF) and Virtual Function (VF) devices. 23 24 ENA devices enable high speed and low overhead network traffic 25 processing by providing multiple Tx/Rx queue pairs (the maximum number 26 is advertised by the device via the Admin Queue), a dedicated MSI-X 27 interrupt vector per Tx/Rx queue pair, adaptive interrupt moderation, 28 and CPU cacheline optimized data placement. 29 30 The ENA driver supports industry standard TCP/IP offload features such as 31 checksum offload. Receive-side scaling (RSS) is supported for multi-core 32 scaling. 33 34 The ENA driver and its corresponding devices implement health 35 monitoring mechanisms such as watchdog, enabling the device and driver 36 to recover in a manner transparent to the application, as well as 37 debug logs. 38 39 Some of the ENA devices support a working mode called Low-latency 40 Queue (LLQ), which saves several more microseconds. 41 42 ENA Source Code Directory Structure 43 =================================== 44 45 ================= ====================================================== 46 ena_com.[ch] Management communication layer. This layer is 47 responsible for the handling all the management 48 (admin) communication between the device and the 49 driver. 50 ena_eth_com.[ch] Tx/Rx data path. 51 ena_admin_defs.h Definition of ENA management interface. 52 ena_eth_io_defs.h Definition of ENA data path interface. 53 ena_common_defs.h Common definitions for ena_com layer. 54 ena_regs_defs.h Definition of ENA PCI memory-mapped (MMIO) registers. 55 ena_netdev.[ch] Main Linux kernel driver. 56 ena_ethtool.c ethtool callbacks. 57 ena_xdp.[ch] XDP files 58 ena_pci_id_tbl.h Supported device IDs. 59 ================= ====================================================== 60 61 Management Interface: 62 ===================== 63 64 ENA management interface is exposed by means of: 65 66 - PCIe Configuration Space 67 - Device Registers 68 - Admin Queue (AQ) and Admin Completion Queue (ACQ) 69 - Asynchronous Event Notification Queue (AENQ) 70 71 ENA device MMIO Registers are accessed only during driver 72 initialization and are not used during further normal device 73 operation. 74 75 AQ is used for submitting management commands, and the 76 results/responses are reported asynchronously through ACQ. 77 78 ENA introduces a small set of management commands with room for 79 vendor-specific extensions. Most of the management operations are 80 framed in a generic Get/Set feature command. 81 82 The following admin queue commands are supported: 83 84 - Create I/O submission queue 85 - Create I/O completion queue 86 - Destroy I/O submission queue 87 - Destroy I/O completion queue 88 - Get feature 89 - Set feature 90 - Configure AENQ 91 - Get statistics 92 93 Refer to ena_admin_defs.h for the list of supported Get/Set Feature 94 properties. 95 96 The Asynchronous Event Notification Queue (AENQ) is a uni-directional 97 queue used by the ENA device to send to the driver events that cannot 98 be reported using ACQ. AENQ events are subdivided into groups. Each 99 group may have multiple syndromes, as shown below 100 101 The events are: 102 103 ==================== =============== 104 Group Syndrome 105 ==================== =============== 106 Link state change **X** 107 Fatal error **X** 108 Notification Suspend traffic 109 Notification Resume traffic 110 Keep-Alive **X** 111 ==================== =============== 112 113 ACQ and AENQ share the same MSI-X vector. 114 115 Keep-Alive is a special mechanism that allows monitoring the device's health. 116 A Keep-Alive event is delivered by the device every second. 117 The driver maintains a watchdog (WD) handler which logs the current state and 118 statistics. If the keep-alive events aren't delivered as expected the WD resets 119 the device and the driver. 120 121 Data Path Interface 122 =================== 123 124 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx 125 SQ correspondingly). Each SQ has a completion queue (CQ) associated 126 with it. 127 128 The SQs and CQs are implemented as descriptor rings in contiguous 129 physical memory. 130 131 The ENA driver supports two Queue Operation modes for Tx SQs: 132 133 - **Regular mode:** 134 In this mode the Tx SQs reside in the host's memory. The ENA 135 device fetches the ENA Tx descriptors and packet data from host 136 memory. 137 138 - **Low Latency Queue (LLQ) mode or "push-mode":** 139 In this mode the driver pushes the transmit descriptors and the 140 first 96 bytes of the packet directly to the ENA device memory 141 space. The rest of the packet payload is fetched by the 142 device. For this operation mode, the driver uses a dedicated PCI 143 device memory BAR, which is mapped with write-combine capability. 144 145 **Note that** not all ENA devices support LLQ, and this feature is negotiated 146 with the device upon initialization. If the ENA device does not 147 support LLQ mode, the driver falls back to the regular mode. 148 149 The Rx SQs support only the regular mode. 150 151 The driver supports multi-queue for both Tx and Rx. This has various 152 benefits: 153 154 - Reduced CPU/thread/process contention on a given Ethernet interface. 155 - Cache miss rate on completion is reduced, particularly for data 156 cache lines that hold the sk_buff structures. 157 - Increased process-level parallelism when handling received packets. 158 - Increased data cache hit rate, by steering kernel processing of 159 packets to the CPU, where the application thread consuming the 160 packet is running. 161 - In hardware interrupt re-direction. 162 163 Interrupt Modes 164 =============== 165 166 The driver assigns a single MSI-X vector per queue pair (for both Tx 167 and Rx directions). The driver assigns an additional dedicated MSI-X vector 168 for management (for ACQ and AENQ). 169 170 Management interrupt registration is performed when the Linux kernel 171 probes the adapter, and it is de-registered when the adapter is 172 removed. I/O queue interrupt registration is performed when the Linux 173 interface of the adapter is opened, and it is de-registered when the 174 interface is closed. 175 176 The management interrupt is named:: 177 178 ena-mgmnt@pci:<PCI domain:bus:slot.function> 179 180 and for each queue pair, an interrupt is named:: 181 182 <interface name>-Tx-Rx-<queue index> 183 184 The ENA device operates in auto-mask and auto-clear interrupt 185 modes. That is, once MSI-X is delivered to the host, its Cause bit is 186 automatically cleared and the interrupt is masked. The interrupt is 187 unmasked by the driver after NAPI processing is complete. 188 189 Interrupt Moderation 190 ==================== 191 192 ENA driver and device can operate in conventional or adaptive interrupt 193 moderation mode. 194 195 **In conventional mode** the driver instructs device to postpone interrupt 196 posting according to static interrupt delay value. The interrupt delay 197 value can be configured through `ethtool(8)`. The following `ethtool` 198 parameters are supported by the driver: ``tx-usecs``, ``rx-usecs`` 199 200 **In adaptive interrupt** moderation mode the interrupt delay value is 201 updated by the driver dynamically and adjusted every NAPI cycle 202 according to the traffic nature. 203 204 Adaptive coalescing can be switched on/off through `ethtool(8)`'s 205 :code:`adaptive_rx on|off` parameter. 206 207 More information about Adaptive Interrupt Moderation (DIM) can be found in 208 Documentation/networking/net_dim.rst 209 210 .. _`RX copybreak`: 211 212 RX copybreak 213 ============ 214 215 The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK 216 and can be configured by the ETHTOOL_STUNABLE command of the 217 SIOCETHTOOL ioctl. 218 219 This option controls the maximum packet length for which the RX 220 descriptor it was received on would be recycled. When a packet smaller 221 than RX copybreak bytes is received, it is copied into a new memory 222 buffer and the RX descriptor is returned to HW. 223 224 Statistics 225 ========== 226 227 The user can obtain ENA device and driver statistics using `ethtool`. 228 The driver can collect regular or extended statistics (including 229 per-queue stats) from the device. 230 231 In addition the driver logs the stats to syslog upon device reset. 232 233 On supported instance types, the statistics will also include the 234 ENA Express data (fields prefixed with `ena_srd`). For a complete 235 documentation of ENA Express data refer to 236 https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-express.html#ena-express-monitor 237 238 MTU 239 === 240 241 The driver supports an arbitrarily large MTU with a maximum that is 242 negotiated with the device. The driver configures MTU using the 243 SetFeature command (ENA_ADMIN_MTU property). The user can change MTU 244 via `ip(8)` and similar legacy tools. 245 246 Stateless Offloads 247 ================== 248 249 The ENA driver supports: 250 251 - IPv4 header checksum offload 252 - TCP/UDP over IPv4/IPv6 checksum offloads 253 254 RSS 255 === 256 257 - The ENA device supports RSS that allows flexible Rx traffic 258 steering. 259 - Toeplitz and CRC32 hash functions are supported. 260 - Different combinations of L2/L3/L4 fields can be configured as 261 inputs for hash functions. 262 - The driver configures RSS settings using the AQ SetFeature command 263 (ENA_ADMIN_RSS_HASH_FUNCTION, ENA_ADMIN_RSS_HASH_INPUT and 264 ENA_ADMIN_RSS_INDIRECTION_TABLE_CONFIG properties). 265 - If the NETIF_F_RXHASH flag is set, the 32-bit result of the hash 266 function delivered in the Rx CQ descriptor is set in the received 267 SKB. 268 - The user can provide a hash key, hash function, and configure the 269 indirection table through `ethtool(8)`. 270 271 DATA PATH 272 ========= 273 274 Tx 275 -- 276 277 :code:`ena_start_xmit()` is called by the stack. This function does the following: 278 279 - Maps data buffers (``skb->data`` and frags). 280 - Populates ``ena_buf`` for the push buffer (if the driver and device are 281 in push mode). 282 - Prepares ENA bufs for the remaining frags. 283 - Allocates a new request ID from the empty ``req_id`` ring. The request 284 ID is the index of the packet in the Tx info. This is used for 285 out-of-order Tx completions. 286 - Adds the packet to the proper place in the Tx ring. 287 - Calls :code:`ena_com_prepare_tx()`, an ENA communication layer that converts 288 the ``ena_bufs`` to ENA descriptors (and adds meta ENA descriptors as 289 needed). 290 291 * This function also copies the ENA descriptors and the push buffer 292 to the Device memory space (if in push mode). 293 294 - Writes a doorbell to the ENA device. 295 - When the ENA device finishes sending the packet, a completion 296 interrupt is raised. 297 - The interrupt handler schedules NAPI. 298 - The :code:`ena_clean_tx_irq()` function is called. This function handles the 299 completion descriptors generated by the ENA, with a single 300 completion descriptor per completed packet. 301 302 * ``req_id`` is retrieved from the completion descriptor. The ``tx_info`` of 303 the packet is retrieved via the ``req_id``. The data buffers are 304 unmapped and ``req_id`` is returned to the empty ``req_id`` ring. 305 * The function stops when the completion descriptors are completed or 306 the budget is reached. 307 308 Rx 309 -- 310 311 - When a packet is received from the ENA device. 312 - The interrupt handler schedules NAPI. 313 - The :code:`ena_clean_rx_irq()` function is called. This function calls 314 :code:`ena_com_rx_pkt()`, an ENA communication layer function, which returns the 315 number of descriptors used for a new packet, and zero if 316 no new packet is found. 317 - :code:`ena_rx_skb()` checks packet length: 318 319 * If the packet is small (len < rx_copybreak), the driver allocates 320 a SKB for the new packet, and copies the packet payload into the 321 SKB data buffer. 322 323 - In this way the original data buffer is not passed to the stack 324 and is reused for future Rx packets. 325 326 * Otherwise the function unmaps the Rx buffer, sets the first 327 descriptor as `skb`'s linear part and the other descriptors as the 328 `skb`'s frags. 329 330 - The new SKB is updated with the necessary information (protocol, 331 checksum hw verify result, etc), and then passed to the network 332 stack, using the NAPI interface function :code:`napi_gro_receive()`. 333 334 Dynamic RX Buffers (DRB) 335 ------------------------ 336 337 Each RX descriptor in the RX ring is a single memory page (which is either 4KB 338 or 16KB long depending on system's configurations). 339 To reduce the memory allocations required when dealing with a high rate of small 340 packets, the driver tries to reuse the remaining RX descriptor's space if more 341 than 2KB of this page remain unused. 342 343 A simple example of this mechanism is the following sequence of events: 344 345 :: 346 347 1. Driver allocates page-sized RX buffer and passes it to hardware 348 +----------------------+ 349 |4KB RX Buffer | 350 +----------------------+ 351 352 2. A 300Bytes packet is received on this buffer 353 354 3. The driver increases the ref count on this page and returns it back to 355 HW as an RX buffer of size 4KB - 300Bytes = 3796 Bytes 356 +----+--------------------+ 357 |****|3796 Bytes RX Buffer| 358 +----+--------------------+ 359 360 This mechanism isn't used when an XDP program is loaded, or when the 361 RX packet is less than rx_copybreak bytes (in which case the packet is 362 copied out of the RX buffer into the linear part of a new skb allocated 363 for it and the RX buffer remains the same size, see `RX copybreak`_).
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.