1 .. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) 2 .. include:: <isonum.txt> 3 4 =========================================== 5 Network Flow Processor (NFP) Kernel Drivers 6 =========================================== 7 8 :Copyright: |copy| 2019, Netronome Systems, Inc. 9 :Copyright: |copy| 2022, Corigine, Inc. 10 11 Contents 12 ======== 13 14 - `Overview`_ 15 - `Acquiring Firmware`_ 16 - `Devlink Info`_ 17 - `Configure Device`_ 18 - `Statistics`_ 19 20 Overview 21 ======== 22 23 This driver supports Netronome and Corigine's line of Network Flow Processor 24 devices, including the NFP3800, NFP4000, NFP5000, and NFP6000 models, which 25 are also incorporated in the companies' family of Agilio SmartNICs. The SR-IOV 26 physical and virtual functions for these devices are supported by the driver. 27 28 Acquiring Firmware 29 ================== 30 31 The NFP3800, NFP4000 and NFP6000 devices require application specific firmware 32 to function. Application firmware can be located either on the host file system 33 or in the device flash (if supported by management firmware). 34 35 Firmware files on the host filesystem contain card type (`AMDA-*` string), media 36 config etc. They should be placed in `/lib/firmware/netronome` directory to 37 load firmware from the host file system. 38 39 Firmware for basic NIC operation is available in the upstream 40 `linux-firmware.git` repository. 41 42 A more comprehensive list of firmware can be downloaded from the 43 `Corigine Support site <https://www.corigine.com/DPUDownload.html>`_. 44 45 Firmware in NVRAM 46 ----------------- 47 48 Recent versions of management firmware supports loading application 49 firmware from flash when the host driver gets probed. The firmware loading 50 policy configuration may be used to configure this feature appropriately. 51 52 Devlink or ethtool can be used to update the application firmware on the device 53 flash by providing the appropriate `nic_AMDA*.nffw` file to the respective 54 command. Users need to take care to write the correct firmware image for the 55 card and media configuration to flash. 56 57 Available storage space in flash depends on the card being used. 58 59 Dealing with multiple projects 60 ------------------------------ 61 62 NFP hardware is fully programmable therefore there can be different 63 firmware images targeting different applications. 64 65 When using application firmware from host, we recommend placing 66 actual firmware files in application-named subdirectories in 67 `/lib/firmware/netronome` and linking the desired files, e.g.:: 68 69 $ tree /lib/firmware/netronome/ 70 /lib/firmware/netronome/ 71 ├── bpf 72 │ ├── nic_AMDA0081-0001_1x40.nffw 73 │ └── nic_AMDA0081-0001_4x10.nffw 74 ├── flower 75 │ ├── nic_AMDA0081-0001_1x40.nffw 76 │ └── nic_AMDA0081-0001_4x10.nffw 77 ├── nic 78 │ ├── nic_AMDA0081-0001_1x40.nffw 79 │ └── nic_AMDA0081-0001_4x10.nffw 80 ├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw 81 └── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw 82 83 3 directories, 8 files 84 85 You may need to use hard instead of symbolic links on distributions 86 which use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu). 87 88 After changing firmware files you may need to regenerate the initramfs 89 image. Initramfs contains drivers and firmware files your system may 90 need to boot. Refer to the documentation of your distribution to find 91 out how to update initramfs. Good indication of stale initramfs 92 is system loading wrong driver or firmware on boot, but when driver is 93 later reloaded manually everything works correctly. 94 95 Selecting firmware per device 96 ----------------------------- 97 98 Most commonly all cards on the system use the same type of firmware. 99 If you want to load a specific firmware image for a specific card, you 100 can use either the PCI bus address or serial number. The driver will 101 print which files it's looking for when it recognizes a NFP device:: 102 103 nfp: Looking for firmware file in order of priority: 104 nfp: netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found 105 nfp: netronome/pci-0000:02:00.0.nffw: not found 106 nfp: netronome/nic_AMDA0081-0001_1x40.nffw: found, loading... 107 108 In this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw* 109 or *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this 110 firmware file will take precedence over `nic_AMDA*` files. 111 112 Note that `serial-*` and `pci-*` files are **not** automatically included 113 in initramfs, you will have to refer to documentation of appropriate tools 114 to find out how to include them. 115 116 Running firmware version 117 ------------------------ 118 119 The version of the loaded firmware for a particular <netdev> interface, 120 (e.g. enp4s0), or an interface's port <netdev port> (e.g. enp4s0np0) can 121 be displayed with the ethtool command:: 122 123 $ ethtool -i <netdev> 124 125 Firmware loading policy 126 ----------------------- 127 128 Firmware loading policy is controlled via three HWinfo parameters 129 stored as key value pairs in the device flash: 130 131 app_fw_from_flash 132 Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or 133 the 'Preferred' (2) firmware. When 'Preferred' is selected, the management 134 firmware makes the decision over which firmware will be loaded by comparing 135 versions of the flash firmware and the host supplied firmware. 136 This variable is configurable using the 'fw_load_policy' 137 devlink parameter. 138 139 abi_drv_reset 140 Defines if the driver should reset the firmware when 141 the driver is probed, either 'Disk' (0) if firmware was found on disk, 142 'Always' (1) reset or 'Never' (2) reset. Note that the device is always 143 reset on driver unload if firmware was loaded when the driver was probed. 144 This variable is configurable using the 'reset_dev_on_drv_probe' 145 devlink parameter. 146 147 abi_drv_load_ifc 148 Defines a list of PF devices allowed to load FW on the device. 149 This variable is not currently user configurable. 150 151 Devlink Info 152 ============ 153 154 The devlink info command displays the running and stored firmware versions 155 on the device, serial number and board information. 156 157 Devlink info command example (replace PCI address):: 158 159 $ devlink dev info pci/0000:03:00.0 160 pci/0000:03:00.0: 161 driver nfp 162 serial_number CSAAMDA2001-1003000111 163 versions: 164 fixed: 165 board.id AMDA2001-1003 166 board.rev 01 167 board.manufacture CSA 168 board.model mozart 169 running: 170 fw.mgmt 22.10.0-rc3 171 fw.cpld 0x1000003 172 fw.app nic-22.09.0 173 chip.init AMDA-2001-1003 1003000111 174 stored: 175 fw.bundle_id bspbundle_1003000111 176 fw.mgmt 22.10.0-rc3 177 fw.cpld 0x0 178 chip.init AMDA-2001-1003 1003000111 179 180 Configure Device 181 ================ 182 183 This section explains how to use Agilio SmartNICs running basic NIC firmware. 184 185 Configure interface link-speed 186 ------------------------------ 187 The following steps explains how to change between 10G mode and 25G mode on 188 Agilio CX 2x25GbE cards. The changing of port speed must be done in order, 189 port 0 (p0) must be set to 10G before port 1 (p1) may be set to 10G. 190 191 Down the respective interface(s):: 192 193 $ ip link set dev <netdev port 0> down 194 $ ip link set dev <netdev port 1> down 195 196 Set interface link-speed to 10G:: 197 198 $ ethtool -s <netdev port 0> speed 10000 199 $ ethtool -s <netdev port 1> speed 10000 200 201 Set interface link-speed to 25G:: 202 203 $ ethtool -s <netdev port 0> speed 25000 204 $ ethtool -s <netdev port 1> speed 25000 205 206 Reload driver for changes to take effect:: 207 208 $ rmmod nfp; modprobe nfp 209 210 Configure interface Maximum Transmission Unit (MTU) 211 --------------------------------------------------- 212 213 The MTU of interfaces can temporarily be set using the iproute2, ip link or 214 ifconfig tools. Note that this change will not persist. Setting this via 215 Network Manager, or another appropriate OS configuration tool, is 216 recommended as changes to the MTU using Network Manager can be made to 217 persist. 218 219 Set interface MTU to 9000 bytes:: 220 221 $ ip link set dev <netdev port> mtu 9000 222 223 It is the responsibility of the user or the orchestration layer to set 224 appropriate MTU values when handling jumbo frames or utilizing tunnels. For 225 example, if packets sent from a VM are to be encapsulated on the card and 226 egress a physical port, then the MTU of the VF should be set to lower than 227 that of the physical port to account for the extra bytes added by the 228 additional header. If a setup is expected to see fallback traffic between 229 the SmartNIC and the kernel then the user should also ensure that the PF MTU 230 is appropriately set to avoid unexpected drops on this path. 231 232 Configure Forward Error Correction (FEC) modes 233 ---------------------------------------------- 234 235 Agilio SmartNICs support FEC mode configuration, e.g. Auto, Firecode Base-R, 236 ReedSolomon and Off modes. Each physical port's FEC mode can be set 237 independently using ethtool. The supported FEC modes for an interface can 238 be viewed using:: 239 240 $ ethtool <netdev> 241 242 The currently configured FEC mode can be viewed using:: 243 244 $ ethtool --show-fec <netdev> 245 246 To force the FEC mode for a particular port, auto-negotiation must be disabled 247 (see the `Auto-negotiation`_ section). An example of how to set the FEC mode 248 to Reed-Solomon is:: 249 250 $ ethtool --set-fec <netdev> encoding rs 251 252 Auto-negotiation 253 ---------------- 254 255 To change auto-negotiation settings, the link must first be put down. After the 256 link is down, auto-negotiation can be enabled or disabled using:: 257 258 ethtool -s <netdev> autoneg <on|off> 259 260 Statistics 261 ========== 262 263 Following device statistics are available through the ``ethtool -S`` interface: 264 265 .. flat-table:: NFP device statistics 266 :header-rows: 1 267 :widths: 3 1 11 268 269 * - Name 270 - ID 271 - Meaning 272 273 * - dev_rx_discards 274 - 1 275 - Packet can be discarded on the RX path for one of the following reasons: 276 277 * The NIC is not in promisc mode, and the destination MAC address 278 doesn't match the interfaces' MAC address. 279 * The received packet is larger than the max buffer size on the host. 280 I.e. it exceeds the Layer 3 MRU. 281 * There is no freelist descriptor available on the host for the packet. 282 It is likely that the NIC couldn't cache one in time. 283 * A BPF program discarded the packet. 284 * The datapath drop action was executed. 285 * The MAC discarded the packet due to lack of ingress buffer space 286 on the NIC. 287 288 * - dev_rx_errors 289 - 2 290 - A packet can be counted (and dropped) as RX error for the following 291 reasons: 292 293 * A problem with the VEB lookup (only when SR-IOV is used). 294 * A physical layer problem that causes Ethernet errors, like FCS or 295 alignment errors. The cause is usually faulty cables or SFPs. 296 297 * - dev_rx_bytes 298 - 3 299 - Total number of bytes received. 300 301 * - dev_rx_uc_bytes 302 - 4 303 - Unicast bytes received. 304 305 * - dev_rx_mc_bytes 306 - 5 307 - Multicast bytes received. 308 309 * - dev_rx_bc_bytes 310 - 6 311 - Broadcast bytes received. 312 313 * - dev_rx_pkts 314 - 7 315 - Total number of packets received. 316 317 * - dev_rx_mc_pkts 318 - 8 319 - Multicast packets received. 320 321 * - dev_rx_bc_pkts 322 - 9 323 - Broadcast packets received. 324 325 * - dev_tx_discards 326 - 10 327 - A packet can be discarded in the TX direction if the MAC is 328 being flow controlled and the NIC runs out of TX queue space. 329 330 * - dev_tx_errors 331 - 11 332 - A packet can be counted as TX error (and dropped) for one for the 333 following reasons: 334 335 * The packet is an LSO segment, but the Layer 3 or Layer 4 offset 336 could not be determined. Therefore LSO could not continue. 337 * An invalid packet descriptor was received over PCIe. 338 * The packet Layer 3 length exceeds the device MTU. 339 * An error on the MAC/physical layer. Usually due to faulty cables or 340 SFPs. 341 * A CTM buffer could not be allocated. 342 * The packet offset was incorrect and could not be fixed by the NIC. 343 344 * - dev_tx_bytes 345 - 12 346 - Total number of bytes transmitted. 347 348 * - dev_tx_uc_bytes 349 - 13 350 - Unicast bytes transmitted. 351 352 * - dev_tx_mc_bytes 353 - 14 354 - Multicast bytes transmitted. 355 356 * - dev_tx_bc_bytes 357 - 15 358 - Broadcast bytes transmitted. 359 360 * - dev_tx_pkts 361 - 16 362 - Total number of packets transmitted. 363 364 * - dev_tx_mc_pkts 365 - 17 366 - Multicast packets transmitted. 367 368 * - dev_tx_bc_pkts 369 - 18 370 - Broadcast packets transmitted. 371 372 Note that statistics unknown to the driver will be displayed as 373 ``dev_unknown_stat$ID``, where ``$ID`` refers to the second column 374 above.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.