1 .. SPDX-License-Identifier: GPL-2.0 2 .. include:: <isonum.txt> 3 4 ============================================= 5 Chelsio N210 10Gb Ethernet Network Controller 6 ============================================= 7 8 Driver Release Notes for Linux 9 10 Version 2.1.1 11 12 June 20, 2005 13 14 .. Contents 15 16 INTRODUCTION 17 FEATURES 18 PERFORMANCE 19 DRIVER MESSAGES 20 KNOWN ISSUES 21 SUPPORT 22 23 24 Introduction 25 ============ 26 27 This document describes the Linux driver for Chelsio 10Gb Ethernet Network 28 Controller. This driver supports the Chelsio N210 NIC and is backward 29 compatible with the Chelsio N110 model 10Gb NICs. 30 31 32 Features 33 ======== 34 35 Adaptive Interrupts (adaptive-rx) 36 --------------------------------- 37 38 This feature provides an adaptive algorithm that adjusts the interrupt 39 coalescing parameters, allowing the driver to dynamically adapt the latency 40 settings to achieve the highest performance during various types of network 41 load. 42 43 The interface used to control this feature is ethtool. Please see the 44 ethtool manpage for additional usage information. 45 46 By default, adaptive-rx is disabled. 47 To enable adaptive-rx:: 48 49 ethtool -C <interface> adaptive-rx on 50 51 To disable adaptive-rx, use ethtool:: 52 53 ethtool -C <interface> adaptive-rx off 54 55 After disabling adaptive-rx, the timer latency value will be set to 50us. 56 You may set the timer latency after disabling adaptive-rx:: 57 58 ethtool -C <interface> rx-usecs <microseconds> 59 60 An example to set the timer latency value to 100us on eth0:: 61 62 ethtool -C eth0 rx-usecs 100 63 64 You may also provide a timer latency value while disabling adaptive-rx:: 65 66 ethtool -C <interface> adaptive-rx off rx-usecs <microseconds> 67 68 If adaptive-rx is disabled and a timer latency value is specified, the timer 69 will be set to the specified value until changed by the user or until 70 adaptive-rx is enabled. 71 72 To view the status of the adaptive-rx and timer latency values:: 73 74 ethtool -c <interface> 75 76 77 TCP Segmentation Offloading (TSO) Support 78 ----------------------------------------- 79 80 This feature, also known as "large send", enables a system's protocol stack 81 to offload portions of outbound TCP processing to a network interface card 82 thereby reducing system CPU utilization and enhancing performance. 83 84 The interface used to control this feature is ethtool version 1.8 or higher. 85 Please see the ethtool manpage for additional usage information. 86 87 By default, TSO is enabled. 88 To disable TSO:: 89 90 ethtool -K <interface> tso off 91 92 To enable TSO:: 93 94 ethtool -K <interface> tso on 95 96 To view the status of TSO:: 97 98 ethtool -k <interface> 99 100 101 Performance 102 =========== 103 104 The following information is provided as an example of how to change system 105 parameters for "performance tuning" an what value to use. You may or may not 106 want to change these system parameters, depending on your server/workstation 107 application. Doing so is not warranted in any way by Chelsio Communications, 108 and is done at "YOUR OWN RISK". Chelsio will not be held responsible for loss 109 of data or damage to equipment. 110 111 Your distribution may have a different way of doing things, or you may prefer 112 a different method. These commands are shown only to provide an example of 113 what to do and are by no means definitive. 114 115 Making any of the following system changes will only last until you reboot 116 your system. You may want to write a script that runs at boot-up which 117 includes the optimal settings for your system. 118 119 Setting PCI Latency Timer:: 120 121 setpci -d 1425:: 122 123 * 0x0c.l=0x0000F800 124 125 Disabling TCP timestamp:: 126 127 sysctl -w net.ipv4.tcp_timestamps=0 128 129 Disabling SACK:: 130 131 sysctl -w net.ipv4.tcp_sack=0 132 133 Setting large number of incoming connection requests:: 134 135 sysctl -w net.ipv4.tcp_max_syn_backlog=3000 136 137 Setting maximum receive socket buffer size:: 138 139 sysctl -w net.core.rmem_max=1024000 140 141 Setting maximum send socket buffer size:: 142 143 sysctl -w net.core.wmem_max=1024000 144 145 Set smp_affinity (on a multiprocessor system) to a single CPU:: 146 147 echo 1 > /proc/irq/<interrupt_number>/smp_affinity 148 149 Setting default receive socket buffer size:: 150 151 sysctl -w net.core.rmem_default=524287 152 153 Setting default send socket buffer size:: 154 155 sysctl -w net.core.wmem_default=524287 156 157 Setting maximum option memory buffers:: 158 159 sysctl -w net.core.optmem_max=524287 160 161 Setting maximum backlog (# of unprocessed packets before kernel drops):: 162 163 sysctl -w net.core.netdev_max_backlog=300000 164 165 Setting TCP read buffers (min/default/max):: 166 167 sysctl -w net.ipv4.tcp_rmem="10000000 10000000 10000000" 168 169 Setting TCP write buffers (min/pressure/max):: 170 171 sysctl -w net.ipv4.tcp_wmem="10000000 10000000 10000000" 172 173 Setting TCP buffer space (min/pressure/max):: 174 175 sysctl -w net.ipv4.tcp_mem="10000000 10000000 10000000" 176 177 TCP window size for single connections: 178 179 The receive buffer (RX_WINDOW) size must be at least as large as the 180 Bandwidth-Delay Product of the communication link between the sender and 181 receiver. Due to the variations of RTT, you may want to increase the buffer 182 size up to 2 times the Bandwidth-Delay Product. Reference page 289 of 183 "TCP/IP Illustrated, Volume 1, The Protocols" by W. Richard Stevens. 184 185 At 10Gb speeds, use the following formula:: 186 187 RX_WINDOW >= 1.25MBytes * RTT(in milliseconds) 188 Example for RTT with 100us: RX_WINDOW = (1,250,000 * 0.1) = 125,000 189 190 RX_WINDOW sizes of 256KB - 512KB should be sufficient. 191 192 Setting the min, max, and default receive buffer (RX_WINDOW) size:: 193 194 sysctl -w net.ipv4.tcp_rmem="<min> <default> <max>" 195 196 TCP window size for multiple connections: 197 The receive buffer (RX_WINDOW) size may be calculated the same as single 198 connections, but should be divided by the number of connections. The 199 smaller window prevents congestion and facilitates better pacing, 200 especially if/when MAC level flow control does not work well or when it is 201 not supported on the machine. Experimentation may be necessary to attain 202 the correct value. This method is provided as a starting point for the 203 correct receive buffer size. 204 205 Setting the min, max, and default receive buffer (RX_WINDOW) size is 206 performed in the same manner as single connection. 207 208 209 Driver Messages 210 =============== 211 212 The following messages are the most common messages logged by syslog. These 213 may be found in /var/log/messages. 214 215 Driver up:: 216 217 Chelsio Network Driver - version 2.1.1 218 219 NIC detected:: 220 221 eth#: Chelsio N210 1x10GBaseX NIC (rev #), PCIX 133MHz/64-bit 222 223 Link up:: 224 225 eth#: link is up at 10 Gbps, full duplex 226 227 Link down:: 228 229 eth#: link is down 230 231 232 Known Issues 233 ============ 234 235 These issues have been identified during testing. The following information 236 is provided as a workaround to the problem. In some cases, this problem is 237 inherent to Linux or to a particular Linux Distribution and/or hardware 238 platform. 239 240 1. Large number of TCP retransmits on a multiprocessor (SMP) system. 241 242 On a system with multiple CPUs, the interrupt (IRQ) for the network 243 controller may be bound to more than one CPU. This will cause TCP 244 retransmits if the packet data were to be split across different CPUs 245 and re-assembled in a different order than expected. 246 247 To eliminate the TCP retransmits, set smp_affinity on the particular 248 interrupt to a single CPU. You can locate the interrupt (IRQ) used on 249 the N110/N210 by using ifconfig:: 250 251 ifconfig <dev_name> | grep Interrupt 252 253 Set the smp_affinity to a single CPU:: 254 255 echo 1 > /proc/irq/<interrupt_number>/smp_affinity 256 257 It is highly suggested that you do not run the irqbalance daemon on your 258 system, as this will change any smp_affinity setting you have applied. 259 The irqbalance daemon runs on a 10 second interval and binds interrupts 260 to the least loaded CPU determined by the daemon. To disable this daemon:: 261 262 chkconfig --level 2345 irqbalance off 263 264 By default, some Linux distributions enable the kernel feature, 265 irqbalance, which performs the same function as the daemon. To disable 266 this feature, add the following line to your bootloader:: 267 268 noirqbalance 269 270 Example using the Grub bootloader:: 271 272 title Red Hat Enterprise Linux AS (2.4.21-27.ELsmp) 273 root (hd0,0) 274 kernel /vmlinuz-2.4.21-27.ELsmp ro root=/dev/hda3 noirqbalance 275 initrd /initrd-2.4.21-27.ELsmp.img 276 277 2. After running insmod, the driver is loaded and the incorrect network 278 interface is brought up without running ifup. 279 280 When using 2.4.x kernels, including RHEL kernels, the Linux kernel 281 invokes a script named "hotplug". This script is primarily used to 282 automatically bring up USB devices when they are plugged in, however, 283 the script also attempts to automatically bring up a network interface 284 after loading the kernel module. The hotplug script does this by scanning 285 the ifcfg-eth# config files in /etc/sysconfig/network-scripts, looking 286 for HWADDR=<mac_address>. 287 288 If the hotplug script does not find the HWADDRR within any of the 289 ifcfg-eth# files, it will bring up the device with the next available 290 interface name. If this interface is already configured for a different 291 network card, your new interface will have incorrect IP address and 292 network settings. 293 294 To solve this issue, you can add the HWADDR=<mac_address> key to the 295 interface config file of your network controller. 296 297 To disable this "hotplug" feature, you may add the driver (module name) 298 to the "blacklist" file located in /etc/hotplug. It has been noted that 299 this does not work for network devices because the net.agent script 300 does not use the blacklist file. Simply remove, or rename, the net.agent 301 script located in /etc/hotplug to disable this feature. 302 303 3. Transport Protocol (TP) hangs when running heavy multi-connection traffic 304 on an AMD Opteron system with HyperTransport PCI-X Tunnel chipset. 305 306 If your AMD Opteron system uses the AMD-8131 HyperTransport PCI-X Tunnel 307 chipset, you may experience the "133-Mhz Mode Split Completion Data 308 Corruption" bug identified by AMD while using a 133Mhz PCI-X card on the 309 bus PCI-X bus. 310 311 AMD states, "Under highly specific conditions, the AMD-8131 PCI-X Tunnel 312 can provide stale data via split completion cycles to a PCI-X card that 313 is operating at 133 Mhz", causing data corruption. 314 315 AMD's provides three workarounds for this problem, however, Chelsio 316 recommends the first option for best performance with this bug: 317 318 For 133Mhz secondary bus operation, limit the transaction length and 319 the number of outstanding transactions, via BIOS configuration 320 programming of the PCI-X card, to the following: 321 322 Data Length (bytes): 1k 323 324 Total allowed outstanding transactions: 2 325 326 Please refer to AMD 8131-HT/PCI-X Errata 26310 Rev 3.08 August 2004, 327 section 56, "133-MHz Mode Split Completion Data Corruption" for more 328 details with this bug and workarounds suggested by AMD. 329 330 It may be possible to work outside AMD's recommended PCI-X settings, try 331 increasing the Data Length to 2k bytes for increased performance. If you 332 have issues with these settings, please revert to the "safe" settings 333 and duplicate the problem before submitting a bug or asking for support. 334 335 .. note:: 336 337 The default setting on most systems is 8 outstanding transactions 338 and 2k bytes data length. 339 340 4. On multiprocessor systems, it has been noted that an application which 341 is handling 10Gb networking can switch between CPUs causing degraded 342 and/or unstable performance. 343 344 If running on an SMP system and taking performance measurements, it 345 is suggested you either run the latest netperf-2.4.0+ or use a binding 346 tool such as Tim Hockin's procstate utilities (runon) 347 <http://www.hockin.org/~thockin/procstate/>. 348 349 Binding netserver and netperf (or other applications) to particular 350 CPUs will have a significant difference in performance measurements. 351 You may need to experiment which CPU to bind the application to in 352 order to achieve the best performance for your system. 353 354 If you are developing an application designed for 10Gb networking, 355 please keep in mind you may want to look at kernel functions 356 sched_setaffinity & sched_getaffinity to bind your application. 357 358 If you are just running user-space applications such as ftp, telnet, 359 etc., you may want to try the runon tool provided by Tim Hockin's 360 procstate utility. You could also try binding the interface to a 361 particular CPU: runon 0 ifup eth0 362 363 364 Support 365 ======= 366 367 If you have problems with the software or hardware, please contact our 368 customer support team via email at support@chelsio.com or check our website 369 at http://www.chelsio.com 370 371 ------------------------------------------------------------------------------- 372 373 :: 374 375 Chelsio Communications 376 370 San Aleso Ave. 377 Suite 100 378 Sunnyvale, CA 94085 379 http://www.chelsio.com 380 381 This program is free software; you can redistribute it and/or modify 382 it under the terms of the GNU General Public License, version 2, as 383 published by the Free Software Foundation. 384 385 You should have received a copy of the GNU General Public License along 386 with this program; if not, write to the Free Software Foundation, Inc., 387 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 388 389 THIS SOFTWARE IS PROVIDED ``AS IS`` AND WITHOUT ANY EXPRESS OR IMPLIED 390 WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF 391 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 392 393 Copyright |copy| 2003-2005 Chelsio Communications. All rights reserved.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.