1 .. SPDX-License-Identifier: GPL-2.0 2 3 ======================================= 4 DSA switch configuration from userspace 5 ======================================= 6 7 The DSA switch configuration is not integrated into the main userspace 8 network configuration suites by now and has to be performed manually. 9 10 .. _dsa-config-showcases: 11 12 Configuration showcases 13 ----------------------- 14 15 To configure a DSA switch a couple of commands need to be executed. In this 16 documentation some common configuration scenarios are handled as showcases: 17 18 *single port* 19 Every switch port acts as a different configurable Ethernet port 20 21 *bridge* 22 Every switch port is part of one configurable Ethernet bridge 23 24 *gateway* 25 Every switch port except one upstream port is part of a configurable 26 Ethernet bridge. 27 The upstream port acts as different configurable Ethernet port. 28 29 All configurations are performed with tools from iproute2, which is available 30 at https://www.kernel.org/pub/linux/utils/net/iproute2/ 31 32 Through DSA every port of a switch is handled like a normal linux Ethernet 33 interface. The CPU port is the switch port connected to an Ethernet MAC chip. 34 The corresponding linux Ethernet interface is called the conduit interface. 35 All other corresponding linux interfaces are called user interfaces. 36 37 The user interfaces depend on the conduit interface being up in order for them 38 to send or receive traffic. Prior to kernel v5.12, the state of the conduit 39 interface had to be managed explicitly by the user. Starting with kernel v5.12, 40 the behavior is as follows: 41 42 - when a DSA user interface is brought up, the conduit interface is 43 automatically brought up. 44 - when the conduit interface is brought down, all DSA user interfaces are 45 automatically brought down. 46 47 In this documentation the following Ethernet interfaces are used: 48 49 *eth0* 50 the conduit interface 51 52 *eth1* 53 another conduit interface 54 55 *lan1* 56 a user interface 57 58 *lan2* 59 another user interface 60 61 *lan3* 62 a third user interface 63 64 *wan* 65 A user interface dedicated for upstream traffic 66 67 Further Ethernet interfaces can be configured similar. 68 The configured IPs and networks are: 69 70 *single port* 71 * lan1: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3) 72 * lan2: 192.0.2.5/30 (192.0.2.4 - 192.0.2.7) 73 * lan3: 192.0.2.9/30 (192.0.2.8 - 192.0.2.11) 74 75 *bridge* 76 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255) 77 78 *gateway* 79 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255) 80 * wan: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3) 81 82 .. _dsa-tagged-configuration: 83 84 Configuration with tagging support 85 ---------------------------------- 86 87 The tagging based configuration is desired and supported by the majority of 88 DSA switches. These switches are capable to tag incoming and outgoing traffic 89 without using a VLAN based configuration. 90 91 *single port* 92 .. code-block:: sh 93 94 # configure each interface 95 ip addr add 192.0.2.1/30 dev lan1 96 ip addr add 192.0.2.5/30 dev lan2 97 ip addr add 192.0.2.9/30 dev lan3 98 99 # For kernels earlier than v5.12, the conduit interface needs to be 100 # brought up manually before the user ports. 101 ip link set eth0 up 102 103 # bring up the user interfaces 104 ip link set lan1 up 105 ip link set lan2 up 106 ip link set lan3 up 107 108 *bridge* 109 .. code-block:: sh 110 111 # For kernels earlier than v5.12, the conduit interface needs to be 112 # brought up manually before the user ports. 113 ip link set eth0 up 114 115 # bring up the user interfaces 116 ip link set lan1 up 117 ip link set lan2 up 118 ip link set lan3 up 119 120 # create bridge 121 ip link add name br0 type bridge 122 123 # add ports to bridge 124 ip link set dev lan1 master br0 125 ip link set dev lan2 master br0 126 ip link set dev lan3 master br0 127 128 # configure the bridge 129 ip addr add 192.0.2.129/25 dev br0 130 131 # bring up the bridge 132 ip link set dev br0 up 133 134 *gateway* 135 .. code-block:: sh 136 137 # For kernels earlier than v5.12, the conduit interface needs to be 138 # brought up manually before the user ports. 139 ip link set eth0 up 140 141 # bring up the user interfaces 142 ip link set wan up 143 ip link set lan1 up 144 ip link set lan2 up 145 146 # configure the upstream port 147 ip addr add 192.0.2.1/30 dev wan 148 149 # create bridge 150 ip link add name br0 type bridge 151 152 # add ports to bridge 153 ip link set dev lan1 master br0 154 ip link set dev lan2 master br0 155 156 # configure the bridge 157 ip addr add 192.0.2.129/25 dev br0 158 159 # bring up the bridge 160 ip link set dev br0 up 161 162 .. _dsa-vlan-configuration: 163 164 Configuration without tagging support 165 ------------------------------------- 166 167 A minority of switches are not capable to use a taging protocol 168 (DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based 169 configuration. 170 171 *single port* 172 The configuration can only be set up via VLAN tagging and bridge setup. 173 174 .. code-block:: sh 175 176 # tag traffic on CPU port 177 ip link add link eth0 name eth0.1 type vlan id 1 178 ip link add link eth0 name eth0.2 type vlan id 2 179 ip link add link eth0 name eth0.3 type vlan id 3 180 181 # For kernels earlier than v5.12, the conduit interface needs to be 182 # brought up manually before the user ports. 183 ip link set eth0 up 184 ip link set eth0.1 up 185 ip link set eth0.2 up 186 ip link set eth0.3 up 187 188 # bring up the user interfaces 189 ip link set lan1 up 190 ip link set lan2 up 191 ip link set lan3 up 192 193 # create bridge 194 ip link add name br0 type bridge 195 196 # activate VLAN filtering 197 ip link set dev br0 type bridge vlan_filtering 1 198 199 # add ports to bridges 200 ip link set dev lan1 master br0 201 ip link set dev lan2 master br0 202 ip link set dev lan3 master br0 203 204 # tag traffic on ports 205 bridge vlan add dev lan1 vid 1 pvid untagged 206 bridge vlan add dev lan2 vid 2 pvid untagged 207 bridge vlan add dev lan3 vid 3 pvid untagged 208 209 # configure the VLANs 210 ip addr add 192.0.2.1/30 dev eth0.1 211 ip addr add 192.0.2.5/30 dev eth0.2 212 ip addr add 192.0.2.9/30 dev eth0.3 213 214 # bring up the bridge devices 215 ip link set br0 up 216 217 218 *bridge* 219 .. code-block:: sh 220 221 # tag traffic on CPU port 222 ip link add link eth0 name eth0.1 type vlan id 1 223 224 # For kernels earlier than v5.12, the conduit interface needs to be 225 # brought up manually before the user ports. 226 ip link set eth0 up 227 ip link set eth0.1 up 228 229 # bring up the user interfaces 230 ip link set lan1 up 231 ip link set lan2 up 232 ip link set lan3 up 233 234 # create bridge 235 ip link add name br0 type bridge 236 237 # activate VLAN filtering 238 ip link set dev br0 type bridge vlan_filtering 1 239 240 # add ports to bridge 241 ip link set dev lan1 master br0 242 ip link set dev lan2 master br0 243 ip link set dev lan3 master br0 244 ip link set eth0.1 master br0 245 246 # tag traffic on ports 247 bridge vlan add dev lan1 vid 1 pvid untagged 248 bridge vlan add dev lan2 vid 1 pvid untagged 249 bridge vlan add dev lan3 vid 1 pvid untagged 250 251 # configure the bridge 252 ip addr add 192.0.2.129/25 dev br0 253 254 # bring up the bridge 255 ip link set dev br0 up 256 257 *gateway* 258 .. code-block:: sh 259 260 # tag traffic on CPU port 261 ip link add link eth0 name eth0.1 type vlan id 1 262 ip link add link eth0 name eth0.2 type vlan id 2 263 264 # For kernels earlier than v5.12, the conduit interface needs to be 265 # brought up manually before the user ports. 266 ip link set eth0 up 267 ip link set eth0.1 up 268 ip link set eth0.2 up 269 270 # bring up the user interfaces 271 ip link set wan up 272 ip link set lan1 up 273 ip link set lan2 up 274 275 # create bridge 276 ip link add name br0 type bridge 277 278 # activate VLAN filtering 279 ip link set dev br0 type bridge vlan_filtering 1 280 281 # add ports to bridges 282 ip link set dev wan master br0 283 ip link set eth0.1 master br0 284 ip link set dev lan1 master br0 285 ip link set dev lan2 master br0 286 287 # tag traffic on ports 288 bridge vlan add dev lan1 vid 1 pvid untagged 289 bridge vlan add dev lan2 vid 1 pvid untagged 290 bridge vlan add dev wan vid 2 pvid untagged 291 292 # configure the VLANs 293 ip addr add 192.0.2.1/30 dev eth0.2 294 ip addr add 192.0.2.129/25 dev br0 295 296 # bring up the bridge devices 297 ip link set br0 up 298 299 Forwarding database (FDB) management 300 ------------------------------------ 301 302 The existing DSA switches do not have the necessary hardware support to keep 303 the software FDB of the bridge in sync with the hardware tables, so the two 304 tables are managed separately (``bridge fdb show`` queries both, and depending 305 on whether the ``self`` or ``master`` flags are being used, a ``bridge fdb 306 add`` or ``bridge fdb del`` command acts upon entries from one or both tables). 307 308 Up until kernel v4.14, DSA only supported user space management of bridge FDB 309 entries using the bridge bypass operations (which do not update the software 310 FDB, just the hardware one) using the ``self`` flag (which is optional and can 311 be omitted). 312 313 .. code-block:: sh 314 315 bridge fdb add dev swp0 00:01:02:03:04:05 self static 316 # or shorthand 317 bridge fdb add dev swp0 00:01:02:03:04:05 static 318 319 Due to a bug, the bridge bypass FDB implementation provided by DSA did not 320 distinguish between ``static`` and ``local`` FDB entries (``static`` are meant 321 to be forwarded, while ``local`` are meant to be locally terminated, i.e. sent 322 to the host port). Instead, all FDB entries with the ``self`` flag (implicit or 323 explicit) are treated by DSA as ``static`` even if they are ``local``. 324 325 .. code-block:: sh 326 327 # This command: 328 bridge fdb add dev swp0 00:01:02:03:04:05 static 329 # behaves the same for DSA as this command: 330 bridge fdb add dev swp0 00:01:02:03:04:05 local 331 # or shorthand, because the 'local' flag is implicit if 'static' is not 332 # specified, it also behaves the same as: 333 bridge fdb add dev swp0 00:01:02:03:04:05 334 335 The last command is an incorrect way of adding a static bridge FDB entry to a 336 DSA switch using the bridge bypass operations, and works by mistake. Other 337 drivers will treat an FDB entry added by the same command as ``local`` and as 338 such, will not forward it, as opposed to DSA. 339 340 Between kernel v4.14 and v5.14, DSA has supported in parallel two modes of 341 adding a bridge FDB entry to the switch: the bridge bypass discussed above, as 342 well as a new mode using the ``master`` flag which installs FDB entries in the 343 software bridge too. 344 345 .. code-block:: sh 346 347 bridge fdb add dev swp0 00:01:02:03:04:05 master static 348 349 Since kernel v5.14, DSA has gained stronger integration with the bridge's 350 software FDB, and the support for its bridge bypass FDB implementation (using 351 the ``self`` flag) has been removed. This results in the following changes: 352 353 .. code-block:: sh 354 355 # This is the only valid way of adding an FDB entry that is supported, 356 # compatible with v4.14 kernels and later: 357 bridge fdb add dev swp0 00:01:02:03:04:05 master static 358 # This command is no longer buggy and the entry is properly treated as 359 # 'local' instead of being forwarded: 360 bridge fdb add dev swp0 00:01:02:03:04:05 361 # This command no longer installs a static FDB entry to hardware: 362 bridge fdb add dev swp0 00:01:02:03:04:05 static 363 364 Script writers are therefore encouraged to use the ``master static`` set of 365 flags when working with bridge FDB entries on DSA switch interfaces. 366 367 Affinity of user ports to CPU ports 368 ----------------------------------- 369 370 Typically, DSA switches are attached to the host via a single Ethernet 371 interface, but in cases where the switch chip is discrete, the hardware design 372 may permit the use of 2 or more ports connected to the host, for an increase in 373 termination throughput. 374 375 DSA can make use of multiple CPU ports in two ways. First, it is possible to 376 statically assign the termination traffic associated with a certain user port 377 to be processed by a certain CPU port. This way, user space can implement 378 custom policies of static load balancing between user ports, by spreading the 379 affinities according to the available CPU ports. 380 381 Secondly, it is possible to perform load balancing between CPU ports on a per 382 packet basis, rather than statically assigning user ports to CPU ports. 383 This can be achieved by placing the DSA conduits under a LAG interface (bonding 384 or team). DSA monitors this operation and creates a mirror of this software LAG 385 on the CPU ports facing the physical DSA conduits that constitute the LAG slave 386 devices. 387 388 To make use of multiple CPU ports, the firmware (device tree) description of 389 the switch must mark all the links between CPU ports and their DSA conduits 390 using the ``ethernet`` reference/phandle. At startup, only a single CPU port 391 and DSA conduit will be used - the numerically first port from the firmware 392 description which has an ``ethernet`` property. It is up to the user to 393 configure the system for the switch to use other conduits. 394 395 DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow 396 changing the DSA conduit of a user port. The ``IFLA_DSA_CONDUIT`` u32 netlink 397 attribute contains the ifindex of the conduit device that handles each user 398 device. The DSA conduit must be a valid candidate based on firmware node 399 information, or a LAG interface which contains only slaves which are valid 400 candidates. 401 402 Using iproute2, the following manipulations are possible: 403 404 .. code-block:: sh 405 406 # See the DSA conduit in current use 407 ip -d link show dev swp0 408 (...) 409 dsa master eth0 410 411 # Static CPU port distribution 412 ip link set swp0 type dsa master eth1 413 ip link set swp1 type dsa master eth0 414 ip link set swp2 type dsa master eth1 415 ip link set swp3 type dsa master eth0 416 417 # CPU ports in LAG, using explicit assignment of the DSA conduit 418 ip link add bond0 type bond mode balance-xor && ip link set bond0 up 419 ip link set eth1 down && ip link set eth1 master bond0 420 ip link set swp0 type dsa master bond0 421 ip link set swp1 type dsa master bond0 422 ip link set swp2 type dsa master bond0 423 ip link set swp3 type dsa master bond0 424 ip link set eth0 down && ip link set eth0 master bond0 425 ip -d link show dev swp0 426 (...) 427 dsa master bond0 428 429 # CPU ports in LAG, relying on implicit migration of the DSA conduit 430 ip link add bond0 type bond mode balance-xor && ip link set bond0 up 431 ip link set eth0 down && ip link set eth0 master bond0 432 ip link set eth1 down && ip link set eth1 master bond0 433 ip -d link show dev swp0 434 (...) 435 dsa master bond0 436 437 Notice that in the case of CPU ports under a LAG, the use of the 438 ``IFLA_DSA_CONDUIT`` netlink attribute is not strictly needed, but rather, DSA 439 reacts to the ``IFLA_MASTER`` attribute change of its present conduit (``eth0``) 440 and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly, 441 when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports 442 that were assigned to this interface to the first physical DSA conduit which is 443 eligible, based on the firmware description (it effectively reverts to the 444 startup configuration). 445 446 In a setup with more than 2 physical CPU ports, it is therefore possible to mix 447 static user to CPU port assignment with LAG between DSA conduits. It is not 448 possible to statically assign a user port towards a DSA conduit that has any 449 upper interfaces (this includes LAG devices - the conduit must always be the LAG 450 in this case). 451 452 Live changing of the DSA conduit (and thus CPU port) affinity of a user port is 453 permitted, in order to allow dynamic redistribution in response to traffic. 454 455 Physical DSA conduits are allowed to join and leave at any time a LAG interface 456 used as a DSA conduit; however, DSA will reject a LAG interface as a valid 457 candidate for being a DSA conduit unless it has at least one physical DSA conduit 458 as a slave device.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.