1 .. SPDX-License-Identifier: GPL-2.0
2
3 ======================
4 USB4 and Thunderbolt
5 ======================
6 USB4 is the public specification based on Thunderbolt 3 protocol with
7 some differences at the register level among other things. Connection
8 manager is an entity running on the host router (host controller)
9 responsible for enumerating routers and establishing tunnels. A
10 connection manager can be implemented either in firmware or software.
11 Typically PCs come with a firmware connection manager for Thunderbolt 3
12 and early USB4 capable systems. Apple systems on the other hand use
13 software connection manager and the later USB4 compliant devices follow
14 the suit.
15
16 The Linux Thunderbolt driver supports both and can detect at runtime which
17 connection manager implementation is to be used. To be on the safe side the
18 software connection manager in Linux also advertises security level
19 ``user`` which means PCIe tunneling is disabled by default. The
20 documentation below applies to both implementations with the exception that
21 the software connection manager only supports ``user`` security level and
22 is expected to be accompanied with an IOMMU based DMA protection.
23
24 Security levels and how to use them
25 -----------------------------------
26 The interface presented here is not meant for end users. Instead there
27 should be a userspace tool that handles all the low-level details, keeps
28 a database of the authorized devices and prompts users for new connections.
29
30 More details about the sysfs interface for Thunderbolt devices can be
31 found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``.
32
33 Those users who just want to connect any device without any sort of
34 manual work can add following line to
35 ``/etc/udev/rules.d/99-local.rules``::
36
37 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"
38
39 This will authorize all devices automatically when they appear. However,
40 keep in mind that this bypasses the security levels and makes the system
41 vulnerable to DMA attacks.
42
43 Starting with Intel Falcon Ridge Thunderbolt controller there are 4
44 security levels available. Intel Titan Ridge added one more security level
45 (usbonly). The reason for these is the fact that the connected devices can
46 be DMA masters and thus read contents of the host memory without CPU and OS
47 knowing about it. There are ways to prevent this by setting up an IOMMU but
48 it is not always available for various reasons.
49
50 Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is
51 treated as another security level (nopcie).
52
53 The security levels are as follows:
54
55 none
56 All devices are automatically connected by the firmware. No user
57 approval is needed. In BIOS settings this is typically called
58 *Legacy mode*.
59
60 user
61 User is asked whether the device is allowed to be connected.
62 Based on the device identification information available through
63 ``/sys/bus/thunderbolt/devices``, the user then can make the decision.
64 In BIOS settings this is typically called *Unique ID*.
65
66 secure
67 User is asked whether the device is allowed to be connected. In
68 addition to UUID the device (if it supports secure connect) is sent
69 a challenge that should match the expected one based on a random key
70 written to the ``key`` sysfs attribute. In BIOS settings this is
71 typically called *One time saved key*.
72
73 dponly
74 The firmware automatically creates tunnels for Display Port and
75 USB. No PCIe tunneling is done. In BIOS settings this is
76 typically called *Display Port Only*.
77
78 usbonly
79 The firmware automatically creates tunnels for the USB controller and
80 Display Port in a dock. All PCIe links downstream of the dock are
81 removed.
82
83 nopcie
84 PCIe tunneling is disabled/forbidden from the BIOS. Available in some
85 USB4 systems.
86
87 The current security level can be read from
88 ``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is
89 the Thunderbolt domain the host controller manages. There is typically
90 one domain per Thunderbolt host controller.
91
92 If the security level reads as ``user`` or ``secure`` the connected
93 device must be authorized by the user before PCIe tunnels are created
94 (e.g the PCIe device appears).
95
96 Each Thunderbolt device plugged in will appear in sysfs under
97 ``/sys/bus/thunderbolt/devices``. The device directory carries
98 information that can be used to identify the particular device,
99 including its name and UUID.
100
101 Authorizing devices when security level is ``user`` or ``secure``
102 -----------------------------------------------------------------
103 When a device is plugged in it will appear in sysfs as follows::
104
105 /sys/bus/thunderbolt/devices/0-1/authorized - 0
106 /sys/bus/thunderbolt/devices/0-1/device - 0x8004
107 /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter
108 /sys/bus/thunderbolt/devices/0-1/vendor - 0x1
109 /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc.
110 /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff
111
112 The ``authorized`` attribute reads 0 which means no PCIe tunnels are
113 created yet. The user can authorize the device by simply entering::
114
115 # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized
116
117 This will create the PCIe tunnels and the device is now connected.
118
119 If the device supports secure connect, and the domain security level is
120 set to ``secure``, it has an additional attribute ``key`` which can hold
121 a random 32-byte value used for authorization and challenging the device in
122 future connects::
123
124 /sys/bus/thunderbolt/devices/0-3/authorized - 0
125 /sys/bus/thunderbolt/devices/0-3/device - 0x305
126 /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box
127 /sys/bus/thunderbolt/devices/0-3/key -
128 /sys/bus/thunderbolt/devices/0-3/vendor - 0x41
129 /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron
130 /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16
131
132 Notice the key is empty by default.
133
134 If the user does not want to use secure connect they can just ``echo 1``
135 to the ``authorized`` attribute and the PCIe tunnels will be created in
136 the same way as in the ``user`` security level.
137
138 If the user wants to use secure connect, the first time the device is
139 plugged a key needs to be created and sent to the device::
140
141 # key=$(openssl rand -hex 32)
142 # echo $key > /sys/bus/thunderbolt/devices/0-3/key
143 # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized
144
145 Now the device is connected (PCIe tunnels are created) and in addition
146 the key is stored on the device NVM.
147
148 Next time the device is plugged in the user can verify (challenge) the
149 device using the same key::
150
151 # echo $key > /sys/bus/thunderbolt/devices/0-3/key
152 # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized
153
154 If the challenge the device returns back matches the one we expect based
155 on the key, the device is connected and the PCIe tunnels are created.
156 However, if the challenge fails no tunnels are created and error is
157 returned to the user.
158
159 If the user still wants to connect the device they can either approve
160 the device without a key or write a new key and write 1 to the
161 ``authorized`` file to get the new key stored on the device NVM.
162
163 De-authorizing devices
164 ----------------------
165 It is possible to de-authorize devices by writing ``0`` to their
166 ``authorized`` attribute. This requires support from the connection
167 manager implementation and can be checked by reading domain
168 ``deauthorization`` attribute. If it reads ``1`` then the feature is
169 supported.
170
171 When a device is de-authorized the PCIe tunnel from the parent device
172 PCIe downstream (or root) port to the device PCIe upstream port is torn
173 down. This is essentially the same thing as PCIe hot-remove and the PCIe
174 toplogy in question will not be accessible anymore until the device is
175 authorized again. If there is storage such as NVMe or similar involved,
176 there is a risk for data loss if the filesystem on that storage is not
177 properly shut down. You have been warned!
178
179 DMA protection utilizing IOMMU
180 ------------------------------
181 Recent systems from 2018 and forward with Thunderbolt ports may natively
182 support IOMMU. This means that Thunderbolt security is handled by an IOMMU
183 so connected devices cannot access memory regions outside of what is
184 allocated for them by drivers. When Linux is running on such system it
185 automatically enables IOMMU if not enabled by the user already. These
186 systems can be identified by reading ``1`` from
187 ``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute.
188
189 The driver does not do anything special in this case but because DMA
190 protection is handled by the IOMMU, security levels (if set) are
191 redundant. For this reason some systems ship with security level set to
192 ``none``. Other systems have security level set to ``user`` in order to
193 support downgrade to older OS, so users who want to automatically
194 authorize devices when IOMMU DMA protection is enabled can use the
195 following ``udev`` rule::
196
197 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1"
198
199 Upgrading NVM on Thunderbolt device, host or retimer
200 ----------------------------------------------------
201 Since most of the functionality is handled in firmware running on a
202 host controller or a device, it is important that the firmware can be
203 upgraded to the latest where possible bugs in it have been fixed.
204 Typically OEMs provide this firmware from their support site.
205
206 There is also a central site which has links where to download firmware
207 for some machines:
208
209 `Thunderbolt Updates <https://thunderbolttechnology.net/updates>`_
210
211 Before you upgrade firmware on a device, host or retimer, please make
212 sure it is a suitable upgrade. Failing to do that may render the device
213 in a state where it cannot be used properly anymore without special
214 tools!
215
216 Host NVM upgrade on Apple Macs is not supported.
217
218 Once the NVM image has been downloaded, you need to plug in a
219 Thunderbolt device so that the host controller appears. It does not
220 matter which device is connected (unless you are upgrading NVM on a
221 device - then you need to connect that particular device).
222
223 Note an OEM-specific method to power the controller up ("force power") may
224 be available for your system in which case there is no need to plug in a
225 Thunderbolt device.
226
227 After that we can write the firmware to the non-active parts of the NVM
228 of the host or device. As an example here is how Intel NUC6i7KYK (Skull
229 Canyon) Thunderbolt controller NVM is upgraded::
230
231 # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem
232
233 Once the operation completes we can trigger NVM authentication and
234 upgrade process as follows::
235
236 # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
237
238 If no errors are returned, the host controller shortly disappears. Once
239 it comes back the driver notices it and initiates a full power cycle.
240 After a while the host controller appears again and this time it should
241 be fully functional.
242
243 We can verify that the new NVM firmware is active by running the following
244 commands::
245
246 # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
247 0x0
248 # cat /sys/bus/thunderbolt/devices/0-0/nvm_version
249 18.0
250
251 If ``nvm_authenticate`` contains anything other than 0x0 it is the error
252 code from the last authentication cycle, which means the authentication
253 of the NVM image failed.
254
255 Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN``
256 depend on the order they are registered in the NVMem subsystem. N in
257 the name is the identifier added by the NVMem subsystem.
258
259 Upgrading on-board retimer NVM when there is no cable connected
260 ---------------------------------------------------------------
261 If the platform supports, it may be possible to upgrade the retimer NVM
262 firmware even when there is nothing connected to the USB4
263 ports. When this is the case the ``usb4_portX`` devices have two special
264 attributes: ``offline`` and ``rescan``. The way to upgrade the firmware
265 is to first put the USB4 port into offline mode::
266
267 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline
268
269 This step makes sure the port does not respond to any hotplug events,
270 and also ensures the retimers are powered on. The next step is to scan
271 for the retimers::
272
273 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan
274
275 This enumerates and adds the on-board retimers. Now retimer NVM can be
276 upgraded in the same way than with cable connected (see previous
277 section). However, the retimer is not disconnected as we are offline
278 mode) so after writing ``1`` to ``nvm_authenticate`` one should wait for
279 5 or more seconds before running rescan again::
280
281 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan
282
283 This point if everything went fine, the port can be put back to
284 functional state again::
285
286 # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline
287
288 Upgrading NVM when host controller is in safe mode
289 --------------------------------------------------
290 If the existing NVM is not properly authenticated (or is missing) the
291 host controller goes into safe mode which means that the only available
292 functionality is flashing a new NVM image. When in this mode, reading
293 ``nvm_version`` fails with ``ENODATA`` and the device identification
294 information is missing.
295
296 To recover from this mode, one needs to flash a valid NVM image to the
297 host controller in the same way it is done in the previous chapter.
298
299 Networking over Thunderbolt cable
300 ---------------------------------
301 Thunderbolt technology allows software communication between two hosts
302 connected by a Thunderbolt cable.
303
304 It is possible to tunnel any kind of traffic over a Thunderbolt link but
305 currently we only support Apple ThunderboltIP protocol.
306
307 If the other host is running Windows or macOS, the only thing you need to
308 do is to connect a Thunderbolt cable between the two hosts; the
309 ``thunderbolt-net`` driver is loaded automatically. If the other host is
310 also Linux you should load ``thunderbolt-net`` manually on one host (it
311 does not matter which one)::
312
313 # modprobe thunderbolt-net
314
315 This triggers module load on the other host automatically. If the driver
316 is built-in to the kernel image, there is no need to do anything.
317
318 The driver will create one virtual ethernet interface per Thunderbolt
319 port which are named like ``thunderbolt0`` and so on. From this point
320 you can either use standard userspace tools like ``ifconfig`` to
321 configure the interface or let your GUI handle it automatically.
322
323 Forcing power
324 -------------
325 Many OEMs include a method that can be used to force the power of a
326 Thunderbolt controller to an "On" state even if nothing is connected.
327 If supported by your machine this will be exposed by the WMI bus with
328 a sysfs attribute called "force_power".
329
330 For example the intel-wmi-thunderbolt driver exposes this attribute in:
331 /sys/bus/wmi/devices/86CCFD48-205E-4A77-9C48-2021CBEDE341/force_power
332
333 To force the power to on, write 1 to this attribute file.
334 To disable force power, write 0 to this attribute file.
335
336 Note: it's currently not possible to query the force power state of a platform.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.