1 .. _embargoed_hardware_issues: 1 .. _embargoed_hardware_issues:
2 2
3 Embargoed hardware issues 3 Embargoed hardware issues
4 ========================= 4 =========================
5 5
6 Scope 6 Scope
7 ----- 7 -----
8 8
9 Hardware issues which result in security probl 9 Hardware issues which result in security problems are a different category
10 of security bugs than pure software bugs which 10 of security bugs than pure software bugs which only affect the Linux
11 kernel. 11 kernel.
12 12
13 Hardware issues like Meltdown, Spectre, L1TF e 13 Hardware issues like Meltdown, Spectre, L1TF etc. must be treated
14 differently because they usually affect all Op 14 differently because they usually affect all Operating Systems ("OS") and
15 therefore need coordination across different O 15 therefore need coordination across different OS vendors, distributions,
16 silicon vendors, hardware integrators, and oth 16 silicon vendors, hardware integrators, and other parties. For some of the
17 issues, software mitigations can depend on mic 17 issues, software mitigations can depend on microcode or firmware updates,
18 which need further coordination. 18 which need further coordination.
19 19
20 .. _Contact: 20 .. _Contact:
21 21
22 Contact 22 Contact
23 ------- 23 -------
24 24
25 The Linux kernel hardware security team is sep 25 The Linux kernel hardware security team is separate from the regular Linux
26 kernel security team. 26 kernel security team.
27 27
28 The team only handles developing fixes for emb 28 The team only handles developing fixes for embargoed hardware security
29 issues. Reports of pure software security bugs 29 issues. Reports of pure software security bugs in the Linux kernel are not
30 handled by this team and the reporter will be 30 handled by this team and the reporter will be guided to contact the regular
31 Linux kernel security team (:ref:`Documentatio 31 Linux kernel security team (:ref:`Documentation/admin-guide/
32 <securitybugs>`) instead. 32 <securitybugs>`) instead.
33 33
34 The team can be contacted by email at <hardware 34 The team can be contacted by email at <hardware-security@kernel.org>. This
35 is a private list of security officers who wil 35 is a private list of security officers who will help you coordinate a fix
36 according to our documented process. 36 according to our documented process.
37 37
38 The list is encrypted and email to the list ca 38 The list is encrypted and email to the list can be sent by either PGP or
39 S/MIME encrypted and must be signed with the r 39 S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
40 certificate. The list's PGP key and S/MIME cer 40 certificate. The list's PGP key and S/MIME certificate are available from
41 the following URLs: 41 the following URLs:
42 42
43 - PGP: https://www.kernel.org/static/files/h 43 - PGP: https://www.kernel.org/static/files/hardware-security.asc
44 - S/MIME: https://www.kernel.org/static/file 44 - S/MIME: https://www.kernel.org/static/files/hardware-security.crt
45 45
46 While hardware security issues are often handl 46 While hardware security issues are often handled by the affected silicon
47 vendor, we welcome contact from researchers or 47 vendor, we welcome contact from researchers or individuals who have
48 identified a potential hardware flaw. 48 identified a potential hardware flaw.
49 49
50 Hardware security officers 50 Hardware security officers
51 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 51 ^^^^^^^^^^^^^^^^^^^^^^^^^^
52 52
53 The current team of hardware security officers 53 The current team of hardware security officers:
54 54
55 - Linus Torvalds (Linux Foundation Fellow) 55 - Linus Torvalds (Linux Foundation Fellow)
56 - Greg Kroah-Hartman (Linux Foundation Fello 56 - Greg Kroah-Hartman (Linux Foundation Fellow)
57 - Thomas Gleixner (Linux Foundation Fellow) 57 - Thomas Gleixner (Linux Foundation Fellow)
58 58
59 Operation of mailing-lists 59 Operation of mailing-lists
60 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 60 ^^^^^^^^^^^^^^^^^^^^^^^^^^
61 61
62 The encrypted mailing-lists which are used in 62 The encrypted mailing-lists which are used in our process are hosted on
63 Linux Foundation's IT infrastructure. By provi 63 Linux Foundation's IT infrastructure. By providing this service, members
64 of Linux Foundation's IT operations personnel 64 of Linux Foundation's IT operations personnel technically have the
65 ability to access the embargoed information, b 65 ability to access the embargoed information, but are obliged to
66 confidentiality by their employment contract. 66 confidentiality by their employment contract. Linux Foundation IT
67 personnel are also responsible for operating a 67 personnel are also responsible for operating and managing the rest of
68 kernel.org's infrastructure. 68 kernel.org's infrastructure.
69 69
70 The Linux Foundation's current director of IT 70 The Linux Foundation's current director of IT Project infrastructure is
71 Konstantin Ryabitsev. 71 Konstantin Ryabitsev.
72 72
73 73
74 Non-disclosure agreements 74 Non-disclosure agreements
75 ------------------------- 75 -------------------------
76 76
77 The Linux kernel hardware security team is not 77 The Linux kernel hardware security team is not a formal body and therefore
78 unable to enter into any non-disclosure agreem 78 unable to enter into any non-disclosure agreements. The kernel community
79 is aware of the sensitive nature of such issue 79 is aware of the sensitive nature of such issues and offers a Memorandum of
80 Understanding instead. 80 Understanding instead.
81 81
82 82
83 Memorandum of Understanding 83 Memorandum of Understanding
84 --------------------------- 84 ---------------------------
85 85
86 The Linux kernel community has a deep understa 86 The Linux kernel community has a deep understanding of the requirement to
87 keep hardware security issues under embargo fo 87 keep hardware security issues under embargo for coordination between
88 different OS vendors, distributors, silicon ve 88 different OS vendors, distributors, silicon vendors, and other parties.
89 89
90 The Linux kernel community has successfully ha 90 The Linux kernel community has successfully handled hardware security
91 issues in the past and has the necessary mecha 91 issues in the past and has the necessary mechanisms in place to allow
92 community compliant development under embargo 92 community compliant development under embargo restrictions.
93 93
94 The Linux kernel community has a dedicated har 94 The Linux kernel community has a dedicated hardware security team for
95 initial contact, which oversees the process of 95 initial contact, which oversees the process of handling such issues under
96 embargo rules. 96 embargo rules.
97 97
98 The hardware security team identifies the deve 98 The hardware security team identifies the developers (domain experts) who
99 will form the initial response team for a part 99 will form the initial response team for a particular issue. The initial
100 response team can bring in further developers 100 response team can bring in further developers (domain experts) to address
101 the issue in the best technical way. 101 the issue in the best technical way.
102 102
103 All involved developers pledge to adhere to th 103 All involved developers pledge to adhere to the embargo rules and to keep
104 the received information confidential. Violati 104 the received information confidential. Violation of the pledge will lead to
105 immediate exclusion from the current issue and 105 immediate exclusion from the current issue and removal from all related
106 mailing lists. In addition, the hardware secur 106 mailing lists. In addition, the hardware security team will also exclude
107 the offender from future issues. The impact of 107 the offender from future issues. The impact of this consequence is a highly
108 effective deterrent in our community. In case 108 effective deterrent in our community. In case a violation happens the
109 hardware security team will inform the involve 109 hardware security team will inform the involved parties immediately. If you
110 or anyone else becomes aware of a potential vi 110 or anyone else becomes aware of a potential violation, please report it
111 immediately to the Hardware security officers. 111 immediately to the Hardware security officers.
112 112
113 113
114 Process 114 Process
115 ^^^^^^^ 115 ^^^^^^^
116 116
117 Due to the globally distributed nature of Linu 117 Due to the globally distributed nature of Linux kernel development,
118 face-to-face meetings are almost impossible to 118 face-to-face meetings are almost impossible to address hardware security
119 issues. Phone conferences are hard to coordin 119 issues. Phone conferences are hard to coordinate due to time zones and
120 other factors and should be only used when abs 120 other factors and should be only used when absolutely necessary. Encrypted
121 email has been proven to be the most effective 121 email has been proven to be the most effective and secure communication
122 method for these types of issues. 122 method for these types of issues.
123 123
124 Start of Disclosure 124 Start of Disclosure
125 """"""""""""""""""" 125 """""""""""""""""""
126 126
127 Disclosure starts by emailing the Linux kernel 127 Disclosure starts by emailing the Linux kernel hardware security team per
128 the Contact section above. This initial conta 128 the Contact section above. This initial contact should contain a
129 description of the problem and a list of any k 129 description of the problem and a list of any known affected silicon. If
130 your organization builds or distributes the af 130 your organization builds or distributes the affected hardware, we encourage
131 you to also consider what other hardware could 131 you to also consider what other hardware could be affected. The disclosing
132 party is responsible for contacting the affect 132 party is responsible for contacting the affected silicon vendors in a
133 timely manner. 133 timely manner.
134 134
135 The hardware security team will provide an inc 135 The hardware security team will provide an incident-specific encrypted
136 mailing list which will be used for initial di 136 mailing list which will be used for initial discussion with the reporter,
137 further disclosure, and coordination of fixes. 137 further disclosure, and coordination of fixes.
138 138
139 The hardware security team will provide the di 139 The hardware security team will provide the disclosing party a list of
140 developers (domain experts) who should be info 140 developers (domain experts) who should be informed initially about the
141 issue after confirming with the developers tha 141 issue after confirming with the developers that they will adhere to this
142 Memorandum of Understanding and the documented 142 Memorandum of Understanding and the documented process. These developers
143 form the initial response team and will be res 143 form the initial response team and will be responsible for handling the
144 issue after initial contact. The hardware secu 144 issue after initial contact. The hardware security team is supporting the
145 response team, but is not necessarily involved 145 response team, but is not necessarily involved in the mitigation
146 development process. 146 development process.
147 147
148 While individual developers might be covered b 148 While individual developers might be covered by a non-disclosure agreement
149 via their employer, they cannot enter individu 149 via their employer, they cannot enter individual non-disclosure agreements
150 in their role as Linux kernel developers. They 150 in their role as Linux kernel developers. They will, however, agree to
151 adhere to this documented process and the Memo 151 adhere to this documented process and the Memorandum of Understanding.
152 152
153 The disclosing party should provide a list of 153 The disclosing party should provide a list of contacts for all other
154 entities who have already been, or should be, 154 entities who have already been, or should be, informed about the issue.
155 This serves several purposes: 155 This serves several purposes:
156 156
157 - The list of disclosed entities allows commu 157 - The list of disclosed entities allows communication across the
158 industry, e.g. other OS vendors, HW vendors 158 industry, e.g. other OS vendors, HW vendors, etc.
159 159
160 - The disclosed entities can be contacted to 160 - The disclosed entities can be contacted to name experts who should
161 participate in the mitigation development. 161 participate in the mitigation development.
162 162
163 - If an expert who is required to handle an i 163 - If an expert who is required to handle an issue is employed by a listed
164 entity or member of an listed entity, then 164 entity or member of an listed entity, then the response teams can
165 request the disclosure of that expert from 165 request the disclosure of that expert from that entity. This ensures
166 that the expert is also part of the entity' 166 that the expert is also part of the entity's response team.
167 167
168 Disclosure 168 Disclosure
169 """""""""" 169 """"""""""
170 170
171 The disclosing party provides detailed informa 171 The disclosing party provides detailed information to the initial response
172 team via the specific encrypted mailing-list. 172 team via the specific encrypted mailing-list.
173 173
174 From our experience, the technical documentati 174 From our experience, the technical documentation of these issues is usually
175 a sufficient starting point, and further techn 175 a sufficient starting point, and further technical clarification is best
176 done via email. 176 done via email.
177 177
178 Mitigation development 178 Mitigation development
179 """""""""""""""""""""" 179 """"""""""""""""""""""
180 180
181 The initial response team sets up an encrypted 181 The initial response team sets up an encrypted mailing-list or repurposes
182 an existing one if appropriate. 182 an existing one if appropriate.
183 183
184 Using a mailing list is close to the normal Li 184 Using a mailing list is close to the normal Linux development process and
185 has been successfully used to develop mitigati 185 has been successfully used to develop mitigations for various hardware
186 security issues in the past. 186 security issues in the past.
187 187
188 The mailing list operates in the same way as n 188 The mailing list operates in the same way as normal Linux development.
189 Patches are posted, discussed, and reviewed an 189 Patches are posted, discussed, and reviewed and if agreed upon, applied to
190 a non-public git repository which is only acce 190 a non-public git repository which is only accessible to the participating
191 developers via a secure connection. The reposi 191 developers via a secure connection. The repository contains the main
192 development branch against the mainline kernel 192 development branch against the mainline kernel and backport branches for
193 stable kernel versions as necessary. 193 stable kernel versions as necessary.
194 194
195 The initial response team will identify furthe 195 The initial response team will identify further experts from the Linux
196 kernel developer community as needed. Any inv 196 kernel developer community as needed. Any involved party can suggest
197 further experts to be included, each of which 197 further experts to be included, each of which will be subject to the same
198 requirements outlined above. 198 requirements outlined above.
199 199
200 Bringing in experts can happen at any time in 200 Bringing in experts can happen at any time in the development process and
201 needs to be handled in a timely manner. 201 needs to be handled in a timely manner.
202 202
203 If an expert is employed by or a member of an 203 If an expert is employed by or a member of an entity on the disclosure list
204 provided by the disclosing party, then partici 204 provided by the disclosing party, then participation will be requested from
205 the relevant entity. 205 the relevant entity.
206 206
207 If not, then the disclosing party will be info 207 If not, then the disclosing party will be informed about the experts'
208 participation. The experts are covered by the 208 participation. The experts are covered by the Memorandum of Understanding
209 and the disclosing party is requested to ackno 209 and the disclosing party is requested to acknowledge their participation.
210 In the case where the disclosing party has a c 210 In the case where the disclosing party has a compelling reason to object,
211 any objection must to be raised within five wo 211 any objection must to be raised within five working days and resolved with
212 the incident team immediately. If the disclosi 212 the incident team immediately. If the disclosing party does not react
213 within five working days this is taken as sile 213 within five working days this is taken as silent acknowledgment.
214 214
215 After the incident team acknowledges or resolv 215 After the incident team acknowledges or resolves an objection, the expert
216 is disclosed and brought into the development 216 is disclosed and brought into the development process.
217 217
218 List participants may not communicate about th 218 List participants may not communicate about the issue outside of the
219 private mailing list. List participants may no 219 private mailing list. List participants may not use any shared resources
220 (e.g. employer build farms, CI systems, etc) w 220 (e.g. employer build farms, CI systems, etc) when working on patches.
221 221
222 Early access 222 Early access
223 """""""""""" 223 """"""""""""
224 224
225 The patches discussed and developed on the lis 225 The patches discussed and developed on the list can neither be distributed
226 to any individual who is not a member of the r 226 to any individual who is not a member of the response team nor to any other
227 organization. 227 organization.
228 228
229 To allow the affected silicon vendors to work 229 To allow the affected silicon vendors to work with their internal teams and
230 industry partners on testing, validation, and 230 industry partners on testing, validation, and logistics, the following
231 exception is provided: 231 exception is provided:
232 232
233 Designated representatives of the affe 233 Designated representatives of the affected silicon vendors are
234 allowed to hand over the patches at an 234 allowed to hand over the patches at any time to the silicon
235 vendor’s response team. The represen 235 vendor’s response team. The representative must notify the kernel
236 response team about the handover. The 236 response team about the handover. The affected silicon vendor must
237 have and maintain their own documented 237 have and maintain their own documented security process for any
238 patches shared with their response tea 238 patches shared with their response team that is consistent with
239 this policy. 239 this policy.
240 240
241 The silicon vendor’s response team c 241 The silicon vendor’s response team can distribute these patches to
242 their industry partners and to their i 242 their industry partners and to their internal teams under the
243 silicon vendor’s documented security 243 silicon vendor’s documented security process. Feedback from the
244 industry partners goes back to the sil 244 industry partners goes back to the silicon vendor and is
245 communicated by the silicon vendor to 245 communicated by the silicon vendor to the kernel response team.
246 246
247 The handover to the silicon vendor’s 247 The handover to the silicon vendor’s response team removes any
248 responsibility or liability from the k 248 responsibility or liability from the kernel response team regarding
249 premature disclosure, which happens du 249 premature disclosure, which happens due to the involvement of the
250 silicon vendor’s internal teams or i 250 silicon vendor’s internal teams or industry partners. The silicon
251 vendor guarantees this release of liab 251 vendor guarantees this release of liability by agreeing to this
252 process. 252 process.
253 253
254 Coordinated release 254 Coordinated release
255 """"""""""""""""""" 255 """""""""""""""""""
256 256
257 The involved parties will negotiate the date a 257 The involved parties will negotiate the date and time when the embargo
258 ends. At that point, the prepared mitigations 258 ends. At that point, the prepared mitigations are published into the
259 relevant kernel trees. There is no pre-notifi 259 relevant kernel trees. There is no pre-notification process: the
260 mitigations are published in public and availa 260 mitigations are published in public and available to everyone at the same
261 time. 261 time.
262 262
263 While we understand that hardware security iss 263 While we understand that hardware security issues need coordinated embargo
264 time, the embargo time should be constrained t 264 time, the embargo time should be constrained to the minimum time that is
265 required for all involved parties to develop, 265 required for all involved parties to develop, test, and prepare their
266 mitigations. Extending embargo time artificial 266 mitigations. Extending embargo time artificially to meet conference talk
267 dates or other non-technical reasons creates m 267 dates or other non-technical reasons creates more work and burden for the
268 involved developers and response teams as the 268 involved developers and response teams as the patches need to be kept up to
269 date in order to follow the ongoing upstream k 269 date in order to follow the ongoing upstream kernel development, which
270 might create conflicting changes. 270 might create conflicting changes.
271 271
272 CVE assignment 272 CVE assignment
273 """""""""""""" 273 """"""""""""""
274 274
275 Neither the hardware security team nor the ini 275 Neither the hardware security team nor the initial response team assign
276 CVEs, nor are CVEs required for the developmen 276 CVEs, nor are CVEs required for the development process. If CVEs are
277 provided by the disclosing party they can be u 277 provided by the disclosing party they can be used for documentation
278 purposes. 278 purposes.
279 279
280 Process ambassadors 280 Process ambassadors
281 ------------------- 281 -------------------
282 282
283 For assistance with this process we have estab 283 For assistance with this process we have established ambassadors in various
284 organizations, who can answer questions about 284 organizations, who can answer questions about or provide guidance on the
285 reporting process and further handling. Ambass 285 reporting process and further handling. Ambassadors are not involved in the
286 disclosure of a particular issue, unless reque 286 disclosure of a particular issue, unless requested by a response team or by
287 an involved disclosed party. The current ambas 287 an involved disclosed party. The current ambassadors list:
288 288
289 ============= ============================== 289 ============= ========================================================
290 AMD Tom Lendacky <thomas.lendacky@a 290 AMD Tom Lendacky <thomas.lendacky@amd.com>
291 Ampere Darren Hart <darren@os.ampereco 291 Ampere Darren Hart <darren@os.amperecomputing.com>
292 ARM Catalin Marinas <catalin.marina 292 ARM Catalin Marinas <catalin.marinas@arm.com>
293 IBM Power Michael Ellerman <ellerman@au.i 293 IBM Power Michael Ellerman <ellerman@au.ibm.com>
294 IBM Z Christian Borntraeger <borntrae 294 IBM Z Christian Borntraeger <borntraeger@de.ibm.com>
295 Intel Tony Luck <tony.luck@intel.com> 295 Intel Tony Luck <tony.luck@intel.com>
296 Qualcomm Trilok Soni <quic_tsoni@quicinc 296 Qualcomm Trilok Soni <quic_tsoni@quicinc.com>
297 RISC-V Palmer Dabbelt <palmer@dabbelt. 297 RISC-V Palmer Dabbelt <palmer@dabbelt.com>
298 Samsung Javier González <javier.gonz@s 298 Samsung Javier González <javier.gonz@samsung.com>
299 299
300 Microsoft James Morris <jamorris@linux.mi 300 Microsoft James Morris <jamorris@linux.microsoft.com>
301 Xen Andrew Cooper <andrew.cooper3@c 301 Xen Andrew Cooper <andrew.cooper3@citrix.com>
302 302
303 Canonical John Johansen <john.johansen@ca 303 Canonical John Johansen <john.johansen@canonical.com>
304 Debian Ben Hutchings <ben@decadent.org 304 Debian Ben Hutchings <ben@decadent.org.uk>
305 Oracle Konrad Rzeszutek Wilk <konrad.w 305 Oracle Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
306 Red Hat Josh Poimboeuf <jpoimboe@redhat 306 Red Hat Josh Poimboeuf <jpoimboe@redhat.com>
307 SUSE Jiri Kosina <jkosina@suse.cz> 307 SUSE Jiri Kosina <jkosina@suse.cz>
308 308
309 Google Kees Cook <keescook@chromium.or 309 Google Kees Cook <keescook@chromium.org>
310 310
311 LLVM Nick Desaulniers <ndesaulniers@ 311 LLVM Nick Desaulniers <ndesaulniers@google.com>
312 ============= ============================== 312 ============= ========================================================
313 313
314 If you want your organization to be added to t 314 If you want your organization to be added to the ambassadors list, please
315 contact the hardware security team. The nomina 315 contact the hardware security team. The nominated ambassador has to
316 understand and support our process fully and i 316 understand and support our process fully and is ideally well-connected in
317 the Linux kernel community. 317 the Linux kernel community.
318 318
319 Encrypted mailing-lists 319 Encrypted mailing-lists
320 ----------------------- 320 -----------------------
321 321
322 We use encrypted mailing lists for communicati 322 We use encrypted mailing lists for communication. The operating principle
323 of these lists is that email sent to the list 323 of these lists is that email sent to the list is encrypted either with the
324 list's PGP key or with the list's S/MIME certi 324 list's PGP key or with the list's S/MIME certificate. The mailing list
325 software decrypts the email and re-encrypts it 325 software decrypts the email and re-encrypts it individually for each
326 subscriber with the subscriber's PGP key or S/ 326 subscriber with the subscriber's PGP key or S/MIME certificate. Details
327 about the mailing list software and the setup 327 about the mailing list software and the setup that is used to ensure the
328 security of the lists and protection of the da 328 security of the lists and protection of the data can be found here:
329 https://korg.wiki.kernel.org/userdoc/remail. 329 https://korg.wiki.kernel.org/userdoc/remail.
330 330
331 List keys 331 List keys
332 ^^^^^^^^^ 332 ^^^^^^^^^
333 333
334 For initial contact see the :ref:`Contact` sec 334 For initial contact see the :ref:`Contact` section above. For incident
335 specific mailing lists, the key and S/MIME cer 335 specific mailing lists, the key and S/MIME certificate are conveyed to the
336 subscribers by email sent from the specific li 336 subscribers by email sent from the specific list.
337 337
338 Subscription to incident-specific lists 338 Subscription to incident-specific lists
339 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 339 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
340 340
341 Subscription to incident-specific lists is han 341 Subscription to incident-specific lists is handled by the response teams.
342 Disclosed parties who want to participate in t 342 Disclosed parties who want to participate in the communication send a list
343 of potential experts to the response team so t 343 of potential experts to the response team so the response team can validate
344 subscription requests. 344 subscription requests.
345 345
346 Each subscriber needs to send a subscription r 346 Each subscriber needs to send a subscription request to the response team
347 by email. The email must be signed with the su 347 by email. The email must be signed with the subscriber's PGP key or S/MIME
348 certificate. If a PGP key is used, it must be 348 certificate. If a PGP key is used, it must be available from a public key
349 server and is ideally connected to the Linux k 349 server and is ideally connected to the Linux kernel's PGP web of trust. See
350 also: https://www.kernel.org/signature.html. 350 also: https://www.kernel.org/signature.html.
351 351
352 The response team verifies that the subscriber 352 The response team verifies that the subscriber request is valid and adds
353 the subscriber to the list. After subscription 353 the subscriber to the list. After subscription the subscriber will receive
354 email from the mailing-list which is signed ei 354 email from the mailing-list which is signed either with the list's PGP key
355 or the list's S/MIME certificate. The subscrib 355 or the list's S/MIME certificate. The subscriber's email client can extract
356 the PGP key or the S/MIME certificate from the 356 the PGP key or the S/MIME certificate from the signature so the subscriber
357 can send encrypted email to the list. 357 can send encrypted email to the list.
358 358
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.