1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY
2 .. [see the bottom of this file for redistribu
3
4 Reporting regressions
5 +++++++++++++++++++++
6
7 "*We don't cause regressions*" is the first ru
8 Linux founder and lead developer Linus Torvald
9 ensures it's obeyed.
10
11 This document describes what the rule means fo
12 development model ensures to address all repor
13 for kernel developers are left to Documentatio
14
15
16 The important bits (aka "TL;DR")
17 ================================
18
19 #. It's a regression if something running fine
20 or not at all with a newer version. Note, t
21 using a similar configuration; the detailed
22 and other fine print in more detail.
23
24 #. Report your issue as outlined in Documentat
25 it already covers all aspects important for
26 below for convenience. Two of them are impo
27 with "[REGRESSION]" and CC or forward it to
28 <https://lore.kernel.org/regressions/>`_ (r
29
30 #. Optional, but recommended: when sending or
31 Linux kernel regression tracking bot "regzb
32 when the regression started like this::
33
34 #regzbot introduced: v5.13..v5.14-rc1
35
36
37 All the details on Linux kernel regressions re
38 ==============================================
39
40
41 The important basics
42 --------------------
43
44
45 What is a "regression" and what is the "no reg
46 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
47
48 It's a regression if some application or pract
49 one Linux kernel works worse or not at all wit
50 similar configuration. The "no regressions" ru
51 it happens by accident, developers that caused
52 the issue.
53
54 It thus is a regression when a WiFi driver fro
55 5.14 doesn't work at all, works significantly
56 It's also a regression if a perfectly working
57 behavior with a newer kernel version; such iss
58 procfs, sysfs, or one of the many other interf
59 software. But keep in mind, as mentioned earli
60 be built from a configuration similar to the o
61 using ``make olddefconfig``, as explained in m
62
63 Note the "practical use case" in the first sen
64 despite the "no regressions" rule are free to
65 and even APIs or ABIs to userland, as long as
66 case breaks.
67
68 Also be aware the "no regressions" rule covers
69 provides to the userland. It thus does not app
70 like the module API, which some externally dev
71 the kernel.
72
73 How do I report a regression?
74 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
75
76 Just report the issue as outlined in
77 Documentation/admin-guide/reporting-issues.rst
78 important points. The following aspects outlin
79 for regressions:
80
81 * When checking for existing reports to join,
82 Linux regressions mailing list <https://lor
83 `regzbot's web-interface <https://linux-reg
84
85 * Start your report's subject with "[REGRESSI
86
87 * In your report, clearly mention the last ke
88 the first broken one. Ideally try to find t
89 regression using a bisection, as explained
90
91 * Remember to let the Linux regressions maili
92 (regressions@lists.linux.dev) know about yo
93
94 * If you report the regression by mail, CC
95
96 * If you report your regression to some bug
97 report by mail to the regressions list wh
98 mailing list for the subsystem in questio
99
100 If it's a regression within a stable or lon
101 v5.15.3..v5.15.5), remember to CC the `Linu
102 <https://lore.kernel.org/stable/>`_ (stable
103
104 In case you performed a successful bisection
105 culprit's commit message mentions in lines s
106
107 When CCing for forwarding your report to the l
108 aforementioned Linux kernel regression trackin
109 that, include a paragraph like this in your ma
110
111 #regzbot introduced: v5.13..v5.14-rc1
112
113 Regzbot will then consider your mail a report
114 specified version range. In above case Linux v
115 v5.14-rc1 was the first version where you enco
116 performed a bisection to find the commit that
117 culprit's commit-id instead::
118
119 #regzbot introduced: 1f2e3d4c5d
120
121 Placing such a "regzbot command" is in your in
122 report won't fall through the cracks unnoticed
123 kernel's regressions tracker will take care of
124 regression, as long as you send a copy to the
125 regression tracker is just one human which som
126 might even enjoy some time away from computers
127 Relying on this person thus will result in an
128 regressions becomes mentioned `on the list of
129 kernel regressions <https://linux-regtracking.
130 weekly regression reports sent by regzbot. Suc
131 Torvalds being unaware of important regression
132 development or call this finished and release
133
134 Are really all regressions fixed?
135 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
136
137 Nearly all of them are, as long as the change
138 "culprit commit") is reliably identified. Some
139 this, but often it's required.
140
141 Who needs to find the root cause of a regressi
142 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
143
144 Developers of the affected code area should tr
145 own. But for them that's often impossible to d
146 a lot of issues only occur in a particular env
147 reach -- for example, a specific hardware plat
148 system's configuration, or application. That's
149 the reporter to locate the culprit commit; som
150 run additional tests afterwards to pinpoint th
151 should offer advice and reasonably help where
152 relatively easy and achievable for typical use
153
154 How can I find the culprit?
155 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
156
157 Perform a bisection, as roughly outlined in
158 Documentation/admin-guide/reporting-issues.rst
159 Documentation/admin-guide/bug-bisect.rst. It m
160 in many cases finds the culprit relatively qui
161 time-consuming to reliably reproduce the issue
162 affected users to narrow down the search range
163
164 Who can I ask for advice when it comes to regr
165 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
166
167 Send a mail to the regressions mailing list (r
168 CCing the Linux kernel's regression tracker (r
169 issue might better be dealt with in private, f
170
171
172 Additional details about regressions
173 ------------------------------------
174
175
176 What is the goal of the "no regressions" rule?
177 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
178
179 Users should feel safe when updating kernel ve
180 something might break. This is in the interest
181 updating attractive: they don't want users to
182 series that are either abandoned or more than
183 in everybody's interest, as `those series migh
184 issues, or other problematic aspects already f
185 <http://www.kroah.com/log/blog/2018/08/24/what
186 Additionally, the kernel developers want to ma
187 users to test the latest pre-release or regula
188 everybody's interest, as it's a lot easier to
189 they are reported shortly after being introduc
190
191 Is the "no regressions" rule really adhered in
192 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
193
194 It's taken really seriously, as can be seen by
195 Linux creator and lead developer Linus Torvald
196 Documentation/process/handling-regressions.rst
197
198 Exceptions to this rule are extremely rare; in
199 turned out to be wrong when they assumed a par
200 an exception.
201
202 Who ensures the "no regressions" rule is actua
203 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
204
205 The subsystem maintainers should take care of
206 supported by the tree maintainers -- e.g. Linu
207 Greg Kroah-Hartman et al. for various stable/l
208
209 All of them are helped by people trying to ens
210 through the cracks. One of them is Thorsten Le
211 the Linux kernel's "regressions tracker"; to f
212 regzbot, the Linux kernel regression tracking
213 your report on the radar of these people by CC
214 the regressions mailing list, ideally with a "
215 get it tracked immediately.
216
217 How quickly are regressions normally fixed?
218 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
219
220 Developers should fix any reported regression
221 affected users with a solution in a timely man
222 running into the issue; nevertheless developer
223 care to ensure regression fixes do not cause a
224
225 The answer thus depends on various factors lik
226 age, or the Linux series in which it occurs. I
227 should be fixed within two weeks.
228
229 Is it a regression, if the issue can be avoide
230 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
231
232 Almost always: yes. If a developer tells you o
233 tracker for advice as outlined above.
234
235 Is it a regression, if a newer kernel works sl
236 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
237
238 Yes, but the difference has to be significant.
239 micro-benchmark thus is unlikely to qualify as
240 influences the results of a broad benchmark by
241 doubt, ask for advice.
242
243 Is it a regression, if an external kernel modu
244 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
245
246 No, as the "no regression" rule is about inter
247 kernel provides to the userland. It thus does
248 externally developed kernel modules, as they r
249 the kernel using internal interfaces occasiona
250
251 How are regressions handled that are caused by
252 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
253
254 In extremely rare situations security issues c
255 regressions; those fixes are given way, as the
256 Luckily this middling almost always can be avo
257 affected area and often Linus Torvalds himself
258 issues without causing regressions.
259
260 If you nevertheless face such a case, check th
261 tried their best to avoid the regression. If n
262 for advice as outlined above.
263
264 What happens if fixing a regression is impossi
265 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
266
267 Sadly these things happen, but luckily not ver
268 developers of the affected code area should lo
269 that avoids regressions or at least their impa
270 situation, do what was outlined already for re
271 fixes: check earlier discussions if people alr
272 advice if in doubt.
273
274 A quick note while at it: these situations cou
275 regularly give mainline pre-releases (say v5.1
276 development cycle a test run. This is best exp
277 integrated between Linux v5.14 and v5.15-rc1 w
278 the same time is a hard requirement for some o
279 5.15-rc1. All these changes often can simply b
280 solved, if someone finds and reports it before
281 weeks later this solution can become impossibl
282 started to rely on aspects introduced by one o
283 all changes would then cause a regression for
284 out of the question.
285
286 Is it a regression, if some feature I relied o
287 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
288
289 It is, but often it's hard to fix such regress
290 in the previous section. It hence needs to be
291 basis. This is another reason why it's in ever
292 mainline pre-releases.
293
294 Does the "no regression" rule apply if I seem
295 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
296
297 It does, but only for practical usage: the Lin
298 remove support for hardware only to be found i
299
300 Note, sometimes regressions can't be avoided t
301 is needed to prevent Linux from stagnation. He
302 to be affected by a regression, it for the gre
303 everyone else's interest to lettings things pa
304 easy way to circumvent the regression somehow,
305 software or using a kernel parameter created j
306
307 Does the regression rule apply for code in the
308 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
309
310 Not according to the `help text for the config
311 staging code <https://git.kernel.org/pub/scm/l
312 which since its early days states::
313
314 Please note that these drivers are unde
315 may not work, and may contain userspace
316 will be changed in the near future.
317
318 The staging developers nevertheless often adhe
319 but sometimes bend it to make progress. That's
320 deal with (often negligible) regressions when
321 tree was replaced by a totally different one w
322
323 Why do later versions have to be "compiled wit
324 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
325
326 Because the Linux kernel developers sometimes
327 regressions, but make them optional and disabl
328 configuration. This trick allows progress, as
329 otherwise would lead to stagnation.
330
331 Consider for example a new security feature bl
332 interfaces often abused by malware, which at t
333 few rarely used applications. The outlined app
334 people using these applications can leave the
335 everyone else can enable it without running in
336
337 How to create a configuration similar to the o
338 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
339
340 Start your machine with a known-good kernel an
341 version with ``make olddefconfig``. This makes
342 up the configuration file (the ".config" file)
343 for the new one you are about to compile; afte
344 configuration options to their default value,
345 that might cause regressions.
346
347 Can I report a regression I found with pre-com
348 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
349
350 You need to ensure the newer kernel was compil
351 file as the older one (see above), as those th
352 some known-to-be incompatible feature for the
353 the matter to the kernel's provider and ask fo
354
355
356 More about regression tracking with "regzbot"
357 ---------------------------------------------
358
359 What is regression tracking and why should I c
360 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
361
362 Rules like "no regressions" need someone to en
363 they are broken either accidentally or on purp
364 true for Linux kernel development as well. Tha
365 Linux Kernel's regression tracker, and some pe
366 are fixed by keeping an eye on them until they
367 paid for this, that's why the work is done on
368
369 Why and how are Linux kernel regressions track
370 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
371
372 Tracking regressions completely manually has p
373 distributed and loosely structured nature of L
374 That's why the Linux kernel's regression track
375 the work, with the long term goal to automate
376 possible for everyone involved.
377
378 Regzbot works by watching for replies to repor
379 Additionally, it's looking out for posted or c
380 reports with "Link:" tags; replies to such pat
381 Combined this data provides good insights into
382 process.
383
384 How to see which regressions regzbot tracks cu
385 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
386
387 Check out `regzbot's web-interface <https://li
388
389 What kind of issues are supposed to be tracked
390 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
391
392 The bot is meant to track regressions, hence p
393 regular issues. But it's okay for the Linux ke
394 involve regzbot to track severe issues, like r
395 data, or internal errors (Panic, Oops, BUG(),
396
397 How to change aspects of a tracked regression?
398 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
399
400 By using a 'regzbot command' in a direct or in
401 report. The easiest way to do that: find the r
402 mailing list archive and reply to it using you
403 In that mail, use one of the following command
404 use blank lines to separate one or multiple of
405 the mail's text).
406
407 * Update when the regression started to happe
408 bisection::
409
410 #regzbot introduced: 1f2e3d4c5d
411
412 * Set or update the title::
413
414 #regzbot title: foo
415
416 * Monitor a discussion or bugzilla.kernel.org
417 the issue or a fix are discussed:::
418
419 #regzbot monitor: https://lore.kernel.o
420 #regzbot monitor: https://bugzilla.kern
421
422 * Point to a place with further details of in
423 or a ticket in a bug tracker that are sligh
424 topic::
425
426 #regzbot link: https://bugzilla.kernel.
427
428 * Mark a regression as invalid::
429
430 #regzbot invalid: wasn't a regression,
431
432 Regzbot supports a few other commands primaril
433 tracking regressions. They and more details ab
434 commands can be found in the `getting started
435 <https://gitlab.com/knurd42/regzbot/-/blob/mai
436 the `reference documentation <https://gitlab.c
437 for regzbot.
438
439 ..
440 end-of-content
441 ..
442 This text is available under GPL-2.0+ or CC
443 of the file. If you want to distribute this
444 please use "The Linux kernel developers" fo
445 this as source:
446 https://git.kernel.org/pub/scm/linux/kernel
447 ..
448 Note: Only the content of this RST file as
449 is available under CC-BY-4.0, as versions o
450 (for example by the kernel's build system)
451 files which use a more restrictive license.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.