1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY 1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
2 .. [see the bottom of this file for redistribu 2 .. [see the bottom of this file for redistribution information]
3 3
4 Reporting regressions 4 Reporting regressions
5 +++++++++++++++++++++ 5 +++++++++++++++++++++
6 6
7 "*We don't cause regressions*" is the first ru 7 "*We don't cause regressions*" is the first rule of Linux kernel development;
8 Linux founder and lead developer Linus Torvald 8 Linux founder and lead developer Linus Torvalds established it himself and
9 ensures it's obeyed. 9 ensures it's obeyed.
10 10
11 This document describes what the rule means fo 11 This document describes what the rule means for users and how the Linux kernel's
12 development model ensures to address all repor 12 development model ensures to address all reported regressions; aspects relevant
13 for kernel developers are left to Documentatio 13 for kernel developers are left to Documentation/process/handling-regressions.rst.
14 14
15 15
16 The important bits (aka "TL;DR") 16 The important bits (aka "TL;DR")
17 ================================ 17 ================================
18 18
19 #. It's a regression if something running fine 19 #. It's a regression if something running fine with one Linux kernel works worse
20 or not at all with a newer version. Note, t 20 or not at all with a newer version. Note, the newer kernel has to be compiled
21 using a similar configuration; the detailed 21 using a similar configuration; the detailed explanations below describes this
22 and other fine print in more detail. 22 and other fine print in more detail.
23 23
24 #. Report your issue as outlined in Documentat 24 #. Report your issue as outlined in Documentation/admin-guide/reporting-issues.rst,
25 it already covers all aspects important for 25 it already covers all aspects important for regressions and repeated
26 below for convenience. Two of them are impo 26 below for convenience. Two of them are important: start your report's subject
27 with "[REGRESSION]" and CC or forward it to 27 with "[REGRESSION]" and CC or forward it to `the regression mailing list
28 <https://lore.kernel.org/regressions/>`_ (r 28 <https://lore.kernel.org/regressions/>`_ (regressions@lists.linux.dev).
29 29
30 #. Optional, but recommended: when sending or 30 #. Optional, but recommended: when sending or forwarding your report, make the
31 Linux kernel regression tracking bot "regzb 31 Linux kernel regression tracking bot "regzbot" track the issue by specifying
32 when the regression started like this:: 32 when the regression started like this::
33 33
34 #regzbot introduced: v5.13..v5.14-rc1 34 #regzbot introduced: v5.13..v5.14-rc1
35 35
36 36
37 All the details on Linux kernel regressions re 37 All the details on Linux kernel regressions relevant for users
38 ============================================== 38 ==============================================================
39 39
40 40
41 The important basics 41 The important basics
42 -------------------- 42 --------------------
43 43
44 44
45 What is a "regression" and what is the "no reg 45 What is a "regression" and what is the "no regressions" rule?
46 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 46 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
47 47
48 It's a regression if some application or pract 48 It's a regression if some application or practical use case running fine with
49 one Linux kernel works worse or not at all wit 49 one Linux kernel works worse or not at all with a newer version compiled using a
50 similar configuration. The "no regressions" ru 50 similar configuration. The "no regressions" rule forbids this to take place; if
51 it happens by accident, developers that caused 51 it happens by accident, developers that caused it are expected to quickly fix
52 the issue. 52 the issue.
53 53
54 It thus is a regression when a WiFi driver fro 54 It thus is a regression when a WiFi driver from Linux 5.13 works fine, but with
55 5.14 doesn't work at all, works significantly 55 5.14 doesn't work at all, works significantly slower, or misbehaves somehow.
56 It's also a regression if a perfectly working 56 It's also a regression if a perfectly working application suddenly shows erratic
57 behavior with a newer kernel version; such iss 57 behavior with a newer kernel version; such issues can be caused by changes in
58 procfs, sysfs, or one of the many other interf 58 procfs, sysfs, or one of the many other interfaces Linux provides to userland
59 software. But keep in mind, as mentioned earli 59 software. But keep in mind, as mentioned earlier: 5.14 in this example needs to
60 be built from a configuration similar to the o 60 be built from a configuration similar to the one from 5.13. This can be achieved
61 using ``make olddefconfig``, as explained in m 61 using ``make olddefconfig``, as explained in more detail below.
62 62
63 Note the "practical use case" in the first sen 63 Note the "practical use case" in the first sentence of this section: developers
64 despite the "no regressions" rule are free to 64 despite the "no regressions" rule are free to change any aspect of the kernel
65 and even APIs or ABIs to userland, as long as 65 and even APIs or ABIs to userland, as long as no existing application or use
66 case breaks. 66 case breaks.
67 67
68 Also be aware the "no regressions" rule covers 68 Also be aware the "no regressions" rule covers only interfaces the kernel
69 provides to the userland. It thus does not app 69 provides to the userland. It thus does not apply to kernel-internal interfaces
70 like the module API, which some externally dev 70 like the module API, which some externally developed drivers use to hook into
71 the kernel. 71 the kernel.
72 72
73 How do I report a regression? 73 How do I report a regression?
74 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 74 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
75 75
76 Just report the issue as outlined in 76 Just report the issue as outlined in
77 Documentation/admin-guide/reporting-issues.rst 77 Documentation/admin-guide/reporting-issues.rst, it already describes the
78 important points. The following aspects outlin 78 important points. The following aspects outlined there are especially relevant
79 for regressions: 79 for regressions:
80 80
81 * When checking for existing reports to join, 81 * When checking for existing reports to join, also search the `archives of the
82 Linux regressions mailing list <https://lor 82 Linux regressions mailing list <https://lore.kernel.org/regressions/>`_ and
83 `regzbot's web-interface <https://linux-reg 83 `regzbot's web-interface <https://linux-regtracking.leemhuis.info/regzbot/>`_.
84 84
85 * Start your report's subject with "[REGRESSI 85 * Start your report's subject with "[REGRESSION]".
86 86
87 * In your report, clearly mention the last ke 87 * In your report, clearly mention the last kernel version that worked fine and
88 the first broken one. Ideally try to find t 88 the first broken one. Ideally try to find the exact change causing the
89 regression using a bisection, as explained 89 regression using a bisection, as explained below in more detail.
90 90
91 * Remember to let the Linux regressions maili 91 * Remember to let the Linux regressions mailing list
92 (regressions@lists.linux.dev) know about yo 92 (regressions@lists.linux.dev) know about your report:
93 93
94 * If you report the regression by mail, CC 94 * If you report the regression by mail, CC the regressions list.
95 95
96 * If you report your regression to some bug 96 * If you report your regression to some bug tracker, forward the submitted
97 report by mail to the regressions list wh 97 report by mail to the regressions list while CCing the maintainer and the
98 mailing list for the subsystem in questio 98 mailing list for the subsystem in question.
99 99
100 If it's a regression within a stable or lon 100 If it's a regression within a stable or longterm series (e.g.
101 v5.15.3..v5.15.5), remember to CC the `Linu 101 v5.15.3..v5.15.5), remember to CC the `Linux stable mailing list
102 <https://lore.kernel.org/stable/>`_ (stable 102 <https://lore.kernel.org/stable/>`_ (stable@vger.kernel.org).
103 103
104 In case you performed a successful bisection 104 In case you performed a successful bisection, add everyone to the CC the
105 culprit's commit message mentions in lines s 105 culprit's commit message mentions in lines starting with "Signed-off-by:".
106 106
107 When CCing for forwarding your report to the l 107 When CCing for forwarding your report to the list, consider directly telling the
108 aforementioned Linux kernel regression trackin 108 aforementioned Linux kernel regression tracking bot about your report. To do
109 that, include a paragraph like this in your ma 109 that, include a paragraph like this in your mail::
110 110
111 #regzbot introduced: v5.13..v5.14-rc1 111 #regzbot introduced: v5.13..v5.14-rc1
112 112
113 Regzbot will then consider your mail a report 113 Regzbot will then consider your mail a report for a regression introduced in the
114 specified version range. In above case Linux v 114 specified version range. In above case Linux v5.13 still worked fine and Linux
115 v5.14-rc1 was the first version where you enco 115 v5.14-rc1 was the first version where you encountered the issue. If you
116 performed a bisection to find the commit that 116 performed a bisection to find the commit that caused the regression, specify the
117 culprit's commit-id instead:: 117 culprit's commit-id instead::
118 118
119 #regzbot introduced: 1f2e3d4c5d 119 #regzbot introduced: 1f2e3d4c5d
120 120
121 Placing such a "regzbot command" is in your in 121 Placing such a "regzbot command" is in your interest, as it will ensure the
122 report won't fall through the cracks unnoticed 122 report won't fall through the cracks unnoticed. If you omit this, the Linux
123 kernel's regressions tracker will take care of 123 kernel's regressions tracker will take care of telling regzbot about your
124 regression, as long as you send a copy to the 124 regression, as long as you send a copy to the regressions mailing lists. But the
125 regression tracker is just one human which som 125 regression tracker is just one human which sometimes has to rest or occasionally
126 might even enjoy some time away from computers 126 might even enjoy some time away from computers (as crazy as that might sound).
127 Relying on this person thus will result in an 127 Relying on this person thus will result in an unnecessary delay before the
128 regressions becomes mentioned `on the list of 128 regressions becomes mentioned `on the list of tracked and unresolved Linux
129 kernel regressions <https://linux-regtracking. 129 kernel regressions <https://linux-regtracking.leemhuis.info/regzbot/>`_ and the
130 weekly regression reports sent by regzbot. Suc 130 weekly regression reports sent by regzbot. Such delays can result in Linus
131 Torvalds being unaware of important regression 131 Torvalds being unaware of important regressions when deciding between "continue
132 development or call this finished and release 132 development or call this finished and release the final?".
133 133
134 Are really all regressions fixed? 134 Are really all regressions fixed?
135 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 135 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
136 136
137 Nearly all of them are, as long as the change 137 Nearly all of them are, as long as the change causing the regression (the
138 "culprit commit") is reliably identified. Some 138 "culprit commit") is reliably identified. Some regressions can be fixed without
139 this, but often it's required. 139 this, but often it's required.
140 140
141 Who needs to find the root cause of a regressi 141 Who needs to find the root cause of a regression?
142 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 142 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
143 143
144 Developers of the affected code area should tr 144 Developers of the affected code area should try to locate the culprit on their
145 own. But for them that's often impossible to d 145 own. But for them that's often impossible to do with reasonable effort, as quite
146 a lot of issues only occur in a particular env 146 a lot of issues only occur in a particular environment outside the developer's
147 reach -- for example, a specific hardware plat 147 reach -- for example, a specific hardware platform, firmware, Linux distro,
148 system's configuration, or application. That's 148 system's configuration, or application. That's why in the end it's often up to
149 the reporter to locate the culprit commit; som 149 the reporter to locate the culprit commit; sometimes users might even need to
150 run additional tests afterwards to pinpoint th 150 run additional tests afterwards to pinpoint the exact root cause. Developers
151 should offer advice and reasonably help where 151 should offer advice and reasonably help where they can, to make this process
152 relatively easy and achievable for typical use 152 relatively easy and achievable for typical users.
153 153
154 How can I find the culprit? 154 How can I find the culprit?
155 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 155 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
156 156
157 Perform a bisection, as roughly outlined in 157 Perform a bisection, as roughly outlined in
158 Documentation/admin-guide/reporting-issues.rst 158 Documentation/admin-guide/reporting-issues.rst and described in more detail by
159 Documentation/admin-guide/bug-bisect.rst. It m 159 Documentation/admin-guide/bug-bisect.rst. It might sound like a lot of work, but
160 in many cases finds the culprit relatively qui 160 in many cases finds the culprit relatively quickly. If it's hard or
161 time-consuming to reliably reproduce the issue 161 time-consuming to reliably reproduce the issue, consider teaming up with other
162 affected users to narrow down the search range 162 affected users to narrow down the search range together.
163 163
164 Who can I ask for advice when it comes to regr 164 Who can I ask for advice when it comes to regressions?
165 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 165 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
166 166
167 Send a mail to the regressions mailing list (r 167 Send a mail to the regressions mailing list (regressions@lists.linux.dev) while
168 CCing the Linux kernel's regression tracker (r 168 CCing the Linux kernel's regression tracker (regressions@leemhuis.info); if the
169 issue might better be dealt with in private, f 169 issue might better be dealt with in private, feel free to omit the list.
170 170
171 171
172 Additional details about regressions 172 Additional details about regressions
173 ------------------------------------ 173 ------------------------------------
174 174
175 175
176 What is the goal of the "no regressions" rule? 176 What is the goal of the "no regressions" rule?
177 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 177 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
178 178
179 Users should feel safe when updating kernel ve 179 Users should feel safe when updating kernel versions and not have to worry
180 something might break. This is in the interest 180 something might break. This is in the interest of the kernel developers to make
181 updating attractive: they don't want users to 181 updating attractive: they don't want users to stay on stable or longterm Linux
182 series that are either abandoned or more than 182 series that are either abandoned or more than one and a half years old. That's
183 in everybody's interest, as `those series migh 183 in everybody's interest, as `those series might have known bugs, security
184 issues, or other problematic aspects already f 184 issues, or other problematic aspects already fixed in later versions
185 <http://www.kroah.com/log/blog/2018/08/24/what 185 <http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/>`_.
186 Additionally, the kernel developers want to ma 186 Additionally, the kernel developers want to make it simple and appealing for
187 users to test the latest pre-release or regula 187 users to test the latest pre-release or regular release. That's also in
188 everybody's interest, as it's a lot easier to 188 everybody's interest, as it's a lot easier to track down and fix problems, if
189 they are reported shortly after being introduc 189 they are reported shortly after being introduced.
190 190
191 Is the "no regressions" rule really adhered in 191 Is the "no regressions" rule really adhered in practice?
192 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 192 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
193 193
194 It's taken really seriously, as can be seen by 194 It's taken really seriously, as can be seen by many mailing list posts from
195 Linux creator and lead developer Linus Torvald 195 Linux creator and lead developer Linus Torvalds, some of which are quoted in
196 Documentation/process/handling-regressions.rst 196 Documentation/process/handling-regressions.rst.
197 197
198 Exceptions to this rule are extremely rare; in 198 Exceptions to this rule are extremely rare; in the past developers almost always
199 turned out to be wrong when they assumed a par 199 turned out to be wrong when they assumed a particular situation was warranting
200 an exception. 200 an exception.
201 201
202 Who ensures the "no regressions" rule is actua 202 Who ensures the "no regressions" rule is actually followed?
203 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 203 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
204 204
205 The subsystem maintainers should take care of 205 The subsystem maintainers should take care of that, which are watched and
206 supported by the tree maintainers -- e.g. Linu 206 supported by the tree maintainers -- e.g. Linus Torvalds for mainline and
207 Greg Kroah-Hartman et al. for various stable/l 207 Greg Kroah-Hartman et al. for various stable/longterm series.
208 208
209 All of them are helped by people trying to ens 209 All of them are helped by people trying to ensure no regression report falls
210 through the cracks. One of them is Thorsten Le 210 through the cracks. One of them is Thorsten Leemhuis, who's currently acting as
211 the Linux kernel's "regressions tracker"; to f 211 the Linux kernel's "regressions tracker"; to facilitate this work he relies on
212 regzbot, the Linux kernel regression tracking 212 regzbot, the Linux kernel regression tracking bot. That's why you want to bring
213 your report on the radar of these people by CC 213 your report on the radar of these people by CCing or forwarding each report to
214 the regressions mailing list, ideally with a " 214 the regressions mailing list, ideally with a "regzbot command" in your mail to
215 get it tracked immediately. 215 get it tracked immediately.
216 216
217 How quickly are regressions normally fixed? 217 How quickly are regressions normally fixed?
218 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 218 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
219 219
220 Developers should fix any reported regression 220 Developers should fix any reported regression as quickly as possible, to provide
221 affected users with a solution in a timely man 221 affected users with a solution in a timely manner and prevent more users from
222 running into the issue; nevertheless developer 222 running into the issue; nevertheless developers need to take enough time and
223 care to ensure regression fixes do not cause a 223 care to ensure regression fixes do not cause additional damage.
224 224
225 The answer thus depends on various factors lik 225 The answer thus depends on various factors like the impact of a regression, its
226 age, or the Linux series in which it occurs. I 226 age, or the Linux series in which it occurs. In the end though, most regressions
227 should be fixed within two weeks. 227 should be fixed within two weeks.
228 228
229 Is it a regression, if the issue can be avoide 229 Is it a regression, if the issue can be avoided by updating some software?
230 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 230 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
231 231
232 Almost always: yes. If a developer tells you o 232 Almost always: yes. If a developer tells you otherwise, ask the regression
233 tracker for advice as outlined above. 233 tracker for advice as outlined above.
234 234
235 Is it a regression, if a newer kernel works sl 235 Is it a regression, if a newer kernel works slower or consumes more energy?
236 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 236 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
237 237
238 Yes, but the difference has to be significant. 238 Yes, but the difference has to be significant. A five percent slow-down in a
239 micro-benchmark thus is unlikely to qualify as 239 micro-benchmark thus is unlikely to qualify as regression, unless it also
240 influences the results of a broad benchmark by 240 influences the results of a broad benchmark by more than one percent. If in
241 doubt, ask for advice. 241 doubt, ask for advice.
242 242
243 Is it a regression, if an external kernel modu 243 Is it a regression, if an external kernel module breaks when updating Linux?
244 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 244 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
245 245
246 No, as the "no regression" rule is about inter 246 No, as the "no regression" rule is about interfaces and services the Linux
247 kernel provides to the userland. It thus does 247 kernel provides to the userland. It thus does not cover building or running
248 externally developed kernel modules, as they r 248 externally developed kernel modules, as they run in kernel-space and hook into
249 the kernel using internal interfaces occasiona 249 the kernel using internal interfaces occasionally changed.
250 250
251 How are regressions handled that are caused by 251 How are regressions handled that are caused by security fixes?
252 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 252 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
253 253
254 In extremely rare situations security issues c 254 In extremely rare situations security issues can't be fixed without causing
255 regressions; those fixes are given way, as the 255 regressions; those fixes are given way, as they are the lesser evil in the end.
256 Luckily this middling almost always can be avo 256 Luckily this middling almost always can be avoided, as key developers for the
257 affected area and often Linus Torvalds himself 257 affected area and often Linus Torvalds himself try very hard to fix security
258 issues without causing regressions. 258 issues without causing regressions.
259 259
260 If you nevertheless face such a case, check th 260 If you nevertheless face such a case, check the mailing list archives if people
261 tried their best to avoid the regression. If n 261 tried their best to avoid the regression. If not, report it; if in doubt, ask
262 for advice as outlined above. 262 for advice as outlined above.
263 263
264 What happens if fixing a regression is impossi 264 What happens if fixing a regression is impossible without causing another?
265 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 265 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
266 266
267 Sadly these things happen, but luckily not ver 267 Sadly these things happen, but luckily not very often; if they occur, expert
268 developers of the affected code area should lo 268 developers of the affected code area should look into the issue to find a fix
269 that avoids regressions or at least their impa 269 that avoids regressions or at least their impact. If you run into such a
270 situation, do what was outlined already for re 270 situation, do what was outlined already for regressions caused by security
271 fixes: check earlier discussions if people alr 271 fixes: check earlier discussions if people already tried their best and ask for
272 advice if in doubt. 272 advice if in doubt.
273 273
274 A quick note while at it: these situations cou 274 A quick note while at it: these situations could be avoided, if people would
275 regularly give mainline pre-releases (say v5.1 275 regularly give mainline pre-releases (say v5.15-rc1 or -rc3) from each
276 development cycle a test run. This is best exp 276 development cycle a test run. This is best explained by imagining a change
277 integrated between Linux v5.14 and v5.15-rc1 w 277 integrated between Linux v5.14 and v5.15-rc1 which causes a regression, but at
278 the same time is a hard requirement for some o 278 the same time is a hard requirement for some other improvement applied for
279 5.15-rc1. All these changes often can simply b 279 5.15-rc1. All these changes often can simply be reverted and the regression thus
280 solved, if someone finds and reports it before 280 solved, if someone finds and reports it before 5.15 is released. A few days or
281 weeks later this solution can become impossibl 281 weeks later this solution can become impossible, as some software might have
282 started to rely on aspects introduced by one o 282 started to rely on aspects introduced by one of the follow-up changes: reverting
283 all changes would then cause a regression for 283 all changes would then cause a regression for users of said software and thus is
284 out of the question. 284 out of the question.
285 285
286 Is it a regression, if some feature I relied o 286 Is it a regression, if some feature I relied on was removed months ago?
287 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 287 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
288 288
289 It is, but often it's hard to fix such regress 289 It is, but often it's hard to fix such regressions due to the aspects outlined
290 in the previous section. It hence needs to be 290 in the previous section. It hence needs to be dealt with on a case-by-case
291 basis. This is another reason why it's in ever 291 basis. This is another reason why it's in everybody's interest to regularly test
292 mainline pre-releases. 292 mainline pre-releases.
293 293
294 Does the "no regression" rule apply if I seem 294 Does the "no regression" rule apply if I seem to be the only affected person?
295 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 295 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
296 296
297 It does, but only for practical usage: the Lin 297 It does, but only for practical usage: the Linux developers want to be free to
298 remove support for hardware only to be found i 298 remove support for hardware only to be found in attics and museums anymore.
299 299
300 Note, sometimes regressions can't be avoided t 300 Note, sometimes regressions can't be avoided to make progress -- and the latter
301 is needed to prevent Linux from stagnation. He 301 is needed to prevent Linux from stagnation. Hence, if only very few users seem
302 to be affected by a regression, it for the gre 302 to be affected by a regression, it for the greater good might be in their and
303 everyone else's interest to lettings things pa 303 everyone else's interest to lettings things pass. Especially if there is an
304 easy way to circumvent the regression somehow, 304 easy way to circumvent the regression somehow, for example by updating some
305 software or using a kernel parameter created j 305 software or using a kernel parameter created just for this purpose.
306 306
307 Does the regression rule apply for code in the 307 Does the regression rule apply for code in the staging tree as well?
308 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 308 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
309 309
310 Not according to the `help text for the config 310 Not according to the `help text for the configuration option covering all
311 staging code <https://git.kernel.org/pub/scm/l 311 staging code <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/Kconfig>`_,
312 which since its early days states:: 312 which since its early days states::
313 313
314 Please note that these drivers are unde 314 Please note that these drivers are under heavy development, may or
315 may not work, and may contain userspace 315 may not work, and may contain userspace interfaces that most likely
316 will be changed in the near future. 316 will be changed in the near future.
317 317
318 The staging developers nevertheless often adhe 318 The staging developers nevertheless often adhere to the "no regressions" rule,
319 but sometimes bend it to make progress. That's 319 but sometimes bend it to make progress. That's for example why some users had to
320 deal with (often negligible) regressions when 320 deal with (often negligible) regressions when a WiFi driver from the staging
321 tree was replaced by a totally different one w 321 tree was replaced by a totally different one written from scratch.
322 322
323 Why do later versions have to be "compiled wit 323 Why do later versions have to be "compiled with a similar configuration"?
324 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 324 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
325 325
326 Because the Linux kernel developers sometimes 326 Because the Linux kernel developers sometimes integrate changes known to cause
327 regressions, but make them optional and disabl 327 regressions, but make them optional and disable them in the kernel's default
328 configuration. This trick allows progress, as 328 configuration. This trick allows progress, as the "no regressions" rule
329 otherwise would lead to stagnation. 329 otherwise would lead to stagnation.
330 330
331 Consider for example a new security feature bl 331 Consider for example a new security feature blocking access to some kernel
332 interfaces often abused by malware, which at t 332 interfaces often abused by malware, which at the same time are required to run a
333 few rarely used applications. The outlined app 333 few rarely used applications. The outlined approach makes both camps happy:
334 people using these applications can leave the 334 people using these applications can leave the new security feature off, while
335 everyone else can enable it without running in 335 everyone else can enable it without running into trouble.
336 336
337 How to create a configuration similar to the o 337 How to create a configuration similar to the one of an older kernel?
338 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 338 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
339 339
340 Start your machine with a known-good kernel an 340 Start your machine with a known-good kernel and configure the newer Linux
341 version with ``make olddefconfig``. This makes 341 version with ``make olddefconfig``. This makes the kernel's build scripts pick
342 up the configuration file (the ".config" file) 342 up the configuration file (the ".config" file) from the running kernel as base
343 for the new one you are about to compile; afte 343 for the new one you are about to compile; afterwards they set all new
344 configuration options to their default value, 344 configuration options to their default value, which should disable new features
345 that might cause regressions. 345 that might cause regressions.
346 346
347 Can I report a regression I found with pre-com 347 Can I report a regression I found with pre-compiled vanilla kernels?
348 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 348 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
349 349
350 You need to ensure the newer kernel was compil 350 You need to ensure the newer kernel was compiled with a similar configuration
351 file as the older one (see above), as those th 351 file as the older one (see above), as those that built them might have enabled
352 some known-to-be incompatible feature for the 352 some known-to-be incompatible feature for the newer kernel. If in doubt, report
353 the matter to the kernel's provider and ask fo 353 the matter to the kernel's provider and ask for advice.
354 354
355 355
356 More about regression tracking with "regzbot" 356 More about regression tracking with "regzbot"
357 --------------------------------------------- 357 ---------------------------------------------
358 358
359 What is regression tracking and why should I c 359 What is regression tracking and why should I care about it?
360 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 360 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
361 361
362 Rules like "no regressions" need someone to en 362 Rules like "no regressions" need someone to ensure they are followed, otherwise
363 they are broken either accidentally or on purp 363 they are broken either accidentally or on purpose. History has shown this to be
364 true for Linux kernel development as well. Tha 364 true for Linux kernel development as well. That's why Thorsten Leemhuis, the
365 Linux Kernel's regression tracker, and some pe 365 Linux Kernel's regression tracker, and some people try to ensure all regression
366 are fixed by keeping an eye on them until they 366 are fixed by keeping an eye on them until they are resolved. Neither of them are
367 paid for this, that's why the work is done on 367 paid for this, that's why the work is done on a best effort basis.
368 368
369 Why and how are Linux kernel regressions track 369 Why and how are Linux kernel regressions tracked using a bot?
370 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 370 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
371 371
372 Tracking regressions completely manually has p 372 Tracking regressions completely manually has proven to be quite hard due to the
373 distributed and loosely structured nature of L 373 distributed and loosely structured nature of Linux kernel development process.
374 That's why the Linux kernel's regression track 374 That's why the Linux kernel's regression tracker developed regzbot to facilitate
375 the work, with the long term goal to automate 375 the work, with the long term goal to automate regression tracking as much as
376 possible for everyone involved. 376 possible for everyone involved.
377 377
378 Regzbot works by watching for replies to repor 378 Regzbot works by watching for replies to reports of tracked regressions.
379 Additionally, it's looking out for posted or c 379 Additionally, it's looking out for posted or committed patches referencing such
380 reports with "Link:" tags; replies to such pat 380 reports with "Link:" tags; replies to such patch postings are tracked as well.
381 Combined this data provides good insights into 381 Combined this data provides good insights into the current state of the fixing
382 process. 382 process.
383 383
384 How to see which regressions regzbot tracks cu 384 How to see which regressions regzbot tracks currently?
385 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 385 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
386 386
387 Check out `regzbot's web-interface <https://li 387 Check out `regzbot's web-interface <https://linux-regtracking.leemhuis.info/regzbot/>`_.
388 388
389 What kind of issues are supposed to be tracked 389 What kind of issues are supposed to be tracked by regzbot?
390 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 390 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
391 391
392 The bot is meant to track regressions, hence p 392 The bot is meant to track regressions, hence please don't involve regzbot for
393 regular issues. But it's okay for the Linux ke 393 regular issues. But it's okay for the Linux kernel's regression tracker if you
394 involve regzbot to track severe issues, like r 394 involve regzbot to track severe issues, like reports about hangs, corrupted
395 data, or internal errors (Panic, Oops, BUG(), 395 data, or internal errors (Panic, Oops, BUG(), warning, ...).
396 396
397 How to change aspects of a tracked regression? 397 How to change aspects of a tracked regression?
398 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 398 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
399 399
400 By using a 'regzbot command' in a direct or in 400 By using a 'regzbot command' in a direct or indirect reply to the mail with the
401 report. The easiest way to do that: find the r 401 report. The easiest way to do that: find the report in your "Sent" folder or the
402 mailing list archive and reply to it using you 402 mailing list archive and reply to it using your mailer's "Reply-all" function.
403 In that mail, use one of the following command 403 In that mail, use one of the following commands in a stand-alone paragraph (IOW:
404 use blank lines to separate one or multiple of 404 use blank lines to separate one or multiple of these commands from the rest of
405 the mail's text). 405 the mail's text).
406 406
407 * Update when the regression started to happe 407 * Update when the regression started to happen, for example after performing a
408 bisection:: 408 bisection::
409 409
410 #regzbot introduced: 1f2e3d4c5d 410 #regzbot introduced: 1f2e3d4c5d
411 411
412 * Set or update the title:: 412 * Set or update the title::
413 413
414 #regzbot title: foo 414 #regzbot title: foo
415 415
416 * Monitor a discussion or bugzilla.kernel.org 416 * Monitor a discussion or bugzilla.kernel.org ticket where additions aspects of
417 the issue or a fix are discussed::: 417 the issue or a fix are discussed:::
418 418
419 #regzbot monitor: https://lore.kernel.o 419 #regzbot monitor: https://lore.kernel.org/r/30th.anniversary.repost@klaava.Helsinki.FI/
420 #regzbot monitor: https://bugzilla.kern 420 #regzbot monitor: https://bugzilla.kernel.org/show_bug.cgi?id=123456789
421 421
422 * Point to a place with further details of in 422 * Point to a place with further details of interest, like a mailing list post
423 or a ticket in a bug tracker that are sligh 423 or a ticket in a bug tracker that are slightly related, but about a different
424 topic:: 424 topic::
425 425
426 #regzbot link: https://bugzilla.kernel. 426 #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=123456789
427 427
428 * Mark a regression as invalid:: 428 * Mark a regression as invalid::
429 429
430 #regzbot invalid: wasn't a regression, 430 #regzbot invalid: wasn't a regression, problem has always existed
431 431
432 Regzbot supports a few other commands primaril 432 Regzbot supports a few other commands primarily used by developers or people
433 tracking regressions. They and more details ab 433 tracking regressions. They and more details about the aforementioned regzbot
434 commands can be found in the `getting started 434 commands can be found in the `getting started guide
435 <https://gitlab.com/knurd42/regzbot/-/blob/mai 435 <https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md>`_ and
436 the `reference documentation <https://gitlab.c 436 the `reference documentation <https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md>`_
437 for regzbot. 437 for regzbot.
438 438
439 .. 439 ..
440 end-of-content 440 end-of-content
441 .. 441 ..
442 This text is available under GPL-2.0+ or CC 442 This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
443 of the file. If you want to distribute this 443 of the file. If you want to distribute this text under CC-BY-4.0 only,
444 please use "The Linux kernel developers" fo 444 please use "The Linux kernel developers" for author attribution and link
445 this as source: 445 this as source:
446 https://git.kernel.org/pub/scm/linux/kernel 446 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-regressions.rst
447 .. 447 ..
448 Note: Only the content of this RST file as 448 Note: Only the content of this RST file as found in the Linux kernel sources
449 is available under CC-BY-4.0, as versions o 449 is available under CC-BY-4.0, as versions of this text that were processed
450 (for example by the kernel's build system) 450 (for example by the kernel's build system) might contain content taken from
451 files which use a more restrictive license. 451 files which use a more restrictive license.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.