1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY 1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
2 .. [see the bottom of this file for redistribu 2 .. [see the bottom of this file for redistribution information]
3 3
4 ========================================= 4 =========================================
5 How to verify bugs and bisect regressions 5 How to verify bugs and bisect regressions
6 ========================================= 6 =========================================
7 7
8 This document describes how to check if some L 8 This document describes how to check if some Linux kernel problem occurs in code
9 currently supported by developers -- to then e 9 currently supported by developers -- to then explain how to locate the change
10 causing the issue, if it is a regression (e.g. 10 causing the issue, if it is a regression (e.g. did not happen with earlier
11 versions). 11 versions).
12 12
13 The text aims at people running kernels from m 13 The text aims at people running kernels from mainstream Linux distributions on
14 commodity hardware who want to report a kernel 14 commodity hardware who want to report a kernel bug to the upstream Linux
15 developers. Despite this intent, the instructi 15 developers. Despite this intent, the instructions work just as well for users
16 who are already familiar with building their o 16 who are already familiar with building their own kernels: they help avoid
17 mistakes occasionally made even by experienced 17 mistakes occasionally made even by experienced developers.
18 18
19 .. 19 ..
20 Note: if you see this note, you are reading 20 Note: if you see this note, you are reading the text's source file. You
21 might want to switch to a rendered version: 21 might want to switch to a rendered version: it makes it a lot easier to
22 read and navigate this document -- especial 22 read and navigate this document -- especially when you want to look something
23 up in the reference section, then jump back 23 up in the reference section, then jump back to where you left off.
24 .. 24 ..
25 Find the latest rendered version of this te 25 Find the latest rendered version of this text here:
26 https://docs.kernel.org/admin-guide/verify- 26 https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
27 27
28 The essence of the process (aka 'TL;DR') 28 The essence of the process (aka 'TL;DR')
29 ======================================== 29 ========================================
30 30
31 *[If you are new to building or bisecting Linu 31 *[If you are new to building or bisecting Linux, ignore this section and head
32 over to the* ':ref:`step-by-step guide <introg 32 over to the* ':ref:`step-by-step guide <introguide_bissbs>`' *below. It utilizes
33 the same commands as this section while descri 33 the same commands as this section while describing them in brief fashion. The
34 steps are nevertheless easy to follow and toge 34 steps are nevertheless easy to follow and together with accompanying entries
35 in a reference section mention many alternativ 35 in a reference section mention many alternatives, pitfalls, and additional
36 aspects, all of which might be essential in yo 36 aspects, all of which might be essential in your present case.]*
37 37
38 **In case you want to check if a bug is presen 38 **In case you want to check if a bug is present in code currently supported by
39 developers**, execute just the *preparations* 39 developers**, execute just the *preparations* and *segment 1*; while doing so,
40 consider the newest Linux kernel you regularly 40 consider the newest Linux kernel you regularly use to be the 'working' kernel.
41 In the following example that's assumed to be 41 In the following example that's assumed to be 6.0, which is why its sources
42 will be used to prepare the .config file. 42 will be used to prepare the .config file.
43 43
44 **In case you face a regression**, follow the 44 **In case you face a regression**, follow the steps at least till the end of
45 *segment 2*. Then you can submit a preliminary 45 *segment 2*. Then you can submit a preliminary report -- or continue with
46 *segment 3*, which describes how to perform a 46 *segment 3*, which describes how to perform a bisection needed for a
47 full-fledged regression report. In the followi 47 full-fledged regression report. In the following example 6.0.13 is assumed to be
48 the 'working' kernel and 6.1.5 to be the first 48 the 'working' kernel and 6.1.5 to be the first 'broken', which is why 6.0
49 will be considered the 'good' release and used 49 will be considered the 'good' release and used to prepare the .config file.
50 50
51 * **Preparations**: set up everything to build 51 * **Preparations**: set up everything to build your own kernels::
52 52
53 # * Remove any software that depends on ex 53 # * Remove any software that depends on externally maintained kernel modules
54 # or builds any automatically during boo 54 # or builds any automatically during bootup.
55 # * Ensure Secure Boot permits booting sel 55 # * Ensure Secure Boot permits booting self-compiled Linux kernels.
56 # * If you are not already running the 'wo 56 # * If you are not already running the 'working' kernel, reboot into it.
57 # * Install compilers and everything else 57 # * Install compilers and everything else needed for building Linux.
58 # * Ensure to have 15 Gigabyte free space 58 # * Ensure to have 15 Gigabyte free space in your home directory.
59 git clone -o mainline --no-checkout \ 59 git clone -o mainline --no-checkout \
60 https://git.kernel.org/pub/scm/linux/ker 60 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
61 cd ~/linux/ 61 cd ~/linux/
62 git remote add -t master stable \ 62 git remote add -t master stable \
63 https://git.kernel.org/pub/scm/linux/ker 63 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
64 git switch --detach v6.0 64 git switch --detach v6.0
65 # * Hint: if you used an existing clone, e 65 # * Hint: if you used an existing clone, ensure no stale .config is around.
66 make olddefconfig 66 make olddefconfig
67 # * Ensure the former command picked the . 67 # * Ensure the former command picked the .config of the 'working' kernel.
68 # * Connect external hardware (USB keys, t 68 # * Connect external hardware (USB keys, tokens, ...), start a VM, bring up
69 # VPNs, mount network shares, and briefl 69 # VPNs, mount network shares, and briefly try the feature that is broken.
70 yes '' | make localmodconfig 70 yes '' | make localmodconfig
71 ./scripts/config --set-str CONFIG_LOCALVER 71 ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
72 ./scripts/config -e CONFIG_LOCALVERSION_AU 72 ./scripts/config -e CONFIG_LOCALVERSION_AUTO
73 # * Note, when short on storage space, che 73 # * Note, when short on storage space, check the guide for an alternative:
74 ./scripts/config -d DEBUG_INFO_NONE -e KAL 74 ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
75 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCH 75 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
76 # * Hint: at this point you might want to 76 # * Hint: at this point you might want to adjust the build configuration;
77 # you'll have to, if you are running Deb 77 # you'll have to, if you are running Debian.
78 make olddefconfig 78 make olddefconfig
79 cp .config ~/kernel-config-working 79 cp .config ~/kernel-config-working
80 80
81 * **Segment 1**: build a kernel from the lates 81 * **Segment 1**: build a kernel from the latest mainline codebase.
82 82
83 This among others checks if the problem was 83 This among others checks if the problem was fixed already and which developers
84 later need to be told about the problem; in 84 later need to be told about the problem; in case of a regression, this rules
85 out a .config change as root of the problem. 85 out a .config change as root of the problem.
86 86
87 a) Checking out latest mainline code:: 87 a) Checking out latest mainline code::
88 88
89 cd ~/linux/ 89 cd ~/linux/
90 git switch --discard-changes --detach m 90 git switch --discard-changes --detach mainline/master
91 91
92 b) Build, install, and boot a kernel:: 92 b) Build, install, and boot a kernel::
93 93
94 cp ~/kernel-config-working .config 94 cp ~/kernel-config-working .config
95 make olddefconfig 95 make olddefconfig
96 make -j $(nproc --all) 96 make -j $(nproc --all)
97 # * Make sure there is enough disk spac 97 # * Make sure there is enough disk space to hold another kernel:
98 df -h /boot/ /lib/modules/ 98 df -h /boot/ /lib/modules/
99 # * Note: on Arch Linux, its derivative 99 # * Note: on Arch Linux, its derivatives and a few other distributions
100 # the following commands will do noth 100 # the following commands will do nothing at all or only part of the
101 # job. See the step-by-step guide for 101 # job. See the step-by-step guide for further details.
102 sudo make modules_install 102 sudo make modules_install
103 command -v installkernel && sudo make i 103 command -v installkernel && sudo make install
104 # * Check how much space your self-buil 104 # * Check how much space your self-built kernel actually needs, which
105 # enables you to make better estimate 105 # enables you to make better estimates later:
106 du -ch /boot/*$(make -s kernelrelease)* 106 du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
107 du -sh /lib/modules/$(make -s kernelrel 107 du -sh /lib/modules/$(make -s kernelrelease)/
108 # * Hint: the output of the following c 108 # * Hint: the output of the following command will help you pick the
109 # right kernel from the boot menu: 109 # right kernel from the boot menu:
110 make -s kernelrelease | tee -a ~/kernel 110 make -s kernelrelease | tee -a ~/kernels-built
111 reboot 111 reboot
112 # * Once booted, ensure you are running 112 # * Once booted, ensure you are running the kernel you just built by
113 # checking if the output of the next 113 # checking if the output of the next two commands matches:
114 tail -n 1 ~/kernels-built 114 tail -n 1 ~/kernels-built
115 uname -r 115 uname -r
116 cat /proc/sys/kernel/tainted 116 cat /proc/sys/kernel/tainted
117 117
118 c) Check if the problem occurs with this ker 118 c) Check if the problem occurs with this kernel as well.
119 119
120 * **Segment 2**: ensure the 'good' kernel is a 120 * **Segment 2**: ensure the 'good' kernel is also a 'working' kernel.
121 121
122 This among others verifies the trimmed .conf 122 This among others verifies the trimmed .config file actually works well, as
123 bisecting with it otherwise would be a waste 123 bisecting with it otherwise would be a waste of time:
124 124
125 a) Start by checking out the sources of the 125 a) Start by checking out the sources of the 'good' version::
126 126
127 cd ~/linux/ 127 cd ~/linux/
128 git switch --discard-changes --detach v 128 git switch --discard-changes --detach v6.0
129 129
130 b) Build, install, and boot a kernel as desc 130 b) Build, install, and boot a kernel as described earlier in *segment 1,
131 section b* -- just feel free to skip the 131 section b* -- just feel free to skip the 'du' commands, as you have a rough
132 estimate already. 132 estimate already.
133 133
134 c) Ensure the feature that regressed with th 134 c) Ensure the feature that regressed with the 'broken' kernel actually works
135 with this one. 135 with this one.
136 136
137 * **Segment 3**: perform and validate the bise 137 * **Segment 3**: perform and validate the bisection.
138 138
139 a) Retrieve the sources for your 'bad' versi 139 a) Retrieve the sources for your 'bad' version::
140 140
141 git remote set-branches --add stable li 141 git remote set-branches --add stable linux-6.1.y
142 git fetch stable 142 git fetch stable
143 143
144 b) Initialize the bisection:: 144 b) Initialize the bisection::
145 145
146 cd ~/linux/ 146 cd ~/linux/
147 git bisect start 147 git bisect start
148 git bisect good v6.0 148 git bisect good v6.0
149 git bisect bad v6.1.5 149 git bisect bad v6.1.5
150 150
151 c) Build, install, and boot a kernel as desc 151 c) Build, install, and boot a kernel as described earlier in *segment 1,
152 section b*. 152 section b*.
153 153
154 In case building or booting the kernel fa 154 In case building or booting the kernel fails for unrelated reasons, run
155 ``git bisect skip``. In all other outcome 155 ``git bisect skip``. In all other outcomes, check if the regressed feature
156 works with the newly built kernel. If it 156 works with the newly built kernel. If it does, tell Git by executing
157 ``git bisect good``; if it does not, run 157 ``git bisect good``; if it does not, run ``git bisect bad`` instead.
158 158
159 All three commands will make Git check ou 159 All three commands will make Git check out another commit; then re-execute
160 this step (e.g. build, install, boot, and 160 this step (e.g. build, install, boot, and test a kernel to then tell Git
161 the outcome). Do so again and again until 161 the outcome). Do so again and again until Git shows which commit broke
162 things. If you run short of disk space du 162 things. If you run short of disk space during this process, check the
163 section 'Complementary tasks: cleanup dur 163 section 'Complementary tasks: cleanup during and after the process'
164 below. 164 below.
165 165
166 d) Once your finished the bisection, put a f 166 d) Once your finished the bisection, put a few things away::
167 167
168 cd ~/linux/ 168 cd ~/linux/
169 git bisect log > ~/bisect-log 169 git bisect log > ~/bisect-log
170 cp .config ~/bisection-config-culprit 170 cp .config ~/bisection-config-culprit
171 git bisect reset 171 git bisect reset
172 172
173 e) Try to verify the bisection result:: 173 e) Try to verify the bisection result::
174 174
175 git switch --discard-changes --detach m 175 git switch --discard-changes --detach mainline/master
176 git revert --no-edit cafec0cacaca0 176 git revert --no-edit cafec0cacaca0
177 cp ~/kernel-config-working .config 177 cp ~/kernel-config-working .config
178 ./scripts/config --set-str CONFIG_LOCAL 178 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
179 179
180 This is optional, as some commits are impo 180 This is optional, as some commits are impossible to revert. But if the
181 second command worked flawlessly, build, i 181 second command worked flawlessly, build, install, and boot one more kernel
182 kernel; just this time skip the first comm 182 kernel; just this time skip the first command copying the base .config file
183 over, as that already has been taken care 183 over, as that already has been taken care off.
184 184
185 * **Complementary tasks**: cleanup during and 185 * **Complementary tasks**: cleanup during and after the process.
186 186
187 a) To avoid running out of disk space during 187 a) To avoid running out of disk space during a bisection, you might need to
188 remove some kernels you built earlier. Yo 188 remove some kernels you built earlier. You most likely want to keep those
189 you built during segment 1 and 2 around f 189 you built during segment 1 and 2 around for a while, but you will most
190 likely no longer need kernels tested duri 190 likely no longer need kernels tested during the actual bisection
191 (Segment 3 c). You can list them in build 191 (Segment 3 c). You can list them in build order using::
192 192
193 ls -ltr /lib/modules/*-local* 193 ls -ltr /lib/modules/*-local*
194 194
195 To then for example erase a kernel that id 195 To then for example erase a kernel that identifies itself as
196 '6.0-rc1-local-gcafec0cacaca0', use this:: 196 '6.0-rc1-local-gcafec0cacaca0', use this::
197 197
198 sudo rm -rf /lib/modules/6.0-rc1-local- 198 sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
199 sudo kernel-install -v remove 6.0-rc1-l 199 sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
200 # * Note, on some distributions kernel- 200 # * Note, on some distributions kernel-install is missing
201 # or does only part of the job. 201 # or does only part of the job.
202 202
203 b) If you performed a bisection and successf 203 b) If you performed a bisection and successfully validated the result, feel
204 free to remove all kernels built during t 204 free to remove all kernels built during the actual bisection (Segment 3 c);
205 the kernels you built earlier and later y 205 the kernels you built earlier and later you might want to keep around for
206 a week or two. 206 a week or two.
207 207
208 * **Optional task**: test a debug patch or a p 208 * **Optional task**: test a debug patch or a proposed fix later::
209 209
210 git fetch mainline 210 git fetch mainline
211 git switch --discard-changes --detach main 211 git switch --discard-changes --detach mainline/master
212 git apply /tmp/foobars-proposed-fix-v1.pat 212 git apply /tmp/foobars-proposed-fix-v1.patch
213 cp ~/kernel-config-working .config 213 cp ~/kernel-config-working .config
214 ./scripts/config --set-str CONFIG_LOCALVER 214 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
215 215
216 Build, install, and boot a kernel as describ 216 Build, install, and boot a kernel as described in *segment 1, section b* --
217 but this time omit the first command copying 217 but this time omit the first command copying the build configuration over,
218 as that has been taken care of already. 218 as that has been taken care of already.
219 219
220 .. _introguide_bissbs: 220 .. _introguide_bissbs:
221 221
222 Step-by-step guide on how to verify bugs and b 222 Step-by-step guide on how to verify bugs and bisect regressions
223 ============================================== 223 ===============================================================
224 224
225 This guide describes how to set up your own Li 225 This guide describes how to set up your own Linux kernels for investigating bugs
226 or regressions you intend to report. How far y 226 or regressions you intend to report. How far you want to follow the instructions
227 depends on your issue: 227 depends on your issue:
228 228
229 Execute all steps till the end of *segment 1* 229 Execute all steps till the end of *segment 1* to **verify if your kernel problem
230 is present in code supported by Linux kernel d 230 is present in code supported by Linux kernel developers**. If it is, you are all
231 set to report the bug -- unless it did not hap 231 set to report the bug -- unless it did not happen with earlier kernel versions,
232 as then your want to at least continue with *s 232 as then your want to at least continue with *segment 2* to **check if the issue
233 qualifies as regression** which receive priori 233 qualifies as regression** which receive priority treatment. Depending on the
234 outcome you then are ready to report a bug or 234 outcome you then are ready to report a bug or submit a preliminary regression
235 report; instead of the latter your could also 235 report; instead of the latter your could also head straight on and follow
236 *segment 3* to **perform a bisection** for a f 236 *segment 3* to **perform a bisection** for a full-fledged regression report
237 developers are obliged to act upon. 237 developers are obliged to act upon.
238 238
239 :ref:`Preparations: set up everything to buil 239 :ref:`Preparations: set up everything to build your own kernels <introprep_bissbs>`.
240 240
241 :ref:`Segment 1: try to reproduce the problem 241 :ref:`Segment 1: try to reproduce the problem with the latest codebase <introlatestcheck_bissbs>`.
242 242
243 :ref:`Segment 2: check if the kernels you bui 243 :ref:`Segment 2: check if the kernels you build work fine <introworkingcheck_bissbs>`.
244 244
245 :ref:`Segment 3: perform a bisection and vali 245 :ref:`Segment 3: perform a bisection and validate the result <introbisect_bissbs>`.
246 246
247 :ref:`Complementary tasks: cleanup during and 247 :ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`.
248 248
249 :ref:`Optional tasks: test reverts, patches, 249 :ref:`Optional tasks: test reverts, patches, or later versions <introoptional_bissbs>`.
250 250
251 The steps in each segment illustrate the impor 251 The steps in each segment illustrate the important aspects of the process, while
252 a comprehensive reference section holds additi 252 a comprehensive reference section holds additional details for almost all of the
253 steps. The reference section sometimes also ou 253 steps. The reference section sometimes also outlines alternative approaches,
254 pitfalls, as well as problems that might occur 254 pitfalls, as well as problems that might occur at the particular step -- and how
255 to get things rolling again. 255 to get things rolling again.
256 256
257 For further details on how to report Linux ker 257 For further details on how to report Linux kernel issues or regressions check
258 out Documentation/admin-guide/reporting-issues 258 out Documentation/admin-guide/reporting-issues.rst, which works in conjunction
259 with this document. It among others explains w 259 with this document. It among others explains why you need to verify bugs with
260 the latest 'mainline' kernel (e.g. versions li 260 the latest 'mainline' kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6),
261 even if you face a problem with a kernel from 261 even if you face a problem with a kernel from a 'stable/longterm' series
262 (say 6.0.13). 262 (say 6.0.13).
263 263
264 For users facing a regression that document al 264 For users facing a regression that document also explains why sending a
265 preliminary report after segment 2 might be wi 265 preliminary report after segment 2 might be wise, as the regression and its
266 culprit might be known already. For further de 266 culprit might be known already. For further details on what actually qualifies
267 as a regression check out Documentation/admin- 267 as a regression check out Documentation/admin-guide/reporting-regressions.rst.
268 268
269 If you run into any problems while following t 269 If you run into any problems while following this guide or have ideas how to
270 improve it, :ref:`please let the kernel develo 270 improve it, :ref:`please let the kernel developers know <submit_improvements>`.
271 271
272 .. _introprep_bissbs: 272 .. _introprep_bissbs:
273 273
274 Preparations: set up everything to build your 274 Preparations: set up everything to build your own kernels
275 ---------------------------------------------- 275 ---------------------------------------------------------
276 276
277 The following steps lay the groundwork for all 277 The following steps lay the groundwork for all further tasks.
278 278
279 Note: the instructions assume you are building 279 Note: the instructions assume you are building and testing on the same
280 machine; if you want to compile the kernel on 280 machine; if you want to compile the kernel on another system, check
281 :ref:`Build kernels on a different machine <bu 281 :ref:`Build kernels on a different machine <buildhost_bis>` below.
282 282
283 .. _backup_bissbs: 283 .. _backup_bissbs:
284 284
285 * Create a fresh backup and put system repair 285 * Create a fresh backup and put system repair and restore tools at hand, just
286 to be prepared for the unlikely case of some 286 to be prepared for the unlikely case of something going sideways.
287 287
288 [:ref:`details <backup_bisref>`] 288 [:ref:`details <backup_bisref>`]
289 289
290 .. _vanilla_bissbs: 290 .. _vanilla_bissbs:
291 291
292 * Remove all software that depends on external 292 * Remove all software that depends on externally developed kernel drivers or
293 builds them automatically. That includes but 293 builds them automatically. That includes but is not limited to DKMS, openZFS,
294 VirtualBox, and Nvidia's graphics drivers (i 294 VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module).
295 295
296 [:ref:`details <vanilla_bisref>`] 296 [:ref:`details <vanilla_bisref>`]
297 297
298 .. _secureboot_bissbs: 298 .. _secureboot_bissbs:
299 299
300 * On platforms with 'Secure Boot' or similar s 300 * On platforms with 'Secure Boot' or similar solutions, prepare everything to
301 ensure the system will permit your self-comp 301 ensure the system will permit your self-compiled kernel to boot. The
302 quickest and easiest way to achieve this on 302 quickest and easiest way to achieve this on commodity x86 systems is to
303 disable such techniques in the BIOS setup ut 303 disable such techniques in the BIOS setup utility; alternatively, remove
304 their restrictions through a process initiat 304 their restrictions through a process initiated by
305 ``mokutil --disable-validation``. 305 ``mokutil --disable-validation``.
306 306
307 [:ref:`details <secureboot_bisref>`] 307 [:ref:`details <secureboot_bisref>`]
308 308
309 .. _rangecheck_bissbs: 309 .. _rangecheck_bissbs:
310 310
311 * Determine the kernel versions considered 'go 311 * Determine the kernel versions considered 'good' and 'bad' throughout this
312 guide: 312 guide:
313 313
314 * Do you follow this guide to verify if a bu 314 * Do you follow this guide to verify if a bug is present in the code the
315 primary developers care for? Then consider 315 primary developers care for? Then consider the version of the newest kernel
316 you regularly use currently as 'good' (e.g 316 you regularly use currently as 'good' (e.g. 6.0, 6.0.13, or 6.1-rc2).
317 317
318 * Do you face a regression, e.g. something b 318 * Do you face a regression, e.g. something broke or works worse after
319 switching to a newer kernel version? In th 319 switching to a newer kernel version? In that case it depends on the version
320 range during which the problem appeared: 320 range during which the problem appeared:
321 321
322 * Something regressed when updating from a 322 * Something regressed when updating from a stable/longterm release
323 (say 6.0.13) to a newer mainline series 323 (say 6.0.13) to a newer mainline series (like 6.1-rc7 or 6.1) or a
324 stable/longterm version based on one (sa 324 stable/longterm version based on one (say 6.1.5)? Then consider the
325 mainline release your working kernel is 325 mainline release your working kernel is based on to be the 'good'
326 version (e.g. 6.0) and the first version 326 version (e.g. 6.0) and the first version to be broken as the 'bad' one
327 (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at 327 (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at this point it is merely assumed
328 that 6.0 is fine; this hypothesis will b 328 that 6.0 is fine; this hypothesis will be checked in segment 2.
329 329
330 * Something regressed when switching from 330 * Something regressed when switching from one mainline version (say 6.0) to
331 a later one (like 6.1-rc1) or a stable/l 331 a later one (like 6.1-rc1) or a stable/longterm release based on it
332 (say 6.1.5)? Then regard the last workin 332 (say 6.1.5)? Then regard the last working version (e.g. 6.0) as 'good' and
333 the first broken (e.g. 6.1-rc1 or 6.1.5) 333 the first broken (e.g. 6.1-rc1 or 6.1.5) as 'bad'.
334 334
335 * Something regressed when updating within 335 * Something regressed when updating within a stable/longterm series (say
336 from 6.0.13 to 6.0.15)? Then consider th 336 from 6.0.13 to 6.0.15)? Then consider those versions as 'good' and 'bad'
337 (e.g. 6.0.13 and 6.0.15), as you need to 337 (e.g. 6.0.13 and 6.0.15), as you need to bisect within that series.
338 338
339 *Note, do not confuse 'good' version with 'w 339 *Note, do not confuse 'good' version with 'working' kernel; the latter term
340 throughout this guide will refer to the last 340 throughout this guide will refer to the last kernel that has been working
341 fine.* 341 fine.*
342 342
343 [:ref:`details <rangecheck_bisref>`] 343 [:ref:`details <rangecheck_bisref>`]
344 344
345 .. _bootworking_bissbs: 345 .. _bootworking_bissbs:
346 346
347 * Boot into the 'working' kernel and briefly u 347 * Boot into the 'working' kernel and briefly use the apparently broken feature.
348 348
349 [:ref:`details <bootworking_bisref>`] 349 [:ref:`details <bootworking_bisref>`]
350 350
351 .. _diskspace_bissbs: 351 .. _diskspace_bissbs:
352 352
353 * Ensure to have enough free space for buildin 353 * Ensure to have enough free space for building Linux. 15 Gigabyte in your home
354 directory should typically suffice. If you h 354 directory should typically suffice. If you have less available, be sure to pay
355 attention to later steps about retrieving th 355 attention to later steps about retrieving the Linux sources and handling of
356 debug symbols: both explain approaches reduc 356 debug symbols: both explain approaches reducing the amount of space, which
357 should allow you to master these tasks with 357 should allow you to master these tasks with about 4 Gigabytes free space.
358 358
359 [:ref:`details <diskspace_bisref>`] 359 [:ref:`details <diskspace_bisref>`]
360 360
361 .. _buildrequires_bissbs: 361 .. _buildrequires_bissbs:
362 362
363 * Install all software required to build a Lin 363 * Install all software required to build a Linux kernel. Often you will need:
364 'bc', 'binutils' ('ld' et al.), 'bison', 'fl 364 'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl',
365 'pahole', 'perl', and the development header 365 'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The
366 reference section shows how to quickly insta 366 reference section shows how to quickly install those on various popular Linux
367 distributions. 367 distributions.
368 368
369 [:ref:`details <buildrequires_bisref>`] 369 [:ref:`details <buildrequires_bisref>`]
370 370
371 .. _sources_bissbs: 371 .. _sources_bissbs:
372 372
373 * Retrieve the mainline Linux sources; then ch 373 * Retrieve the mainline Linux sources; then change into the directory holding
374 them, as all further commands in this guide 374 them, as all further commands in this guide are meant to be executed from
375 there. 375 there.
376 376
377 *Note, the following describe how to retriev 377 *Note, the following describe how to retrieve the sources using a full
378 mainline clone, which downloads about 2,75 G 378 mainline clone, which downloads about 2,75 GByte as of early 2024. The*
379 :ref:`reference section describes two altern 379 :ref:`reference section describes two alternatives <sources_bisref>` *:
380 one downloads less than 500 MByte, the other 380 one downloads less than 500 MByte, the other works better with unreliable
381 internet connections.* 381 internet connections.*
382 382
383 Execute the following command to retrieve a 383 Execute the following command to retrieve a fresh mainline codebase while
384 preparing things to add branches for stable/ 384 preparing things to add branches for stable/longterm series later::
385 385
386 git clone -o mainline --no-checkout \ 386 git clone -o mainline --no-checkout \
387 https://git.kernel.org/pub/scm/linux/ker 387 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
388 cd ~/linux/ 388 cd ~/linux/
389 git remote add -t master stable \ 389 git remote add -t master stable \
390 https://git.kernel.org/pub/scm/linux/ker 390 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
391 391
392 [:ref:`details <sources_bisref>`] 392 [:ref:`details <sources_bisref>`]
393 393
394 .. _stablesources_bissbs: 394 .. _stablesources_bissbs:
395 395
396 * Is one of the versions you earlier establish 396 * Is one of the versions you earlier established as 'good' or 'bad' a stable or
397 longterm release (say 6.1.5)? Then download 397 longterm release (say 6.1.5)? Then download the code for the series it belongs
398 to ('linux-6.1.y' in this example):: 398 to ('linux-6.1.y' in this example)::
399 399
400 git remote set-branches --add stable linux 400 git remote set-branches --add stable linux-6.1.y
401 git fetch stable 401 git fetch stable
402 402
403 .. _oldconfig_bissbs: 403 .. _oldconfig_bissbs:
404 404
405 * Start preparing a kernel build configuration 405 * Start preparing a kernel build configuration (the '.config' file).
406 406
407 Before doing so, ensure you are still runnin 407 Before doing so, ensure you are still running the 'working' kernel an earlier
408 step told you to boot; if you are unsure, ch 408 step told you to boot; if you are unsure, check the current kernelrelease
409 identifier using ``uname -r``. 409 identifier using ``uname -r``.
410 410
411 Afterwards check out the source code for the 411 Afterwards check out the source code for the version earlier established as
412 'good'. In the following example command thi 412 'good'. In the following example command this is assumed to be 6.0; note that
413 the version number in this and all later Git 413 the version number in this and all later Git commands needs to be prefixed
414 with a 'v':: 414 with a 'v'::
415 415
416 git switch --discard-changes --detach v6.0 416 git switch --discard-changes --detach v6.0
417 417
418 Now create a build configuration file:: 418 Now create a build configuration file::
419 419
420 make olddefconfig 420 make olddefconfig
421 421
422 The kernel build scripts then will try to lo 422 The kernel build scripts then will try to locate the build configuration file
423 for the running kernel and then adjust it fo 423 for the running kernel and then adjust it for the needs of the kernel sources
424 you checked out. While doing so, it will pri 424 you checked out. While doing so, it will print a few lines you need to check.
425 425
426 Look out for a line starting with '# using d 426 Look out for a line starting with '# using defaults found in'. It should be
427 followed by a path to a file in '/boot/' tha 427 followed by a path to a file in '/boot/' that contains the release identifier
428 of your currently working kernel. If the lin 428 of your currently working kernel. If the line instead continues with something
429 like 'arch/x86/configs/x86_64_defconfig', th 429 like 'arch/x86/configs/x86_64_defconfig', then the build infra failed to find
430 the .config file for your running kernel -- 430 the .config file for your running kernel -- in which case you have to put one
431 there manually, as explained in the referenc 431 there manually, as explained in the reference section.
432 432
433 In case you can not find such a line, look f 433 In case you can not find such a line, look for one containing '# configuration
434 written to .config'. If that's the case you 434 written to .config'. If that's the case you have a stale build configuration
435 lying around. Unless you intend to use it, d 435 lying around. Unless you intend to use it, delete it; afterwards run
436 'make olddefconfig' again and check if it no 436 'make olddefconfig' again and check if it now picked up the right config file
437 as base. 437 as base.
438 438
439 [:ref:`details <oldconfig_bisref>`] 439 [:ref:`details <oldconfig_bisref>`]
440 440
441 .. _localmodconfig_bissbs: 441 .. _localmodconfig_bissbs:
442 442
443 * Disable any kernel modules apparently superf 443 * Disable any kernel modules apparently superfluous for your setup. This is
444 optional, but especially wise for bisections 444 optional, but especially wise for bisections, as it speeds up the build
445 process enormously -- at least unless the .c 445 process enormously -- at least unless the .config file picked up in the
446 previous step was already tailored to your a 446 previous step was already tailored to your and your hardware needs, in which
447 case you should skip this step. 447 case you should skip this step.
448 448
449 To prepare the trimming, connect external ha 449 To prepare the trimming, connect external hardware you occasionally use (USB
450 keys, tokens, ...), quickly start a VM, and 450 keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted
451 since you started that guide, ensure that yo 451 since you started that guide, ensure that you tried using the feature causing
452 trouble since you started the system. Only t 452 trouble since you started the system. Only then trim your .config::
453 453
454 yes '' | make localmodconfig 454 yes '' | make localmodconfig
455 455
456 There is a catch to this, as the 'apparently 456 There is a catch to this, as the 'apparently' in initial sentence of this step
457 and the preparation instructions already hin 457 and the preparation instructions already hinted at:
458 458
459 The 'localmodconfig' target easily disables 459 The 'localmodconfig' target easily disables kernel modules for features only
460 used occasionally -- like modules for extern 460 used occasionally -- like modules for external peripherals not yet connected
461 since booting, virtualization software not y 461 since booting, virtualization software not yet utilized, VPN tunnels, and a
462 few other things. That's because some tasks 462 few other things. That's because some tasks rely on kernel modules Linux only
463 loads when you execute tasks like the aforem 463 loads when you execute tasks like the aforementioned ones for the first time.
464 464
465 This drawback of localmodconfig is nothing y 465 This drawback of localmodconfig is nothing you should lose sleep over, but
466 something to keep in mind: if something is m 466 something to keep in mind: if something is misbehaving with the kernels built
467 during this guide, this is most likely the r 467 during this guide, this is most likely the reason. You can reduce or nearly
468 eliminate the risk with tricks outlined in t 468 eliminate the risk with tricks outlined in the reference section; but when
469 building a kernel just for quick testing pur 469 building a kernel just for quick testing purposes this is usually not worth
470 spending much effort on, as long as it boots 470 spending much effort on, as long as it boots and allows to properly test the
471 feature that causes trouble. 471 feature that causes trouble.
472 472
473 [:ref:`details <localmodconfig_bisref>`] 473 [:ref:`details <localmodconfig_bisref>`]
474 474
475 .. _tagging_bissbs: 475 .. _tagging_bissbs:
476 476
477 * Ensure all the kernels you will build are cl 477 * Ensure all the kernels you will build are clearly identifiable using a special
478 tag and a unique version number:: 478 tag and a unique version number::
479 479
480 ./scripts/config --set-str CONFIG_LOCALVER 480 ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
481 ./scripts/config -e CONFIG_LOCALVERSION_AU 481 ./scripts/config -e CONFIG_LOCALVERSION_AUTO
482 482
483 [:ref:`details <tagging_bisref>`] 483 [:ref:`details <tagging_bisref>`]
484 484
485 .. _debugsymbols_bissbs: 485 .. _debugsymbols_bissbs:
486 486
487 * Decide how to handle debug symbols. 487 * Decide how to handle debug symbols.
488 488
489 In the context of this document it is often 489 In the context of this document it is often wise to enable them, as there is a
490 decent chance you will need to decode a stac 490 decent chance you will need to decode a stack trace from a 'panic', 'Oops',
491 'warning', or 'BUG':: 491 'warning', or 'BUG'::
492 492
493 ./scripts/config -d DEBUG_INFO_NONE -e KAL 493 ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
494 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCH 494 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
495 495
496 But if you are extremely short on storage sp 496 But if you are extremely short on storage space, you might want to disable
497 debug symbols instead:: 497 debug symbols instead::
498 498
499 ./scripts/config -d DEBUG_INFO -d DEBUG_IN 499 ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
500 -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF 500 -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE
501 501
502 [:ref:`details <debugsymbols_bisref>`] 502 [:ref:`details <debugsymbols_bisref>`]
503 503
504 .. _configmods_bissbs: 504 .. _configmods_bissbs:
505 505
506 * Check if you may want or need to adjust some 506 * Check if you may want or need to adjust some other kernel configuration
507 options: 507 options:
508 508
509 * Are you running Debian? Then you want to a 509 * Are you running Debian? Then you want to avoid known problems by performing
510 additional adjustments explained in the re 510 additional adjustments explained in the reference section.
511 511
512 [:ref:`details <configmods_distros_bisref> 512 [:ref:`details <configmods_distros_bisref>`].
513 513
514 * If you want to influence other aspects of 514 * If you want to influence other aspects of the configuration, do so now using
515 your preferred tool. Note, to use make tar 515 your preferred tool. Note, to use make targets like 'menuconfig' or
516 'nconfig', you will need to install the de 516 'nconfig', you will need to install the development files of ncurses; for
517 'xconfig' you likewise need the Qt5 or Qt6 517 'xconfig' you likewise need the Qt5 or Qt6 headers.
518 518
519 [:ref:`details <configmods_individual_bisr 519 [:ref:`details <configmods_individual_bisref>`].
520 520
521 .. _saveconfig_bissbs: 521 .. _saveconfig_bissbs:
522 522
523 * Reprocess the .config after the latest adjus 523 * Reprocess the .config after the latest adjustments and store it in a safe
524 place:: 524 place::
525 525
526 make olddefconfig 526 make olddefconfig
527 cp .config ~/kernel-config-working 527 cp .config ~/kernel-config-working
528 528
529 [:ref:`details <saveconfig_bisref>`] 529 [:ref:`details <saveconfig_bisref>`]
530 530
531 .. _introlatestcheck_bissbs: 531 .. _introlatestcheck_bissbs:
532 532
533 Segment 1: try to reproduce the problem with t 533 Segment 1: try to reproduce the problem with the latest codebase
534 ---------------------------------------------- 534 ----------------------------------------------------------------
535 535
536 The following steps verify if the problem occu 536 The following steps verify if the problem occurs with the code currently
537 supported by developers. In case you face a re 537 supported by developers. In case you face a regression, it also checks that the
538 problem is not caused by some .config change, 538 problem is not caused by some .config change, as reporting the issue then would
539 be a waste of time. [:ref:`details <introlates 539 be a waste of time. [:ref:`details <introlatestcheck_bisref>`]
540 540
541 .. _checkoutmaster_bissbs: 541 .. _checkoutmaster_bissbs:
542 542
543 * Check out the latest Linux codebase. 543 * Check out the latest Linux codebase.
544 544
545 * Are your 'good' and 'bad' versions from th 545 * Are your 'good' and 'bad' versions from the same stable or longterm series?
546 Then check the `front page of kernel.org < 546 Then check the `front page of kernel.org <https://kernel.org/>`_: if it
547 lists a release from that series without a 547 lists a release from that series without an '[EOL]' tag, checkout the series
548 latest version ('linux-6.1.y' in the follo 548 latest version ('linux-6.1.y' in the following example)::
549 549
550 cd ~/linux/ 550 cd ~/linux/
551 git switch --discard-changes --detach st 551 git switch --discard-changes --detach stable/linux-6.1.y
552 552
553 Your series is unsupported, if is not list 553 Your series is unsupported, if is not listed or carrying a 'end of life'
554 tag. In that case you might want to check 554 tag. In that case you might want to check if a successor series (say
555 linux-6.2.y) or mainline (see next point) 555 linux-6.2.y) or mainline (see next point) fix the bug.
556 556
557 * In all other cases, run:: 557 * In all other cases, run::
558 558
559 cd ~/linux/ 559 cd ~/linux/
560 git switch --discard-changes --detach ma 560 git switch --discard-changes --detach mainline/master
561 561
562 [:ref:`details <checkoutmaster_bisref>`] 562 [:ref:`details <checkoutmaster_bisref>`]
563 563
564 .. _build_bissbs: 564 .. _build_bissbs:
565 565
566 * Build the image and the modules of your firs 566 * Build the image and the modules of your first kernel using the config file you
567 prepared:: 567 prepared::
568 568
569 cp ~/kernel-config-working .config 569 cp ~/kernel-config-working .config
570 make olddefconfig 570 make olddefconfig
571 make -j $(nproc --all) 571 make -j $(nproc --all)
572 572
573 If you want your kernel packaged up as deb, 573 If you want your kernel packaged up as deb, rpm, or tar file, see the
574 reference section for alternatives, which ob 574 reference section for alternatives, which obviously will require other
575 steps to install as well. 575 steps to install as well.
576 576
577 [:ref:`details <build_bisref>`] 577 [:ref:`details <build_bisref>`]
578 578
579 .. _install_bissbs: 579 .. _install_bissbs:
580 580
581 * Install your newly built kernel. 581 * Install your newly built kernel.
582 582
583 Before doing so, consider checking if there 583 Before doing so, consider checking if there is still enough space for it::
584 584
585 df -h /boot/ /lib/modules/ 585 df -h /boot/ /lib/modules/
586 586
587 For now assume 150 MByte in /boot/ and 200 i 587 For now assume 150 MByte in /boot/ and 200 in /lib/modules/ will suffice; how
588 much your kernels actually require will be d 588 much your kernels actually require will be determined later during this guide.
589 589
590 Now install the kernel's modules and its ima 590 Now install the kernel's modules and its image, which will be stored in
591 parallel to the your Linux distribution's ke 591 parallel to the your Linux distribution's kernels::
592 592
593 sudo make modules_install 593 sudo make modules_install
594 command -v installkernel && sudo make inst 594 command -v installkernel && sudo make install
595 595
596 The second command ideally will take care of 596 The second command ideally will take care of three steps required at this
597 point: copying the kernel's image to /boot/, 597 point: copying the kernel's image to /boot/, generating an initramfs, and
598 adding an entry for both to the boot loader' 598 adding an entry for both to the boot loader's configuration.
599 599
600 Sadly some distributions (among them Arch Li 600 Sadly some distributions (among them Arch Linux, its derivatives, and many
601 immutable Linux distributions) will perform 601 immutable Linux distributions) will perform none or only some of those tasks.
602 You therefore want to check if all of them w 602 You therefore want to check if all of them were taken care of and manually
603 perform those that were not. The reference s 603 perform those that were not. The reference section provides further details on
604 that; your distribution's documentation migh 604 that; your distribution's documentation might help, too.
605 605
606 Once you figured out the steps needed at thi 606 Once you figured out the steps needed at this point, consider writing them
607 down: if you will build more kernels as desc 607 down: if you will build more kernels as described in segment 2 and 3, you will
608 have to perform those again after executing 608 have to perform those again after executing ``command -v installkernel [...]``.
609 609
610 [:ref:`details <install_bisref>`] 610 [:ref:`details <install_bisref>`]
611 611
612 .. _storagespace_bissbs: 612 .. _storagespace_bissbs:
613 613
614 * In case you plan to follow this guide furthe 614 * In case you plan to follow this guide further, check how much storage space
615 the kernel, its modules, and other related f 615 the kernel, its modules, and other related files like the initramfs consume::
616 616
617 du -ch /boot/*$(make -s kernelrelease)* | 617 du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
618 du -sh /lib/modules/$(make -s kernelreleas 618 du -sh /lib/modules/$(make -s kernelrelease)/
619 619
620 Write down or remember those two values for 620 Write down or remember those two values for later: they enable you to prevent
621 running out of disk space accidentally durin 621 running out of disk space accidentally during a bisection.
622 622
623 [:ref:`details <storagespace_bisref>`] 623 [:ref:`details <storagespace_bisref>`]
624 624
625 .. _kernelrelease_bissbs: 625 .. _kernelrelease_bissbs:
626 626
627 * Show and store the kernelrelease identifier 627 * Show and store the kernelrelease identifier of the kernel you just built::
628 628
629 make -s kernelrelease | tee -a ~/kernels-b 629 make -s kernelrelease | tee -a ~/kernels-built
630 630
631 Remember the identifier momentarily, as it w 631 Remember the identifier momentarily, as it will help you pick the right kernel
632 from the boot menu upon restarting. 632 from the boot menu upon restarting.
633 633
634 * Reboot into your newly built kernel. To ensu 634 * Reboot into your newly built kernel. To ensure your actually started the one
635 you just built, you might want to verify if 635 you just built, you might want to verify if the output of these commands
636 matches:: 636 matches::
637 637
638 tail -n 1 ~/kernels-built 638 tail -n 1 ~/kernels-built
639 uname -r 639 uname -r
640 640
641 .. _tainted_bissbs: 641 .. _tainted_bissbs:
642 642
643 * Check if the kernel marked itself as 'tainte 643 * Check if the kernel marked itself as 'tainted'::
644 644
645 cat /proc/sys/kernel/tainted 645 cat /proc/sys/kernel/tainted
646 646
647 If that command does not return '0', check t 647 If that command does not return '0', check the reference section, as the cause
648 for this might interfere with your testing. 648 for this might interfere with your testing.
649 649
650 [:ref:`details <tainted_bisref>`] 650 [:ref:`details <tainted_bisref>`]
651 651
652 .. _recheckbroken_bissbs: 652 .. _recheckbroken_bissbs:
653 653
654 * Verify if your bug occurs with the newly bui 654 * Verify if your bug occurs with the newly built kernel. If it does not, check
655 out the instructions in the reference sectio 655 out the instructions in the reference section to ensure nothing went sideways
656 during your tests. 656 during your tests.
657 657
658 [:ref:`details <recheckbroken_bisref>`] 658 [:ref:`details <recheckbroken_bisref>`]
659 659
660 .. _recheckstablebroken_bissbs: 660 .. _recheckstablebroken_bissbs:
661 661
662 * Did you just built a stable or longterm kern 662 * Did you just built a stable or longterm kernel? And were you able to reproduce
663 the regression with it? Then you should test 663 the regression with it? Then you should test the latest mainline codebase as
664 well, because the result determines which de 664 well, because the result determines which developers the bug must be submitted
665 to. 665 to.
666 666
667 To prepare that test, check out current main 667 To prepare that test, check out current mainline::
668 668
669 cd ~/linux/ 669 cd ~/linux/
670 git switch --discard-changes --detach main 670 git switch --discard-changes --detach mainline/master
671 671
672 Now use the checked out code to build and in 672 Now use the checked out code to build and install another kernel using the
673 commands the earlier steps already described 673 commands the earlier steps already described in more detail::
674 674
675 cp ~/kernel-config-working .config 675 cp ~/kernel-config-working .config
676 make olddefconfig 676 make olddefconfig
677 make -j $(nproc --all) 677 make -j $(nproc --all)
678 # * Check if the free space suffices holdi 678 # * Check if the free space suffices holding another kernel:
679 df -h /boot/ /lib/modules/ 679 df -h /boot/ /lib/modules/
680 sudo make modules_install 680 sudo make modules_install
681 command -v installkernel && sudo make inst 681 command -v installkernel && sudo make install
682 make -s kernelrelease | tee -a ~/kernels-b 682 make -s kernelrelease | tee -a ~/kernels-built
683 reboot 683 reboot
684 684
685 Confirm you booted the kernel you intended t 685 Confirm you booted the kernel you intended to start and check its tainted
686 status:: 686 status::
687 687
688 tail -n 1 ~/kernels-built 688 tail -n 1 ~/kernels-built
689 uname -r 689 uname -r
690 cat /proc/sys/kernel/tainted 690 cat /proc/sys/kernel/tainted
691 691
692 Now verify if this kernel is showing the pro 692 Now verify if this kernel is showing the problem. If it does, then you need
693 to report the bug to the primary developers; 693 to report the bug to the primary developers; if it does not, report it to the
694 stable team. See Documentation/admin-guide/r 694 stable team. See Documentation/admin-guide/reporting-issues.rst for details.
695 695
696 [:ref:`details <recheckstablebroken_bisref>` 696 [:ref:`details <recheckstablebroken_bisref>`]
697 697
698 Do you follow this guide to verify if a proble 698 Do you follow this guide to verify if a problem is present in the code
699 currently supported by Linux kernel developers 699 currently supported by Linux kernel developers? Then you are done at this
700 point. If you later want to remove the kernel 700 point. If you later want to remove the kernel you just built, check out
701 :ref:`Complementary tasks: cleanup during and 701 :ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`.
702 702
703 In case you face a regression, move on and exe 703 In case you face a regression, move on and execute at least the next segment
704 as well. 704 as well.
705 705
706 .. _introworkingcheck_bissbs: 706 .. _introworkingcheck_bissbs:
707 707
708 Segment 2: check if the kernels you build work 708 Segment 2: check if the kernels you build work fine
709 ---------------------------------------------- 709 ---------------------------------------------------
710 710
711 In case of a regression, you now want to ensur 711 In case of a regression, you now want to ensure the trimmed configuration file
712 you created earlier works as expected; a bisec 712 you created earlier works as expected; a bisection with the .config file
713 otherwise would be a waste of time. [:ref:`det 713 otherwise would be a waste of time. [:ref:`details <introworkingcheck_bisref>`]
714 714
715 .. _recheckworking_bissbs: 715 .. _recheckworking_bissbs:
716 716
717 * Build your own variant of the 'working' kern 717 * Build your own variant of the 'working' kernel and check if the feature that
718 regressed works as expected with it. 718 regressed works as expected with it.
719 719
720 Start by checking out the sources for the ve 720 Start by checking out the sources for the version earlier established as
721 'good' (once again assumed to be 6.0 here):: 721 'good' (once again assumed to be 6.0 here)::
722 722
723 cd ~/linux/ 723 cd ~/linux/
724 git switch --discard-changes --detach v6.0 724 git switch --discard-changes --detach v6.0
725 725
726 Now use the checked out code to configure, b 726 Now use the checked out code to configure, build, and install another kernel
727 using the commands the previous subsection e 727 using the commands the previous subsection explained in more detail::
728 728
729 cp ~/kernel-config-working .config 729 cp ~/kernel-config-working .config
730 make olddefconfig 730 make olddefconfig
731 make -j $(nproc --all) 731 make -j $(nproc --all)
732 # * Check if the free space suffices holdi 732 # * Check if the free space suffices holding another kernel:
733 df -h /boot/ /lib/modules/ 733 df -h /boot/ /lib/modules/
734 sudo make modules_install 734 sudo make modules_install
735 command -v installkernel && sudo make inst 735 command -v installkernel && sudo make install
736 make -s kernelrelease | tee -a ~/kernels-b 736 make -s kernelrelease | tee -a ~/kernels-built
737 reboot 737 reboot
738 738
739 When the system booted, you may want to veri 739 When the system booted, you may want to verify once again that the
740 kernel you started is the one you just built 740 kernel you started is the one you just built::
741 741
742 tail -n 1 ~/kernels-built 742 tail -n 1 ~/kernels-built
743 uname -r 743 uname -r
744 744
745 Now check if this kernel works as expected; 745 Now check if this kernel works as expected; if not, consult the reference
746 section for further instructions. 746 section for further instructions.
747 747
748 [:ref:`details <recheckworking_bisref>`] 748 [:ref:`details <recheckworking_bisref>`]
749 749
750 .. _introbisect_bissbs: 750 .. _introbisect_bissbs:
751 751
752 Segment 3: perform the bisection and validate 752 Segment 3: perform the bisection and validate the result
753 ---------------------------------------------- 753 --------------------------------------------------------
754 754
755 With all the preparations and precaution build 755 With all the preparations and precaution builds taken care of, you are now ready
756 to begin the bisection. This will make you bui 756 to begin the bisection. This will make you build quite a few kernels -- usually
757 about 15 in case you encountered a regression 757 about 15 in case you encountered a regression when updating to a newer series
758 (say from 6.0.13 to 6.1.5). But do not worry, 758 (say from 6.0.13 to 6.1.5). But do not worry, due to the trimmed build
759 configuration created earlier this works a lot 759 configuration created earlier this works a lot faster than many people assume:
760 overall on average it will often just take abo 760 overall on average it will often just take about 10 to 15 minutes to compile
761 each kernel on commodity x86 machines. 761 each kernel on commodity x86 machines.
762 762
763 .. _bisectstart_bissbs: 763 .. _bisectstart_bissbs:
764 764
765 * Start the bisection and tell Git about the v 765 * Start the bisection and tell Git about the versions earlier established as
766 'good' (6.0 in the following example command 766 'good' (6.0 in the following example command) and 'bad' (6.1.5)::
767 767
768 cd ~/linux/ 768 cd ~/linux/
769 git bisect start 769 git bisect start
770 git bisect good v6.0 770 git bisect good v6.0
771 git bisect bad v6.1.5 771 git bisect bad v6.1.5
772 772
773 [:ref:`details <bisectstart_bisref>`] 773 [:ref:`details <bisectstart_bisref>`]
774 774
775 .. _bisectbuild_bissbs: 775 .. _bisectbuild_bissbs:
776 776
777 * Now use the code Git checked out to build, i 777 * Now use the code Git checked out to build, install, and boot a kernel using
778 the commands introduced earlier:: 778 the commands introduced earlier::
779 779
780 cp ~/kernel-config-working .config 780 cp ~/kernel-config-working .config
781 make olddefconfig 781 make olddefconfig
782 make -j $(nproc --all) 782 make -j $(nproc --all)
783 # * Check if the free space suffices holdi 783 # * Check if the free space suffices holding another kernel:
784 df -h /boot/ /lib/modules/ 784 df -h /boot/ /lib/modules/
785 sudo make modules_install 785 sudo make modules_install
786 command -v installkernel && sudo make inst 786 command -v installkernel && sudo make install
787 make -s kernelrelease | tee -a ~/kernels-b 787 make -s kernelrelease | tee -a ~/kernels-built
788 reboot 788 reboot
789 789
790 If compilation fails for some reason, run `` 790 If compilation fails for some reason, run ``git bisect skip`` and restart
791 executing the stack of commands from the beg 791 executing the stack of commands from the beginning.
792 792
793 In case you skipped the 'test latest codebas 793 In case you skipped the 'test latest codebase' step in the guide, check its
794 description as for why the 'df [...]' and 'm 794 description as for why the 'df [...]' and 'make -s kernelrelease [...]'
795 commands are here. 795 commands are here.
796 796
797 Important note: the latter command from this 797 Important note: the latter command from this point on will print release
798 identifiers that might look odd or wrong to 798 identifiers that might look odd or wrong to you -- which they are not, as it's
799 totally normal to see release identifiers li 799 totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0'
800 if you bisect between versions 6.1 and 6.2 f 800 if you bisect between versions 6.1 and 6.2 for example.
801 801
802 [:ref:`details <bisectbuild_bisref>`] 802 [:ref:`details <bisectbuild_bisref>`]
803 803
804 .. _bisecttest_bissbs: 804 .. _bisecttest_bissbs:
805 805
806 * Now check if the feature that regressed work 806 * Now check if the feature that regressed works in the kernel you just built.
807 807
808 You again might want to start by making sure 808 You again might want to start by making sure the kernel you booted is the one
809 you just built:: 809 you just built::
810 810
811 cd ~/linux/ 811 cd ~/linux/
812 tail -n 1 ~/kernels-built 812 tail -n 1 ~/kernels-built
813 uname -r 813 uname -r
814 814
815 Now verify if the feature that regressed wor 815 Now verify if the feature that regressed works at this kernel bisection point.
816 If it does, run this:: 816 If it does, run this::
817 817
818 git bisect good 818 git bisect good
819 819
820 If it does not, run this:: 820 If it does not, run this::
821 821
822 git bisect bad 822 git bisect bad
823 823
824 Be sure about what you tell Git, as getting 824 Be sure about what you tell Git, as getting this wrong just once will send the
825 rest of the bisection totally off course. 825 rest of the bisection totally off course.
826 826
827 While the bisection is ongoing, Git will use 827 While the bisection is ongoing, Git will use the information you provided to
828 find and check out another bisection point f 828 find and check out another bisection point for you to test. While doing so, it
829 will print something like 'Bisecting: 675 re 829 will print something like 'Bisecting: 675 revisions left to test after this
830 (roughly 10 steps)' to indicate how many fur 830 (roughly 10 steps)' to indicate how many further changes it expects to be
831 tested. Now build and install another kernel 831 tested. Now build and install another kernel using the instructions from the
832 previous step; afterwards follow the instruc 832 previous step; afterwards follow the instructions in this step again.
833 833
834 Repeat this again and again until you finish 834 Repeat this again and again until you finish the bisection -- that's the case
835 when Git after tagging a change as 'good' or 835 when Git after tagging a change as 'good' or 'bad' prints something like
836 'cafecaca0c0dacafecaca0c0dacafecaca0c0da is 836 'cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit'; right
837 afterwards it will show some details about t 837 afterwards it will show some details about the culprit including the patch
838 description of the change. The latter might 838 description of the change. The latter might fill your terminal screen, so you
839 might need to scroll up to see the message m 839 might need to scroll up to see the message mentioning the culprit;
840 alternatively, run ``git bisect log > ~/bise 840 alternatively, run ``git bisect log > ~/bisection-log``.
841 841
842 [:ref:`details <bisecttest_bisref>`] 842 [:ref:`details <bisecttest_bisref>`]
843 843
844 .. _bisectlog_bissbs: 844 .. _bisectlog_bissbs:
845 845
846 * Store Git's bisection log and the current .c 846 * Store Git's bisection log and the current .config file in a safe place before
847 telling Git to reset the sources to the stat 847 telling Git to reset the sources to the state before the bisection::
848 848
849 cd ~/linux/ 849 cd ~/linux/
850 git bisect log > ~/bisection-log 850 git bisect log > ~/bisection-log
851 cp .config ~/bisection-config-culprit 851 cp .config ~/bisection-config-culprit
852 git bisect reset 852 git bisect reset
853 853
854 [:ref:`details <bisectlog_bisref>`] 854 [:ref:`details <bisectlog_bisref>`]
855 855
856 .. _revert_bissbs: 856 .. _revert_bissbs:
857 857
858 * Try reverting the culprit on top of latest m 858 * Try reverting the culprit on top of latest mainline to see if this fixes your
859 regression. 859 regression.
860 860
861 This is optional, as it might be impossible 861 This is optional, as it might be impossible or hard to realize. The former is
862 the case, if the bisection determined a merg 862 the case, if the bisection determined a merge commit as the culprit; the
863 latter happens if other changes depend on th 863 latter happens if other changes depend on the culprit. But if the revert
864 succeeds, it is worth building another kerne 864 succeeds, it is worth building another kernel, as it validates the result of
865 a bisection, which can easily deroute; it fu 865 a bisection, which can easily deroute; it furthermore will let kernel
866 developers know, if they can resolve the reg 866 developers know, if they can resolve the regression with a quick revert.
867 867
868 Begin by checking out the latest codebase de 868 Begin by checking out the latest codebase depending on the range you bisected:
869 869
870 * Did you face a regression within a stable/ 870 * Did you face a regression within a stable/longterm series (say between
871 6.0.13 and 6.0.15) that does not happen in 871 6.0.13 and 6.0.15) that does not happen in mainline? Then check out the
872 latest codebase for the affected series li 872 latest codebase for the affected series like this::
873 873
874 git fetch stable 874 git fetch stable
875 git switch --discard-changes --detach li 875 git switch --discard-changes --detach linux-6.0.y
876 876
877 * In all other cases check out latest mainli 877 * In all other cases check out latest mainline::
878 878
879 git fetch mainline 879 git fetch mainline
880 git switch --discard-changes --detach ma 880 git switch --discard-changes --detach mainline/master
881 881
882 If you bisected a regression within a stab 882 If you bisected a regression within a stable/longterm series that also
883 happens in mainline, there is one more thi 883 happens in mainline, there is one more thing to do: look up the mainline
884 commit-id. To do so, use a command like `` 884 commit-id. To do so, use a command like ``git show abcdcafecabcd`` to
885 view the patch description of the culprit. 885 view the patch description of the culprit. There will be a line near
886 the top which looks like 'commit cafec0cac 886 the top which looks like 'commit cafec0cacaca0 upstream.' or
887 'Upstream commit cafec0cacaca0'; use that 887 'Upstream commit cafec0cacaca0'; use that commit-id in the next command
888 and not the one the bisection blamed. 888 and not the one the bisection blamed.
889 889
890 Now try reverting the culprit by specifying 890 Now try reverting the culprit by specifying its commit id::
891 891
892 git revert --no-edit cafec0cacaca0 892 git revert --no-edit cafec0cacaca0
893 893
894 If that fails, give up trying and move on to 894 If that fails, give up trying and move on to the next step; if it works,
895 adjust the tag to facilitate the identificat 895 adjust the tag to facilitate the identification and prevent accidentally
896 overwriting another kernel:: 896 overwriting another kernel::
897 897
898 cp ~/kernel-config-working .config 898 cp ~/kernel-config-working .config
899 ./scripts/config --set-str CONFIG_LOCALVER 899 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
900 900
901 Build a kernel using the familiar command se 901 Build a kernel using the familiar command sequence, just without copying the
902 the base .config over:: 902 the base .config over::
903 903
904 make olddefconfig && 904 make olddefconfig &&
905 make -j $(nproc --all) 905 make -j $(nproc --all)
906 # * Check if the free space suffices holdi 906 # * Check if the free space suffices holding another kernel:
907 df -h /boot/ /lib/modules/ 907 df -h /boot/ /lib/modules/
908 sudo make modules_install 908 sudo make modules_install
909 command -v installkernel && sudo make inst 909 command -v installkernel && sudo make install
910 make -s kernelrelease | tee -a ~/kernels-b 910 make -s kernelrelease | tee -a ~/kernels-built
911 reboot 911 reboot
912 912
913 Now check one last time if the feature that 913 Now check one last time if the feature that made you perform a bisection works
914 with that kernel: if everything went well, i 914 with that kernel: if everything went well, it should not show the regression.
915 915
916 [:ref:`details <revert_bisref>`] 916 [:ref:`details <revert_bisref>`]
917 917
918 .. _introclosure_bissbs: 918 .. _introclosure_bissbs:
919 919
920 Complementary tasks: cleanup during and after 920 Complementary tasks: cleanup during and after the bisection
921 ---------------------------------------------- 921 -----------------------------------------------------------
922 922
923 During and after following this guide you migh 923 During and after following this guide you might want or need to remove some of
924 the kernels you installed: the boot menu other 924 the kernels you installed: the boot menu otherwise will become confusing or
925 space might run out. 925 space might run out.
926 926
927 .. _makeroom_bissbs: 927 .. _makeroom_bissbs:
928 928
929 * To remove one of the kernels you installed, 929 * To remove one of the kernels you installed, look up its 'kernelrelease'
930 identifier. This guide stores them in '~/ker 930 identifier. This guide stores them in '~/kernels-built', but the following
931 command will print them as well:: 931 command will print them as well::
932 932
933 ls -ltr /lib/modules/*-local* 933 ls -ltr /lib/modules/*-local*
934 934
935 You in most situations want to remove the ol 935 You in most situations want to remove the oldest kernels built during the
936 actual bisection (e.g. segment 3 of this gui 936 actual bisection (e.g. segment 3 of this guide). The two ones you created
937 beforehand (e.g. to test the latest codebase 937 beforehand (e.g. to test the latest codebase and the version considered
938 'good') might become handy to verify somethi 938 'good') might become handy to verify something later -- thus better keep them
939 around, unless you are really short on stora 939 around, unless you are really short on storage space.
940 940
941 To remove the modules of a kernel with the k 941 To remove the modules of a kernel with the kernelrelease identifier
942 '*6.0-rc1-local-gcafec0cacaca0*', start by r 942 '*6.0-rc1-local-gcafec0cacaca0*', start by removing the directory holding its
943 modules:: 943 modules::
944 944
945 sudo rm -rf /lib/modules/6.0-rc1-local-gca 945 sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
946 946
947 Afterwards try the following command:: 947 Afterwards try the following command::
948 948
949 sudo kernel-install -v remove 6.0-rc1-loca 949 sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
950 950
951 On quite a few distributions this will delet 951 On quite a few distributions this will delete all other kernel files installed
952 while also removing the kernel's entry from 952 while also removing the kernel's entry from the boot menu. But on some
953 distributions kernel-install does not exist 953 distributions kernel-install does not exist or leaves boot-loader entries or
954 kernel image and related files behind; in th 954 kernel image and related files behind; in that case remove them as described
955 in the reference section. 955 in the reference section.
956 956
957 [:ref:`details <makeroom_bisref>`] 957 [:ref:`details <makeroom_bisref>`]
958 958
959 .. _finishingtouch_bissbs: 959 .. _finishingtouch_bissbs:
960 960
961 * Once you have finished the bisection, do not 961 * Once you have finished the bisection, do not immediately remove anything you
962 set up, as you might need a few things again 962 set up, as you might need a few things again. What is safe to remove depends
963 on the outcome of the bisection: 963 on the outcome of the bisection:
964 964
965 * Could you initially reproduce the regressi 965 * Could you initially reproduce the regression with the latest codebase and
966 after the bisection were able to fix the p 966 after the bisection were able to fix the problem by reverting the culprit on
967 top of the latest codebase? Then you want 967 top of the latest codebase? Then you want to keep those two kernels around
968 for a while, but safely remove all others 968 for a while, but safely remove all others with a '-local' in the release
969 identifier. 969 identifier.
970 970
971 * Did the bisection end on a merge-commit or 971 * Did the bisection end on a merge-commit or seems questionable for other
972 reasons? Then you want to keep as many ker 972 reasons? Then you want to keep as many kernels as possible around for a few
973 days: it's pretty likely that you will be 973 days: it's pretty likely that you will be asked to recheck something.
974 974
975 * In other cases it likely is a good idea to 975 * In other cases it likely is a good idea to keep the following kernels around
976 for some time: the one built from the late 976 for some time: the one built from the latest codebase, the one created from
977 the version considered 'good', and the las 977 the version considered 'good', and the last three or four you compiled
978 during the actual bisection process. 978 during the actual bisection process.
979 979
980 [:ref:`details <finishingtouch_bisref>`] 980 [:ref:`details <finishingtouch_bisref>`]
981 981
982 .. _introoptional_bissbs: 982 .. _introoptional_bissbs:
983 983
984 Optional: test reverts, patches, or later vers 984 Optional: test reverts, patches, or later versions
985 ---------------------------------------------- 985 --------------------------------------------------
986 986
987 While or after reporting a bug, you might want 987 While or after reporting a bug, you might want or potentially will be asked to
988 test reverts, debug patches, proposed fixes, o 988 test reverts, debug patches, proposed fixes, or other versions. In that case
989 follow these instructions. 989 follow these instructions.
990 990
991 * Update your Git clone and check out the late 991 * Update your Git clone and check out the latest code.
992 992
993 * In case you want to test mainline, fetch i 993 * In case you want to test mainline, fetch its latest changes before checking
994 its code out:: 994 its code out::
995 995
996 git fetch mainline 996 git fetch mainline
997 git switch --discard-changes --detach ma 997 git switch --discard-changes --detach mainline/master
998 998
999 * In case you want to test a stable or longt 999 * In case you want to test a stable or longterm kernel, first add the branch
1000 holding the series you are interested in 1000 holding the series you are interested in (6.2 in the example), unless you
1001 already did so earlier:: 1001 already did so earlier::
1002 1002
1003 git remote set-branches --add stable li 1003 git remote set-branches --add stable linux-6.2.y
1004 1004
1005 Then fetch the latest changes and check o 1005 Then fetch the latest changes and check out the latest version from the
1006 series:: 1006 series::
1007 1007
1008 git fetch stable 1008 git fetch stable
1009 git switch --discard-changes --detach s 1009 git switch --discard-changes --detach stable/linux-6.2.y
1010 1010
1011 * Copy your kernel build configuration over:: 1011 * Copy your kernel build configuration over::
1012 1012
1013 cp ~/kernel-config-working .config 1013 cp ~/kernel-config-working .config
1014 1014
1015 * Your next step depends on what you want to 1015 * Your next step depends on what you want to do:
1016 1016
1017 * In case you just want to test the latest 1017 * In case you just want to test the latest codebase, head to the next step,
1018 you are already all set. 1018 you are already all set.
1019 1019
1020 * In case you want to test if a revert fixe 1020 * In case you want to test if a revert fixes an issue, revert one or multiple
1021 changes by specifying their commit ids:: 1021 changes by specifying their commit ids::
1022 1022
1023 git revert --no-edit cafec0cacaca0 1023 git revert --no-edit cafec0cacaca0
1024 1024
1025 Now give that kernel a special tag to fac 1025 Now give that kernel a special tag to facilitates its identification and
1026 prevent accidentally overwriting another 1026 prevent accidentally overwriting another kernel::
1027 1027
1028 ./scripts/config --set-str CONFIG_LOCAL 1028 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
1029 1029
1030 * In case you want to test a patch, store t 1030 * In case you want to test a patch, store the patch in a file like
1031 '/tmp/foobars-proposed-fix-v1.patch' and 1031 '/tmp/foobars-proposed-fix-v1.patch' and apply it like this::
1032 1032
1033 git apply /tmp/foobars-proposed-fix-v1. 1033 git apply /tmp/foobars-proposed-fix-v1.patch
1034 1034
1035 In case of multiple patches, repeat this 1035 In case of multiple patches, repeat this step with the others.
1036 1036
1037 Now give that kernel a special tag to fac 1037 Now give that kernel a special tag to facilitates its identification and
1038 prevent accidentally overwriting another 1038 prevent accidentally overwriting another kernel::
1039 1039
1040 ./scripts/config --set-str CONFIG_LOCALVE 1040 ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
1041 1041
1042 * Build a kernel using the familiar commands, 1042 * Build a kernel using the familiar commands, just without copying the kernel
1043 build configuration over, as that has been 1043 build configuration over, as that has been taken care of already::
1044 1044
1045 make olddefconfig && 1045 make olddefconfig &&
1046 make -j $(nproc --all) 1046 make -j $(nproc --all)
1047 # * Check if the free space suffices hold 1047 # * Check if the free space suffices holding another kernel:
1048 df -h /boot/ /lib/modules/ 1048 df -h /boot/ /lib/modules/
1049 sudo make modules_install 1049 sudo make modules_install
1050 command -v installkernel && sudo make ins 1050 command -v installkernel && sudo make install
1051 make -s kernelrelease | tee -a ~/kernels- 1051 make -s kernelrelease | tee -a ~/kernels-built
1052 reboot 1052 reboot
1053 1053
1054 * Now verify you booted the newly built kerne 1054 * Now verify you booted the newly built kernel and check it.
1055 1055
1056 [:ref:`details <introoptional_bisref>`] 1056 [:ref:`details <introoptional_bisref>`]
1057 1057
1058 .. _submit_improvements: 1058 .. _submit_improvements:
1059 1059
1060 Conclusion 1060 Conclusion
1061 ---------- 1061 ----------
1062 1062
1063 You have reached the end of the step-by-step 1063 You have reached the end of the step-by-step guide.
1064 1064
1065 Did you run into trouble following any of the 1065 Did you run into trouble following any of the above steps not cleared up by the
1066 reference section below? Did you spot errors? 1066 reference section below? Did you spot errors? Or do you have ideas how to
1067 improve the guide? 1067 improve the guide?
1068 1068
1069 If any of that applies, please take a moment 1069 If any of that applies, please take a moment and let the maintainer of this
1070 document know by email (Thorsten Leemhuis <lin 1070 document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while
1071 CCing the Linux docs mailing list (linux-doc@ 1071 CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is
1072 vital to improve this text further, which is 1072 vital to improve this text further, which is in everybody's interest, as it
1073 will enable more people to master the task de 1073 will enable more people to master the task described here -- and hopefully also
1074 improve similar guides inspired by this one. 1074 improve similar guides inspired by this one.
1075 1075
1076 1076
1077 Reference section for the step-by-step guide 1077 Reference section for the step-by-step guide
1078 ============================================ 1078 ============================================
1079 1079
1080 This section holds additional information for 1080 This section holds additional information for almost all the items in the above
1081 step-by-step guide. 1081 step-by-step guide.
1082 1082
1083 Preparations for building your own kernels 1083 Preparations for building your own kernels
1084 ------------------------------------------ 1084 ------------------------------------------
1085 1085
1086 *The steps in this section lay the groundwo 1086 *The steps in this section lay the groundwork for all further tests.*
1087 [:ref:`... <introprep_bissbs>`] 1087 [:ref:`... <introprep_bissbs>`]
1088 1088
1089 The steps in all later sections of this guide 1089 The steps in all later sections of this guide depend on those described here.
1090 1090
1091 [:ref:`back to step-by-step guide <introprep_ 1091 [:ref:`back to step-by-step guide <introprep_bissbs>`].
1092 1092
1093 .. _backup_bisref: 1093 .. _backup_bisref:
1094 1094
1095 Prepare for emergencies 1095 Prepare for emergencies
1096 ~~~~~~~~~~~~~~~~~~~~~~~ 1096 ~~~~~~~~~~~~~~~~~~~~~~~
1097 1097
1098 *Create a fresh backup and put system repai 1098 *Create a fresh backup and put system repair and restore tools at hand.*
1099 [:ref:`... <backup_bissbs>`] 1099 [:ref:`... <backup_bissbs>`]
1100 1100
1101 Remember, you are dealing with computers, whi 1101 Remember, you are dealing with computers, which sometimes do unexpected things
1102 -- especially if you fiddle with crucial part 1102 -- especially if you fiddle with crucial parts like the kernel of an operating
1103 system. That's what you are about to do in th 1103 system. That's what you are about to do in this process. Hence, better prepare
1104 for something going sideways, even if that sh 1104 for something going sideways, even if that should not happen.
1105 1105
1106 [:ref:`back to step-by-step guide <backup_bis 1106 [:ref:`back to step-by-step guide <backup_bissbs>`]
1107 1107
1108 .. _vanilla_bisref: 1108 .. _vanilla_bisref:
1109 1109
1110 Remove anything related to externally maintai 1110 Remove anything related to externally maintained kernel modules
1111 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1111 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1112 1112
1113 *Remove all software that depends on extern 1113 *Remove all software that depends on externally developed kernel drivers or
1114 builds them automatically.* [:ref:`...<vani 1114 builds them automatically.* [:ref:`...<vanilla_bissbs>`]
1115 1115
1116 Externally developed kernel modules can easil 1116 Externally developed kernel modules can easily cause trouble during a bisection.
1117 1117
1118 But there is a more important reason why this 1118 But there is a more important reason why this guide contains this step: most
1119 kernel developers will not care about reports 1119 kernel developers will not care about reports about regressions occurring with
1120 kernels that utilize such modules. That's bec 1120 kernels that utilize such modules. That's because such kernels are not
1121 considered 'vanilla' anymore, as Documentatio 1121 considered 'vanilla' anymore, as Documentation/admin-guide/reporting-issues.rst
1122 explains in more detail. 1122 explains in more detail.
1123 1123
1124 [:ref:`back to step-by-step guide <vanilla_bi 1124 [:ref:`back to step-by-step guide <vanilla_bissbs>`]
1125 1125
1126 .. _secureboot_bisref: 1126 .. _secureboot_bisref:
1127 1127
1128 Deal with techniques like Secure Boot 1128 Deal with techniques like Secure Boot
1129 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1129 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1130 1130
1131 *On platforms with 'Secure Boot' or similar 1131 *On platforms with 'Secure Boot' or similar techniques, prepare everything to
1132 ensure the system will permit your self-com 1132 ensure the system will permit your self-compiled kernel to boot later.*
1133 [:ref:`... <secureboot_bissbs>`] 1133 [:ref:`... <secureboot_bissbs>`]
1134 1134
1135 Many modern systems allow only certain operat 1135 Many modern systems allow only certain operating systems to start; that's why
1136 they reject booting self-compiled kernels by 1136 they reject booting self-compiled kernels by default.
1137 1137
1138 You ideally deal with this by making your pla 1138 You ideally deal with this by making your platform trust your self-built kernels
1139 with the help of a certificate. How to do tha 1139 with the help of a certificate. How to do that is not described
1140 here, as it requires various steps that would 1140 here, as it requires various steps that would take the text too far away from
1141 its purpose; 'Documentation/admin-guide/modul 1141 its purpose; 'Documentation/admin-guide/module-signing.rst' and various web
1142 sides already explain everything needed in mo 1142 sides already explain everything needed in more detail.
1143 1143
1144 Temporarily disabling solutions like Secure B 1144 Temporarily disabling solutions like Secure Boot is another way to make your own
1145 Linux boot. On commodity x86 systems it is po 1145 Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup
1146 utility; the required steps vary a lot betwee 1146 utility; the required steps vary a lot between machines and therefore cannot be
1147 described here. 1147 described here.
1148 1148
1149 On mainstream x86 Linux distributions there i 1149 On mainstream x86 Linux distributions there is a third and universal option:
1150 disable all Secure Boot restrictions for your 1150 disable all Secure Boot restrictions for your Linux environment. You can
1151 initiate this process by running ``mokutil -- 1151 initiate this process by running ``mokutil --disable-validation``; this will
1152 tell you to create a one-time password, which 1152 tell you to create a one-time password, which is safe to write down. Now
1153 restart; right after your BIOS performed all 1153 restart; right after your BIOS performed all self-tests the bootloader Shim will
1154 show a blue box with a message 'Press any key 1154 show a blue box with a message 'Press any key to perform MOK management'. Hit
1155 some key before the countdown exposes, which 1155 some key before the countdown exposes, which will open a menu. Choose 'Change
1156 Secure Boot state'. Shim's 'MokManager' will 1156 Secure Boot state'. Shim's 'MokManager' will now ask you to enter three
1157 randomly chosen characters from the one-time 1157 randomly chosen characters from the one-time password specified earlier. Once
1158 you provided them, confirm you really want to 1158 you provided them, confirm you really want to disable the validation.
1159 Afterwards, permit MokManager to reboot the m 1159 Afterwards, permit MokManager to reboot the machine.
1160 1160
1161 [:ref:`back to step-by-step guide <secureboot 1161 [:ref:`back to step-by-step guide <secureboot_bissbs>`]
1162 1162
1163 .. _bootworking_bisref: 1163 .. _bootworking_bisref:
1164 1164
1165 Boot the last kernel that was working 1165 Boot the last kernel that was working
1166 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1166 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1167 1167
1168 *Boot into the last working kernel and brie 1168 *Boot into the last working kernel and briefly recheck if the feature that
1169 regressed really works.* [:ref:`...<bootwor 1169 regressed really works.* [:ref:`...<bootworking_bissbs>`]
1170 1170
1171 This will make later steps that cover creatin 1171 This will make later steps that cover creating and trimming the configuration do
1172 the right thing. 1172 the right thing.
1173 1173
1174 [:ref:`back to step-by-step guide <bootworkin 1174 [:ref:`back to step-by-step guide <bootworking_bissbs>`]
1175 1175
1176 .. _diskspace_bisref: 1176 .. _diskspace_bisref:
1177 1177
1178 Space requirements 1178 Space requirements
1179 ~~~~~~~~~~~~~~~~~~ 1179 ~~~~~~~~~~~~~~~~~~
1180 1180
1181 *Ensure to have enough free space for build 1181 *Ensure to have enough free space for building Linux.*
1182 [:ref:`... <diskspace_bissbs>`] 1182 [:ref:`... <diskspace_bissbs>`]
1183 1183
1184 The numbers mentioned are rough estimates wit 1184 The numbers mentioned are rough estimates with a big extra charge to be on the
1185 safe side, so often you will need less. 1185 safe side, so often you will need less.
1186 1186
1187 If you have space constraints, be sure to hay 1187 If you have space constraints, be sure to hay attention to the :ref:`step about
1188 debug symbols' <debugsymbols_bissbs>` and its 1188 debug symbols' <debugsymbols_bissbs>` and its :ref:`accompanying reference
1189 section' <debugsymbols_bisref>`, as disabling 1189 section' <debugsymbols_bisref>`, as disabling then will reduce the consumed disk
1190 space by quite a few gigabytes. 1190 space by quite a few gigabytes.
1191 1191
1192 [:ref:`back to step-by-step guide <diskspace_ 1192 [:ref:`back to step-by-step guide <diskspace_bissbs>`]
1193 1193
1194 .. _rangecheck_bisref: 1194 .. _rangecheck_bisref:
1195 1195
1196 Bisection range 1196 Bisection range
1197 ~~~~~~~~~~~~~~~ 1197 ~~~~~~~~~~~~~~~
1198 1198
1199 *Determine the kernel versions considered ' 1199 *Determine the kernel versions considered 'good' and 'bad' throughout this
1200 guide.* [:ref:`...<rangecheck_bissbs>`] 1200 guide.* [:ref:`...<rangecheck_bissbs>`]
1201 1201
1202 Establishing the range of commits to be check 1202 Establishing the range of commits to be checked is mostly straightforward,
1203 except when a regression occurred when switch 1203 except when a regression occurred when switching from a release of one stable
1204 series to a release of a later series (e.g. f 1204 series to a release of a later series (e.g. from 6.0.13 to 6.1.5). In that case
1205 Git will need some hand holding, as there is 1205 Git will need some hand holding, as there is no straight line of descent.
1206 1206
1207 That's because with the release of 6.0 mainli 1207 That's because with the release of 6.0 mainline carried on to 6.1 while the
1208 stable series 6.0.y branched to the side. It' 1208 stable series 6.0.y branched to the side. It's therefore theoretically possible
1209 that the issue you face with 6.1.5 only worke 1209 that the issue you face with 6.1.5 only worked in 6.0.13, as it was fixed by a
1210 commit that went into one of the 6.0.y releas 1210 commit that went into one of the 6.0.y releases, but never hit mainline or the
1211 6.1.y series. Thankfully that normally should 1211 6.1.y series. Thankfully that normally should not happen due to the way the
1212 stable/longterm maintainers maintain the code 1212 stable/longterm maintainers maintain the code. It's thus pretty safe to assume
1213 6.0 as a 'good' kernel. That assumption will 1213 6.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel
1214 will be built and tested in the segment '2' o 1214 will be built and tested in the segment '2' of this guide; Git would force you
1215 to do this as well, if you tried bisecting be 1215 to do this as well, if you tried bisecting between 6.0.13 and 6.1.15.
1216 1216
1217 [:ref:`back to step-by-step guide <rangecheck 1217 [:ref:`back to step-by-step guide <rangecheck_bissbs>`]
1218 1218
1219 .. _buildrequires_bisref: 1219 .. _buildrequires_bisref:
1220 1220
1221 Install build requirements 1221 Install build requirements
1222 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1222 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1223 1223
1224 *Install all software required to build a L 1224 *Install all software required to build a Linux kernel.*
1225 [:ref:`...<buildrequires_bissbs>`] 1225 [:ref:`...<buildrequires_bissbs>`]
1226 1226
1227 The kernel is pretty stand-alone, but besides 1227 The kernel is pretty stand-alone, but besides tools like the compiler you will
1228 sometimes need a few libraries to build one. 1228 sometimes need a few libraries to build one. How to install everything needed
1229 depends on your Linux distribution and the co 1229 depends on your Linux distribution and the configuration of the kernel you are
1230 about to build. 1230 about to build.
1231 1231
1232 Here are a few examples what you typically ne 1232 Here are a few examples what you typically need on some mainstream
1233 distributions: 1233 distributions:
1234 1234
1235 * Arch Linux and derivatives:: 1235 * Arch Linux and derivatives::
1236 1236
1237 sudo pacman --needed -S bc binutils bison 1237 sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \
1238 pahole perl zlib ncurses qt6-base 1238 pahole perl zlib ncurses qt6-base
1239 1239
1240 * Debian, Ubuntu, and derivatives:: 1240 * Debian, Ubuntu, and derivatives::
1241 1241
1242 sudo apt install bc binutils bison dwarve 1242 sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \
1243 libssl-dev make openssl pahole perl-bas 1243 libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \
1244 libncurses-dev qt6-base-dev g++ 1244 libncurses-dev qt6-base-dev g++
1245 1245
1246 * Fedora and derivatives:: 1246 * Fedora and derivatives::
1247 1247
1248 sudo dnf install binutils \ 1248 sudo dnf install binutils \
1249 /usr/bin/{bc,bison,flex,gcc,git,openssl 1249 /usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \
1250 /usr/include/{libelf.h,openssl/pkcs7.h, 1250 /usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction}
1251 1251
1252 * openSUSE and derivatives:: 1252 * openSUSE and derivatives::
1253 1253
1254 sudo zypper install bc binutils bison dwa 1254 sudo zypper install bc binutils bison dwarves flex gcc git \
1255 kernel-install-tools libelf-devel make 1255 kernel-install-tools libelf-devel make modutils openssl openssl-devel \
1256 perl-base zlib-devel rpm-build ncurses- 1256 perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel
1257 1257
1258 These commands install a few packages that ar 1258 These commands install a few packages that are often, but not always needed. You
1259 for example might want to skip installing the 1259 for example might want to skip installing the development headers for ncurses,
1260 which you will only need in case you later mi 1260 which you will only need in case you later might want to adjust the kernel build
1261 configuration using make the targets 'menucon 1261 configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit
1262 the headers of Qt6 if you do not plan to adju 1262 the headers of Qt6 if you do not plan to adjust the .config using 'xconfig'.
1263 1263
1264 You furthermore might need additional librari 1264 You furthermore might need additional libraries and their development headers
1265 for tasks not covered in this guide -- for ex 1265 for tasks not covered in this guide -- for example when building utilities from
1266 the kernel's tools/ directory. 1266 the kernel's tools/ directory.
1267 1267
1268 [:ref:`back to step-by-step guide <buildrequi 1268 [:ref:`back to step-by-step guide <buildrequires_bissbs>`]
1269 1269
1270 .. _sources_bisref: 1270 .. _sources_bisref:
1271 1271
1272 Download the sources using Git 1272 Download the sources using Git
1273 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1273 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1274 1274
1275 *Retrieve the Linux mainline sources.* 1275 *Retrieve the Linux mainline sources.*
1276 [:ref:`...<sources_bissbs>`] 1276 [:ref:`...<sources_bissbs>`]
1277 1277
1278 The step-by-step guide outlines how to downlo 1278 The step-by-step guide outlines how to download the Linux sources using a full
1279 Git clone of Linus' mainline repository. Ther 1279 Git clone of Linus' mainline repository. There is nothing more to say about
1280 that -- but there are two alternatives ways t 1280 that -- but there are two alternatives ways to retrieve the sources that might
1281 work better for you: 1281 work better for you:
1282 1282
1283 * If you have an unreliable internet connecti 1283 * If you have an unreliable internet connection, consider
1284 :ref:`using a 'Git bundle'<sources_bundle_b 1284 :ref:`using a 'Git bundle'<sources_bundle_bisref>`.
1285 1285
1286 * If downloading the complete repository woul 1286 * If downloading the complete repository would take too long or requires too
1287 much storage space, consider :ref:`using a 1287 much storage space, consider :ref:`using a 'shallow
1288 clone'<sources_shallow_bisref>`. 1288 clone'<sources_shallow_bisref>`.
1289 1289
1290 .. _sources_bundle_bisref: 1290 .. _sources_bundle_bisref:
1291 1291
1292 Downloading Linux mainline sources using a bu 1292 Downloading Linux mainline sources using a bundle
1293 """"""""""""""""""""""""""""""""""""""""""""" 1293 """""""""""""""""""""""""""""""""""""""""""""""""
1294 1294
1295 Use the following commands to retrieve the Li 1295 Use the following commands to retrieve the Linux mainline sources using a
1296 bundle:: 1296 bundle::
1297 1297
1298 wget -c \ 1298 wget -c \
1299 https://git.kernel.org/pub/scm/linux/ke 1299 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
1300 git clone --no-checkout clone.bundle ~/li 1300 git clone --no-checkout clone.bundle ~/linux/
1301 cd ~/linux/ 1301 cd ~/linux/
1302 git remote remove origin 1302 git remote remove origin
1303 git remote add mainline \ 1303 git remote add mainline \
1304 https://git.kernel.org/pub/scm/linux/ke 1304 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
1305 git fetch mainline 1305 git fetch mainline
1306 git remote add -t master stable \ 1306 git remote add -t master stable \
1307 https://git.kernel.org/pub/scm/linux/ke 1307 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1308 1308
1309 In case the 'wget' command fails, just re-exe 1309 In case the 'wget' command fails, just re-execute it, it will pick up where
1310 it left off. 1310 it left off.
1311 1311
1312 [:ref:`back to step-by-step guide <sources_bi 1312 [:ref:`back to step-by-step guide <sources_bissbs>`]
1313 [:ref:`back to section intro <sources_bisref> 1313 [:ref:`back to section intro <sources_bisref>`]
1314 1314
1315 .. _sources_shallow_bisref: 1315 .. _sources_shallow_bisref:
1316 1316
1317 Downloading Linux mainline sources using a sh 1317 Downloading Linux mainline sources using a shallow clone
1318 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1318 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1319 1319
1320 First, execute the following command to retri 1320 First, execute the following command to retrieve the latest mainline codebase::
1321 1321
1322 git clone -o mainline --no-checkout --dep 1322 git clone -o mainline --no-checkout --depth 1 -b master \
1323 https://git.kernel.org/pub/scm/linux/ke 1323 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
1324 cd ~/linux/ 1324 cd ~/linux/
1325 git remote add -t master stable \ 1325 git remote add -t master stable \
1326 https://git.kernel.org/pub/scm/linux/ke 1326 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1327 1327
1328 Now deepen your clone's history to the second 1328 Now deepen your clone's history to the second predecessor of the mainline
1329 release of your 'good' version. In case the l 1329 release of your 'good' version. In case the latter are 6.0 or 6.0.13, 5.19 would
1330 be the first predecessor and 5.18 the second 1330 be the first predecessor and 5.18 the second -- hence deepen the history up to
1331 that version:: 1331 that version::
1332 1332
1333 git fetch --shallow-exclude=v5.18 mainlin 1333 git fetch --shallow-exclude=v5.18 mainline
1334 1334
1335 Afterwards add the stable Git repository as r 1335 Afterwards add the stable Git repository as remote and all required stable
1336 branches as explained in the step-by-step gui 1336 branches as explained in the step-by-step guide.
1337 1337
1338 Note, shallow clones have a few peculiar char 1338 Note, shallow clones have a few peculiar characteristics:
1339 1339
1340 * For bisections the history needs to be deep 1340 * For bisections the history needs to be deepened a few mainline versions
1341 farther than it seems necessary, as explain 1341 farther than it seems necessary, as explained above already. That's because
1342 Git otherwise will be unable to revert or d 1342 Git otherwise will be unable to revert or describe most of the commits within
1343 a range (say 6.1..6.2), as they are interna 1343 a range (say 6.1..6.2), as they are internally based on earlier kernels
1344 releases (like 6.0-rc2 or 5.19-rc3). 1344 releases (like 6.0-rc2 or 5.19-rc3).
1345 1345
1346 * This document in most places uses ``git fet 1346 * This document in most places uses ``git fetch`` with ``--shallow-exclude=``
1347 to specify the earliest version you care ab 1347 to specify the earliest version you care about (or to be precise: its git
1348 tag). You alternatively can use the paramet 1348 tag). You alternatively can use the parameter ``--shallow-since=`` to specify
1349 an absolute (say ``'2023-07-15'``) or relat 1349 an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to
1350 define the depth of the history you want to 1350 define the depth of the history you want to download. When using them while
1351 bisecting mainline, ensure to deepen the hi 1351 bisecting mainline, ensure to deepen the history to at least 7 months before
1352 the release of the mainline release your 'g 1352 the release of the mainline release your 'good' kernel is based on.
1353 1353
1354 * Be warned, when deepening your clone you mi 1354 * Be warned, when deepening your clone you might encounter an error like
1355 'fatal: error in object: unshallow cafecaca 1355 'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'.
1356 In that case run ``git repack -d`` and try 1356 In that case run ``git repack -d`` and try again.
1357 1357
1358 [:ref:`back to step-by-step guide <sources_bi 1358 [:ref:`back to step-by-step guide <sources_bissbs>`]
1359 [:ref:`back to section intro <sources_bisref> 1359 [:ref:`back to section intro <sources_bisref>`]
1360 1360
1361 .. _oldconfig_bisref: 1361 .. _oldconfig_bisref:
1362 1362
1363 Start defining the build configuration for yo 1363 Start defining the build configuration for your kernel
1364 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1364 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1365 1365
1366 *Start preparing a kernel build configurati 1366 *Start preparing a kernel build configuration (the '.config' file).*
1367 [:ref:`... <oldconfig_bissbs>`] 1367 [:ref:`... <oldconfig_bissbs>`]
1368 1368
1369 *Note, this is the first of multiple steps in 1369 *Note, this is the first of multiple steps in this guide that create or modify
1370 build artifacts. The commands used in this gu 1370 build artifacts. The commands used in this guide store them right in the source
1371 tree to keep things simple. In case you prefe 1371 tree to keep things simple. In case you prefer storing the build artifacts
1372 separately, create a directory like '~/linux- 1372 separately, create a directory like '~/linux-builddir/' and add the parameter
1373 ``O=~/linux-builddir/`` to all make calls use 1373 ``O=~/linux-builddir/`` to all make calls used throughout this guide. You will
1374 have to point other commands there as well -- 1374 have to point other commands there as well -- among them the ``./scripts/config
1375 [...]`` commands, which will require ``--file 1375 [...]`` commands, which will require ``--file ~/linux-builddir/.config`` to
1376 locate the right build configuration.* 1376 locate the right build configuration.*
1377 1377
1378 Two things can easily go wrong when creating 1378 Two things can easily go wrong when creating a .config file as advised:
1379 1379
1380 * The oldconfig target will use a .config fil 1380 * The oldconfig target will use a .config file from your build directory, if
1381 one is already present there (e.g. '~/linux 1381 one is already present there (e.g. '~/linux/.config'). That's totally fine if
1382 that's what you intend (see next step), but 1382 that's what you intend (see next step), but in all other cases you want to
1383 delete it. This for example is important in 1383 delete it. This for example is important in case you followed this guide
1384 further, but due to problems come back here 1384 further, but due to problems come back here to redo the configuration from
1385 scratch. 1385 scratch.
1386 1386
1387 * Sometimes olddefconfig is unable to locate 1387 * Sometimes olddefconfig is unable to locate the .config file for your running
1388 kernel and will use defaults, as briefly ou 1388 kernel and will use defaults, as briefly outlined in the guide. In that case
1389 check if your distribution ships the config 1389 check if your distribution ships the configuration somewhere and manually put
1390 it in the right place (e.g. '~/linux/.confi 1390 it in the right place (e.g. '~/linux/.config') if it does. On distributions
1391 where /proc/config.gz exists this can be ac 1391 where /proc/config.gz exists this can be achieved using this command::
1392 1392
1393 zcat /proc/config.gz > .config 1393 zcat /proc/config.gz > .config
1394 1394
1395 Once you put it there, run ``make olddefcon 1395 Once you put it there, run ``make olddefconfig`` again to adjust it to the
1396 needs of the kernel about to be built. 1396 needs of the kernel about to be built.
1397 1397
1398 Note, the olddefconfig target will set any un 1398 Note, the olddefconfig target will set any undefined build options to their
1399 default value. If you prefer to set such conf 1399 default value. If you prefer to set such configuration options manually, use
1400 ``make oldconfig`` instead. Then for each und 1400 ``make oldconfig`` instead. Then for each undefined configuration option you
1401 will be asked how to proceed; in case you are 1401 will be asked how to proceed; in case you are unsure what to answer, simply hit
1402 'enter' to apply the default value. Note thou 1402 'enter' to apply the default value. Note though that for bisections you normally
1403 want to go with the defaults, as you otherwis 1403 want to go with the defaults, as you otherwise might enable a new feature that
1404 causes a problem looking like regressions (fo 1404 causes a problem looking like regressions (for example due to security
1405 restrictions). 1405 restrictions).
1406 1406
1407 Occasionally odd things happen when trying to 1407 Occasionally odd things happen when trying to use a config file prepared for one
1408 kernel (say 6.1) on an older mainline release 1408 kernel (say 6.1) on an older mainline release -- especially if it is much older
1409 (say 5.15). That's one of the reasons why the 1409 (say 5.15). That's one of the reasons why the previous step in the guide told
1410 you to boot the kernel where everything works 1410 you to boot the kernel where everything works. If you manually add a .config
1411 file you thus want to ensure it's from the wo 1411 file you thus want to ensure it's from the working kernel and not from a one
1412 that shows the regression. 1412 that shows the regression.
1413 1413
1414 In case you want to build kernels for another 1414 In case you want to build kernels for another machine, locate its kernel build
1415 configuration; usually ``ls /boot/config-$(un 1415 configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy
1416 that file to the build machine and store it a 1416 that file to the build machine and store it as ~/linux/.config; afterwards run
1417 ``make olddefconfig`` to adjust it. 1417 ``make olddefconfig`` to adjust it.
1418 1418
1419 [:ref:`back to step-by-step guide <oldconfig_ 1419 [:ref:`back to step-by-step guide <oldconfig_bissbs>`]
1420 1420
1421 .. _localmodconfig_bisref: 1421 .. _localmodconfig_bisref:
1422 1422
1423 Trim the build configuration for your kernel 1423 Trim the build configuration for your kernel
1424 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1424 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1425 1425
1426 *Disable any kernel modules apparently supe 1426 *Disable any kernel modules apparently superfluous for your setup.*
1427 [:ref:`... <localmodconfig_bissbs>`] 1427 [:ref:`... <localmodconfig_bissbs>`]
1428 1428
1429 As explained briefly in the step-by-step guid 1429 As explained briefly in the step-by-step guide already: with localmodconfig it
1430 can easily happen that your self-built kernel 1430 can easily happen that your self-built kernels will lack modules for tasks you
1431 did not perform at least once before utilizin 1431 did not perform at least once before utilizing this make target. That happens
1432 when a task requires kernel modules which are 1432 when a task requires kernel modules which are only autoloaded when you execute
1433 it for the first time. So when you never perf 1433 it for the first time. So when you never performed that task since starting your
1434 kernel the modules will not have been loaded 1434 kernel the modules will not have been loaded -- and from localmodonfig's point
1435 of view look superfluous, which thus disables 1435 of view look superfluous, which thus disables them to reduce the amount of code
1436 to be compiled. 1436 to be compiled.
1437 1437
1438 You can try to avoid this by performing typic 1438 You can try to avoid this by performing typical tasks that often will autoload
1439 additional kernel modules: start a VM, establ 1439 additional kernel modules: start a VM, establish VPN connections, loop-mount a
1440 CD/DVD ISO, mount network shares (CIFS, NFS, 1440 CD/DVD ISO, mount network shares (CIFS, NFS, ...), and connect all external
1441 devices (2FA keys, headsets, webcams, ...) as 1441 devices (2FA keys, headsets, webcams, ...) as well as storage devices with file
1442 systems you otherwise do not utilize (btrfs, 1442 systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, ...). But it
1443 is hard to think of everything that might be 1443 is hard to think of everything that might be needed -- even kernel developers
1444 often forget one thing or another at this poi 1444 often forget one thing or another at this point.
1445 1445
1446 Do not let that risk bother you, especially w 1446 Do not let that risk bother you, especially when compiling a kernel only for
1447 testing purposes: everything typically crucia 1447 testing purposes: everything typically crucial will be there. And if you forget
1448 something important you can turn on a missing 1448 something important you can turn on a missing feature manually later and quickly
1449 run the commands again to compile and install 1449 run the commands again to compile and install a kernel that has everything you
1450 need. 1450 need.
1451 1451
1452 But if you plan to build and use self-built k 1452 But if you plan to build and use self-built kernels regularly, you might want to
1453 reduce the risk by recording which modules yo 1453 reduce the risk by recording which modules your system loads over the course of
1454 a few weeks. You can automate this with `modp 1454 a few weeks. You can automate this with `modprobed-db
1455 <https://github.com/graysky2/modprobed-db>`_. 1455 <https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to
1456 point localmodconfig to the list of modules m 1456 point localmodconfig to the list of modules modprobed-db noticed being used::
1457 1457
1458 yes '' | make LSMOD='${HOME}'/.config/modpr 1458 yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig
1459 1459
1460 That parameter also allows you to build trimm 1460 That parameter also allows you to build trimmed kernels for another machine in
1461 case you copied a suitable .config over to us 1461 case you copied a suitable .config over to use as base (see previous step). Just
1462 run ``lsmod > lsmod_foo-machine`` on that sys 1462 run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to
1463 your build's host home directory. Then run th 1463 your build's host home directory. Then run these commands instead of the one the
1464 step-by-step guide mentions:: 1464 step-by-step guide mentions::
1465 1465
1466 yes '' | make LSMOD=~/lsmod_foo-machine loc 1466 yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig
1467 1467
1468 [:ref:`back to step-by-step guide <localmodco 1468 [:ref:`back to step-by-step guide <localmodconfig_bissbs>`]
1469 1469
1470 .. _tagging_bisref: 1470 .. _tagging_bisref:
1471 1471
1472 Tag the kernels about to be build 1472 Tag the kernels about to be build
1473 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1473 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1474 1474
1475 *Ensure all the kernels you will build are 1475 *Ensure all the kernels you will build are clearly identifiable using a
1476 special tag and a unique version identifier 1476 special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`]
1477 1477
1478 This allows you to differentiate your distrib 1478 This allows you to differentiate your distribution's kernels from those created
1479 during this process, as the file or directori 1479 during this process, as the file or directories for the latter will contain
1480 '-local' in the name; it also helps picking t 1480 '-local' in the name; it also helps picking the right entry in the boot menu and
1481 not lose track of you kernels, as their versi 1481 not lose track of you kernels, as their version numbers will look slightly
1482 confusing during the bisection. 1482 confusing during the bisection.
1483 1483
1484 [:ref:`back to step-by-step guide <tagging_bi 1484 [:ref:`back to step-by-step guide <tagging_bissbs>`]
1485 1485
1486 .. _debugsymbols_bisref: 1486 .. _debugsymbols_bisref:
1487 1487
1488 Decide to enable or disable debug symbols 1488 Decide to enable or disable debug symbols
1489 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1489 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1490 1490
1491 *Decide how to handle debug symbols.* [:ref 1491 *Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`]
1492 1492
1493 Having debug symbols available can be importa 1493 Having debug symbols available can be important when your kernel throws a
1494 'panic', 'Oops', 'warning', or 'BUG' later wh 1494 'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be
1495 able to find the exact place where the proble 1495 able to find the exact place where the problem occurred in the code. But
1496 collecting and embedding the needed debug inf 1496 collecting and embedding the needed debug information takes time and consumes
1497 quite a bit of space: in late 2022 the build 1497 quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel
1498 trimmed with localmodconfig consumed around 5 1498 trimmed with localmodconfig consumed around 5 Gigabyte of space with debug
1499 symbols, but less than 1 when they were disab 1499 symbols, but less than 1 when they were disabled. The resulting kernel image and
1500 modules are bigger as well, which increases s 1500 modules are bigger as well, which increases storage requirements for /boot/ and
1501 load times. 1501 load times.
1502 1502
1503 In case you want a small kernel and are unlik 1503 In case you want a small kernel and are unlikely to decode a stack trace later,
1504 you thus might want to disable debug symbols 1504 you thus might want to disable debug symbols to avoid those downsides. If it
1505 later turns out that you need them, just enab 1505 later turns out that you need them, just enable them as shown and rebuild the
1506 kernel. 1506 kernel.
1507 1507
1508 You on the other hand definitely want to enab 1508 You on the other hand definitely want to enable them for this process, if there
1509 is a decent chance that you need to decode a 1509 is a decent chance that you need to decode a stack trace later. The section
1510 'Decode failure messages' in Documentation/ad 1510 'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst
1511 explains this process in more detail. 1511 explains this process in more detail.
1512 1512
1513 [:ref:`back to step-by-step guide <debugsymbo 1513 [:ref:`back to step-by-step guide <debugsymbols_bissbs>`]
1514 1514
1515 .. _configmods_bisref: 1515 .. _configmods_bisref:
1516 1516
1517 Adjust build configuration 1517 Adjust build configuration
1518 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1518 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1519 1519
1520 *Check if you may want or need to adjust so 1520 *Check if you may want or need to adjust some other kernel configuration
1521 options:* 1521 options:*
1522 1522
1523 Depending on your needs you at this point mig 1523 Depending on your needs you at this point might want or have to adjust some
1524 kernel configuration options. 1524 kernel configuration options.
1525 1525
1526 .. _configmods_distros_bisref: 1526 .. _configmods_distros_bisref:
1527 1527
1528 Distro specific adjustments 1528 Distro specific adjustments
1529 """"""""""""""""""""""""""" 1529 """""""""""""""""""""""""""
1530 1530
1531 *Are you running* [:ref:`... <configmods_bi 1531 *Are you running* [:ref:`... <configmods_bissbs>`]
1532 1532
1533 The following sections help you to avoid buil 1533 The following sections help you to avoid build problems that are known to occur
1534 when following this guide on a few commodity 1534 when following this guide on a few commodity distributions.
1535 1535
1536 **Debian:** 1536 **Debian:**
1537 1537
1538 * Remove a stale reference to a certificate f 1538 * Remove a stale reference to a certificate file that would cause your build to
1539 fail:: 1539 fail::
1540 1540
1541 ./scripts/config --set-str SYSTEM_TRUSTED_ 1541 ./scripts/config --set-str SYSTEM_TRUSTED_KEYS ''
1542 1542
1543 Alternatively, download the needed certific 1543 Alternatively, download the needed certificate and make that configuration
1544 option point to it, as `the Debian handbook 1544 option point to it, as `the Debian handbook explains in more detail
1545 <https://debian-handbook.info/browse/stable 1545 <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_
1546 -- or generate your own, as explained in 1546 -- or generate your own, as explained in
1547 Documentation/admin-guide/module-signing.rs 1547 Documentation/admin-guide/module-signing.rst.
1548 1548
1549 [:ref:`back to step-by-step guide <configmods 1549 [:ref:`back to step-by-step guide <configmods_bissbs>`]
1550 1550
1551 .. _configmods_individual_bisref: 1551 .. _configmods_individual_bisref:
1552 1552
1553 Individual adjustments 1553 Individual adjustments
1554 """""""""""""""""""""" 1554 """"""""""""""""""""""
1555 1555
1556 *If you want to influence the other aspects 1556 *If you want to influence the other aspects of the configuration, do so
1557 now.* [:ref:`... <configmods_bissbs>`] 1557 now.* [:ref:`... <configmods_bissbs>`]
1558 1558
1559 At this point you can use a command like ``ma 1559 At this point you can use a command like ``make menuconfig`` or ``make nconfig``
1560 to enable or disable certain features using a 1560 to enable or disable certain features using a text-based user interface; to use
1561 a graphical configuration utility, run ``make 1561 a graphical configuration utility, run ``make xconfig`` instead. Both of them
1562 require development libraries from toolkits t 1562 require development libraries from toolkits they are rely on (ncurses
1563 respectively Qt5 or Qt6); an error message wi 1563 respectively Qt5 or Qt6); an error message will tell you if something required
1564 is missing. 1564 is missing.
1565 1565
1566 [:ref:`back to step-by-step guide <configmods 1566 [:ref:`back to step-by-step guide <configmods_bissbs>`]
1567 1567
1568 .. _saveconfig_bisref: 1568 .. _saveconfig_bisref:
1569 1569
1570 Put the .config file aside 1570 Put the .config file aside
1571 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1571 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1572 1572
1573 *Reprocess the .config after the latest cha 1573 *Reprocess the .config after the latest changes and store it in a safe place.*
1574 [:ref:`... <saveconfig_bissbs>`] 1574 [:ref:`... <saveconfig_bissbs>`]
1575 1575
1576 Put the .config you prepared aside, as you wa 1576 Put the .config you prepared aside, as you want to copy it back to the build
1577 directory every time during this guide before 1577 directory every time during this guide before you start building another
1578 kernel. That's because going back and forth b 1578 kernel. That's because going back and forth between different versions can alter
1579 .config files in odd ways; those occasionally 1579 .config files in odd ways; those occasionally cause side effects that could
1580 confuse testing or in some cases render the r 1580 confuse testing or in some cases render the result of your bisection
1581 meaningless. 1581 meaningless.
1582 1582
1583 [:ref:`back to step-by-step guide <saveconfig 1583 [:ref:`back to step-by-step guide <saveconfig_bissbs>`]
1584 1584
1585 .. _introlatestcheck_bisref: 1585 .. _introlatestcheck_bisref:
1586 1586
1587 Try to reproduce the problem with the latest 1587 Try to reproduce the problem with the latest codebase
1588 --------------------------------------------- 1588 -----------------------------------------------------
1589 1589
1590 *Verify the regression is not caused by som 1590 *Verify the regression is not caused by some .config change and check if it
1591 still occurs with the latest codebase.* [:r 1591 still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`]
1592 1592
1593 For some readers it might seem unnecessary to 1593 For some readers it might seem unnecessary to check the latest codebase at this
1594 point, especially if you did that already wit 1594 point, especially if you did that already with a kernel prepared by your
1595 distributor or face a regression within a sta 1595 distributor or face a regression within a stable/longterm series. But it's
1596 highly recommended for these reasons: 1596 highly recommended for these reasons:
1597 1597
1598 * You will run into any problems caused by yo 1598 * You will run into any problems caused by your setup before you actually begin
1599 a bisection. That will make it a lot easier 1599 a bisection. That will make it a lot easier to differentiate between 'this
1600 most likely is some problem in my setup' an 1600 most likely is some problem in my setup' and 'this change needs to be skipped
1601 during the bisection, as the kernel sources 1601 during the bisection, as the kernel sources at that stage contain an unrelated
1602 problem that causes building or booting to 1602 problem that causes building or booting to fail'.
1603 1603
1604 * These steps will rule out if your problem i 1604 * These steps will rule out if your problem is caused by some change in the
1605 build configuration between the 'working' a 1605 build configuration between the 'working' and the 'broken' kernel. This for
1606 example can happen when your distributor en 1606 example can happen when your distributor enabled an additional security
1607 feature in the newer kernel which was disab 1607 feature in the newer kernel which was disabled or not yet supported by the
1608 older kernel. That security feature might g 1608 older kernel. That security feature might get into the way of something you
1609 do -- in which case your problem from the p 1609 do -- in which case your problem from the perspective of the Linux kernel
1610 upstream developers is not a regression, as 1610 upstream developers is not a regression, as
1611 Documentation/admin-guide/reporting-regress 1611 Documentation/admin-guide/reporting-regressions.rst explains in more detail.
1612 You thus would waste your time if you'd try 1612 You thus would waste your time if you'd try to bisect this.
1613 1613
1614 * If the cause for your regression was alread 1614 * If the cause for your regression was already fixed in the latest mainline
1615 codebase, you'd perform the bisection for n 1615 codebase, you'd perform the bisection for nothing. This holds true for a
1616 regression you encountered with a stable/lo 1616 regression you encountered with a stable/longterm release as well, as they are
1617 often caused by problems in mainline change 1617 often caused by problems in mainline changes that were backported -- in which
1618 case the problem will have to be fixed in m 1618 case the problem will have to be fixed in mainline first. Maybe it already was
1619 fixed there and the fix is already in the p 1619 fixed there and the fix is already in the process of being backported.
1620 1620
1621 * For regressions within a stable/longterm se 1621 * For regressions within a stable/longterm series it's furthermore crucial to
1622 know if the issue is specific to that serie 1622 know if the issue is specific to that series or also happens in the mainline
1623 kernel, as the report needs to be sent to d 1623 kernel, as the report needs to be sent to different people:
1624 1624
1625 * Regressions specific to a stable/longterm 1625 * Regressions specific to a stable/longterm series are the stable team's
1626 responsibility; mainline Linux developers 1626 responsibility; mainline Linux developers might or might not care.
1627 1627
1628 * Regressions also happening in mainline ar 1628 * Regressions also happening in mainline are something the regular Linux
1629 developers and maintainers have to handle 1629 developers and maintainers have to handle; the stable team does not care
1630 and does not need to be involved in the r 1630 and does not need to be involved in the report, they just should be told
1631 to backport the fix once it's ready. 1631 to backport the fix once it's ready.
1632 1632
1633 Your report might be ignored if you send it 1633 Your report might be ignored if you send it to the wrong party -- and even
1634 when you get a reply there is a decent chan 1634 when you get a reply there is a decent chance that developers tell you to
1635 evaluate which of the two cases it is befor 1635 evaluate which of the two cases it is before they take a closer look.
1636 1636
1637 [:ref:`back to step-by-step guide <introlates 1637 [:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]
1638 1638
1639 .. _checkoutmaster_bisref: 1639 .. _checkoutmaster_bisref:
1640 1640
1641 Check out the latest Linux codebase 1641 Check out the latest Linux codebase
1642 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1642 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1643 1643
1644 *Check out the latest Linux codebase.* 1644 *Check out the latest Linux codebase.*
1645 [:ref:`... <checkoutmaster_bissbs>`] 1645 [:ref:`... <checkoutmaster_bissbs>`]
1646 1646
1647 In case you later want to recheck if an ever 1647 In case you later want to recheck if an ever newer codebase might fix the
1648 problem, remember to run that ``git fetch --s 1648 problem, remember to run that ``git fetch --shallow-exclude [...]`` command
1649 again mentioned earlier to update your local 1649 again mentioned earlier to update your local Git repository.
1650 1650
1651 [:ref:`back to step-by-step guide <checkoutma 1651 [:ref:`back to step-by-step guide <checkoutmaster_bissbs>`]
1652 1652
1653 .. _build_bisref: 1653 .. _build_bisref:
1654 1654
1655 Build your kernel 1655 Build your kernel
1656 ~~~~~~~~~~~~~~~~~ 1656 ~~~~~~~~~~~~~~~~~
1657 1657
1658 *Build the image and the modules of your fi 1658 *Build the image and the modules of your first kernel using the config file
1659 you prepared.* [:ref:`... <build_bissbs>`] 1659 you prepared.* [:ref:`... <build_bissbs>`]
1660 1660
1661 A lot can go wrong at this stage, but the ins 1661 A lot can go wrong at this stage, but the instructions below will help you help
1662 yourself. Another subsection explains how to 1662 yourself. Another subsection explains how to directly package your kernel up as
1663 deb, rpm or tar file. 1663 deb, rpm or tar file.
1664 1664
1665 Dealing with build errors 1665 Dealing with build errors
1666 """"""""""""""""""""""""" 1666 """""""""""""""""""""""""
1667 1667
1668 When a build error occurs, it might be caused 1668 When a build error occurs, it might be caused by some aspect of your machine's
1669 setup that often can be fixed quickly; other 1669 setup that often can be fixed quickly; other times though the problem lies in
1670 the code and can only be fixed by a developer 1670 the code and can only be fixed by a developer. A close examination of the
1671 failure messages coupled with some research o 1671 failure messages coupled with some research on the internet will often tell you
1672 which of the two it is. To perform such inves 1672 which of the two it is. To perform such investigation, restart the build
1673 process like this:: 1673 process like this::
1674 1674
1675 make V=1 1675 make V=1
1676 1676
1677 The ``V=1`` activates verbose output, which m 1677 The ``V=1`` activates verbose output, which might be needed to see the actual
1678 error. To make it easier to spot, this comman 1678 error. To make it easier to spot, this command also omits the ``-j $(nproc
1679 --all)`` used earlier to utilize every CPU co 1679 --all)`` used earlier to utilize every CPU core in the system for the job -- but
1680 this parallelism also results in some clutter 1680 this parallelism also results in some clutter when failures occur.
1681 1681
1682 After a few seconds the build process should 1682 After a few seconds the build process should run into the error again. Now try
1683 to find the most crucial line describing the 1683 to find the most crucial line describing the problem. Then search the internet
1684 for the most important and non-generic sectio 1684 for the most important and non-generic section of that line (say 4 to 8 words);
1685 avoid or remove anything that looks remotely 1685 avoid or remove anything that looks remotely system-specific, like your username
1686 or local path names like ``/home/username/lin 1686 or local path names like ``/home/username/linux/``. First try your regular
1687 internet search engine with that string, afte 1687 internet search engine with that string, afterwards search Linux kernel mailing
1688 lists via `lore.kernel.org/all/ <https://lore 1688 lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_.
1689 1689
1690 This most of the time will find something tha 1690 This most of the time will find something that will explain what is wrong; quite
1691 often one of the hits will provide a solution 1691 often one of the hits will provide a solution for your problem, too. If you
1692 do not find anything that matches your proble 1692 do not find anything that matches your problem, try again from a different angle
1693 by modifying your search terms or using anoth 1693 by modifying your search terms or using another line from the error messages.
1694 1694
1695 In the end, most issues you run into have lik 1695 In the end, most issues you run into have likely been encountered and
1696 reported by others already. That includes iss 1696 reported by others already. That includes issues where the cause is not your
1697 system, but lies in the code. If you run into 1697 system, but lies in the code. If you run into one of those, you might thus find
1698 a solution (e.g. a patch) or workaround for y 1698 a solution (e.g. a patch) or workaround for your issue, too.
1699 1699
1700 Package your kernel up 1700 Package your kernel up
1701 """""""""""""""""""""" 1701 """"""""""""""""""""""
1702 1702
1703 The step-by-step guide uses the default make 1703 The step-by-step guide uses the default make targets (e.g. 'bzImage' and
1704 'modules' on x86) to build the image and the 1704 'modules' on x86) to build the image and the modules of your kernel, which later
1705 steps of the guide then install. You instead 1705 steps of the guide then install. You instead can also directly build everything
1706 and directly package it up by using one of th 1706 and directly package it up by using one of the following targets:
1707 1707
1708 * ``make -j $(nproc --all) bindeb-pkg`` to ge 1708 * ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package
1709 1709
1710 * ``make -j $(nproc --all) binrpm-pkg`` to ge 1710 * ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package
1711 1711
1712 * ``make -j $(nproc --all) tarbz2-pkg`` to ge 1712 * ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball
1713 1713
1714 This is just a selection of available make ta 1714 This is just a selection of available make targets for this purpose, see
1715 ``make help`` for others. You can also use th 1715 ``make help`` for others. You can also use these targets after running
1716 ``make -j $(nproc --all)``, as they will pick 1716 ``make -j $(nproc --all)``, as they will pick up everything already built.
1717 1717
1718 If you employ the targets to generate deb or 1718 If you employ the targets to generate deb or rpm packages, ignore the
1719 step-by-step guide's instructions on installi 1719 step-by-step guide's instructions on installing and removing your kernel;
1720 instead install and remove the packages using 1720 instead install and remove the packages using the package utility for the format
1721 (e.g. dpkg and rpm) or a package management u 1721 (e.g. dpkg and rpm) or a package management utility build on top of them (apt,
1722 aptitude, dnf/yum, zypper, ...). Be aware tha 1722 aptitude, dnf/yum, zypper, ...). Be aware that the packages generated using
1723 these two make targets are designed to work o 1723 these two make targets are designed to work on various distributions utilizing
1724 those formats, they thus will sometimes behav 1724 those formats, they thus will sometimes behave differently than your
1725 distribution's kernel packages. 1725 distribution's kernel packages.
1726 1726
1727 [:ref:`back to step-by-step guide <build_biss 1727 [:ref:`back to step-by-step guide <build_bissbs>`]
1728 1728
1729 .. _install_bisref: 1729 .. _install_bisref:
1730 1730
1731 Put the kernel in place 1731 Put the kernel in place
1732 ~~~~~~~~~~~~~~~~~~~~~~~ 1732 ~~~~~~~~~~~~~~~~~~~~~~~
1733 1733
1734 *Install the kernel you just built.* [:ref: 1734 *Install the kernel you just built.* [:ref:`... <install_bissbs>`]
1735 1735
1736 What you need to do after executing the comma 1736 What you need to do after executing the command in the step-by-step guide
1737 depends on the existence and the implementati 1737 depends on the existence and the implementation of ``/sbin/installkernel``
1738 executable on your distribution. 1738 executable on your distribution.
1739 1739
1740 If installkernel is found, the kernel's build 1740 If installkernel is found, the kernel's build system will delegate the actual
1741 installation of your kernel image to this exe 1741 installation of your kernel image to this executable, which then performs some
1742 or all of these tasks: 1742 or all of these tasks:
1743 1743
1744 * On almost all Linux distributions installke 1744 * On almost all Linux distributions installkernel will store your kernel's
1745 image in /boot/, usually as '/boot/vmlinuz- 1745 image in /boot/, usually as '/boot/vmlinuz-<kernelrelease_id>'; often it will
1746 put a 'System.map-<kernelrelease_id>' along 1746 put a 'System.map-<kernelrelease_id>' alongside it.
1747 1747
1748 * On most distributions installkernel will th 1748 * On most distributions installkernel will then generate an 'initramfs'
1749 (sometimes also called 'initrd'), which usu 1749 (sometimes also called 'initrd'), which usually are stored as
1750 '/boot/initramfs-<kernelrelease_id>.img' or 1750 '/boot/initramfs-<kernelrelease_id>.img' or
1751 '/boot/initrd-<kernelrelease_id>'. Commodit 1751 '/boot/initrd-<kernelrelease_id>'. Commodity distributions rely on this file
1752 for booting, hence ensure to execute the ma 1752 for booting, hence ensure to execute the make target 'modules_install' first,
1753 as your distribution's initramfs generator 1753 as your distribution's initramfs generator otherwise will be unable to find
1754 the modules that go into the image. 1754 the modules that go into the image.
1755 1755
1756 * On some distributions installkernel will th 1756 * On some distributions installkernel will then add an entry for your kernel
1757 to your bootloader's configuration. 1757 to your bootloader's configuration.
1758 1758
1759 You have to take care of some or all of the t 1759 You have to take care of some or all of the tasks yourself, if your
1760 distribution lacks a installkernel script or 1760 distribution lacks a installkernel script or does only handle part of them.
1761 Consult the distribution's documentation for 1761 Consult the distribution's documentation for details. If in doubt, install the
1762 kernel manually:: 1762 kernel manually::
1763 1763
1764 sudo install -m 0600 $(make -s image_name) 1764 sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)
1765 sudo install -m 0600 System.map /boot/Syst 1765 sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)
1766 1766
1767 Now generate your initramfs using the tools y 1767 Now generate your initramfs using the tools your distribution provides for this
1768 process. Afterwards add your kernel to your b 1768 process. Afterwards add your kernel to your bootloader configuration and reboot.
1769 1769
1770 [:ref:`back to step-by-step guide <install_bi 1770 [:ref:`back to step-by-step guide <install_bissbs>`]
1771 1771
1772 .. _storagespace_bisref: 1772 .. _storagespace_bisref:
1773 1773
1774 Storage requirements per kernel 1774 Storage requirements per kernel
1775 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1775 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1776 1776
1777 *Check how much storage space the kernel, i 1777 *Check how much storage space the kernel, its modules, and other related files
1778 like the initramfs consume.* [:ref:`... <st 1778 like the initramfs consume.* [:ref:`... <storagespace_bissbs>`]
1779 1779
1780 The kernels built during a bisection consume 1780 The kernels built during a bisection consume quite a bit of space in /boot/ and
1781 /lib/modules/, especially if you enabled debu 1781 /lib/modules/, especially if you enabled debug symbols. That makes it easy to
1782 fill up volumes during a bisection -- and due 1782 fill up volumes during a bisection -- and due to that even kernels which used to
1783 work earlier might fail to boot. To prevent t 1783 work earlier might fail to boot. To prevent that you will need to know how much
1784 space each installed kernel typically require 1784 space each installed kernel typically requires.
1785 1785
1786 Note, most of the time the pattern '/boot/*$( 1786 Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in
1787 the guide will match all files needed to boot 1787 the guide will match all files needed to boot your kernel -- but neither the
1788 path nor the naming scheme are mandatory. On 1788 path nor the naming scheme are mandatory. On some distributions you thus will
1789 need to look in different places. 1789 need to look in different places.
1790 1790
1791 [:ref:`back to step-by-step guide <storagespa 1791 [:ref:`back to step-by-step guide <storagespace_bissbs>`]
1792 1792
1793 .. _tainted_bisref: 1793 .. _tainted_bisref:
1794 1794
1795 Check if your newly built kernel considers it 1795 Check if your newly built kernel considers itself 'tainted'
1796 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1796 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1797 1797
1798 *Check if the kernel marked itself as 'tain 1798 *Check if the kernel marked itself as 'tainted'.*
1799 [:ref:`... <tainted_bissbs>`] 1799 [:ref:`... <tainted_bissbs>`]
1800 1800
1801 Linux marks itself as tainted when something 1801 Linux marks itself as tainted when something happens that potentially leads to
1802 follow-up errors that look totally unrelated. 1802 follow-up errors that look totally unrelated. That is why developers might
1803 ignore or react scantly to reports from taint 1803 ignore or react scantly to reports from tainted kernels -- unless of course the
1804 kernel set the flag right when the reported b 1804 kernel set the flag right when the reported bug occurred.
1805 1805
1806 That's why you want check why a kernel is tai 1806 That's why you want check why a kernel is tainted as explained in
1807 Documentation/admin-guide/tainted-kernels.rst 1807 Documentation/admin-guide/tainted-kernels.rst; doing so is also in your own
1808 interest, as your testing might be flawed oth 1808 interest, as your testing might be flawed otherwise.
1809 1809
1810 [:ref:`back to step-by-step guide <tainted_bi 1810 [:ref:`back to step-by-step guide <tainted_bissbs>`]
1811 1811
1812 .. _recheckbroken_bisref: 1812 .. _recheckbroken_bisref:
1813 1813
1814 Check the kernel built from a recent mainline 1814 Check the kernel built from a recent mainline codebase
1815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1816 1816
1817 *Verify if your bug occurs with the newly b 1817 *Verify if your bug occurs with the newly built kernel.*
1818 [:ref:`... <recheckbroken_bissbs>`] 1818 [:ref:`... <recheckbroken_bissbs>`]
1819 1819
1820 There are a couple of reasons why your bug or 1820 There are a couple of reasons why your bug or regression might not show up with
1821 the kernel you built from the latest codebase 1821 the kernel you built from the latest codebase. These are the most frequent:
1822 1822
1823 * The bug was fixed meanwhile. 1823 * The bug was fixed meanwhile.
1824 1824
1825 * What you suspected to be a regression was c 1825 * What you suspected to be a regression was caused by a change in the build
1826 configuration the provider of your kernel c 1826 configuration the provider of your kernel carried out.
1827 1827
1828 * Your problem might be a race condition that 1828 * Your problem might be a race condition that does not show up with your kernel;
1829 the trimmed build configuration, a differen 1829 the trimmed build configuration, a different setting for debug symbols, the
1830 compiler used, and various other things can 1830 compiler used, and various other things can cause this.
1831 1831
1832 * In case you encountered the regression with 1832 * In case you encountered the regression with a stable/longterm kernel it might
1833 be a problem that is specific to that serie 1833 be a problem that is specific to that series; the next step in this guide will
1834 check this. 1834 check this.
1835 1835
1836 [:ref:`back to step-by-step guide <recheckbro 1836 [:ref:`back to step-by-step guide <recheckbroken_bissbs>`]
1837 1837
1838 .. _recheckstablebroken_bisref: 1838 .. _recheckstablebroken_bisref:
1839 1839
1840 Check the kernel built from the latest stable 1840 Check the kernel built from the latest stable/longterm codebase
1841 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1841 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1842 1842
1843 *Are you facing a regression within a stabl 1843 *Are you facing a regression within a stable/longterm release, but failed to
1844 reproduce it with the kernel you just built 1844 reproduce it with the kernel you just built using the latest mainline sources?
1845 Then check if the latest codebase for the p 1845 Then check if the latest codebase for the particular series might already fix
1846 the problem.* [:ref:`... <recheckstablebrok 1846 the problem.* [:ref:`... <recheckstablebroken_bissbs>`]
1847 1847
1848 If this kernel does not show the regression e 1848 If this kernel does not show the regression either, there most likely is no need
1849 for a bisection. 1849 for a bisection.
1850 1850
1851 [:ref:`back to step-by-step guide <rechecksta 1851 [:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`]
1852 1852
1853 .. _introworkingcheck_bisref: 1853 .. _introworkingcheck_bisref:
1854 1854
1855 Ensure the 'good' version is really working w 1855 Ensure the 'good' version is really working well
1856 --------------------------------------------- 1856 ------------------------------------------------
1857 1857
1858 *Check if the kernels you build work fine.* 1858 *Check if the kernels you build work fine.*
1859 [:ref:`... <introworkingcheck_bissbs>`] 1859 [:ref:`... <introworkingcheck_bissbs>`]
1860 1860
1861 This section will reestablish a known working 1861 This section will reestablish a known working base. Skipping it might be
1862 appealing, but is usually a bad idea, as it d 1862 appealing, but is usually a bad idea, as it does something important:
1863 1863
1864 It will ensure the .config file you prepared 1864 It will ensure the .config file you prepared earlier actually works as expected.
1865 That is in your own interest, as trimming the 1865 That is in your own interest, as trimming the configuration is not foolproof --
1866 and you might be building and testing ten or 1866 and you might be building and testing ten or more kernels for nothing before
1867 starting to suspect something might be wrong 1867 starting to suspect something might be wrong with the build configuration.
1868 1868
1869 That alone is reason enough to spend the time 1869 That alone is reason enough to spend the time on this, but not the only reason.
1870 1870
1871 Many readers of this guide normally run kerne 1871 Many readers of this guide normally run kernels that are patched, use add-on
1872 modules, or both. Those kernels thus are not 1872 modules, or both. Those kernels thus are not considered 'vanilla' -- therefore
1873 it's possible that the thing that regressed m 1873 it's possible that the thing that regressed might never have worked in vanilla
1874 builds of the 'good' version in the first pla 1874 builds of the 'good' version in the first place.
1875 1875
1876 There is a third reason for those that notice 1876 There is a third reason for those that noticed a regression between
1877 stable/longterm kernels of different series ( 1877 stable/longterm kernels of different series (e.g. 6.0.13..6.1.5): it will
1878 ensure the kernel version you assumed to be ' 1878 ensure the kernel version you assumed to be 'good' earlier in the process (e.g.
1879 6.0) actually is working. 1879 6.0) actually is working.
1880 1880
1881 [:ref:`back to step-by-step guide <introworki 1881 [:ref:`back to step-by-step guide <introworkingcheck_bissbs>`]
1882 1882
1883 .. _recheckworking_bisref: 1883 .. _recheckworking_bisref:
1884 1884
1885 Build your own version of the 'good' kernel 1885 Build your own version of the 'good' kernel
1886 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1886 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1887 1887
1888 *Build your own variant of the working kern 1888 *Build your own variant of the working kernel and check if the feature that
1889 regressed works as expected with it.* [:ref 1889 regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`]
1890 1890
1891 In case the feature that broke with newer ker 1891 In case the feature that broke with newer kernels does not work with your first
1892 self-built kernel, find and resolve the cause 1892 self-built kernel, find and resolve the cause before moving on. There are a
1893 multitude of reasons why this might happen. S 1893 multitude of reasons why this might happen. Some ideas where to look:
1894 1894
1895 * Check the taint status and the output of `` 1895 * Check the taint status and the output of ``dmesg``, maybe something unrelated
1896 went wrong. 1896 went wrong.
1897 1897
1898 * Maybe localmodconfig did something odd and 1898 * Maybe localmodconfig did something odd and disabled the module required to
1899 test the feature? Then you might want to re 1899 test the feature? Then you might want to recreate a .config file based on the
1900 one from the last working kernel and skip t 1900 one from the last working kernel and skip trimming it down; manually disabling
1901 some features in the .config might work as 1901 some features in the .config might work as well to reduce the build time.
1902 1902
1903 * Maybe it's not a kernel regression and some 1903 * Maybe it's not a kernel regression and something that is caused by some fluke,
1904 a broken initramfs (also known as initrd), 1904 a broken initramfs (also known as initrd), new firmware files, or an updated
1905 userland software? 1905 userland software?
1906 1906
1907 * Maybe it was a feature added to your distri 1907 * Maybe it was a feature added to your distributor's kernel which vanilla Linux
1908 at that point never supported? 1908 at that point never supported?
1909 1909
1910 Note, if you found and fixed problems with th 1910 Note, if you found and fixed problems with the .config file, you want to use it
1911 to build another kernel from the latest codeb 1911 to build another kernel from the latest codebase, as your earlier tests with
1912 mainline and the latest version from an affec 1912 mainline and the latest version from an affected stable/longterm series were
1913 most likely flawed. 1913 most likely flawed.
1914 1914
1915 [:ref:`back to step-by-step guide <recheckwor 1915 [:ref:`back to step-by-step guide <recheckworking_bissbs>`]
1916 1916
1917 Perform a bisection and validate the result 1917 Perform a bisection and validate the result
1918 ------------------------------------------- 1918 -------------------------------------------
1919 1919
1920 *With all the preparations and precaution b 1920 *With all the preparations and precaution builds taken care of, you are now
1921 ready to begin the bisection.* [:ref:`... < 1921 ready to begin the bisection.* [:ref:`... <introbisect_bissbs>`]
1922 1922
1923 The steps in this segment perform and validat 1923 The steps in this segment perform and validate the bisection.
1924 1924
1925 [:ref:`back to step-by-step guide <introbisec 1925 [:ref:`back to step-by-step guide <introbisect_bissbs>`].
1926 1926
1927 .. _bisectstart_bisref: 1927 .. _bisectstart_bisref:
1928 1928
1929 Start the bisection 1929 Start the bisection
1930 ~~~~~~~~~~~~~~~~~~~ 1930 ~~~~~~~~~~~~~~~~~~~
1931 1931
1932 *Start the bisection and tell Git about the 1932 *Start the bisection and tell Git about the versions earlier established as
1933 'good' and 'bad'.* [:ref:`... <bisectstart_ 1933 'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`]
1934 1934
1935 This will start the bisection process; the la 1935 This will start the bisection process; the last of the commands will make Git
1936 check out a commit round about half-way betwe 1936 check out a commit round about half-way between the 'good' and the 'bad' changes
1937 for you to test. 1937 for you to test.
1938 1938
1939 [:ref:`back to step-by-step guide <bisectstar 1939 [:ref:`back to step-by-step guide <bisectstart_bissbs>`]
1940 1940
1941 .. _bisectbuild_bisref: 1941 .. _bisectbuild_bisref:
1942 1942
1943 Build a kernel from the bisection point 1943 Build a kernel from the bisection point
1944 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1944 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1945 1945
1946 *Build, install, and boot a kernel from the 1946 *Build, install, and boot a kernel from the code Git checked out using the
1947 same commands you used earlier.* [:ref:`... 1947 same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`]
1948 1948
1949 There are two things worth of note here: 1949 There are two things worth of note here:
1950 1950
1951 * Occasionally building the kernel will fail 1951 * Occasionally building the kernel will fail or it might not boot due some
1952 problem in the code at the bisection point. 1952 problem in the code at the bisection point. In that case run this command::
1953 1953
1954 git bisect skip 1954 git bisect skip
1955 1955
1956 Git will then check out another commit near 1956 Git will then check out another commit nearby which with a bit of luck should
1957 work better. Afterwards restart executing t 1957 work better. Afterwards restart executing this step.
1958 1958
1959 * Those slightly odd looking version identifi 1959 * Those slightly odd looking version identifiers can happen during bisections,
1960 because the Linux kernel subsystems prepare 1960 because the Linux kernel subsystems prepare their changes for a new mainline
1961 release (say 6.2) before its predecessor (e 1961 release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus
1962 base them on a somewhat earlier point like 1962 base them on a somewhat earlier point like 6.1-rc1 or even 6.0 -- and then
1963 get merged for 6.2 without rebasing nor squ 1963 get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This
1964 leads to those slightly odd looking version 1964 leads to those slightly odd looking version identifiers coming up during
1965 bisections. 1965 bisections.
1966 1966
1967 [:ref:`back to step-by-step guide <bisectbuil 1967 [:ref:`back to step-by-step guide <bisectbuild_bissbs>`]
1968 1968
1969 .. _bisecttest_bisref: 1969 .. _bisecttest_bisref:
1970 1970
1971 Bisection checkpoint 1971 Bisection checkpoint
1972 ~~~~~~~~~~~~~~~~~~~~ 1972 ~~~~~~~~~~~~~~~~~~~~
1973 1973
1974 *Check if the feature that regressed works 1974 *Check if the feature that regressed works in the kernel you just built.*
1975 [:ref:`... <bisecttest_bissbs>`] 1975 [:ref:`... <bisecttest_bissbs>`]
1976 1976
1977 Ensure what you tell Git is accurate: getting 1977 Ensure what you tell Git is accurate: getting it wrong just one time will bring
1978 the rest of the bisection totally off course, 1978 the rest of the bisection totally off course, hence all testing after that point
1979 will be for nothing. 1979 will be for nothing.
1980 1980
1981 [:ref:`back to step-by-step guide <bisecttest 1981 [:ref:`back to step-by-step guide <bisecttest_bissbs>`]
1982 1982
1983 .. _bisectlog_bisref: 1983 .. _bisectlog_bisref:
1984 1984
1985 Put the bisection log away 1985 Put the bisection log away
1986 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1986 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1987 1987
1988 *Store Git's bisection log and the current 1988 *Store Git's bisection log and the current .config file in a safe place.*
1989 [:ref:`... <bisectlog_bissbs>`] 1989 [:ref:`... <bisectlog_bissbs>`]
1990 1990
1991 As indicated above: declaring just one kernel 1991 As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will
1992 render the end result of a bisection useless. 1992 render the end result of a bisection useless. In that case you'd normally have
1993 to restart the bisection from scratch. The lo 1993 to restart the bisection from scratch. The log can prevent that, as it might
1994 allow someone to point out where a bisection 1994 allow someone to point out where a bisection likely went sideways -- and then
1995 instead of testing ten or more kernels you mi 1995 instead of testing ten or more kernels you might only have to build a few to
1996 resolve things. 1996 resolve things.
1997 1997
1998 The .config file is put aside, as there is a 1998 The .config file is put aside, as there is a decent chance that developers might
1999 ask for it after you report the regression. 1999 ask for it after you report the regression.
2000 2000
2001 [:ref:`back to step-by-step guide <bisectlog_ 2001 [:ref:`back to step-by-step guide <bisectlog_bissbs>`]
2002 2002
2003 .. _revert_bisref: 2003 .. _revert_bisref:
2004 2004
2005 Try reverting the culprit 2005 Try reverting the culprit
2006 ~~~~~~~~~~~~~~~~~~~~~~~~~ 2006 ~~~~~~~~~~~~~~~~~~~~~~~~~
2007 2007
2008 *Try reverting the culprit on top of the la 2008 *Try reverting the culprit on top of the latest codebase to see if this fixes
2009 your regression.* [:ref:`... <revert_bissbs 2009 your regression.* [:ref:`... <revert_bissbs>`]
2010 2010
2011 This is an optional step, but whenever possib 2011 This is an optional step, but whenever possible one you should try: there is a
2012 decent chance that developers will ask you to 2012 decent chance that developers will ask you to perform this step when you bring
2013 the bisection result up. So give it a try, yo 2013 the bisection result up. So give it a try, you are in the flow already, building
2014 one more kernel shouldn't be a big deal at th 2014 one more kernel shouldn't be a big deal at this point.
2015 2015
2016 The step-by-step guide covers everything rele 2016 The step-by-step guide covers everything relevant already except one slightly
2017 rare thing: did you bisected a regression tha 2017 rare thing: did you bisected a regression that also happened with mainline using
2018 a stable/longterm series, but Git failed to r 2018 a stable/longterm series, but Git failed to revert the commit in mainline? Then
2019 try to revert the culprit in the affected sta 2019 try to revert the culprit in the affected stable/longterm series -- and if that
2020 succeeds, test that kernel version instead. 2020 succeeds, test that kernel version instead.
2021 2021
2022 [:ref:`back to step-by-step guide <revert_bis 2022 [:ref:`back to step-by-step guide <revert_bissbs>`]
2023 2023
2024 Cleanup steps during and after following this 2024 Cleanup steps during and after following this guide
2025 --------------------------------------------- 2025 ---------------------------------------------------
2026 2026
2027 *During and after following this guide you 2027 *During and after following this guide you might want or need to remove some
2028 of the kernels you installed.* [:ref:`... < 2028 of the kernels you installed.* [:ref:`... <introclosure_bissbs>`]
2029 2029
2030 The steps in this section describe clean-up p 2030 The steps in this section describe clean-up procedures.
2031 2031
2032 [:ref:`back to step-by-step guide <introclosu 2032 [:ref:`back to step-by-step guide <introclosure_bissbs>`].
2033 2033
2034 .. _makeroom_bisref: 2034 .. _makeroom_bisref:
2035 2035
2036 Cleaning up during the bisection 2036 Cleaning up during the bisection
2037 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2037 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2038 2038
2039 *To remove one of the kernels you installed 2039 *To remove one of the kernels you installed, look up its 'kernelrelease'
2040 identifier.* [:ref:`... <makeroom_bissbs>`] 2040 identifier.* [:ref:`... <makeroom_bissbs>`]
2041 2041
2042 The kernels you install during this process a 2042 The kernels you install during this process are easy to remove later, as its
2043 parts are only stored in two places and clear 2043 parts are only stored in two places and clearly identifiable. You thus do not
2044 need to worry to mess up your machine when yo 2044 need to worry to mess up your machine when you install a kernel manually (and
2045 thus bypass your distribution's packaging sys 2045 thus bypass your distribution's packaging system): all parts of your kernels are
2046 relatively easy to remove later. 2046 relatively easy to remove later.
2047 2047
2048 One of the two places is a directory in /lib/ 2048 One of the two places is a directory in /lib/modules/, which holds the modules
2049 for each installed kernel. This directory is 2049 for each installed kernel. This directory is named after the kernel's release
2050 identifier; hence, to remove all modules for 2050 identifier; hence, to remove all modules for one of the kernels you built,
2051 simply remove its modules directory in /lib/m 2051 simply remove its modules directory in /lib/modules/.
2052 2052
2053 The other place is /boot/, where typically tw 2053 The other place is /boot/, where typically two up to five files will be placed
2054 during installation of a kernel. All of them 2054 during installation of a kernel. All of them usually contain the release name in
2055 their file name, but how many files and their 2055 their file name, but how many files and their exact names depend somewhat on
2056 your distribution's installkernel executable 2056 your distribution's installkernel executable and its initramfs generator. On
2057 some distributions the ``kernel-install remov 2057 some distributions the ``kernel-install remove...`` command mentioned in the
2058 step-by-step guide will delete all of these f 2058 step-by-step guide will delete all of these files for you while also removing
2059 the menu entry for the kernel from your bootl 2059 the menu entry for the kernel from your bootloader configuration. On others you
2060 have to take care of these two tasks yourself 2060 have to take care of these two tasks yourself. The following command should
2061 interactively remove the three main files of 2061 interactively remove the three main files of a kernel with the release name
2062 '6.0-rc1-local-gcafec0cacaca0':: 2062 '6.0-rc1-local-gcafec0cacaca0'::
2063 2063
2064 rm -i /boot/{System.map,vmlinuz,initr}-6.0- 2064 rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0
2065 2065
2066 Afterwards check for other files in /boot/ th 2066 Afterwards check for other files in /boot/ that have
2067 '6.0-rc1-local-gcafec0cacaca0' in their name 2067 '6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well.
2068 Now remove the boot entry for the kernel from 2068 Now remove the boot entry for the kernel from your bootloader's configuration;
2069 the steps to do that vary quite a bit between 2069 the steps to do that vary quite a bit between Linux distributions.
2070 2070
2071 Note, be careful with wildcards like '*' when 2071 Note, be careful with wildcards like '*' when deleting files or directories
2072 for kernels manually: you might accidentally 2072 for kernels manually: you might accidentally remove files of a 6.0.13 kernel
2073 when all you want is to remove 6.0 or 6.0.1. 2073 when all you want is to remove 6.0 or 6.0.1.
2074 2074
2075 [:ref:`back to step-by-step guide <makeroom_b 2075 [:ref:`back to step-by-step guide <makeroom_bissbs>`]
2076 2076
2077 Cleaning up after the bisection 2077 Cleaning up after the bisection
2078 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2078 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2079 2079
2080 .. _finishingtouch_bisref: 2080 .. _finishingtouch_bisref:
2081 2081
2082 *Once you have finished the bisection, do n 2082 *Once you have finished the bisection, do not immediately remove anything
2083 you set up, as you might need a few things 2083 you set up, as you might need a few things again.*
2084 [:ref:`... <finishingtouch_bissbs>`] 2084 [:ref:`... <finishingtouch_bissbs>`]
2085 2085
2086 When you are really short of storage space re 2086 When you are really short of storage space removing the kernels as described in
2087 the step-by-step guide might not free as much 2087 the step-by-step guide might not free as much space as you would like. In that
2088 case consider running ``rm -rf ~/linux/*`` as 2088 case consider running ``rm -rf ~/linux/*`` as well now. This will remove the
2089 build artifacts and the Linux sources, but wi 2089 build artifacts and the Linux sources, but will leave the Git repository
2090 (~/linux/.git/) behind -- a simple ``git rese 2090 (~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the
2091 sources back. 2091 sources back.
2092 2092
2093 Removing the repository as well would likely 2093 Removing the repository as well would likely be unwise at this point: there
2094 is a decent chance developers will ask you to 2094 is a decent chance developers will ask you to build another kernel to
2095 perform additional tests -- like testing a de 2095 perform additional tests -- like testing a debug patch or a proposed fix.
2096 Details on how to perform those can be found 2096 Details on how to perform those can be found in the section :ref:`Optional
2097 tasks: test reverts, patches, or later versio 2097 tasks: test reverts, patches, or later versions <introoptional_bissbs>`.
2098 2098
2099 Additional tests are also the reason why you 2099 Additional tests are also the reason why you want to keep the
2100 ~/kernel-config-working file around for a few 2100 ~/kernel-config-working file around for a few weeks.
2101 2101
2102 [:ref:`back to step-by-step guide <finishingt 2102 [:ref:`back to step-by-step guide <finishingtouch_bissbs>`]
2103 2103
2104 .. _introoptional_bisref: 2104 .. _introoptional_bisref:
2105 2105
2106 Test reverts, patches, or later versions 2106 Test reverts, patches, or later versions
2107 ---------------------------------------- 2107 ----------------------------------------
2108 2108
2109 *While or after reporting a bug, you might 2109 *While or after reporting a bug, you might want or potentially will be asked
2110 to test reverts, patches, proposed fixes, o 2110 to test reverts, patches, proposed fixes, or other versions.*
2111 [:ref:`... <introoptional_bissbs>`] 2111 [:ref:`... <introoptional_bissbs>`]
2112 2112
2113 All the commands used in this section should 2113 All the commands used in this section should be pretty straight forward, so
2114 there is not much to add except one thing: wh 2114 there is not much to add except one thing: when setting a kernel tag as
2115 instructed, ensure it is not much longer than 2115 instructed, ensure it is not much longer than the one used in the example, as
2116 problems will arise if the kernelrelease iden 2116 problems will arise if the kernelrelease identifier exceeds 63 characters.
2117 2117
2118 [:ref:`back to step-by-step guide <introoptio 2118 [:ref:`back to step-by-step guide <introoptional_bissbs>`].
2119 2119
2120 2120
2121 Additional information 2121 Additional information
2122 ====================== 2122 ======================
2123 2123
2124 .. _buildhost_bis: 2124 .. _buildhost_bis:
2125 2125
2126 Build kernels on a different machine 2126 Build kernels on a different machine
2127 ------------------------------------ 2127 ------------------------------------
2128 2128
2129 To compile kernels on another system, slightl 2129 To compile kernels on another system, slightly alter the step-by-step guide's
2130 instructions: 2130 instructions:
2131 2131
2132 * Start following the guide on the machine wh 2132 * Start following the guide on the machine where you want to install and test
2133 the kernels later. 2133 the kernels later.
2134 2134
2135 * After executing ':ref:`Boot into the workin 2135 * After executing ':ref:`Boot into the working kernel and briefly use the
2136 apparently broken feature <bootworking_biss 2136 apparently broken feature <bootworking_bissbs>`', save the list of loaded
2137 modules to a file using ``lsmod > ~/test-ma 2137 modules to a file using ``lsmod > ~/test-machine-lsmod``. Then locate the
2138 build configuration for the running kernel 2138 build configuration for the running kernel (see ':ref:`Start defining the
2139 build configuration for your kernel <oldcon 2139 build configuration for your kernel <oldconfig_bisref>`' for hints on where
2140 to find it) and store it as '~/test-machine 2140 to find it) and store it as '~/test-machine-config-working'. Transfer both
2141 files to the home directory of your build h 2141 files to the home directory of your build host.
2142 2142
2143 * Continue the guide on the build host (e.g. 2143 * Continue the guide on the build host (e.g. with ':ref:`Ensure to have enough
2144 free space for building [...] <diskspace_bi 2144 free space for building [...] <diskspace_bissbs>`').
2145 2145
2146 * When you reach ':ref:`Start preparing a ker 2146 * When you reach ':ref:`Start preparing a kernel build configuration[...]
2147 <oldconfig_bissbs>`': before running ``make 2147 <oldconfig_bissbs>`': before running ``make olddefconfig`` for the first time,
2148 execute the following command to base your 2148 execute the following command to base your configuration on the one from the
2149 test machine's 'working' kernel:: 2149 test machine's 'working' kernel::
2150 2150
2151 cp ~/test-machine-config-working ~/linux/ 2151 cp ~/test-machine-config-working ~/linux/.config
2152 2152
2153 * During the next step to ':ref:`disable any 2153 * During the next step to ':ref:`disable any apparently superfluous kernel
2154 modules <localmodconfig_bissbs>`' use the f 2154 modules <localmodconfig_bissbs>`' use the following command instead::
2155 2155
2156 yes '' | make localmodconfig LSMOD=~/lsmo 2156 yes '' | make localmodconfig LSMOD=~/lsmod_foo-machine localmodconfig
2157 2157
2158 * Continue the guide, but ignore the instruct 2158 * Continue the guide, but ignore the instructions outlining how to compile,
2159 install, and reboot into a kernel every tim 2159 install, and reboot into a kernel every time they come up. Instead build
2160 like this:: 2160 like this::
2161 2161
2162 cp ~/kernel-config-working .config 2162 cp ~/kernel-config-working .config
2163 make olddefconfig && 2163 make olddefconfig &&
2164 make -j $(nproc --all) targz-pkg 2164 make -j $(nproc --all) targz-pkg
2165 2165
2166 This will generate a gzipped tar file whose 2166 This will generate a gzipped tar file whose name is printed in the last
2167 line shown; for example, a kernel with the 2167 line shown; for example, a kernel with the kernelrelease identifier
2168 '6.0.0-rc1-local-g928a87efa423' built for x 2168 '6.0.0-rc1-local-g928a87efa423' built for x86 machines usually will
2169 be stored as '~/linux/linux-6.0.0-rc1-local 2169 be stored as '~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz'.
2170 2170
2171 Copy that file to your test machine's home 2171 Copy that file to your test machine's home directory.
2172 2172
2173 * Switch to the test machine to check if you 2173 * Switch to the test machine to check if you have enough space to hold another
2174 kernel. Then extract the file you transferr 2174 kernel. Then extract the file you transferred::
2175 2175
2176 sudo tar -xvzf ~/linux-6.0.0-rc1-local-g9 2176 sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C /
2177 2177
2178 Afterwards :ref:`generate the initramfs and 2178 Afterwards :ref:`generate the initramfs and add the kernel to your boot
2179 loader's configuration <install_bisref>`; o 2179 loader's configuration <install_bisref>`; on some distributions the following
2180 command will take care of both these tasks: 2180 command will take care of both these tasks::
2181 2181
2182 sudo /sbin/installkernel 6.0.0-rc1-local- 2182 sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6.0.0-rc1-local-g928a87efa423
2183 2183
2184 Now reboot and ensure you started the inten 2184 Now reboot and ensure you started the intended kernel.
2185 2185
2186 This approach even works when building for an 2186 This approach even works when building for another architecture: just install
2187 cross-compilers and add the appropriate param 2187 cross-compilers and add the appropriate parameters to every invocation of make
2188 (e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64 2188 (e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]``).
2189 2189
2190 Additional reading material 2190 Additional reading material
2191 --------------------------- 2191 ---------------------------
2192 2192
2193 * The `man page for 'git bisect' <https://git 2193 * The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
2194 `fighting regressions with 'git bisect' <ht 2194 `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
2195 in the Git documentation. 2195 in the Git documentation.
2196 * `Working with git bisect <https://nathancha 2196 * `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
2197 from kernel developer Nathan Chancellor. 2197 from kernel developer Nathan Chancellor.
2198 * `Using Git bisect to figure out when broken 2198 * `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
2199 * `Fully automated bisecting with 'git bisect 2199 * `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.
2200 2200
2201 .. 2201 ..
2202 end-of-content 2202 end-of-content
2203 .. 2203 ..
2204 This document is maintained by Thorsten Le< 2204 This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
2205 you spot a typo or small mistake, feel fre 2205 you spot a typo or small mistake, feel free to let him know directly and
2206 he'll fix it. You are free to do the same 2206 he'll fix it. You are free to do the same in a mostly informal way if you
2207 want to contribute changes to the text -- 2207 want to contribute changes to the text -- but for copyright reasons please CC
2208 linux-doc@vger.kernel.org and 'sign-off' y 2208 linux-doc@vger.kernel.org and 'sign-off' your contribution as
2209 Documentation/process/submitting-patches.r 2209 Documentation/process/submitting-patches.rst explains in the section 'Sign
2210 your work - the Developer's Certificate of 2210 your work - the Developer's Certificate of Origin'.
2211 .. 2211 ..
2212 This text is available under GPL-2.0+ or C 2212 This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
2213 of the file. If you want to distribute thi 2213 of the file. If you want to distribute this text under CC-BY-4.0 only,
2214 please use 'The Linux kernel development c 2214 please use 'The Linux kernel development community' for author attribution
2215 and link this as source: 2215 and link this as source:
2216 https://git.kernel.org/pub/scm/linux/kerne 2216 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
2217 2217
2218 .. 2218 ..
2219 Note: Only the content of this RST file as 2219 Note: Only the content of this RST file as found in the Linux kernel sources
2220 is available under CC-BY-4.0, as versions 2220 is available under CC-BY-4.0, as versions of this text that were processed
2221 (for example by the kernel's build system) 2221 (for example by the kernel's build system) might contain content taken from
2222 files which use a more restrictive license 2222 files which use a more restrictive license.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.