1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY
2 .. [see the bottom of this file for redistribu
3
4 =========================================
5 How to verify bugs and bisect regressions
6 =========================================
7
8 This document describes how to check if some L
9 currently supported by developers -- to then e
10 causing the issue, if it is a regression (e.g.
11 versions).
12
13 The text aims at people running kernels from m
14 commodity hardware who want to report a kernel
15 developers. Despite this intent, the instructi
16 who are already familiar with building their o
17 mistakes occasionally made even by experienced
18
19 ..
20 Note: if you see this note, you are reading
21 might want to switch to a rendered version:
22 read and navigate this document -- especial
23 up in the reference section, then jump back
24 ..
25 Find the latest rendered version of this te
26 https://docs.kernel.org/admin-guide/verify-
27
28 The essence of the process (aka 'TL;DR')
29 ========================================
30
31 *[If you are new to building or bisecting Linu
32 over to the* ':ref:`step-by-step guide <introg
33 the same commands as this section while descri
34 steps are nevertheless easy to follow and toge
35 in a reference section mention many alternativ
36 aspects, all of which might be essential in yo
37
38 **In case you want to check if a bug is presen
39 developers**, execute just the *preparations*
40 consider the newest Linux kernel you regularly
41 In the following example that's assumed to be
42 will be used to prepare the .config file.
43
44 **In case you face a regression**, follow the
45 *segment 2*. Then you can submit a preliminary
46 *segment 3*, which describes how to perform a
47 full-fledged regression report. In the followi
48 the 'working' kernel and 6.1.5 to be the first
49 will be considered the 'good' release and used
50
51 * **Preparations**: set up everything to build
52
53 # * Remove any software that depends on ex
54 # or builds any automatically during boo
55 # * Ensure Secure Boot permits booting sel
56 # * If you are not already running the 'wo
57 # * Install compilers and everything else
58 # * Ensure to have 15 Gigabyte free space
59 git clone -o mainline --no-checkout \
60 https://git.kernel.org/pub/scm/linux/ker
61 cd ~/linux/
62 git remote add -t master stable \
63 https://git.kernel.org/pub/scm/linux/ker
64 git switch --detach v6.0
65 # * Hint: if you used an existing clone, e
66 make olddefconfig
67 # * Ensure the former command picked the .
68 # * Connect external hardware (USB keys, t
69 # VPNs, mount network shares, and briefl
70 yes '' | make localmodconfig
71 ./scripts/config --set-str CONFIG_LOCALVER
72 ./scripts/config -e CONFIG_LOCALVERSION_AU
73 # * Note, when short on storage space, che
74 ./scripts/config -d DEBUG_INFO_NONE -e KAL
75 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCH
76 # * Hint: at this point you might want to
77 # you'll have to, if you are running Deb
78 make olddefconfig
79 cp .config ~/kernel-config-working
80
81 * **Segment 1**: build a kernel from the lates
82
83 This among others checks if the problem was
84 later need to be told about the problem; in
85 out a .config change as root of the problem.
86
87 a) Checking out latest mainline code::
88
89 cd ~/linux/
90 git switch --discard-changes --detach m
91
92 b) Build, install, and boot a kernel::
93
94 cp ~/kernel-config-working .config
95 make olddefconfig
96 make -j $(nproc --all)
97 # * Make sure there is enough disk spac
98 df -h /boot/ /lib/modules/
99 # * Note: on Arch Linux, its derivative
100 # the following commands will do noth
101 # job. See the step-by-step guide for
102 sudo make modules_install
103 command -v installkernel && sudo make i
104 # * Check how much space your self-buil
105 # enables you to make better estimate
106 du -ch /boot/*$(make -s kernelrelease)*
107 du -sh /lib/modules/$(make -s kernelrel
108 # * Hint: the output of the following c
109 # right kernel from the boot menu:
110 make -s kernelrelease | tee -a ~/kernel
111 reboot
112 # * Once booted, ensure you are running
113 # checking if the output of the next
114 tail -n 1 ~/kernels-built
115 uname -r
116 cat /proc/sys/kernel/tainted
117
118 c) Check if the problem occurs with this ker
119
120 * **Segment 2**: ensure the 'good' kernel is a
121
122 This among others verifies the trimmed .conf
123 bisecting with it otherwise would be a waste
124
125 a) Start by checking out the sources of the
126
127 cd ~/linux/
128 git switch --discard-changes --detach v
129
130 b) Build, install, and boot a kernel as desc
131 section b* -- just feel free to skip the
132 estimate already.
133
134 c) Ensure the feature that regressed with th
135 with this one.
136
137 * **Segment 3**: perform and validate the bise
138
139 a) Retrieve the sources for your 'bad' versi
140
141 git remote set-branches --add stable li
142 git fetch stable
143
144 b) Initialize the bisection::
145
146 cd ~/linux/
147 git bisect start
148 git bisect good v6.0
149 git bisect bad v6.1.5
150
151 c) Build, install, and boot a kernel as desc
152 section b*.
153
154 In case building or booting the kernel fa
155 ``git bisect skip``. In all other outcome
156 works with the newly built kernel. If it
157 ``git bisect good``; if it does not, run
158
159 All three commands will make Git check ou
160 this step (e.g. build, install, boot, and
161 the outcome). Do so again and again until
162 things. If you run short of disk space du
163 section 'Complementary tasks: cleanup dur
164 below.
165
166 d) Once your finished the bisection, put a f
167
168 cd ~/linux/
169 git bisect log > ~/bisect-log
170 cp .config ~/bisection-config-culprit
171 git bisect reset
172
173 e) Try to verify the bisection result::
174
175 git switch --discard-changes --detach m
176 git revert --no-edit cafec0cacaca0
177 cp ~/kernel-config-working .config
178 ./scripts/config --set-str CONFIG_LOCAL
179
180 This is optional, as some commits are impo
181 second command worked flawlessly, build, i
182 kernel; just this time skip the first comm
183 over, as that already has been taken care
184
185 * **Complementary tasks**: cleanup during and
186
187 a) To avoid running out of disk space during
188 remove some kernels you built earlier. Yo
189 you built during segment 1 and 2 around f
190 likely no longer need kernels tested duri
191 (Segment 3 c). You can list them in build
192
193 ls -ltr /lib/modules/*-local*
194
195 To then for example erase a kernel that id
196 '6.0-rc1-local-gcafec0cacaca0', use this::
197
198 sudo rm -rf /lib/modules/6.0-rc1-local-
199 sudo kernel-install -v remove 6.0-rc1-l
200 # * Note, on some distributions kernel-
201 # or does only part of the job.
202
203 b) If you performed a bisection and successf
204 free to remove all kernels built during t
205 the kernels you built earlier and later y
206 a week or two.
207
208 * **Optional task**: test a debug patch or a p
209
210 git fetch mainline
211 git switch --discard-changes --detach main
212 git apply /tmp/foobars-proposed-fix-v1.pat
213 cp ~/kernel-config-working .config
214 ./scripts/config --set-str CONFIG_LOCALVER
215
216 Build, install, and boot a kernel as describ
217 but this time omit the first command copying
218 as that has been taken care of already.
219
220 .. _introguide_bissbs:
221
222 Step-by-step guide on how to verify bugs and b
223 ==============================================
224
225 This guide describes how to set up your own Li
226 or regressions you intend to report. How far y
227 depends on your issue:
228
229 Execute all steps till the end of *segment 1*
230 is present in code supported by Linux kernel d
231 set to report the bug -- unless it did not hap
232 as then your want to at least continue with *s
233 qualifies as regression** which receive priori
234 outcome you then are ready to report a bug or
235 report; instead of the latter your could also
236 *segment 3* to **perform a bisection** for a f
237 developers are obliged to act upon.
238
239 :ref:`Preparations: set up everything to buil
240
241 :ref:`Segment 1: try to reproduce the problem
242
243 :ref:`Segment 2: check if the kernels you bui
244
245 :ref:`Segment 3: perform a bisection and vali
246
247 :ref:`Complementary tasks: cleanup during and
248
249 :ref:`Optional tasks: test reverts, patches,
250
251 The steps in each segment illustrate the impor
252 a comprehensive reference section holds additi
253 steps. The reference section sometimes also ou
254 pitfalls, as well as problems that might occur
255 to get things rolling again.
256
257 For further details on how to report Linux ker
258 out Documentation/admin-guide/reporting-issues
259 with this document. It among others explains w
260 the latest 'mainline' kernel (e.g. versions li
261 even if you face a problem with a kernel from
262 (say 6.0.13).
263
264 For users facing a regression that document al
265 preliminary report after segment 2 might be wi
266 culprit might be known already. For further de
267 as a regression check out Documentation/admin-
268
269 If you run into any problems while following t
270 improve it, :ref:`please let the kernel develo
271
272 .. _introprep_bissbs:
273
274 Preparations: set up everything to build your
275 ----------------------------------------------
276
277 The following steps lay the groundwork for all
278
279 Note: the instructions assume you are building
280 machine; if you want to compile the kernel on
281 :ref:`Build kernels on a different machine <bu
282
283 .. _backup_bissbs:
284
285 * Create a fresh backup and put system repair
286 to be prepared for the unlikely case of some
287
288 [:ref:`details <backup_bisref>`]
289
290 .. _vanilla_bissbs:
291
292 * Remove all software that depends on external
293 builds them automatically. That includes but
294 VirtualBox, and Nvidia's graphics drivers (i
295
296 [:ref:`details <vanilla_bisref>`]
297
298 .. _secureboot_bissbs:
299
300 * On platforms with 'Secure Boot' or similar s
301 ensure the system will permit your self-comp
302 quickest and easiest way to achieve this on
303 disable such techniques in the BIOS setup ut
304 their restrictions through a process initiat
305 ``mokutil --disable-validation``.
306
307 [:ref:`details <secureboot_bisref>`]
308
309 .. _rangecheck_bissbs:
310
311 * Determine the kernel versions considered 'go
312 guide:
313
314 * Do you follow this guide to verify if a bu
315 primary developers care for? Then consider
316 you regularly use currently as 'good' (e.g
317
318 * Do you face a regression, e.g. something b
319 switching to a newer kernel version? In th
320 range during which the problem appeared:
321
322 * Something regressed when updating from a
323 (say 6.0.13) to a newer mainline series
324 stable/longterm version based on one (sa
325 mainline release your working kernel is
326 version (e.g. 6.0) and the first version
327 (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at
328 that 6.0 is fine; this hypothesis will b
329
330 * Something regressed when switching from
331 a later one (like 6.1-rc1) or a stable/l
332 (say 6.1.5)? Then regard the last workin
333 the first broken (e.g. 6.1-rc1 or 6.1.5)
334
335 * Something regressed when updating within
336 from 6.0.13 to 6.0.15)? Then consider th
337 (e.g. 6.0.13 and 6.0.15), as you need to
338
339 *Note, do not confuse 'good' version with 'w
340 throughout this guide will refer to the last
341 fine.*
342
343 [:ref:`details <rangecheck_bisref>`]
344
345 .. _bootworking_bissbs:
346
347 * Boot into the 'working' kernel and briefly u
348
349 [:ref:`details <bootworking_bisref>`]
350
351 .. _diskspace_bissbs:
352
353 * Ensure to have enough free space for buildin
354 directory should typically suffice. If you h
355 attention to later steps about retrieving th
356 debug symbols: both explain approaches reduc
357 should allow you to master these tasks with
358
359 [:ref:`details <diskspace_bisref>`]
360
361 .. _buildrequires_bissbs:
362
363 * Install all software required to build a Lin
364 'bc', 'binutils' ('ld' et al.), 'bison', 'fl
365 'pahole', 'perl', and the development header
366 reference section shows how to quickly insta
367 distributions.
368
369 [:ref:`details <buildrequires_bisref>`]
370
371 .. _sources_bissbs:
372
373 * Retrieve the mainline Linux sources; then ch
374 them, as all further commands in this guide
375 there.
376
377 *Note, the following describe how to retriev
378 mainline clone, which downloads about 2,75 G
379 :ref:`reference section describes two altern
380 one downloads less than 500 MByte, the other
381 internet connections.*
382
383 Execute the following command to retrieve a
384 preparing things to add branches for stable/
385
386 git clone -o mainline --no-checkout \
387 https://git.kernel.org/pub/scm/linux/ker
388 cd ~/linux/
389 git remote add -t master stable \
390 https://git.kernel.org/pub/scm/linux/ker
391
392 [:ref:`details <sources_bisref>`]
393
394 .. _stablesources_bissbs:
395
396 * Is one of the versions you earlier establish
397 longterm release (say 6.1.5)? Then download
398 to ('linux-6.1.y' in this example)::
399
400 git remote set-branches --add stable linux
401 git fetch stable
402
403 .. _oldconfig_bissbs:
404
405 * Start preparing a kernel build configuration
406
407 Before doing so, ensure you are still runnin
408 step told you to boot; if you are unsure, ch
409 identifier using ``uname -r``.
410
411 Afterwards check out the source code for the
412 'good'. In the following example command thi
413 the version number in this and all later Git
414 with a 'v'::
415
416 git switch --discard-changes --detach v6.0
417
418 Now create a build configuration file::
419
420 make olddefconfig
421
422 The kernel build scripts then will try to lo
423 for the running kernel and then adjust it fo
424 you checked out. While doing so, it will pri
425
426 Look out for a line starting with '# using d
427 followed by a path to a file in '/boot/' tha
428 of your currently working kernel. If the lin
429 like 'arch/x86/configs/x86_64_defconfig', th
430 the .config file for your running kernel --
431 there manually, as explained in the referenc
432
433 In case you can not find such a line, look f
434 written to .config'. If that's the case you
435 lying around. Unless you intend to use it, d
436 'make olddefconfig' again and check if it no
437 as base.
438
439 [:ref:`details <oldconfig_bisref>`]
440
441 .. _localmodconfig_bissbs:
442
443 * Disable any kernel modules apparently superf
444 optional, but especially wise for bisections
445 process enormously -- at least unless the .c
446 previous step was already tailored to your a
447 case you should skip this step.
448
449 To prepare the trimming, connect external ha
450 keys, tokens, ...), quickly start a VM, and
451 since you started that guide, ensure that yo
452 trouble since you started the system. Only t
453
454 yes '' | make localmodconfig
455
456 There is a catch to this, as the 'apparently
457 and the preparation instructions already hin
458
459 The 'localmodconfig' target easily disables
460 used occasionally -- like modules for extern
461 since booting, virtualization software not y
462 few other things. That's because some tasks
463 loads when you execute tasks like the aforem
464
465 This drawback of localmodconfig is nothing y
466 something to keep in mind: if something is m
467 during this guide, this is most likely the r
468 eliminate the risk with tricks outlined in t
469 building a kernel just for quick testing pur
470 spending much effort on, as long as it boots
471 feature that causes trouble.
472
473 [:ref:`details <localmodconfig_bisref>`]
474
475 .. _tagging_bissbs:
476
477 * Ensure all the kernels you will build are cl
478 tag and a unique version number::
479
480 ./scripts/config --set-str CONFIG_LOCALVER
481 ./scripts/config -e CONFIG_LOCALVERSION_AU
482
483 [:ref:`details <tagging_bisref>`]
484
485 .. _debugsymbols_bissbs:
486
487 * Decide how to handle debug symbols.
488
489 In the context of this document it is often
490 decent chance you will need to decode a stac
491 'warning', or 'BUG'::
492
493 ./scripts/config -d DEBUG_INFO_NONE -e KAL
494 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCH
495
496 But if you are extremely short on storage sp
497 debug symbols instead::
498
499 ./scripts/config -d DEBUG_INFO -d DEBUG_IN
500 -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF
501
502 [:ref:`details <debugsymbols_bisref>`]
503
504 .. _configmods_bissbs:
505
506 * Check if you may want or need to adjust some
507 options:
508
509 * Are you running Debian? Then you want to a
510 additional adjustments explained in the re
511
512 [:ref:`details <configmods_distros_bisref>
513
514 * If you want to influence other aspects of
515 your preferred tool. Note, to use make tar
516 'nconfig', you will need to install the de
517 'xconfig' you likewise need the Qt5 or Qt6
518
519 [:ref:`details <configmods_individual_bisr
520
521 .. _saveconfig_bissbs:
522
523 * Reprocess the .config after the latest adjus
524 place::
525
526 make olddefconfig
527 cp .config ~/kernel-config-working
528
529 [:ref:`details <saveconfig_bisref>`]
530
531 .. _introlatestcheck_bissbs:
532
533 Segment 1: try to reproduce the problem with t
534 ----------------------------------------------
535
536 The following steps verify if the problem occu
537 supported by developers. In case you face a re
538 problem is not caused by some .config change,
539 be a waste of time. [:ref:`details <introlates
540
541 .. _checkoutmaster_bissbs:
542
543 * Check out the latest Linux codebase.
544
545 * Are your 'good' and 'bad' versions from th
546 Then check the `front page of kernel.org <
547 lists a release from that series without a
548 latest version ('linux-6.1.y' in the follo
549
550 cd ~/linux/
551 git switch --discard-changes --detach st
552
553 Your series is unsupported, if is not list
554 tag. In that case you might want to check
555 linux-6.2.y) or mainline (see next point)
556
557 * In all other cases, run::
558
559 cd ~/linux/
560 git switch --discard-changes --detach ma
561
562 [:ref:`details <checkoutmaster_bisref>`]
563
564 .. _build_bissbs:
565
566 * Build the image and the modules of your firs
567 prepared::
568
569 cp ~/kernel-config-working .config
570 make olddefconfig
571 make -j $(nproc --all)
572
573 If you want your kernel packaged up as deb,
574 reference section for alternatives, which ob
575 steps to install as well.
576
577 [:ref:`details <build_bisref>`]
578
579 .. _install_bissbs:
580
581 * Install your newly built kernel.
582
583 Before doing so, consider checking if there
584
585 df -h /boot/ /lib/modules/
586
587 For now assume 150 MByte in /boot/ and 200 i
588 much your kernels actually require will be d
589
590 Now install the kernel's modules and its ima
591 parallel to the your Linux distribution's ke
592
593 sudo make modules_install
594 command -v installkernel && sudo make inst
595
596 The second command ideally will take care of
597 point: copying the kernel's image to /boot/,
598 adding an entry for both to the boot loader'
599
600 Sadly some distributions (among them Arch Li
601 immutable Linux distributions) will perform
602 You therefore want to check if all of them w
603 perform those that were not. The reference s
604 that; your distribution's documentation migh
605
606 Once you figured out the steps needed at thi
607 down: if you will build more kernels as desc
608 have to perform those again after executing
609
610 [:ref:`details <install_bisref>`]
611
612 .. _storagespace_bissbs:
613
614 * In case you plan to follow this guide furthe
615 the kernel, its modules, and other related f
616
617 du -ch /boot/*$(make -s kernelrelease)* |
618 du -sh /lib/modules/$(make -s kernelreleas
619
620 Write down or remember those two values for
621 running out of disk space accidentally durin
622
623 [:ref:`details <storagespace_bisref>`]
624
625 .. _kernelrelease_bissbs:
626
627 * Show and store the kernelrelease identifier
628
629 make -s kernelrelease | tee -a ~/kernels-b
630
631 Remember the identifier momentarily, as it w
632 from the boot menu upon restarting.
633
634 * Reboot into your newly built kernel. To ensu
635 you just built, you might want to verify if
636 matches::
637
638 tail -n 1 ~/kernels-built
639 uname -r
640
641 .. _tainted_bissbs:
642
643 * Check if the kernel marked itself as 'tainte
644
645 cat /proc/sys/kernel/tainted
646
647 If that command does not return '0', check t
648 for this might interfere with your testing.
649
650 [:ref:`details <tainted_bisref>`]
651
652 .. _recheckbroken_bissbs:
653
654 * Verify if your bug occurs with the newly bui
655 out the instructions in the reference sectio
656 during your tests.
657
658 [:ref:`details <recheckbroken_bisref>`]
659
660 .. _recheckstablebroken_bissbs:
661
662 * Did you just built a stable or longterm kern
663 the regression with it? Then you should test
664 well, because the result determines which de
665 to.
666
667 To prepare that test, check out current main
668
669 cd ~/linux/
670 git switch --discard-changes --detach main
671
672 Now use the checked out code to build and in
673 commands the earlier steps already described
674
675 cp ~/kernel-config-working .config
676 make olddefconfig
677 make -j $(nproc --all)
678 # * Check if the free space suffices holdi
679 df -h /boot/ /lib/modules/
680 sudo make modules_install
681 command -v installkernel && sudo make inst
682 make -s kernelrelease | tee -a ~/kernels-b
683 reboot
684
685 Confirm you booted the kernel you intended t
686 status::
687
688 tail -n 1 ~/kernels-built
689 uname -r
690 cat /proc/sys/kernel/tainted
691
692 Now verify if this kernel is showing the pro
693 to report the bug to the primary developers;
694 stable team. See Documentation/admin-guide/r
695
696 [:ref:`details <recheckstablebroken_bisref>`
697
698 Do you follow this guide to verify if a proble
699 currently supported by Linux kernel developers
700 point. If you later want to remove the kernel
701 :ref:`Complementary tasks: cleanup during and
702
703 In case you face a regression, move on and exe
704 as well.
705
706 .. _introworkingcheck_bissbs:
707
708 Segment 2: check if the kernels you build work
709 ----------------------------------------------
710
711 In case of a regression, you now want to ensur
712 you created earlier works as expected; a bisec
713 otherwise would be a waste of time. [:ref:`det
714
715 .. _recheckworking_bissbs:
716
717 * Build your own variant of the 'working' kern
718 regressed works as expected with it.
719
720 Start by checking out the sources for the ve
721 'good' (once again assumed to be 6.0 here)::
722
723 cd ~/linux/
724 git switch --discard-changes --detach v6.0
725
726 Now use the checked out code to configure, b
727 using the commands the previous subsection e
728
729 cp ~/kernel-config-working .config
730 make olddefconfig
731 make -j $(nproc --all)
732 # * Check if the free space suffices holdi
733 df -h /boot/ /lib/modules/
734 sudo make modules_install
735 command -v installkernel && sudo make inst
736 make -s kernelrelease | tee -a ~/kernels-b
737 reboot
738
739 When the system booted, you may want to veri
740 kernel you started is the one you just built
741
742 tail -n 1 ~/kernels-built
743 uname -r
744
745 Now check if this kernel works as expected;
746 section for further instructions.
747
748 [:ref:`details <recheckworking_bisref>`]
749
750 .. _introbisect_bissbs:
751
752 Segment 3: perform the bisection and validate
753 ----------------------------------------------
754
755 With all the preparations and precaution build
756 to begin the bisection. This will make you bui
757 about 15 in case you encountered a regression
758 (say from 6.0.13 to 6.1.5). But do not worry,
759 configuration created earlier this works a lot
760 overall on average it will often just take abo
761 each kernel on commodity x86 machines.
762
763 .. _bisectstart_bissbs:
764
765 * Start the bisection and tell Git about the v
766 'good' (6.0 in the following example command
767
768 cd ~/linux/
769 git bisect start
770 git bisect good v6.0
771 git bisect bad v6.1.5
772
773 [:ref:`details <bisectstart_bisref>`]
774
775 .. _bisectbuild_bissbs:
776
777 * Now use the code Git checked out to build, i
778 the commands introduced earlier::
779
780 cp ~/kernel-config-working .config
781 make olddefconfig
782 make -j $(nproc --all)
783 # * Check if the free space suffices holdi
784 df -h /boot/ /lib/modules/
785 sudo make modules_install
786 command -v installkernel && sudo make inst
787 make -s kernelrelease | tee -a ~/kernels-b
788 reboot
789
790 If compilation fails for some reason, run ``
791 executing the stack of commands from the beg
792
793 In case you skipped the 'test latest codebas
794 description as for why the 'df [...]' and 'm
795 commands are here.
796
797 Important note: the latter command from this
798 identifiers that might look odd or wrong to
799 totally normal to see release identifiers li
800 if you bisect between versions 6.1 and 6.2 f
801
802 [:ref:`details <bisectbuild_bisref>`]
803
804 .. _bisecttest_bissbs:
805
806 * Now check if the feature that regressed work
807
808 You again might want to start by making sure
809 you just built::
810
811 cd ~/linux/
812 tail -n 1 ~/kernels-built
813 uname -r
814
815 Now verify if the feature that regressed wor
816 If it does, run this::
817
818 git bisect good
819
820 If it does not, run this::
821
822 git bisect bad
823
824 Be sure about what you tell Git, as getting
825 rest of the bisection totally off course.
826
827 While the bisection is ongoing, Git will use
828 find and check out another bisection point f
829 will print something like 'Bisecting: 675 re
830 (roughly 10 steps)' to indicate how many fur
831 tested. Now build and install another kernel
832 previous step; afterwards follow the instruc
833
834 Repeat this again and again until you finish
835 when Git after tagging a change as 'good' or
836 'cafecaca0c0dacafecaca0c0dacafecaca0c0da is
837 afterwards it will show some details about t
838 description of the change. The latter might
839 might need to scroll up to see the message m
840 alternatively, run ``git bisect log > ~/bise
841
842 [:ref:`details <bisecttest_bisref>`]
843
844 .. _bisectlog_bissbs:
845
846 * Store Git's bisection log and the current .c
847 telling Git to reset the sources to the stat
848
849 cd ~/linux/
850 git bisect log > ~/bisection-log
851 cp .config ~/bisection-config-culprit
852 git bisect reset
853
854 [:ref:`details <bisectlog_bisref>`]
855
856 .. _revert_bissbs:
857
858 * Try reverting the culprit on top of latest m
859 regression.
860
861 This is optional, as it might be impossible
862 the case, if the bisection determined a merg
863 latter happens if other changes depend on th
864 succeeds, it is worth building another kerne
865 a bisection, which can easily deroute; it fu
866 developers know, if they can resolve the reg
867
868 Begin by checking out the latest codebase de
869
870 * Did you face a regression within a stable/
871 6.0.13 and 6.0.15) that does not happen in
872 latest codebase for the affected series li
873
874 git fetch stable
875 git switch --discard-changes --detach li
876
877 * In all other cases check out latest mainli
878
879 git fetch mainline
880 git switch --discard-changes --detach ma
881
882 If you bisected a regression within a stab
883 happens in mainline, there is one more thi
884 commit-id. To do so, use a command like ``
885 view the patch description of the culprit.
886 the top which looks like 'commit cafec0cac
887 'Upstream commit cafec0cacaca0'; use that
888 and not the one the bisection blamed.
889
890 Now try reverting the culprit by specifying
891
892 git revert --no-edit cafec0cacaca0
893
894 If that fails, give up trying and move on to
895 adjust the tag to facilitate the identificat
896 overwriting another kernel::
897
898 cp ~/kernel-config-working .config
899 ./scripts/config --set-str CONFIG_LOCALVER
900
901 Build a kernel using the familiar command se
902 the base .config over::
903
904 make olddefconfig &&
905 make -j $(nproc --all)
906 # * Check if the free space suffices holdi
907 df -h /boot/ /lib/modules/
908 sudo make modules_install
909 command -v installkernel && sudo make inst
910 make -s kernelrelease | tee -a ~/kernels-b
911 reboot
912
913 Now check one last time if the feature that
914 with that kernel: if everything went well, i
915
916 [:ref:`details <revert_bisref>`]
917
918 .. _introclosure_bissbs:
919
920 Complementary tasks: cleanup during and after
921 ----------------------------------------------
922
923 During and after following this guide you migh
924 the kernels you installed: the boot menu other
925 space might run out.
926
927 .. _makeroom_bissbs:
928
929 * To remove one of the kernels you installed,
930 identifier. This guide stores them in '~/ker
931 command will print them as well::
932
933 ls -ltr /lib/modules/*-local*
934
935 You in most situations want to remove the ol
936 actual bisection (e.g. segment 3 of this gui
937 beforehand (e.g. to test the latest codebase
938 'good') might become handy to verify somethi
939 around, unless you are really short on stora
940
941 To remove the modules of a kernel with the k
942 '*6.0-rc1-local-gcafec0cacaca0*', start by r
943 modules::
944
945 sudo rm -rf /lib/modules/6.0-rc1-local-gca
946
947 Afterwards try the following command::
948
949 sudo kernel-install -v remove 6.0-rc1-loca
950
951 On quite a few distributions this will delet
952 while also removing the kernel's entry from
953 distributions kernel-install does not exist
954 kernel image and related files behind; in th
955 in the reference section.
956
957 [:ref:`details <makeroom_bisref>`]
958
959 .. _finishingtouch_bissbs:
960
961 * Once you have finished the bisection, do not
962 set up, as you might need a few things again
963 on the outcome of the bisection:
964
965 * Could you initially reproduce the regressi
966 after the bisection were able to fix the p
967 top of the latest codebase? Then you want
968 for a while, but safely remove all others
969 identifier.
970
971 * Did the bisection end on a merge-commit or
972 reasons? Then you want to keep as many ker
973 days: it's pretty likely that you will be
974
975 * In other cases it likely is a good idea to
976 for some time: the one built from the late
977 the version considered 'good', and the las
978 during the actual bisection process.
979
980 [:ref:`details <finishingtouch_bisref>`]
981
982 .. _introoptional_bissbs:
983
984 Optional: test reverts, patches, or later vers
985 ----------------------------------------------
986
987 While or after reporting a bug, you might want
988 test reverts, debug patches, proposed fixes, o
989 follow these instructions.
990
991 * Update your Git clone and check out the late
992
993 * In case you want to test mainline, fetch i
994 its code out::
995
996 git fetch mainline
997 git switch --discard-changes --detach ma
998
999 * In case you want to test a stable or longt
1000 holding the series you are interested in
1001 already did so earlier::
1002
1003 git remote set-branches --add stable li
1004
1005 Then fetch the latest changes and check o
1006 series::
1007
1008 git fetch stable
1009 git switch --discard-changes --detach s
1010
1011 * Copy your kernel build configuration over::
1012
1013 cp ~/kernel-config-working .config
1014
1015 * Your next step depends on what you want to
1016
1017 * In case you just want to test the latest
1018 you are already all set.
1019
1020 * In case you want to test if a revert fixe
1021 changes by specifying their commit ids::
1022
1023 git revert --no-edit cafec0cacaca0
1024
1025 Now give that kernel a special tag to fac
1026 prevent accidentally overwriting another
1027
1028 ./scripts/config --set-str CONFIG_LOCAL
1029
1030 * In case you want to test a patch, store t
1031 '/tmp/foobars-proposed-fix-v1.patch' and
1032
1033 git apply /tmp/foobars-proposed-fix-v1.
1034
1035 In case of multiple patches, repeat this
1036
1037 Now give that kernel a special tag to fac
1038 prevent accidentally overwriting another
1039
1040 ./scripts/config --set-str CONFIG_LOCALVE
1041
1042 * Build a kernel using the familiar commands,
1043 build configuration over, as that has been
1044
1045 make olddefconfig &&
1046 make -j $(nproc --all)
1047 # * Check if the free space suffices hold
1048 df -h /boot/ /lib/modules/
1049 sudo make modules_install
1050 command -v installkernel && sudo make ins
1051 make -s kernelrelease | tee -a ~/kernels-
1052 reboot
1053
1054 * Now verify you booted the newly built kerne
1055
1056 [:ref:`details <introoptional_bisref>`]
1057
1058 .. _submit_improvements:
1059
1060 Conclusion
1061 ----------
1062
1063 You have reached the end of the step-by-step
1064
1065 Did you run into trouble following any of the
1066 reference section below? Did you spot errors?
1067 improve the guide?
1068
1069 If any of that applies, please take a moment
1070 document know by email (Thorsten Leemhuis <lin
1071 CCing the Linux docs mailing list (linux-doc@
1072 vital to improve this text further, which is
1073 will enable more people to master the task de
1074 improve similar guides inspired by this one.
1075
1076
1077 Reference section for the step-by-step guide
1078 ============================================
1079
1080 This section holds additional information for
1081 step-by-step guide.
1082
1083 Preparations for building your own kernels
1084 ------------------------------------------
1085
1086 *The steps in this section lay the groundwo
1087 [:ref:`... <introprep_bissbs>`]
1088
1089 The steps in all later sections of this guide
1090
1091 [:ref:`back to step-by-step guide <introprep_
1092
1093 .. _backup_bisref:
1094
1095 Prepare for emergencies
1096 ~~~~~~~~~~~~~~~~~~~~~~~
1097
1098 *Create a fresh backup and put system repai
1099 [:ref:`... <backup_bissbs>`]
1100
1101 Remember, you are dealing with computers, whi
1102 -- especially if you fiddle with crucial part
1103 system. That's what you are about to do in th
1104 for something going sideways, even if that sh
1105
1106 [:ref:`back to step-by-step guide <backup_bis
1107
1108 .. _vanilla_bisref:
1109
1110 Remove anything related to externally maintai
1111 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1112
1113 *Remove all software that depends on extern
1114 builds them automatically.* [:ref:`...<vani
1115
1116 Externally developed kernel modules can easil
1117
1118 But there is a more important reason why this
1119 kernel developers will not care about reports
1120 kernels that utilize such modules. That's bec
1121 considered 'vanilla' anymore, as Documentatio
1122 explains in more detail.
1123
1124 [:ref:`back to step-by-step guide <vanilla_bi
1125
1126 .. _secureboot_bisref:
1127
1128 Deal with techniques like Secure Boot
1129 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1130
1131 *On platforms with 'Secure Boot' or similar
1132 ensure the system will permit your self-com
1133 [:ref:`... <secureboot_bissbs>`]
1134
1135 Many modern systems allow only certain operat
1136 they reject booting self-compiled kernels by
1137
1138 You ideally deal with this by making your pla
1139 with the help of a certificate. How to do tha
1140 here, as it requires various steps that would
1141 its purpose; 'Documentation/admin-guide/modul
1142 sides already explain everything needed in mo
1143
1144 Temporarily disabling solutions like Secure B
1145 Linux boot. On commodity x86 systems it is po
1146 utility; the required steps vary a lot betwee
1147 described here.
1148
1149 On mainstream x86 Linux distributions there i
1150 disable all Secure Boot restrictions for your
1151 initiate this process by running ``mokutil --
1152 tell you to create a one-time password, which
1153 restart; right after your BIOS performed all
1154 show a blue box with a message 'Press any key
1155 some key before the countdown exposes, which
1156 Secure Boot state'. Shim's 'MokManager' will
1157 randomly chosen characters from the one-time
1158 you provided them, confirm you really want to
1159 Afterwards, permit MokManager to reboot the m
1160
1161 [:ref:`back to step-by-step guide <secureboot
1162
1163 .. _bootworking_bisref:
1164
1165 Boot the last kernel that was working
1166 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1167
1168 *Boot into the last working kernel and brie
1169 regressed really works.* [:ref:`...<bootwor
1170
1171 This will make later steps that cover creatin
1172 the right thing.
1173
1174 [:ref:`back to step-by-step guide <bootworkin
1175
1176 .. _diskspace_bisref:
1177
1178 Space requirements
1179 ~~~~~~~~~~~~~~~~~~
1180
1181 *Ensure to have enough free space for build
1182 [:ref:`... <diskspace_bissbs>`]
1183
1184 The numbers mentioned are rough estimates wit
1185 safe side, so often you will need less.
1186
1187 If you have space constraints, be sure to hay
1188 debug symbols' <debugsymbols_bissbs>` and its
1189 section' <debugsymbols_bisref>`, as disabling
1190 space by quite a few gigabytes.
1191
1192 [:ref:`back to step-by-step guide <diskspace_
1193
1194 .. _rangecheck_bisref:
1195
1196 Bisection range
1197 ~~~~~~~~~~~~~~~
1198
1199 *Determine the kernel versions considered '
1200 guide.* [:ref:`...<rangecheck_bissbs>`]
1201
1202 Establishing the range of commits to be check
1203 except when a regression occurred when switch
1204 series to a release of a later series (e.g. f
1205 Git will need some hand holding, as there is
1206
1207 That's because with the release of 6.0 mainli
1208 stable series 6.0.y branched to the side. It'
1209 that the issue you face with 6.1.5 only worke
1210 commit that went into one of the 6.0.y releas
1211 6.1.y series. Thankfully that normally should
1212 stable/longterm maintainers maintain the code
1213 6.0 as a 'good' kernel. That assumption will
1214 will be built and tested in the segment '2' o
1215 to do this as well, if you tried bisecting be
1216
1217 [:ref:`back to step-by-step guide <rangecheck
1218
1219 .. _buildrequires_bisref:
1220
1221 Install build requirements
1222 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1223
1224 *Install all software required to build a L
1225 [:ref:`...<buildrequires_bissbs>`]
1226
1227 The kernel is pretty stand-alone, but besides
1228 sometimes need a few libraries to build one.
1229 depends on your Linux distribution and the co
1230 about to build.
1231
1232 Here are a few examples what you typically ne
1233 distributions:
1234
1235 * Arch Linux and derivatives::
1236
1237 sudo pacman --needed -S bc binutils bison
1238 pahole perl zlib ncurses qt6-base
1239
1240 * Debian, Ubuntu, and derivatives::
1241
1242 sudo apt install bc binutils bison dwarve
1243 libssl-dev make openssl pahole perl-bas
1244 libncurses-dev qt6-base-dev g++
1245
1246 * Fedora and derivatives::
1247
1248 sudo dnf install binutils \
1249 /usr/bin/{bc,bison,flex,gcc,git,openssl
1250 /usr/include/{libelf.h,openssl/pkcs7.h,
1251
1252 * openSUSE and derivatives::
1253
1254 sudo zypper install bc binutils bison dwa
1255 kernel-install-tools libelf-devel make
1256 perl-base zlib-devel rpm-build ncurses-
1257
1258 These commands install a few packages that ar
1259 for example might want to skip installing the
1260 which you will only need in case you later mi
1261 configuration using make the targets 'menucon
1262 the headers of Qt6 if you do not plan to adju
1263
1264 You furthermore might need additional librari
1265 for tasks not covered in this guide -- for ex
1266 the kernel's tools/ directory.
1267
1268 [:ref:`back to step-by-step guide <buildrequi
1269
1270 .. _sources_bisref:
1271
1272 Download the sources using Git
1273 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1274
1275 *Retrieve the Linux mainline sources.*
1276 [:ref:`...<sources_bissbs>`]
1277
1278 The step-by-step guide outlines how to downlo
1279 Git clone of Linus' mainline repository. Ther
1280 that -- but there are two alternatives ways t
1281 work better for you:
1282
1283 * If you have an unreliable internet connecti
1284 :ref:`using a 'Git bundle'<sources_bundle_b
1285
1286 * If downloading the complete repository woul
1287 much storage space, consider :ref:`using a
1288 clone'<sources_shallow_bisref>`.
1289
1290 .. _sources_bundle_bisref:
1291
1292 Downloading Linux mainline sources using a bu
1293 """""""""""""""""""""""""""""""""""""""""""""
1294
1295 Use the following commands to retrieve the Li
1296 bundle::
1297
1298 wget -c \
1299 https://git.kernel.org/pub/scm/linux/ke
1300 git clone --no-checkout clone.bundle ~/li
1301 cd ~/linux/
1302 git remote remove origin
1303 git remote add mainline \
1304 https://git.kernel.org/pub/scm/linux/ke
1305 git fetch mainline
1306 git remote add -t master stable \
1307 https://git.kernel.org/pub/scm/linux/ke
1308
1309 In case the 'wget' command fails, just re-exe
1310 it left off.
1311
1312 [:ref:`back to step-by-step guide <sources_bi
1313 [:ref:`back to section intro <sources_bisref>
1314
1315 .. _sources_shallow_bisref:
1316
1317 Downloading Linux mainline sources using a sh
1318 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1319
1320 First, execute the following command to retri
1321
1322 git clone -o mainline --no-checkout --dep
1323 https://git.kernel.org/pub/scm/linux/ke
1324 cd ~/linux/
1325 git remote add -t master stable \
1326 https://git.kernel.org/pub/scm/linux/ke
1327
1328 Now deepen your clone's history to the second
1329 release of your 'good' version. In case the l
1330 be the first predecessor and 5.18 the second
1331 that version::
1332
1333 git fetch --shallow-exclude=v5.18 mainlin
1334
1335 Afterwards add the stable Git repository as r
1336 branches as explained in the step-by-step gui
1337
1338 Note, shallow clones have a few peculiar char
1339
1340 * For bisections the history needs to be deep
1341 farther than it seems necessary, as explain
1342 Git otherwise will be unable to revert or d
1343 a range (say 6.1..6.2), as they are interna
1344 releases (like 6.0-rc2 or 5.19-rc3).
1345
1346 * This document in most places uses ``git fet
1347 to specify the earliest version you care ab
1348 tag). You alternatively can use the paramet
1349 an absolute (say ``'2023-07-15'``) or relat
1350 define the depth of the history you want to
1351 bisecting mainline, ensure to deepen the hi
1352 the release of the mainline release your 'g
1353
1354 * Be warned, when deepening your clone you mi
1355 'fatal: error in object: unshallow cafecaca
1356 In that case run ``git repack -d`` and try
1357
1358 [:ref:`back to step-by-step guide <sources_bi
1359 [:ref:`back to section intro <sources_bisref>
1360
1361 .. _oldconfig_bisref:
1362
1363 Start defining the build configuration for yo
1364 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1365
1366 *Start preparing a kernel build configurati
1367 [:ref:`... <oldconfig_bissbs>`]
1368
1369 *Note, this is the first of multiple steps in
1370 build artifacts. The commands used in this gu
1371 tree to keep things simple. In case you prefe
1372 separately, create a directory like '~/linux-
1373 ``O=~/linux-builddir/`` to all make calls use
1374 have to point other commands there as well --
1375 [...]`` commands, which will require ``--file
1376 locate the right build configuration.*
1377
1378 Two things can easily go wrong when creating
1379
1380 * The oldconfig target will use a .config fil
1381 one is already present there (e.g. '~/linux
1382 that's what you intend (see next step), but
1383 delete it. This for example is important in
1384 further, but due to problems come back here
1385 scratch.
1386
1387 * Sometimes olddefconfig is unable to locate
1388 kernel and will use defaults, as briefly ou
1389 check if your distribution ships the config
1390 it in the right place (e.g. '~/linux/.confi
1391 where /proc/config.gz exists this can be ac
1392
1393 zcat /proc/config.gz > .config
1394
1395 Once you put it there, run ``make olddefcon
1396 needs of the kernel about to be built.
1397
1398 Note, the olddefconfig target will set any un
1399 default value. If you prefer to set such conf
1400 ``make oldconfig`` instead. Then for each und
1401 will be asked how to proceed; in case you are
1402 'enter' to apply the default value. Note thou
1403 want to go with the defaults, as you otherwis
1404 causes a problem looking like regressions (fo
1405 restrictions).
1406
1407 Occasionally odd things happen when trying to
1408 kernel (say 6.1) on an older mainline release
1409 (say 5.15). That's one of the reasons why the
1410 you to boot the kernel where everything works
1411 file you thus want to ensure it's from the wo
1412 that shows the regression.
1413
1414 In case you want to build kernels for another
1415 configuration; usually ``ls /boot/config-$(un
1416 that file to the build machine and store it a
1417 ``make olddefconfig`` to adjust it.
1418
1419 [:ref:`back to step-by-step guide <oldconfig_
1420
1421 .. _localmodconfig_bisref:
1422
1423 Trim the build configuration for your kernel
1424 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1425
1426 *Disable any kernel modules apparently supe
1427 [:ref:`... <localmodconfig_bissbs>`]
1428
1429 As explained briefly in the step-by-step guid
1430 can easily happen that your self-built kernel
1431 did not perform at least once before utilizin
1432 when a task requires kernel modules which are
1433 it for the first time. So when you never perf
1434 kernel the modules will not have been loaded
1435 of view look superfluous, which thus disables
1436 to be compiled.
1437
1438 You can try to avoid this by performing typic
1439 additional kernel modules: start a VM, establ
1440 CD/DVD ISO, mount network shares (CIFS, NFS,
1441 devices (2FA keys, headsets, webcams, ...) as
1442 systems you otherwise do not utilize (btrfs,
1443 is hard to think of everything that might be
1444 often forget one thing or another at this poi
1445
1446 Do not let that risk bother you, especially w
1447 testing purposes: everything typically crucia
1448 something important you can turn on a missing
1449 run the commands again to compile and install
1450 need.
1451
1452 But if you plan to build and use self-built k
1453 reduce the risk by recording which modules yo
1454 a few weeks. You can automate this with `modp
1455 <https://github.com/graysky2/modprobed-db>`_.
1456 point localmodconfig to the list of modules m
1457
1458 yes '' | make LSMOD='${HOME}'/.config/modpr
1459
1460 That parameter also allows you to build trimm
1461 case you copied a suitable .config over to us
1462 run ``lsmod > lsmod_foo-machine`` on that sys
1463 your build's host home directory. Then run th
1464 step-by-step guide mentions::
1465
1466 yes '' | make LSMOD=~/lsmod_foo-machine loc
1467
1468 [:ref:`back to step-by-step guide <localmodco
1469
1470 .. _tagging_bisref:
1471
1472 Tag the kernels about to be build
1473 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1474
1475 *Ensure all the kernels you will build are
1476 special tag and a unique version identifier
1477
1478 This allows you to differentiate your distrib
1479 during this process, as the file or directori
1480 '-local' in the name; it also helps picking t
1481 not lose track of you kernels, as their versi
1482 confusing during the bisection.
1483
1484 [:ref:`back to step-by-step guide <tagging_bi
1485
1486 .. _debugsymbols_bisref:
1487
1488 Decide to enable or disable debug symbols
1489 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1490
1491 *Decide how to handle debug symbols.* [:ref
1492
1493 Having debug symbols available can be importa
1494 'panic', 'Oops', 'warning', or 'BUG' later wh
1495 able to find the exact place where the proble
1496 collecting and embedding the needed debug inf
1497 quite a bit of space: in late 2022 the build
1498 trimmed with localmodconfig consumed around 5
1499 symbols, but less than 1 when they were disab
1500 modules are bigger as well, which increases s
1501 load times.
1502
1503 In case you want a small kernel and are unlik
1504 you thus might want to disable debug symbols
1505 later turns out that you need them, just enab
1506 kernel.
1507
1508 You on the other hand definitely want to enab
1509 is a decent chance that you need to decode a
1510 'Decode failure messages' in Documentation/ad
1511 explains this process in more detail.
1512
1513 [:ref:`back to step-by-step guide <debugsymbo
1514
1515 .. _configmods_bisref:
1516
1517 Adjust build configuration
1518 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1519
1520 *Check if you may want or need to adjust so
1521 options:*
1522
1523 Depending on your needs you at this point mig
1524 kernel configuration options.
1525
1526 .. _configmods_distros_bisref:
1527
1528 Distro specific adjustments
1529 """""""""""""""""""""""""""
1530
1531 *Are you running* [:ref:`... <configmods_bi
1532
1533 The following sections help you to avoid buil
1534 when following this guide on a few commodity
1535
1536 **Debian:**
1537
1538 * Remove a stale reference to a certificate f
1539 fail::
1540
1541 ./scripts/config --set-str SYSTEM_TRUSTED_
1542
1543 Alternatively, download the needed certific
1544 option point to it, as `the Debian handbook
1545 <https://debian-handbook.info/browse/stable
1546 -- or generate your own, as explained in
1547 Documentation/admin-guide/module-signing.rs
1548
1549 [:ref:`back to step-by-step guide <configmods
1550
1551 .. _configmods_individual_bisref:
1552
1553 Individual adjustments
1554 """"""""""""""""""""""
1555
1556 *If you want to influence the other aspects
1557 now.* [:ref:`... <configmods_bissbs>`]
1558
1559 At this point you can use a command like ``ma
1560 to enable or disable certain features using a
1561 a graphical configuration utility, run ``make
1562 require development libraries from toolkits t
1563 respectively Qt5 or Qt6); an error message wi
1564 is missing.
1565
1566 [:ref:`back to step-by-step guide <configmods
1567
1568 .. _saveconfig_bisref:
1569
1570 Put the .config file aside
1571 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1572
1573 *Reprocess the .config after the latest cha
1574 [:ref:`... <saveconfig_bissbs>`]
1575
1576 Put the .config you prepared aside, as you wa
1577 directory every time during this guide before
1578 kernel. That's because going back and forth b
1579 .config files in odd ways; those occasionally
1580 confuse testing or in some cases render the r
1581 meaningless.
1582
1583 [:ref:`back to step-by-step guide <saveconfig
1584
1585 .. _introlatestcheck_bisref:
1586
1587 Try to reproduce the problem with the latest
1588 ---------------------------------------------
1589
1590 *Verify the regression is not caused by som
1591 still occurs with the latest codebase.* [:r
1592
1593 For some readers it might seem unnecessary to
1594 point, especially if you did that already wit
1595 distributor or face a regression within a sta
1596 highly recommended for these reasons:
1597
1598 * You will run into any problems caused by yo
1599 a bisection. That will make it a lot easier
1600 most likely is some problem in my setup' an
1601 during the bisection, as the kernel sources
1602 problem that causes building or booting to
1603
1604 * These steps will rule out if your problem i
1605 build configuration between the 'working' a
1606 example can happen when your distributor en
1607 feature in the newer kernel which was disab
1608 older kernel. That security feature might g
1609 do -- in which case your problem from the p
1610 upstream developers is not a regression, as
1611 Documentation/admin-guide/reporting-regress
1612 You thus would waste your time if you'd try
1613
1614 * If the cause for your regression was alread
1615 codebase, you'd perform the bisection for n
1616 regression you encountered with a stable/lo
1617 often caused by problems in mainline change
1618 case the problem will have to be fixed in m
1619 fixed there and the fix is already in the p
1620
1621 * For regressions within a stable/longterm se
1622 know if the issue is specific to that serie
1623 kernel, as the report needs to be sent to d
1624
1625 * Regressions specific to a stable/longterm
1626 responsibility; mainline Linux developers
1627
1628 * Regressions also happening in mainline ar
1629 developers and maintainers have to handle
1630 and does not need to be involved in the r
1631 to backport the fix once it's ready.
1632
1633 Your report might be ignored if you send it
1634 when you get a reply there is a decent chan
1635 evaluate which of the two cases it is befor
1636
1637 [:ref:`back to step-by-step guide <introlates
1638
1639 .. _checkoutmaster_bisref:
1640
1641 Check out the latest Linux codebase
1642 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1643
1644 *Check out the latest Linux codebase.*
1645 [:ref:`... <checkoutmaster_bissbs>`]
1646
1647 In case you later want to recheck if an ever
1648 problem, remember to run that ``git fetch --s
1649 again mentioned earlier to update your local
1650
1651 [:ref:`back to step-by-step guide <checkoutma
1652
1653 .. _build_bisref:
1654
1655 Build your kernel
1656 ~~~~~~~~~~~~~~~~~
1657
1658 *Build the image and the modules of your fi
1659 you prepared.* [:ref:`... <build_bissbs>`]
1660
1661 A lot can go wrong at this stage, but the ins
1662 yourself. Another subsection explains how to
1663 deb, rpm or tar file.
1664
1665 Dealing with build errors
1666 """""""""""""""""""""""""
1667
1668 When a build error occurs, it might be caused
1669 setup that often can be fixed quickly; other
1670 the code and can only be fixed by a developer
1671 failure messages coupled with some research o
1672 which of the two it is. To perform such inves
1673 process like this::
1674
1675 make V=1
1676
1677 The ``V=1`` activates verbose output, which m
1678 error. To make it easier to spot, this comman
1679 --all)`` used earlier to utilize every CPU co
1680 this parallelism also results in some clutter
1681
1682 After a few seconds the build process should
1683 to find the most crucial line describing the
1684 for the most important and non-generic sectio
1685 avoid or remove anything that looks remotely
1686 or local path names like ``/home/username/lin
1687 internet search engine with that string, afte
1688 lists via `lore.kernel.org/all/ <https://lore
1689
1690 This most of the time will find something tha
1691 often one of the hits will provide a solution
1692 do not find anything that matches your proble
1693 by modifying your search terms or using anoth
1694
1695 In the end, most issues you run into have lik
1696 reported by others already. That includes iss
1697 system, but lies in the code. If you run into
1698 a solution (e.g. a patch) or workaround for y
1699
1700 Package your kernel up
1701 """"""""""""""""""""""
1702
1703 The step-by-step guide uses the default make
1704 'modules' on x86) to build the image and the
1705 steps of the guide then install. You instead
1706 and directly package it up by using one of th
1707
1708 * ``make -j $(nproc --all) bindeb-pkg`` to ge
1709
1710 * ``make -j $(nproc --all) binrpm-pkg`` to ge
1711
1712 * ``make -j $(nproc --all) tarbz2-pkg`` to ge
1713
1714 This is just a selection of available make ta
1715 ``make help`` for others. You can also use th
1716 ``make -j $(nproc --all)``, as they will pick
1717
1718 If you employ the targets to generate deb or
1719 step-by-step guide's instructions on installi
1720 instead install and remove the packages using
1721 (e.g. dpkg and rpm) or a package management u
1722 aptitude, dnf/yum, zypper, ...). Be aware tha
1723 these two make targets are designed to work o
1724 those formats, they thus will sometimes behav
1725 distribution's kernel packages.
1726
1727 [:ref:`back to step-by-step guide <build_biss
1728
1729 .. _install_bisref:
1730
1731 Put the kernel in place
1732 ~~~~~~~~~~~~~~~~~~~~~~~
1733
1734 *Install the kernel you just built.* [:ref:
1735
1736 What you need to do after executing the comma
1737 depends on the existence and the implementati
1738 executable on your distribution.
1739
1740 If installkernel is found, the kernel's build
1741 installation of your kernel image to this exe
1742 or all of these tasks:
1743
1744 * On almost all Linux distributions installke
1745 image in /boot/, usually as '/boot/vmlinuz-
1746 put a 'System.map-<kernelrelease_id>' along
1747
1748 * On most distributions installkernel will th
1749 (sometimes also called 'initrd'), which usu
1750 '/boot/initramfs-<kernelrelease_id>.img' or
1751 '/boot/initrd-<kernelrelease_id>'. Commodit
1752 for booting, hence ensure to execute the ma
1753 as your distribution's initramfs generator
1754 the modules that go into the image.
1755
1756 * On some distributions installkernel will th
1757 to your bootloader's configuration.
1758
1759 You have to take care of some or all of the t
1760 distribution lacks a installkernel script or
1761 Consult the distribution's documentation for
1762 kernel manually::
1763
1764 sudo install -m 0600 $(make -s image_name)
1765 sudo install -m 0600 System.map /boot/Syst
1766
1767 Now generate your initramfs using the tools y
1768 process. Afterwards add your kernel to your b
1769
1770 [:ref:`back to step-by-step guide <install_bi
1771
1772 .. _storagespace_bisref:
1773
1774 Storage requirements per kernel
1775 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1776
1777 *Check how much storage space the kernel, i
1778 like the initramfs consume.* [:ref:`... <st
1779
1780 The kernels built during a bisection consume
1781 /lib/modules/, especially if you enabled debu
1782 fill up volumes during a bisection -- and due
1783 work earlier might fail to boot. To prevent t
1784 space each installed kernel typically require
1785
1786 Note, most of the time the pattern '/boot/*$(
1787 the guide will match all files needed to boot
1788 path nor the naming scheme are mandatory. On
1789 need to look in different places.
1790
1791 [:ref:`back to step-by-step guide <storagespa
1792
1793 .. _tainted_bisref:
1794
1795 Check if your newly built kernel considers it
1796 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1797
1798 *Check if the kernel marked itself as 'tain
1799 [:ref:`... <tainted_bissbs>`]
1800
1801 Linux marks itself as tainted when something
1802 follow-up errors that look totally unrelated.
1803 ignore or react scantly to reports from taint
1804 kernel set the flag right when the reported b
1805
1806 That's why you want check why a kernel is tai
1807 Documentation/admin-guide/tainted-kernels.rst
1808 interest, as your testing might be flawed oth
1809
1810 [:ref:`back to step-by-step guide <tainted_bi
1811
1812 .. _recheckbroken_bisref:
1813
1814 Check the kernel built from a recent mainline
1815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1816
1817 *Verify if your bug occurs with the newly b
1818 [:ref:`... <recheckbroken_bissbs>`]
1819
1820 There are a couple of reasons why your bug or
1821 the kernel you built from the latest codebase
1822
1823 * The bug was fixed meanwhile.
1824
1825 * What you suspected to be a regression was c
1826 configuration the provider of your kernel c
1827
1828 * Your problem might be a race condition that
1829 the trimmed build configuration, a differen
1830 compiler used, and various other things can
1831
1832 * In case you encountered the regression with
1833 be a problem that is specific to that serie
1834 check this.
1835
1836 [:ref:`back to step-by-step guide <recheckbro
1837
1838 .. _recheckstablebroken_bisref:
1839
1840 Check the kernel built from the latest stable
1841 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1842
1843 *Are you facing a regression within a stabl
1844 reproduce it with the kernel you just built
1845 Then check if the latest codebase for the p
1846 the problem.* [:ref:`... <recheckstablebrok
1847
1848 If this kernel does not show the regression e
1849 for a bisection.
1850
1851 [:ref:`back to step-by-step guide <rechecksta
1852
1853 .. _introworkingcheck_bisref:
1854
1855 Ensure the 'good' version is really working w
1856 ---------------------------------------------
1857
1858 *Check if the kernels you build work fine.*
1859 [:ref:`... <introworkingcheck_bissbs>`]
1860
1861 This section will reestablish a known working
1862 appealing, but is usually a bad idea, as it d
1863
1864 It will ensure the .config file you prepared
1865 That is in your own interest, as trimming the
1866 and you might be building and testing ten or
1867 starting to suspect something might be wrong
1868
1869 That alone is reason enough to spend the time
1870
1871 Many readers of this guide normally run kerne
1872 modules, or both. Those kernels thus are not
1873 it's possible that the thing that regressed m
1874 builds of the 'good' version in the first pla
1875
1876 There is a third reason for those that notice
1877 stable/longterm kernels of different series (
1878 ensure the kernel version you assumed to be '
1879 6.0) actually is working.
1880
1881 [:ref:`back to step-by-step guide <introworki
1882
1883 .. _recheckworking_bisref:
1884
1885 Build your own version of the 'good' kernel
1886 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1887
1888 *Build your own variant of the working kern
1889 regressed works as expected with it.* [:ref
1890
1891 In case the feature that broke with newer ker
1892 self-built kernel, find and resolve the cause
1893 multitude of reasons why this might happen. S
1894
1895 * Check the taint status and the output of ``
1896 went wrong.
1897
1898 * Maybe localmodconfig did something odd and
1899 test the feature? Then you might want to re
1900 one from the last working kernel and skip t
1901 some features in the .config might work as
1902
1903 * Maybe it's not a kernel regression and some
1904 a broken initramfs (also known as initrd),
1905 userland software?
1906
1907 * Maybe it was a feature added to your distri
1908 at that point never supported?
1909
1910 Note, if you found and fixed problems with th
1911 to build another kernel from the latest codeb
1912 mainline and the latest version from an affec
1913 most likely flawed.
1914
1915 [:ref:`back to step-by-step guide <recheckwor
1916
1917 Perform a bisection and validate the result
1918 -------------------------------------------
1919
1920 *With all the preparations and precaution b
1921 ready to begin the bisection.* [:ref:`... <
1922
1923 The steps in this segment perform and validat
1924
1925 [:ref:`back to step-by-step guide <introbisec
1926
1927 .. _bisectstart_bisref:
1928
1929 Start the bisection
1930 ~~~~~~~~~~~~~~~~~~~
1931
1932 *Start the bisection and tell Git about the
1933 'good' and 'bad'.* [:ref:`... <bisectstart_
1934
1935 This will start the bisection process; the la
1936 check out a commit round about half-way betwe
1937 for you to test.
1938
1939 [:ref:`back to step-by-step guide <bisectstar
1940
1941 .. _bisectbuild_bisref:
1942
1943 Build a kernel from the bisection point
1944 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1945
1946 *Build, install, and boot a kernel from the
1947 same commands you used earlier.* [:ref:`...
1948
1949 There are two things worth of note here:
1950
1951 * Occasionally building the kernel will fail
1952 problem in the code at the bisection point.
1953
1954 git bisect skip
1955
1956 Git will then check out another commit near
1957 work better. Afterwards restart executing t
1958
1959 * Those slightly odd looking version identifi
1960 because the Linux kernel subsystems prepare
1961 release (say 6.2) before its predecessor (e
1962 base them on a somewhat earlier point like
1963 get merged for 6.2 without rebasing nor squ
1964 leads to those slightly odd looking version
1965 bisections.
1966
1967 [:ref:`back to step-by-step guide <bisectbuil
1968
1969 .. _bisecttest_bisref:
1970
1971 Bisection checkpoint
1972 ~~~~~~~~~~~~~~~~~~~~
1973
1974 *Check if the feature that regressed works
1975 [:ref:`... <bisecttest_bissbs>`]
1976
1977 Ensure what you tell Git is accurate: getting
1978 the rest of the bisection totally off course,
1979 will be for nothing.
1980
1981 [:ref:`back to step-by-step guide <bisecttest
1982
1983 .. _bisectlog_bisref:
1984
1985 Put the bisection log away
1986 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1987
1988 *Store Git's bisection log and the current
1989 [:ref:`... <bisectlog_bissbs>`]
1990
1991 As indicated above: declaring just one kernel
1992 render the end result of a bisection useless.
1993 to restart the bisection from scratch. The lo
1994 allow someone to point out where a bisection
1995 instead of testing ten or more kernels you mi
1996 resolve things.
1997
1998 The .config file is put aside, as there is a
1999 ask for it after you report the regression.
2000
2001 [:ref:`back to step-by-step guide <bisectlog_
2002
2003 .. _revert_bisref:
2004
2005 Try reverting the culprit
2006 ~~~~~~~~~~~~~~~~~~~~~~~~~
2007
2008 *Try reverting the culprit on top of the la
2009 your regression.* [:ref:`... <revert_bissbs
2010
2011 This is an optional step, but whenever possib
2012 decent chance that developers will ask you to
2013 the bisection result up. So give it a try, yo
2014 one more kernel shouldn't be a big deal at th
2015
2016 The step-by-step guide covers everything rele
2017 rare thing: did you bisected a regression tha
2018 a stable/longterm series, but Git failed to r
2019 try to revert the culprit in the affected sta
2020 succeeds, test that kernel version instead.
2021
2022 [:ref:`back to step-by-step guide <revert_bis
2023
2024 Cleanup steps during and after following this
2025 ---------------------------------------------
2026
2027 *During and after following this guide you
2028 of the kernels you installed.* [:ref:`... <
2029
2030 The steps in this section describe clean-up p
2031
2032 [:ref:`back to step-by-step guide <introclosu
2033
2034 .. _makeroom_bisref:
2035
2036 Cleaning up during the bisection
2037 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2038
2039 *To remove one of the kernels you installed
2040 identifier.* [:ref:`... <makeroom_bissbs>`]
2041
2042 The kernels you install during this process a
2043 parts are only stored in two places and clear
2044 need to worry to mess up your machine when yo
2045 thus bypass your distribution's packaging sys
2046 relatively easy to remove later.
2047
2048 One of the two places is a directory in /lib/
2049 for each installed kernel. This directory is
2050 identifier; hence, to remove all modules for
2051 simply remove its modules directory in /lib/m
2052
2053 The other place is /boot/, where typically tw
2054 during installation of a kernel. All of them
2055 their file name, but how many files and their
2056 your distribution's installkernel executable
2057 some distributions the ``kernel-install remov
2058 step-by-step guide will delete all of these f
2059 the menu entry for the kernel from your bootl
2060 have to take care of these two tasks yourself
2061 interactively remove the three main files of
2062 '6.0-rc1-local-gcafec0cacaca0'::
2063
2064 rm -i /boot/{System.map,vmlinuz,initr}-6.0-
2065
2066 Afterwards check for other files in /boot/ th
2067 '6.0-rc1-local-gcafec0cacaca0' in their name
2068 Now remove the boot entry for the kernel from
2069 the steps to do that vary quite a bit between
2070
2071 Note, be careful with wildcards like '*' when
2072 for kernels manually: you might accidentally
2073 when all you want is to remove 6.0 or 6.0.1.
2074
2075 [:ref:`back to step-by-step guide <makeroom_b
2076
2077 Cleaning up after the bisection
2078 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2079
2080 .. _finishingtouch_bisref:
2081
2082 *Once you have finished the bisection, do n
2083 you set up, as you might need a few things
2084 [:ref:`... <finishingtouch_bissbs>`]
2085
2086 When you are really short of storage space re
2087 the step-by-step guide might not free as much
2088 case consider running ``rm -rf ~/linux/*`` as
2089 build artifacts and the Linux sources, but wi
2090 (~/linux/.git/) behind -- a simple ``git rese
2091 sources back.
2092
2093 Removing the repository as well would likely
2094 is a decent chance developers will ask you to
2095 perform additional tests -- like testing a de
2096 Details on how to perform those can be found
2097 tasks: test reverts, patches, or later versio
2098
2099 Additional tests are also the reason why you
2100 ~/kernel-config-working file around for a few
2101
2102 [:ref:`back to step-by-step guide <finishingt
2103
2104 .. _introoptional_bisref:
2105
2106 Test reverts, patches, or later versions
2107 ----------------------------------------
2108
2109 *While or after reporting a bug, you might
2110 to test reverts, patches, proposed fixes, o
2111 [:ref:`... <introoptional_bissbs>`]
2112
2113 All the commands used in this section should
2114 there is not much to add except one thing: wh
2115 instructed, ensure it is not much longer than
2116 problems will arise if the kernelrelease iden
2117
2118 [:ref:`back to step-by-step guide <introoptio
2119
2120
2121 Additional information
2122 ======================
2123
2124 .. _buildhost_bis:
2125
2126 Build kernels on a different machine
2127 ------------------------------------
2128
2129 To compile kernels on another system, slightl
2130 instructions:
2131
2132 * Start following the guide on the machine wh
2133 the kernels later.
2134
2135 * After executing ':ref:`Boot into the workin
2136 apparently broken feature <bootworking_biss
2137 modules to a file using ``lsmod > ~/test-ma
2138 build configuration for the running kernel
2139 build configuration for your kernel <oldcon
2140 to find it) and store it as '~/test-machine
2141 files to the home directory of your build h
2142
2143 * Continue the guide on the build host (e.g.
2144 free space for building [...] <diskspace_bi
2145
2146 * When you reach ':ref:`Start preparing a ker
2147 <oldconfig_bissbs>`': before running ``make
2148 execute the following command to base your
2149 test machine's 'working' kernel::
2150
2151 cp ~/test-machine-config-working ~/linux/
2152
2153 * During the next step to ':ref:`disable any
2154 modules <localmodconfig_bissbs>`' use the f
2155
2156 yes '' | make localmodconfig LSMOD=~/lsmo
2157
2158 * Continue the guide, but ignore the instruct
2159 install, and reboot into a kernel every tim
2160 like this::
2161
2162 cp ~/kernel-config-working .config
2163 make olddefconfig &&
2164 make -j $(nproc --all) targz-pkg
2165
2166 This will generate a gzipped tar file whose
2167 line shown; for example, a kernel with the
2168 '6.0.0-rc1-local-g928a87efa423' built for x
2169 be stored as '~/linux/linux-6.0.0-rc1-local
2170
2171 Copy that file to your test machine's home
2172
2173 * Switch to the test machine to check if you
2174 kernel. Then extract the file you transferr
2175
2176 sudo tar -xvzf ~/linux-6.0.0-rc1-local-g9
2177
2178 Afterwards :ref:`generate the initramfs and
2179 loader's configuration <install_bisref>`; o
2180 command will take care of both these tasks:
2181
2182 sudo /sbin/installkernel 6.0.0-rc1-local-
2183
2184 Now reboot and ensure you started the inten
2185
2186 This approach even works when building for an
2187 cross-compilers and add the appropriate param
2188 (e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64
2189
2190 Additional reading material
2191 ---------------------------
2192
2193 * The `man page for 'git bisect' <https://git
2194 `fighting regressions with 'git bisect' <ht
2195 in the Git documentation.
2196 * `Working with git bisect <https://nathancha
2197 from kernel developer Nathan Chancellor.
2198 * `Using Git bisect to figure out when broken
2199 * `Fully automated bisecting with 'git bisect
2200
2201 ..
2202 end-of-content
2203 ..
2204 This document is maintained by Thorsten Le<
2205 you spot a typo or small mistake, feel fre
2206 he'll fix it. You are free to do the same
2207 want to contribute changes to the text --
2208 linux-doc@vger.kernel.org and 'sign-off' y
2209 Documentation/process/submitting-patches.r
2210 your work - the Developer's Certificate of
2211 ..
2212 This text is available under GPL-2.0+ or C
2213 of the file. If you want to distribute thi
2214 please use 'The Linux kernel development c
2215 and link this as source:
2216 https://git.kernel.org/pub/scm/linux/kerne
2217
2218 ..
2219 Note: Only the content of this RST file as
2220 is available under CC-BY-4.0, as versions
2221 (for example by the kernel's build system)
2222 files which use a more restrictive license
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.