1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY 2 .. [see the bottom of this file for redistribu 3 4 ========================================= 5 How to verify bugs and bisect regressions 6 ========================================= 7 8 This document describes how to check if some L 9 currently supported by developers -- to then e 10 causing the issue, if it is a regression (e.g. 11 versions). 12 13 The text aims at people running kernels from m 14 commodity hardware who want to report a kernel 15 developers. Despite this intent, the instructi 16 who are already familiar with building their o 17 mistakes occasionally made even by experienced 18 19 .. 20 Note: if you see this note, you are reading 21 might want to switch to a rendered version: 22 read and navigate this document -- especial 23 up in the reference section, then jump back 24 .. 25 Find the latest rendered version of this te 26 https://docs.kernel.org/admin-guide/verify- 27 28 The essence of the process (aka 'TL;DR') 29 ======================================== 30 31 *[If you are new to building or bisecting Linu 32 over to the* ':ref:`step-by-step guide <introg 33 the same commands as this section while descri 34 steps are nevertheless easy to follow and toge 35 in a reference section mention many alternativ 36 aspects, all of which might be essential in yo 37 38 **In case you want to check if a bug is presen 39 developers**, execute just the *preparations* 40 consider the newest Linux kernel you regularly 41 In the following example that's assumed to be 42 will be used to prepare the .config file. 43 44 **In case you face a regression**, follow the 45 *segment 2*. Then you can submit a preliminary 46 *segment 3*, which describes how to perform a 47 full-fledged regression report. In the followi 48 the 'working' kernel and 6.1.5 to be the first 49 will be considered the 'good' release and used 50 51 * **Preparations**: set up everything to build 52 53 # * Remove any software that depends on ex 54 # or builds any automatically during boo 55 # * Ensure Secure Boot permits booting sel 56 # * If you are not already running the 'wo 57 # * Install compilers and everything else 58 # * Ensure to have 15 Gigabyte free space 59 git clone -o mainline --no-checkout \ 60 https://git.kernel.org/pub/scm/linux/ker 61 cd ~/linux/ 62 git remote add -t master stable \ 63 https://git.kernel.org/pub/scm/linux/ker 64 git switch --detach v6.0 65 # * Hint: if you used an existing clone, e 66 make olddefconfig 67 # * Ensure the former command picked the . 68 # * Connect external hardware (USB keys, t 69 # VPNs, mount network shares, and briefl 70 yes '' | make localmodconfig 71 ./scripts/config --set-str CONFIG_LOCALVER 72 ./scripts/config -e CONFIG_LOCALVERSION_AU 73 # * Note, when short on storage space, che 74 ./scripts/config -d DEBUG_INFO_NONE -e KAL 75 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCH 76 # * Hint: at this point you might want to 77 # you'll have to, if you are running Deb 78 make olddefconfig 79 cp .config ~/kernel-config-working 80 81 * **Segment 1**: build a kernel from the lates 82 83 This among others checks if the problem was 84 later need to be told about the problem; in 85 out a .config change as root of the problem. 86 87 a) Checking out latest mainline code:: 88 89 cd ~/linux/ 90 git switch --discard-changes --detach m 91 92 b) Build, install, and boot a kernel:: 93 94 cp ~/kernel-config-working .config 95 make olddefconfig 96 make -j $(nproc --all) 97 # * Make sure there is enough disk spac 98 df -h /boot/ /lib/modules/ 99 # * Note: on Arch Linux, its derivative 100 # the following commands will do noth 101 # job. See the step-by-step guide for 102 sudo make modules_install 103 command -v installkernel && sudo make i 104 # * Check how much space your self-buil 105 # enables you to make better estimate 106 du -ch /boot/*$(make -s kernelrelease)* 107 du -sh /lib/modules/$(make -s kernelrel 108 # * Hint: the output of the following c 109 # right kernel from the boot menu: 110 make -s kernelrelease | tee -a ~/kernel 111 reboot 112 # * Once booted, ensure you are running 113 # checking if the output of the next 114 tail -n 1 ~/kernels-built 115 uname -r 116 cat /proc/sys/kernel/tainted 117 118 c) Check if the problem occurs with this ker 119 120 * **Segment 2**: ensure the 'good' kernel is a 121 122 This among others verifies the trimmed .conf 123 bisecting with it otherwise would be a waste 124 125 a) Start by checking out the sources of the 126 127 cd ~/linux/ 128 git switch --discard-changes --detach v 129 130 b) Build, install, and boot a kernel as desc 131 section b* -- just feel free to skip the 132 estimate already. 133 134 c) Ensure the feature that regressed with th 135 with this one. 136 137 * **Segment 3**: perform and validate the bise 138 139 a) Retrieve the sources for your 'bad' versi 140 141 git remote set-branches --add stable li 142 git fetch stable 143 144 b) Initialize the bisection:: 145 146 cd ~/linux/ 147 git bisect start 148 git bisect good v6.0 149 git bisect bad v6.1.5 150 151 c) Build, install, and boot a kernel as desc 152 section b*. 153 154 In case building or booting the kernel fa 155 ``git bisect skip``. In all other outcome 156 works with the newly built kernel. If it 157 ``git bisect good``; if it does not, run 158 159 All three commands will make Git check ou 160 this step (e.g. build, install, boot, and 161 the outcome). Do so again and again until 162 things. If you run short of disk space du 163 section 'Complementary tasks: cleanup dur 164 below. 165 166 d) Once your finished the bisection, put a f 167 168 cd ~/linux/ 169 git bisect log > ~/bisect-log 170 cp .config ~/bisection-config-culprit 171 git bisect reset 172 173 e) Try to verify the bisection result:: 174 175 git switch --discard-changes --detach m 176 git revert --no-edit cafec0cacaca0 177 cp ~/kernel-config-working .config 178 ./scripts/config --set-str CONFIG_LOCAL 179 180 This is optional, as some commits are impo 181 second command worked flawlessly, build, i 182 kernel; just this time skip the first comm 183 over, as that already has been taken care 184 185 * **Complementary tasks**: cleanup during and 186 187 a) To avoid running out of disk space during 188 remove some kernels you built earlier. Yo 189 you built during segment 1 and 2 around f 190 likely no longer need kernels tested duri 191 (Segment 3 c). You can list them in build 192 193 ls -ltr /lib/modules/*-local* 194 195 To then for example erase a kernel that id 196 '6.0-rc1-local-gcafec0cacaca0', use this:: 197 198 sudo rm -rf /lib/modules/6.0-rc1-local- 199 sudo kernel-install -v remove 6.0-rc1-l 200 # * Note, on some distributions kernel- 201 # or does only part of the job. 202 203 b) If you performed a bisection and successf 204 free to remove all kernels built during t 205 the kernels you built earlier and later y 206 a week or two. 207 208 * **Optional task**: test a debug patch or a p 209 210 git fetch mainline 211 git switch --discard-changes --detach main 212 git apply /tmp/foobars-proposed-fix-v1.pat 213 cp ~/kernel-config-working .config 214 ./scripts/config --set-str CONFIG_LOCALVER 215 216 Build, install, and boot a kernel as describ 217 but this time omit the first command copying 218 as that has been taken care of already. 219 220 .. _introguide_bissbs: 221 222 Step-by-step guide on how to verify bugs and b 223 ============================================== 224 225 This guide describes how to set up your own Li 226 or regressions you intend to report. How far y 227 depends on your issue: 228 229 Execute all steps till the end of *segment 1* 230 is present in code supported by Linux kernel d 231 set to report the bug -- unless it did not hap 232 as then your want to at least continue with *s 233 qualifies as regression** which receive priori 234 outcome you then are ready to report a bug or 235 report; instead of the latter your could also 236 *segment 3* to **perform a bisection** for a f 237 developers are obliged to act upon. 238 239 :ref:`Preparations: set up everything to buil 240 241 :ref:`Segment 1: try to reproduce the problem 242 243 :ref:`Segment 2: check if the kernels you bui 244 245 :ref:`Segment 3: perform a bisection and vali 246 247 :ref:`Complementary tasks: cleanup during and 248 249 :ref:`Optional tasks: test reverts, patches, 250 251 The steps in each segment illustrate the impor 252 a comprehensive reference section holds additi 253 steps. The reference section sometimes also ou 254 pitfalls, as well as problems that might occur 255 to get things rolling again. 256 257 For further details on how to report Linux ker 258 out Documentation/admin-guide/reporting-issues 259 with this document. It among others explains w 260 the latest 'mainline' kernel (e.g. versions li 261 even if you face a problem with a kernel from 262 (say 6.0.13). 263 264 For users facing a regression that document al 265 preliminary report after segment 2 might be wi 266 culprit might be known already. For further de 267 as a regression check out Documentation/admin- 268 269 If you run into any problems while following t 270 improve it, :ref:`please let the kernel develo 271 272 .. _introprep_bissbs: 273 274 Preparations: set up everything to build your 275 ---------------------------------------------- 276 277 The following steps lay the groundwork for all 278 279 Note: the instructions assume you are building 280 machine; if you want to compile the kernel on 281 :ref:`Build kernels on a different machine <bu 282 283 .. _backup_bissbs: 284 285 * Create a fresh backup and put system repair 286 to be prepared for the unlikely case of some 287 288 [:ref:`details <backup_bisref>`] 289 290 .. _vanilla_bissbs: 291 292 * Remove all software that depends on external 293 builds them automatically. That includes but 294 VirtualBox, and Nvidia's graphics drivers (i 295 296 [:ref:`details <vanilla_bisref>`] 297 298 .. _secureboot_bissbs: 299 300 * On platforms with 'Secure Boot' or similar s 301 ensure the system will permit your self-comp 302 quickest and easiest way to achieve this on 303 disable such techniques in the BIOS setup ut 304 their restrictions through a process initiat 305 ``mokutil --disable-validation``. 306 307 [:ref:`details <secureboot_bisref>`] 308 309 .. _rangecheck_bissbs: 310 311 * Determine the kernel versions considered 'go 312 guide: 313 314 * Do you follow this guide to verify if a bu 315 primary developers care for? Then consider 316 you regularly use currently as 'good' (e.g 317 318 * Do you face a regression, e.g. something b 319 switching to a newer kernel version? In th 320 range during which the problem appeared: 321 322 * Something regressed when updating from a 323 (say 6.0.13) to a newer mainline series 324 stable/longterm version based on one (sa 325 mainline release your working kernel is 326 version (e.g. 6.0) and the first version 327 (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at 328 that 6.0 is fine; this hypothesis will b 329 330 * Something regressed when switching from 331 a later one (like 6.1-rc1) or a stable/l 332 (say 6.1.5)? Then regard the last workin 333 the first broken (e.g. 6.1-rc1 or 6.1.5) 334 335 * Something regressed when updating within 336 from 6.0.13 to 6.0.15)? Then consider th 337 (e.g. 6.0.13 and 6.0.15), as you need to 338 339 *Note, do not confuse 'good' version with 'w 340 throughout this guide will refer to the last 341 fine.* 342 343 [:ref:`details <rangecheck_bisref>`] 344 345 .. _bootworking_bissbs: 346 347 * Boot into the 'working' kernel and briefly u 348 349 [:ref:`details <bootworking_bisref>`] 350 351 .. _diskspace_bissbs: 352 353 * Ensure to have enough free space for buildin 354 directory should typically suffice. If you h 355 attention to later steps about retrieving th 356 debug symbols: both explain approaches reduc 357 should allow you to master these tasks with 358 359 [:ref:`details <diskspace_bisref>`] 360 361 .. _buildrequires_bissbs: 362 363 * Install all software required to build a Lin 364 'bc', 'binutils' ('ld' et al.), 'bison', 'fl 365 'pahole', 'perl', and the development header 366 reference section shows how to quickly insta 367 distributions. 368 369 [:ref:`details <buildrequires_bisref>`] 370 371 .. _sources_bissbs: 372 373 * Retrieve the mainline Linux sources; then ch 374 them, as all further commands in this guide 375 there. 376 377 *Note, the following describe how to retriev 378 mainline clone, which downloads about 2,75 G 379 :ref:`reference section describes two altern 380 one downloads less than 500 MByte, the other 381 internet connections.* 382 383 Execute the following command to retrieve a 384 preparing things to add branches for stable/ 385 386 git clone -o mainline --no-checkout \ 387 https://git.kernel.org/pub/scm/linux/ker 388 cd ~/linux/ 389 git remote add -t master stable \ 390 https://git.kernel.org/pub/scm/linux/ker 391 392 [:ref:`details <sources_bisref>`] 393 394 .. _stablesources_bissbs: 395 396 * Is one of the versions you earlier establish 397 longterm release (say 6.1.5)? Then download 398 to ('linux-6.1.y' in this example):: 399 400 git remote set-branches --add stable linux 401 git fetch stable 402 403 .. _oldconfig_bissbs: 404 405 * Start preparing a kernel build configuration 406 407 Before doing so, ensure you are still runnin 408 step told you to boot; if you are unsure, ch 409 identifier using ``uname -r``. 410 411 Afterwards check out the source code for the 412 'good'. In the following example command thi 413 the version number in this and all later Git 414 with a 'v':: 415 416 git switch --discard-changes --detach v6.0 417 418 Now create a build configuration file:: 419 420 make olddefconfig 421 422 The kernel build scripts then will try to lo 423 for the running kernel and then adjust it fo 424 you checked out. While doing so, it will pri 425 426 Look out for a line starting with '# using d 427 followed by a path to a file in '/boot/' tha 428 of your currently working kernel. If the lin 429 like 'arch/x86/configs/x86_64_defconfig', th 430 the .config file for your running kernel -- 431 there manually, as explained in the referenc 432 433 In case you can not find such a line, look f 434 written to .config'. If that's the case you 435 lying around. Unless you intend to use it, d 436 'make olddefconfig' again and check if it no 437 as base. 438 439 [:ref:`details <oldconfig_bisref>`] 440 441 .. _localmodconfig_bissbs: 442 443 * Disable any kernel modules apparently superf 444 optional, but especially wise for bisections 445 process enormously -- at least unless the .c 446 previous step was already tailored to your a 447 case you should skip this step. 448 449 To prepare the trimming, connect external ha 450 keys, tokens, ...), quickly start a VM, and 451 since you started that guide, ensure that yo 452 trouble since you started the system. Only t 453 454 yes '' | make localmodconfig 455 456 There is a catch to this, as the 'apparently 457 and the preparation instructions already hin 458 459 The 'localmodconfig' target easily disables 460 used occasionally -- like modules for extern 461 since booting, virtualization software not y 462 few other things. That's because some tasks 463 loads when you execute tasks like the aforem 464 465 This drawback of localmodconfig is nothing y 466 something to keep in mind: if something is m 467 during this guide, this is most likely the r 468 eliminate the risk with tricks outlined in t 469 building a kernel just for quick testing pur 470 spending much effort on, as long as it boots 471 feature that causes trouble. 472 473 [:ref:`details <localmodconfig_bisref>`] 474 475 .. _tagging_bissbs: 476 477 * Ensure all the kernels you will build are cl 478 tag and a unique version number:: 479 480 ./scripts/config --set-str CONFIG_LOCALVER 481 ./scripts/config -e CONFIG_LOCALVERSION_AU 482 483 [:ref:`details <tagging_bisref>`] 484 485 .. _debugsymbols_bissbs: 486 487 * Decide how to handle debug symbols. 488 489 In the context of this document it is often 490 decent chance you will need to decode a stac 491 'warning', or 'BUG':: 492 493 ./scripts/config -d DEBUG_INFO_NONE -e KAL 494 -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCH 495 496 But if you are extremely short on storage sp 497 debug symbols instead:: 498 499 ./scripts/config -d DEBUG_INFO -d DEBUG_IN 500 -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF 501 502 [:ref:`details <debugsymbols_bisref>`] 503 504 .. _configmods_bissbs: 505 506 * Check if you may want or need to adjust some 507 options: 508 509 * Are you running Debian? Then you want to a 510 additional adjustments explained in the re 511 512 [:ref:`details <configmods_distros_bisref> 513 514 * If you want to influence other aspects of 515 your preferred tool. Note, to use make tar 516 'nconfig', you will need to install the de 517 'xconfig' you likewise need the Qt5 or Qt6 518 519 [:ref:`details <configmods_individual_bisr 520 521 .. _saveconfig_bissbs: 522 523 * Reprocess the .config after the latest adjus 524 place:: 525 526 make olddefconfig 527 cp .config ~/kernel-config-working 528 529 [:ref:`details <saveconfig_bisref>`] 530 531 .. _introlatestcheck_bissbs: 532 533 Segment 1: try to reproduce the problem with t 534 ---------------------------------------------- 535 536 The following steps verify if the problem occu 537 supported by developers. In case you face a re 538 problem is not caused by some .config change, 539 be a waste of time. [:ref:`details <introlates 540 541 .. _checkoutmaster_bissbs: 542 543 * Check out the latest Linux codebase. 544 545 * Are your 'good' and 'bad' versions from th 546 Then check the `front page of kernel.org < 547 lists a release from that series without a 548 latest version ('linux-6.1.y' in the follo 549 550 cd ~/linux/ 551 git switch --discard-changes --detach st 552 553 Your series is unsupported, if is not list 554 tag. In that case you might want to check 555 linux-6.2.y) or mainline (see next point) 556 557 * In all other cases, run:: 558 559 cd ~/linux/ 560 git switch --discard-changes --detach ma 561 562 [:ref:`details <checkoutmaster_bisref>`] 563 564 .. _build_bissbs: 565 566 * Build the image and the modules of your firs 567 prepared:: 568 569 cp ~/kernel-config-working .config 570 make olddefconfig 571 make -j $(nproc --all) 572 573 If you want your kernel packaged up as deb, 574 reference section for alternatives, which ob 575 steps to install as well. 576 577 [:ref:`details <build_bisref>`] 578 579 .. _install_bissbs: 580 581 * Install your newly built kernel. 582 583 Before doing so, consider checking if there 584 585 df -h /boot/ /lib/modules/ 586 587 For now assume 150 MByte in /boot/ and 200 i 588 much your kernels actually require will be d 589 590 Now install the kernel's modules and its ima 591 parallel to the your Linux distribution's ke 592 593 sudo make modules_install 594 command -v installkernel && sudo make inst 595 596 The second command ideally will take care of 597 point: copying the kernel's image to /boot/, 598 adding an entry for both to the boot loader' 599 600 Sadly some distributions (among them Arch Li 601 immutable Linux distributions) will perform 602 You therefore want to check if all of them w 603 perform those that were not. The reference s 604 that; your distribution's documentation migh 605 606 Once you figured out the steps needed at thi 607 down: if you will build more kernels as desc 608 have to perform those again after executing 609 610 [:ref:`details <install_bisref>`] 611 612 .. _storagespace_bissbs: 613 614 * In case you plan to follow this guide furthe 615 the kernel, its modules, and other related f 616 617 du -ch /boot/*$(make -s kernelrelease)* | 618 du -sh /lib/modules/$(make -s kernelreleas 619 620 Write down or remember those two values for 621 running out of disk space accidentally durin 622 623 [:ref:`details <storagespace_bisref>`] 624 625 .. _kernelrelease_bissbs: 626 627 * Show and store the kernelrelease identifier 628 629 make -s kernelrelease | tee -a ~/kernels-b 630 631 Remember the identifier momentarily, as it w 632 from the boot menu upon restarting. 633 634 * Reboot into your newly built kernel. To ensu 635 you just built, you might want to verify if 636 matches:: 637 638 tail -n 1 ~/kernels-built 639 uname -r 640 641 .. _tainted_bissbs: 642 643 * Check if the kernel marked itself as 'tainte 644 645 cat /proc/sys/kernel/tainted 646 647 If that command does not return '0', check t 648 for this might interfere with your testing. 649 650 [:ref:`details <tainted_bisref>`] 651 652 .. _recheckbroken_bissbs: 653 654 * Verify if your bug occurs with the newly bui 655 out the instructions in the reference sectio 656 during your tests. 657 658 [:ref:`details <recheckbroken_bisref>`] 659 660 .. _recheckstablebroken_bissbs: 661 662 * Did you just built a stable or longterm kern 663 the regression with it? Then you should test 664 well, because the result determines which de 665 to. 666 667 To prepare that test, check out current main 668 669 cd ~/linux/ 670 git switch --discard-changes --detach main 671 672 Now use the checked out code to build and in 673 commands the earlier steps already described 674 675 cp ~/kernel-config-working .config 676 make olddefconfig 677 make -j $(nproc --all) 678 # * Check if the free space suffices holdi 679 df -h /boot/ /lib/modules/ 680 sudo make modules_install 681 command -v installkernel && sudo make inst 682 make -s kernelrelease | tee -a ~/kernels-b 683 reboot 684 685 Confirm you booted the kernel you intended t 686 status:: 687 688 tail -n 1 ~/kernels-built 689 uname -r 690 cat /proc/sys/kernel/tainted 691 692 Now verify if this kernel is showing the pro 693 to report the bug to the primary developers; 694 stable team. See Documentation/admin-guide/r 695 696 [:ref:`details <recheckstablebroken_bisref>` 697 698 Do you follow this guide to verify if a proble 699 currently supported by Linux kernel developers 700 point. If you later want to remove the kernel 701 :ref:`Complementary tasks: cleanup during and 702 703 In case you face a regression, move on and exe 704 as well. 705 706 .. _introworkingcheck_bissbs: 707 708 Segment 2: check if the kernels you build work 709 ---------------------------------------------- 710 711 In case of a regression, you now want to ensur 712 you created earlier works as expected; a bisec 713 otherwise would be a waste of time. [:ref:`det 714 715 .. _recheckworking_bissbs: 716 717 * Build your own variant of the 'working' kern 718 regressed works as expected with it. 719 720 Start by checking out the sources for the ve 721 'good' (once again assumed to be 6.0 here):: 722 723 cd ~/linux/ 724 git switch --discard-changes --detach v6.0 725 726 Now use the checked out code to configure, b 727 using the commands the previous subsection e 728 729 cp ~/kernel-config-working .config 730 make olddefconfig 731 make -j $(nproc --all) 732 # * Check if the free space suffices holdi 733 df -h /boot/ /lib/modules/ 734 sudo make modules_install 735 command -v installkernel && sudo make inst 736 make -s kernelrelease | tee -a ~/kernels-b 737 reboot 738 739 When the system booted, you may want to veri 740 kernel you started is the one you just built 741 742 tail -n 1 ~/kernels-built 743 uname -r 744 745 Now check if this kernel works as expected; 746 section for further instructions. 747 748 [:ref:`details <recheckworking_bisref>`] 749 750 .. _introbisect_bissbs: 751 752 Segment 3: perform the bisection and validate 753 ---------------------------------------------- 754 755 With all the preparations and precaution build 756 to begin the bisection. This will make you bui 757 about 15 in case you encountered a regression 758 (say from 6.0.13 to 6.1.5). But do not worry, 759 configuration created earlier this works a lot 760 overall on average it will often just take abo 761 each kernel on commodity x86 machines. 762 763 .. _bisectstart_bissbs: 764 765 * Start the bisection and tell Git about the v 766 'good' (6.0 in the following example command 767 768 cd ~/linux/ 769 git bisect start 770 git bisect good v6.0 771 git bisect bad v6.1.5 772 773 [:ref:`details <bisectstart_bisref>`] 774 775 .. _bisectbuild_bissbs: 776 777 * Now use the code Git checked out to build, i 778 the commands introduced earlier:: 779 780 cp ~/kernel-config-working .config 781 make olddefconfig 782 make -j $(nproc --all) 783 # * Check if the free space suffices holdi 784 df -h /boot/ /lib/modules/ 785 sudo make modules_install 786 command -v installkernel && sudo make inst 787 make -s kernelrelease | tee -a ~/kernels-b 788 reboot 789 790 If compilation fails for some reason, run `` 791 executing the stack of commands from the beg 792 793 In case you skipped the 'test latest codebas 794 description as for why the 'df [...]' and 'm 795 commands are here. 796 797 Important note: the latter command from this 798 identifiers that might look odd or wrong to 799 totally normal to see release identifiers li 800 if you bisect between versions 6.1 and 6.2 f 801 802 [:ref:`details <bisectbuild_bisref>`] 803 804 .. _bisecttest_bissbs: 805 806 * Now check if the feature that regressed work 807 808 You again might want to start by making sure 809 you just built:: 810 811 cd ~/linux/ 812 tail -n 1 ~/kernels-built 813 uname -r 814 815 Now verify if the feature that regressed wor 816 If it does, run this:: 817 818 git bisect good 819 820 If it does not, run this:: 821 822 git bisect bad 823 824 Be sure about what you tell Git, as getting 825 rest of the bisection totally off course. 826 827 While the bisection is ongoing, Git will use 828 find and check out another bisection point f 829 will print something like 'Bisecting: 675 re 830 (roughly 10 steps)' to indicate how many fur 831 tested. Now build and install another kernel 832 previous step; afterwards follow the instruc 833 834 Repeat this again and again until you finish 835 when Git after tagging a change as 'good' or 836 'cafecaca0c0dacafecaca0c0dacafecaca0c0da is 837 afterwards it will show some details about t 838 description of the change. The latter might 839 might need to scroll up to see the message m 840 alternatively, run ``git bisect log > ~/bise 841 842 [:ref:`details <bisecttest_bisref>`] 843 844 .. _bisectlog_bissbs: 845 846 * Store Git's bisection log and the current .c 847 telling Git to reset the sources to the stat 848 849 cd ~/linux/ 850 git bisect log > ~/bisection-log 851 cp .config ~/bisection-config-culprit 852 git bisect reset 853 854 [:ref:`details <bisectlog_bisref>`] 855 856 .. _revert_bissbs: 857 858 * Try reverting the culprit on top of latest m 859 regression. 860 861 This is optional, as it might be impossible 862 the case, if the bisection determined a merg 863 latter happens if other changes depend on th 864 succeeds, it is worth building another kerne 865 a bisection, which can easily deroute; it fu 866 developers know, if they can resolve the reg 867 868 Begin by checking out the latest codebase de 869 870 * Did you face a regression within a stable/ 871 6.0.13 and 6.0.15) that does not happen in 872 latest codebase for the affected series li 873 874 git fetch stable 875 git switch --discard-changes --detach li 876 877 * In all other cases check out latest mainli 878 879 git fetch mainline 880 git switch --discard-changes --detach ma 881 882 If you bisected a regression within a stab 883 happens in mainline, there is one more thi 884 commit-id. To do so, use a command like `` 885 view the patch description of the culprit. 886 the top which looks like 'commit cafec0cac 887 'Upstream commit cafec0cacaca0'; use that 888 and not the one the bisection blamed. 889 890 Now try reverting the culprit by specifying 891 892 git revert --no-edit cafec0cacaca0 893 894 If that fails, give up trying and move on to 895 adjust the tag to facilitate the identificat 896 overwriting another kernel:: 897 898 cp ~/kernel-config-working .config 899 ./scripts/config --set-str CONFIG_LOCALVER 900 901 Build a kernel using the familiar command se 902 the base .config over:: 903 904 make olddefconfig && 905 make -j $(nproc --all) 906 # * Check if the free space suffices holdi 907 df -h /boot/ /lib/modules/ 908 sudo make modules_install 909 command -v installkernel && sudo make inst 910 make -s kernelrelease | tee -a ~/kernels-b 911 reboot 912 913 Now check one last time if the feature that 914 with that kernel: if everything went well, i 915 916 [:ref:`details <revert_bisref>`] 917 918 .. _introclosure_bissbs: 919 920 Complementary tasks: cleanup during and after 921 ---------------------------------------------- 922 923 During and after following this guide you migh 924 the kernels you installed: the boot menu other 925 space might run out. 926 927 .. _makeroom_bissbs: 928 929 * To remove one of the kernels you installed, 930 identifier. This guide stores them in '~/ker 931 command will print them as well:: 932 933 ls -ltr /lib/modules/*-local* 934 935 You in most situations want to remove the ol 936 actual bisection (e.g. segment 3 of this gui 937 beforehand (e.g. to test the latest codebase 938 'good') might become handy to verify somethi 939 around, unless you are really short on stora 940 941 To remove the modules of a kernel with the k 942 '*6.0-rc1-local-gcafec0cacaca0*', start by r 943 modules:: 944 945 sudo rm -rf /lib/modules/6.0-rc1-local-gca 946 947 Afterwards try the following command:: 948 949 sudo kernel-install -v remove 6.0-rc1-loca 950 951 On quite a few distributions this will delet 952 while also removing the kernel's entry from 953 distributions kernel-install does not exist 954 kernel image and related files behind; in th 955 in the reference section. 956 957 [:ref:`details <makeroom_bisref>`] 958 959 .. _finishingtouch_bissbs: 960 961 * Once you have finished the bisection, do not 962 set up, as you might need a few things again 963 on the outcome of the bisection: 964 965 * Could you initially reproduce the regressi 966 after the bisection were able to fix the p 967 top of the latest codebase? Then you want 968 for a while, but safely remove all others 969 identifier. 970 971 * Did the bisection end on a merge-commit or 972 reasons? Then you want to keep as many ker 973 days: it's pretty likely that you will be 974 975 * In other cases it likely is a good idea to 976 for some time: the one built from the late 977 the version considered 'good', and the las 978 during the actual bisection process. 979 980 [:ref:`details <finishingtouch_bisref>`] 981 982 .. _introoptional_bissbs: 983 984 Optional: test reverts, patches, or later vers 985 ---------------------------------------------- 986 987 While or after reporting a bug, you might want 988 test reverts, debug patches, proposed fixes, o 989 follow these instructions. 990 991 * Update your Git clone and check out the late 992 993 * In case you want to test mainline, fetch i 994 its code out:: 995 996 git fetch mainline 997 git switch --discard-changes --detach ma 998 999 * In case you want to test a stable or longt 1000 holding the series you are interested in 1001 already did so earlier:: 1002 1003 git remote set-branches --add stable li 1004 1005 Then fetch the latest changes and check o 1006 series:: 1007 1008 git fetch stable 1009 git switch --discard-changes --detach s 1010 1011 * Copy your kernel build configuration over:: 1012 1013 cp ~/kernel-config-working .config 1014 1015 * Your next step depends on what you want to 1016 1017 * In case you just want to test the latest 1018 you are already all set. 1019 1020 * In case you want to test if a revert fixe 1021 changes by specifying their commit ids:: 1022 1023 git revert --no-edit cafec0cacaca0 1024 1025 Now give that kernel a special tag to fac 1026 prevent accidentally overwriting another 1027 1028 ./scripts/config --set-str CONFIG_LOCAL 1029 1030 * In case you want to test a patch, store t 1031 '/tmp/foobars-proposed-fix-v1.patch' and 1032 1033 git apply /tmp/foobars-proposed-fix-v1. 1034 1035 In case of multiple patches, repeat this 1036 1037 Now give that kernel a special tag to fac 1038 prevent accidentally overwriting another 1039 1040 ./scripts/config --set-str CONFIG_LOCALVE 1041 1042 * Build a kernel using the familiar commands, 1043 build configuration over, as that has been 1044 1045 make olddefconfig && 1046 make -j $(nproc --all) 1047 # * Check if the free space suffices hold 1048 df -h /boot/ /lib/modules/ 1049 sudo make modules_install 1050 command -v installkernel && sudo make ins 1051 make -s kernelrelease | tee -a ~/kernels- 1052 reboot 1053 1054 * Now verify you booted the newly built kerne 1055 1056 [:ref:`details <introoptional_bisref>`] 1057 1058 .. _submit_improvements: 1059 1060 Conclusion 1061 ---------- 1062 1063 You have reached the end of the step-by-step 1064 1065 Did you run into trouble following any of the 1066 reference section below? Did you spot errors? 1067 improve the guide? 1068 1069 If any of that applies, please take a moment 1070 document know by email (Thorsten Leemhuis <lin 1071 CCing the Linux docs mailing list (linux-doc@ 1072 vital to improve this text further, which is 1073 will enable more people to master the task de 1074 improve similar guides inspired by this one. 1075 1076 1077 Reference section for the step-by-step guide 1078 ============================================ 1079 1080 This section holds additional information for 1081 step-by-step guide. 1082 1083 Preparations for building your own kernels 1084 ------------------------------------------ 1085 1086 *The steps in this section lay the groundwo 1087 [:ref:`... <introprep_bissbs>`] 1088 1089 The steps in all later sections of this guide 1090 1091 [:ref:`back to step-by-step guide <introprep_ 1092 1093 .. _backup_bisref: 1094 1095 Prepare for emergencies 1096 ~~~~~~~~~~~~~~~~~~~~~~~ 1097 1098 *Create a fresh backup and put system repai 1099 [:ref:`... <backup_bissbs>`] 1100 1101 Remember, you are dealing with computers, whi 1102 -- especially if you fiddle with crucial part 1103 system. That's what you are about to do in th 1104 for something going sideways, even if that sh 1105 1106 [:ref:`back to step-by-step guide <backup_bis 1107 1108 .. _vanilla_bisref: 1109 1110 Remove anything related to externally maintai 1111 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1112 1113 *Remove all software that depends on extern 1114 builds them automatically.* [:ref:`...<vani 1115 1116 Externally developed kernel modules can easil 1117 1118 But there is a more important reason why this 1119 kernel developers will not care about reports 1120 kernels that utilize such modules. That's bec 1121 considered 'vanilla' anymore, as Documentatio 1122 explains in more detail. 1123 1124 [:ref:`back to step-by-step guide <vanilla_bi 1125 1126 .. _secureboot_bisref: 1127 1128 Deal with techniques like Secure Boot 1129 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1130 1131 *On platforms with 'Secure Boot' or similar 1132 ensure the system will permit your self-com 1133 [:ref:`... <secureboot_bissbs>`] 1134 1135 Many modern systems allow only certain operat 1136 they reject booting self-compiled kernels by 1137 1138 You ideally deal with this by making your pla 1139 with the help of a certificate. How to do tha 1140 here, as it requires various steps that would 1141 its purpose; 'Documentation/admin-guide/modul 1142 sides already explain everything needed in mo 1143 1144 Temporarily disabling solutions like Secure B 1145 Linux boot. On commodity x86 systems it is po 1146 utility; the required steps vary a lot betwee 1147 described here. 1148 1149 On mainstream x86 Linux distributions there i 1150 disable all Secure Boot restrictions for your 1151 initiate this process by running ``mokutil -- 1152 tell you to create a one-time password, which 1153 restart; right after your BIOS performed all 1154 show a blue box with a message 'Press any key 1155 some key before the countdown exposes, which 1156 Secure Boot state'. Shim's 'MokManager' will 1157 randomly chosen characters from the one-time 1158 you provided them, confirm you really want to 1159 Afterwards, permit MokManager to reboot the m 1160 1161 [:ref:`back to step-by-step guide <secureboot 1162 1163 .. _bootworking_bisref: 1164 1165 Boot the last kernel that was working 1166 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1167 1168 *Boot into the last working kernel and brie 1169 regressed really works.* [:ref:`...<bootwor 1170 1171 This will make later steps that cover creatin 1172 the right thing. 1173 1174 [:ref:`back to step-by-step guide <bootworkin 1175 1176 .. _diskspace_bisref: 1177 1178 Space requirements 1179 ~~~~~~~~~~~~~~~~~~ 1180 1181 *Ensure to have enough free space for build 1182 [:ref:`... <diskspace_bissbs>`] 1183 1184 The numbers mentioned are rough estimates wit 1185 safe side, so often you will need less. 1186 1187 If you have space constraints, be sure to hay 1188 debug symbols' <debugsymbols_bissbs>` and its 1189 section' <debugsymbols_bisref>`, as disabling 1190 space by quite a few gigabytes. 1191 1192 [:ref:`back to step-by-step guide <diskspace_ 1193 1194 .. _rangecheck_bisref: 1195 1196 Bisection range 1197 ~~~~~~~~~~~~~~~ 1198 1199 *Determine the kernel versions considered ' 1200 guide.* [:ref:`...<rangecheck_bissbs>`] 1201 1202 Establishing the range of commits to be check 1203 except when a regression occurred when switch 1204 series to a release of a later series (e.g. f 1205 Git will need some hand holding, as there is 1206 1207 That's because with the release of 6.0 mainli 1208 stable series 6.0.y branched to the side. It' 1209 that the issue you face with 6.1.5 only worke 1210 commit that went into one of the 6.0.y releas 1211 6.1.y series. Thankfully that normally should 1212 stable/longterm maintainers maintain the code 1213 6.0 as a 'good' kernel. That assumption will 1214 will be built and tested in the segment '2' o 1215 to do this as well, if you tried bisecting be 1216 1217 [:ref:`back to step-by-step guide <rangecheck 1218 1219 .. _buildrequires_bisref: 1220 1221 Install build requirements 1222 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1223 1224 *Install all software required to build a L 1225 [:ref:`...<buildrequires_bissbs>`] 1226 1227 The kernel is pretty stand-alone, but besides 1228 sometimes need a few libraries to build one. 1229 depends on your Linux distribution and the co 1230 about to build. 1231 1232 Here are a few examples what you typically ne 1233 distributions: 1234 1235 * Arch Linux and derivatives:: 1236 1237 sudo pacman --needed -S bc binutils bison 1238 pahole perl zlib ncurses qt6-base 1239 1240 * Debian, Ubuntu, and derivatives:: 1241 1242 sudo apt install bc binutils bison dwarve 1243 libssl-dev make openssl pahole perl-bas 1244 libncurses-dev qt6-base-dev g++ 1245 1246 * Fedora and derivatives:: 1247 1248 sudo dnf install binutils \ 1249 /usr/bin/{bc,bison,flex,gcc,git,openssl 1250 /usr/include/{libelf.h,openssl/pkcs7.h, 1251 1252 * openSUSE and derivatives:: 1253 1254 sudo zypper install bc binutils bison dwa 1255 kernel-install-tools libelf-devel make 1256 perl-base zlib-devel rpm-build ncurses- 1257 1258 These commands install a few packages that ar 1259 for example might want to skip installing the 1260 which you will only need in case you later mi 1261 configuration using make the targets 'menucon 1262 the headers of Qt6 if you do not plan to adju 1263 1264 You furthermore might need additional librari 1265 for tasks not covered in this guide -- for ex 1266 the kernel's tools/ directory. 1267 1268 [:ref:`back to step-by-step guide <buildrequi 1269 1270 .. _sources_bisref: 1271 1272 Download the sources using Git 1273 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1274 1275 *Retrieve the Linux mainline sources.* 1276 [:ref:`...<sources_bissbs>`] 1277 1278 The step-by-step guide outlines how to downlo 1279 Git clone of Linus' mainline repository. Ther 1280 that -- but there are two alternatives ways t 1281 work better for you: 1282 1283 * If you have an unreliable internet connecti 1284 :ref:`using a 'Git bundle'<sources_bundle_b 1285 1286 * If downloading the complete repository woul 1287 much storage space, consider :ref:`using a 1288 clone'<sources_shallow_bisref>`. 1289 1290 .. _sources_bundle_bisref: 1291 1292 Downloading Linux mainline sources using a bu 1293 """"""""""""""""""""""""""""""""""""""""""""" 1294 1295 Use the following commands to retrieve the Li 1296 bundle:: 1297 1298 wget -c \ 1299 https://git.kernel.org/pub/scm/linux/ke 1300 git clone --no-checkout clone.bundle ~/li 1301 cd ~/linux/ 1302 git remote remove origin 1303 git remote add mainline \ 1304 https://git.kernel.org/pub/scm/linux/ke 1305 git fetch mainline 1306 git remote add -t master stable \ 1307 https://git.kernel.org/pub/scm/linux/ke 1308 1309 In case the 'wget' command fails, just re-exe 1310 it left off. 1311 1312 [:ref:`back to step-by-step guide <sources_bi 1313 [:ref:`back to section intro <sources_bisref> 1314 1315 .. _sources_shallow_bisref: 1316 1317 Downloading Linux mainline sources using a sh 1318 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1319 1320 First, execute the following command to retri 1321 1322 git clone -o mainline --no-checkout --dep 1323 https://git.kernel.org/pub/scm/linux/ke 1324 cd ~/linux/ 1325 git remote add -t master stable \ 1326 https://git.kernel.org/pub/scm/linux/ke 1327 1328 Now deepen your clone's history to the second 1329 release of your 'good' version. In case the l 1330 be the first predecessor and 5.18 the second 1331 that version:: 1332 1333 git fetch --shallow-exclude=v5.18 mainlin 1334 1335 Afterwards add the stable Git repository as r 1336 branches as explained in the step-by-step gui 1337 1338 Note, shallow clones have a few peculiar char 1339 1340 * For bisections the history needs to be deep 1341 farther than it seems necessary, as explain 1342 Git otherwise will be unable to revert or d 1343 a range (say 6.1..6.2), as they are interna 1344 releases (like 6.0-rc2 or 5.19-rc3). 1345 1346 * This document in most places uses ``git fet 1347 to specify the earliest version you care ab 1348 tag). You alternatively can use the paramet 1349 an absolute (say ``'2023-07-15'``) or relat 1350 define the depth of the history you want to 1351 bisecting mainline, ensure to deepen the hi 1352 the release of the mainline release your 'g 1353 1354 * Be warned, when deepening your clone you mi 1355 'fatal: error in object: unshallow cafecaca 1356 In that case run ``git repack -d`` and try 1357 1358 [:ref:`back to step-by-step guide <sources_bi 1359 [:ref:`back to section intro <sources_bisref> 1360 1361 .. _oldconfig_bisref: 1362 1363 Start defining the build configuration for yo 1364 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1365 1366 *Start preparing a kernel build configurati 1367 [:ref:`... <oldconfig_bissbs>`] 1368 1369 *Note, this is the first of multiple steps in 1370 build artifacts. The commands used in this gu 1371 tree to keep things simple. In case you prefe 1372 separately, create a directory like '~/linux- 1373 ``O=~/linux-builddir/`` to all make calls use 1374 have to point other commands there as well -- 1375 [...]`` commands, which will require ``--file 1376 locate the right build configuration.* 1377 1378 Two things can easily go wrong when creating 1379 1380 * The oldconfig target will use a .config fil 1381 one is already present there (e.g. '~/linux 1382 that's what you intend (see next step), but 1383 delete it. This for example is important in 1384 further, but due to problems come back here 1385 scratch. 1386 1387 * Sometimes olddefconfig is unable to locate 1388 kernel and will use defaults, as briefly ou 1389 check if your distribution ships the config 1390 it in the right place (e.g. '~/linux/.confi 1391 where /proc/config.gz exists this can be ac 1392 1393 zcat /proc/config.gz > .config 1394 1395 Once you put it there, run ``make olddefcon 1396 needs of the kernel about to be built. 1397 1398 Note, the olddefconfig target will set any un 1399 default value. If you prefer to set such conf 1400 ``make oldconfig`` instead. Then for each und 1401 will be asked how to proceed; in case you are 1402 'enter' to apply the default value. Note thou 1403 want to go with the defaults, as you otherwis 1404 causes a problem looking like regressions (fo 1405 restrictions). 1406 1407 Occasionally odd things happen when trying to 1408 kernel (say 6.1) on an older mainline release 1409 (say 5.15). That's one of the reasons why the 1410 you to boot the kernel where everything works 1411 file you thus want to ensure it's from the wo 1412 that shows the regression. 1413 1414 In case you want to build kernels for another 1415 configuration; usually ``ls /boot/config-$(un 1416 that file to the build machine and store it a 1417 ``make olddefconfig`` to adjust it. 1418 1419 [:ref:`back to step-by-step guide <oldconfig_ 1420 1421 .. _localmodconfig_bisref: 1422 1423 Trim the build configuration for your kernel 1424 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1425 1426 *Disable any kernel modules apparently supe 1427 [:ref:`... <localmodconfig_bissbs>`] 1428 1429 As explained briefly in the step-by-step guid 1430 can easily happen that your self-built kernel 1431 did not perform at least once before utilizin 1432 when a task requires kernel modules which are 1433 it for the first time. So when you never perf 1434 kernel the modules will not have been loaded 1435 of view look superfluous, which thus disables 1436 to be compiled. 1437 1438 You can try to avoid this by performing typic 1439 additional kernel modules: start a VM, establ 1440 CD/DVD ISO, mount network shares (CIFS, NFS, 1441 devices (2FA keys, headsets, webcams, ...) as 1442 systems you otherwise do not utilize (btrfs, 1443 is hard to think of everything that might be 1444 often forget one thing or another at this poi 1445 1446 Do not let that risk bother you, especially w 1447 testing purposes: everything typically crucia 1448 something important you can turn on a missing 1449 run the commands again to compile and install 1450 need. 1451 1452 But if you plan to build and use self-built k 1453 reduce the risk by recording which modules yo 1454 a few weeks. You can automate this with `modp 1455 <https://github.com/graysky2/modprobed-db>`_. 1456 point localmodconfig to the list of modules m 1457 1458 yes '' | make LSMOD='${HOME}'/.config/modpr 1459 1460 That parameter also allows you to build trimm 1461 case you copied a suitable .config over to us 1462 run ``lsmod > lsmod_foo-machine`` on that sys 1463 your build's host home directory. Then run th 1464 step-by-step guide mentions:: 1465 1466 yes '' | make LSMOD=~/lsmod_foo-machine loc 1467 1468 [:ref:`back to step-by-step guide <localmodco 1469 1470 .. _tagging_bisref: 1471 1472 Tag the kernels about to be build 1473 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1474 1475 *Ensure all the kernels you will build are 1476 special tag and a unique version identifier 1477 1478 This allows you to differentiate your distrib 1479 during this process, as the file or directori 1480 '-local' in the name; it also helps picking t 1481 not lose track of you kernels, as their versi 1482 confusing during the bisection. 1483 1484 [:ref:`back to step-by-step guide <tagging_bi 1485 1486 .. _debugsymbols_bisref: 1487 1488 Decide to enable or disable debug symbols 1489 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1490 1491 *Decide how to handle debug symbols.* [:ref 1492 1493 Having debug symbols available can be importa 1494 'panic', 'Oops', 'warning', or 'BUG' later wh 1495 able to find the exact place where the proble 1496 collecting and embedding the needed debug inf 1497 quite a bit of space: in late 2022 the build 1498 trimmed with localmodconfig consumed around 5 1499 symbols, but less than 1 when they were disab 1500 modules are bigger as well, which increases s 1501 load times. 1502 1503 In case you want a small kernel and are unlik 1504 you thus might want to disable debug symbols 1505 later turns out that you need them, just enab 1506 kernel. 1507 1508 You on the other hand definitely want to enab 1509 is a decent chance that you need to decode a 1510 'Decode failure messages' in Documentation/ad 1511 explains this process in more detail. 1512 1513 [:ref:`back to step-by-step guide <debugsymbo 1514 1515 .. _configmods_bisref: 1516 1517 Adjust build configuration 1518 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1519 1520 *Check if you may want or need to adjust so 1521 options:* 1522 1523 Depending on your needs you at this point mig 1524 kernel configuration options. 1525 1526 .. _configmods_distros_bisref: 1527 1528 Distro specific adjustments 1529 """"""""""""""""""""""""""" 1530 1531 *Are you running* [:ref:`... <configmods_bi 1532 1533 The following sections help you to avoid buil 1534 when following this guide on a few commodity 1535 1536 **Debian:** 1537 1538 * Remove a stale reference to a certificate f 1539 fail:: 1540 1541 ./scripts/config --set-str SYSTEM_TRUSTED_ 1542 1543 Alternatively, download the needed certific 1544 option point to it, as `the Debian handbook 1545 <https://debian-handbook.info/browse/stable 1546 -- or generate your own, as explained in 1547 Documentation/admin-guide/module-signing.rs 1548 1549 [:ref:`back to step-by-step guide <configmods 1550 1551 .. _configmods_individual_bisref: 1552 1553 Individual adjustments 1554 """""""""""""""""""""" 1555 1556 *If you want to influence the other aspects 1557 now.* [:ref:`... <configmods_bissbs>`] 1558 1559 At this point you can use a command like ``ma 1560 to enable or disable certain features using a 1561 a graphical configuration utility, run ``make 1562 require development libraries from toolkits t 1563 respectively Qt5 or Qt6); an error message wi 1564 is missing. 1565 1566 [:ref:`back to step-by-step guide <configmods 1567 1568 .. _saveconfig_bisref: 1569 1570 Put the .config file aside 1571 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1572 1573 *Reprocess the .config after the latest cha 1574 [:ref:`... <saveconfig_bissbs>`] 1575 1576 Put the .config you prepared aside, as you wa 1577 directory every time during this guide before 1578 kernel. That's because going back and forth b 1579 .config files in odd ways; those occasionally 1580 confuse testing or in some cases render the r 1581 meaningless. 1582 1583 [:ref:`back to step-by-step guide <saveconfig 1584 1585 .. _introlatestcheck_bisref: 1586 1587 Try to reproduce the problem with the latest 1588 --------------------------------------------- 1589 1590 *Verify the regression is not caused by som 1591 still occurs with the latest codebase.* [:r 1592 1593 For some readers it might seem unnecessary to 1594 point, especially if you did that already wit 1595 distributor or face a regression within a sta 1596 highly recommended for these reasons: 1597 1598 * You will run into any problems caused by yo 1599 a bisection. That will make it a lot easier 1600 most likely is some problem in my setup' an 1601 during the bisection, as the kernel sources 1602 problem that causes building or booting to 1603 1604 * These steps will rule out if your problem i 1605 build configuration between the 'working' a 1606 example can happen when your distributor en 1607 feature in the newer kernel which was disab 1608 older kernel. That security feature might g 1609 do -- in which case your problem from the p 1610 upstream developers is not a regression, as 1611 Documentation/admin-guide/reporting-regress 1612 You thus would waste your time if you'd try 1613 1614 * If the cause for your regression was alread 1615 codebase, you'd perform the bisection for n 1616 regression you encountered with a stable/lo 1617 often caused by problems in mainline change 1618 case the problem will have to be fixed in m 1619 fixed there and the fix is already in the p 1620 1621 * For regressions within a stable/longterm se 1622 know if the issue is specific to that serie 1623 kernel, as the report needs to be sent to d 1624 1625 * Regressions specific to a stable/longterm 1626 responsibility; mainline Linux developers 1627 1628 * Regressions also happening in mainline ar 1629 developers and maintainers have to handle 1630 and does not need to be involved in the r 1631 to backport the fix once it's ready. 1632 1633 Your report might be ignored if you send it 1634 when you get a reply there is a decent chan 1635 evaluate which of the two cases it is befor 1636 1637 [:ref:`back to step-by-step guide <introlates 1638 1639 .. _checkoutmaster_bisref: 1640 1641 Check out the latest Linux codebase 1642 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1643 1644 *Check out the latest Linux codebase.* 1645 [:ref:`... <checkoutmaster_bissbs>`] 1646 1647 In case you later want to recheck if an ever 1648 problem, remember to run that ``git fetch --s 1649 again mentioned earlier to update your local 1650 1651 [:ref:`back to step-by-step guide <checkoutma 1652 1653 .. _build_bisref: 1654 1655 Build your kernel 1656 ~~~~~~~~~~~~~~~~~ 1657 1658 *Build the image and the modules of your fi 1659 you prepared.* [:ref:`... <build_bissbs>`] 1660 1661 A lot can go wrong at this stage, but the ins 1662 yourself. Another subsection explains how to 1663 deb, rpm or tar file. 1664 1665 Dealing with build errors 1666 """"""""""""""""""""""""" 1667 1668 When a build error occurs, it might be caused 1669 setup that often can be fixed quickly; other 1670 the code and can only be fixed by a developer 1671 failure messages coupled with some research o 1672 which of the two it is. To perform such inves 1673 process like this:: 1674 1675 make V=1 1676 1677 The ``V=1`` activates verbose output, which m 1678 error. To make it easier to spot, this comman 1679 --all)`` used earlier to utilize every CPU co 1680 this parallelism also results in some clutter 1681 1682 After a few seconds the build process should 1683 to find the most crucial line describing the 1684 for the most important and non-generic sectio 1685 avoid or remove anything that looks remotely 1686 or local path names like ``/home/username/lin 1687 internet search engine with that string, afte 1688 lists via `lore.kernel.org/all/ <https://lore 1689 1690 This most of the time will find something tha 1691 often one of the hits will provide a solution 1692 do not find anything that matches your proble 1693 by modifying your search terms or using anoth 1694 1695 In the end, most issues you run into have lik 1696 reported by others already. That includes iss 1697 system, but lies in the code. If you run into 1698 a solution (e.g. a patch) or workaround for y 1699 1700 Package your kernel up 1701 """""""""""""""""""""" 1702 1703 The step-by-step guide uses the default make 1704 'modules' on x86) to build the image and the 1705 steps of the guide then install. You instead 1706 and directly package it up by using one of th 1707 1708 * ``make -j $(nproc --all) bindeb-pkg`` to ge 1709 1710 * ``make -j $(nproc --all) binrpm-pkg`` to ge 1711 1712 * ``make -j $(nproc --all) tarbz2-pkg`` to ge 1713 1714 This is just a selection of available make ta 1715 ``make help`` for others. You can also use th 1716 ``make -j $(nproc --all)``, as they will pick 1717 1718 If you employ the targets to generate deb or 1719 step-by-step guide's instructions on installi 1720 instead install and remove the packages using 1721 (e.g. dpkg and rpm) or a package management u 1722 aptitude, dnf/yum, zypper, ...). Be aware tha 1723 these two make targets are designed to work o 1724 those formats, they thus will sometimes behav 1725 distribution's kernel packages. 1726 1727 [:ref:`back to step-by-step guide <build_biss 1728 1729 .. _install_bisref: 1730 1731 Put the kernel in place 1732 ~~~~~~~~~~~~~~~~~~~~~~~ 1733 1734 *Install the kernel you just built.* [:ref: 1735 1736 What you need to do after executing the comma 1737 depends on the existence and the implementati 1738 executable on your distribution. 1739 1740 If installkernel is found, the kernel's build 1741 installation of your kernel image to this exe 1742 or all of these tasks: 1743 1744 * On almost all Linux distributions installke 1745 image in /boot/, usually as '/boot/vmlinuz- 1746 put a 'System.map-<kernelrelease_id>' along 1747 1748 * On most distributions installkernel will th 1749 (sometimes also called 'initrd'), which usu 1750 '/boot/initramfs-<kernelrelease_id>.img' or 1751 '/boot/initrd-<kernelrelease_id>'. Commodit 1752 for booting, hence ensure to execute the ma 1753 as your distribution's initramfs generator 1754 the modules that go into the image. 1755 1756 * On some distributions installkernel will th 1757 to your bootloader's configuration. 1758 1759 You have to take care of some or all of the t 1760 distribution lacks a installkernel script or 1761 Consult the distribution's documentation for 1762 kernel manually:: 1763 1764 sudo install -m 0600 $(make -s image_name) 1765 sudo install -m 0600 System.map /boot/Syst 1766 1767 Now generate your initramfs using the tools y 1768 process. Afterwards add your kernel to your b 1769 1770 [:ref:`back to step-by-step guide <install_bi 1771 1772 .. _storagespace_bisref: 1773 1774 Storage requirements per kernel 1775 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1776 1777 *Check how much storage space the kernel, i 1778 like the initramfs consume.* [:ref:`... <st 1779 1780 The kernels built during a bisection consume 1781 /lib/modules/, especially if you enabled debu 1782 fill up volumes during a bisection -- and due 1783 work earlier might fail to boot. To prevent t 1784 space each installed kernel typically require 1785 1786 Note, most of the time the pattern '/boot/*$( 1787 the guide will match all files needed to boot 1788 path nor the naming scheme are mandatory. On 1789 need to look in different places. 1790 1791 [:ref:`back to step-by-step guide <storagespa 1792 1793 .. _tainted_bisref: 1794 1795 Check if your newly built kernel considers it 1796 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1797 1798 *Check if the kernel marked itself as 'tain 1799 [:ref:`... <tainted_bissbs>`] 1800 1801 Linux marks itself as tainted when something 1802 follow-up errors that look totally unrelated. 1803 ignore or react scantly to reports from taint 1804 kernel set the flag right when the reported b 1805 1806 That's why you want check why a kernel is tai 1807 Documentation/admin-guide/tainted-kernels.rst 1808 interest, as your testing might be flawed oth 1809 1810 [:ref:`back to step-by-step guide <tainted_bi 1811 1812 .. _recheckbroken_bisref: 1813 1814 Check the kernel built from a recent mainline 1815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1816 1817 *Verify if your bug occurs with the newly b 1818 [:ref:`... <recheckbroken_bissbs>`] 1819 1820 There are a couple of reasons why your bug or 1821 the kernel you built from the latest codebase 1822 1823 * The bug was fixed meanwhile. 1824 1825 * What you suspected to be a regression was c 1826 configuration the provider of your kernel c 1827 1828 * Your problem might be a race condition that 1829 the trimmed build configuration, a differen 1830 compiler used, and various other things can 1831 1832 * In case you encountered the regression with 1833 be a problem that is specific to that serie 1834 check this. 1835 1836 [:ref:`back to step-by-step guide <recheckbro 1837 1838 .. _recheckstablebroken_bisref: 1839 1840 Check the kernel built from the latest stable 1841 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1842 1843 *Are you facing a regression within a stabl 1844 reproduce it with the kernel you just built 1845 Then check if the latest codebase for the p 1846 the problem.* [:ref:`... <recheckstablebrok 1847 1848 If this kernel does not show the regression e 1849 for a bisection. 1850 1851 [:ref:`back to step-by-step guide <rechecksta 1852 1853 .. _introworkingcheck_bisref: 1854 1855 Ensure the 'good' version is really working w 1856 --------------------------------------------- 1857 1858 *Check if the kernels you build work fine.* 1859 [:ref:`... <introworkingcheck_bissbs>`] 1860 1861 This section will reestablish a known working 1862 appealing, but is usually a bad idea, as it d 1863 1864 It will ensure the .config file you prepared 1865 That is in your own interest, as trimming the 1866 and you might be building and testing ten or 1867 starting to suspect something might be wrong 1868 1869 That alone is reason enough to spend the time 1870 1871 Many readers of this guide normally run kerne 1872 modules, or both. Those kernels thus are not 1873 it's possible that the thing that regressed m 1874 builds of the 'good' version in the first pla 1875 1876 There is a third reason for those that notice 1877 stable/longterm kernels of different series ( 1878 ensure the kernel version you assumed to be ' 1879 6.0) actually is working. 1880 1881 [:ref:`back to step-by-step guide <introworki 1882 1883 .. _recheckworking_bisref: 1884 1885 Build your own version of the 'good' kernel 1886 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1887 1888 *Build your own variant of the working kern 1889 regressed works as expected with it.* [:ref 1890 1891 In case the feature that broke with newer ker 1892 self-built kernel, find and resolve the cause 1893 multitude of reasons why this might happen. S 1894 1895 * Check the taint status and the output of `` 1896 went wrong. 1897 1898 * Maybe localmodconfig did something odd and 1899 test the feature? Then you might want to re 1900 one from the last working kernel and skip t 1901 some features in the .config might work as 1902 1903 * Maybe it's not a kernel regression and some 1904 a broken initramfs (also known as initrd), 1905 userland software? 1906 1907 * Maybe it was a feature added to your distri 1908 at that point never supported? 1909 1910 Note, if you found and fixed problems with th 1911 to build another kernel from the latest codeb 1912 mainline and the latest version from an affec 1913 most likely flawed. 1914 1915 [:ref:`back to step-by-step guide <recheckwor 1916 1917 Perform a bisection and validate the result 1918 ------------------------------------------- 1919 1920 *With all the preparations and precaution b 1921 ready to begin the bisection.* [:ref:`... < 1922 1923 The steps in this segment perform and validat 1924 1925 [:ref:`back to step-by-step guide <introbisec 1926 1927 .. _bisectstart_bisref: 1928 1929 Start the bisection 1930 ~~~~~~~~~~~~~~~~~~~ 1931 1932 *Start the bisection and tell Git about the 1933 'good' and 'bad'.* [:ref:`... <bisectstart_ 1934 1935 This will start the bisection process; the la 1936 check out a commit round about half-way betwe 1937 for you to test. 1938 1939 [:ref:`back to step-by-step guide <bisectstar 1940 1941 .. _bisectbuild_bisref: 1942 1943 Build a kernel from the bisection point 1944 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1945 1946 *Build, install, and boot a kernel from the 1947 same commands you used earlier.* [:ref:`... 1948 1949 There are two things worth of note here: 1950 1951 * Occasionally building the kernel will fail 1952 problem in the code at the bisection point. 1953 1954 git bisect skip 1955 1956 Git will then check out another commit near 1957 work better. Afterwards restart executing t 1958 1959 * Those slightly odd looking version identifi 1960 because the Linux kernel subsystems prepare 1961 release (say 6.2) before its predecessor (e 1962 base them on a somewhat earlier point like 1963 get merged for 6.2 without rebasing nor squ 1964 leads to those slightly odd looking version 1965 bisections. 1966 1967 [:ref:`back to step-by-step guide <bisectbuil 1968 1969 .. _bisecttest_bisref: 1970 1971 Bisection checkpoint 1972 ~~~~~~~~~~~~~~~~~~~~ 1973 1974 *Check if the feature that regressed works 1975 [:ref:`... <bisecttest_bissbs>`] 1976 1977 Ensure what you tell Git is accurate: getting 1978 the rest of the bisection totally off course, 1979 will be for nothing. 1980 1981 [:ref:`back to step-by-step guide <bisecttest 1982 1983 .. _bisectlog_bisref: 1984 1985 Put the bisection log away 1986 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1987 1988 *Store Git's bisection log and the current 1989 [:ref:`... <bisectlog_bissbs>`] 1990 1991 As indicated above: declaring just one kernel 1992 render the end result of a bisection useless. 1993 to restart the bisection from scratch. The lo 1994 allow someone to point out where a bisection 1995 instead of testing ten or more kernels you mi 1996 resolve things. 1997 1998 The .config file is put aside, as there is a 1999 ask for it after you report the regression. 2000 2001 [:ref:`back to step-by-step guide <bisectlog_ 2002 2003 .. _revert_bisref: 2004 2005 Try reverting the culprit 2006 ~~~~~~~~~~~~~~~~~~~~~~~~~ 2007 2008 *Try reverting the culprit on top of the la 2009 your regression.* [:ref:`... <revert_bissbs 2010 2011 This is an optional step, but whenever possib 2012 decent chance that developers will ask you to 2013 the bisection result up. So give it a try, yo 2014 one more kernel shouldn't be a big deal at th 2015 2016 The step-by-step guide covers everything rele 2017 rare thing: did you bisected a regression tha 2018 a stable/longterm series, but Git failed to r 2019 try to revert the culprit in the affected sta 2020 succeeds, test that kernel version instead. 2021 2022 [:ref:`back to step-by-step guide <revert_bis 2023 2024 Cleanup steps during and after following this 2025 --------------------------------------------- 2026 2027 *During and after following this guide you 2028 of the kernels you installed.* [:ref:`... < 2029 2030 The steps in this section describe clean-up p 2031 2032 [:ref:`back to step-by-step guide <introclosu 2033 2034 .. _makeroom_bisref: 2035 2036 Cleaning up during the bisection 2037 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2038 2039 *To remove one of the kernels you installed 2040 identifier.* [:ref:`... <makeroom_bissbs>`] 2041 2042 The kernels you install during this process a 2043 parts are only stored in two places and clear 2044 need to worry to mess up your machine when yo 2045 thus bypass your distribution's packaging sys 2046 relatively easy to remove later. 2047 2048 One of the two places is a directory in /lib/ 2049 for each installed kernel. This directory is 2050 identifier; hence, to remove all modules for 2051 simply remove its modules directory in /lib/m 2052 2053 The other place is /boot/, where typically tw 2054 during installation of a kernel. All of them 2055 their file name, but how many files and their 2056 your distribution's installkernel executable 2057 some distributions the ``kernel-install remov 2058 step-by-step guide will delete all of these f 2059 the menu entry for the kernel from your bootl 2060 have to take care of these two tasks yourself 2061 interactively remove the three main files of 2062 '6.0-rc1-local-gcafec0cacaca0':: 2063 2064 rm -i /boot/{System.map,vmlinuz,initr}-6.0- 2065 2066 Afterwards check for other files in /boot/ th 2067 '6.0-rc1-local-gcafec0cacaca0' in their name 2068 Now remove the boot entry for the kernel from 2069 the steps to do that vary quite a bit between 2070 2071 Note, be careful with wildcards like '*' when 2072 for kernels manually: you might accidentally 2073 when all you want is to remove 6.0 or 6.0.1. 2074 2075 [:ref:`back to step-by-step guide <makeroom_b 2076 2077 Cleaning up after the bisection 2078 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2079 2080 .. _finishingtouch_bisref: 2081 2082 *Once you have finished the bisection, do n 2083 you set up, as you might need a few things 2084 [:ref:`... <finishingtouch_bissbs>`] 2085 2086 When you are really short of storage space re 2087 the step-by-step guide might not free as much 2088 case consider running ``rm -rf ~/linux/*`` as 2089 build artifacts and the Linux sources, but wi 2090 (~/linux/.git/) behind -- a simple ``git rese 2091 sources back. 2092 2093 Removing the repository as well would likely 2094 is a decent chance developers will ask you to 2095 perform additional tests -- like testing a de 2096 Details on how to perform those can be found 2097 tasks: test reverts, patches, or later versio 2098 2099 Additional tests are also the reason why you 2100 ~/kernel-config-working file around for a few 2101 2102 [:ref:`back to step-by-step guide <finishingt 2103 2104 .. _introoptional_bisref: 2105 2106 Test reverts, patches, or later versions 2107 ---------------------------------------- 2108 2109 *While or after reporting a bug, you might 2110 to test reverts, patches, proposed fixes, o 2111 [:ref:`... <introoptional_bissbs>`] 2112 2113 All the commands used in this section should 2114 there is not much to add except one thing: wh 2115 instructed, ensure it is not much longer than 2116 problems will arise if the kernelrelease iden 2117 2118 [:ref:`back to step-by-step guide <introoptio 2119 2120 2121 Additional information 2122 ====================== 2123 2124 .. _buildhost_bis: 2125 2126 Build kernels on a different machine 2127 ------------------------------------ 2128 2129 To compile kernels on another system, slightl 2130 instructions: 2131 2132 * Start following the guide on the machine wh 2133 the kernels later. 2134 2135 * After executing ':ref:`Boot into the workin 2136 apparently broken feature <bootworking_biss 2137 modules to a file using ``lsmod > ~/test-ma 2138 build configuration for the running kernel 2139 build configuration for your kernel <oldcon 2140 to find it) and store it as '~/test-machine 2141 files to the home directory of your build h 2142 2143 * Continue the guide on the build host (e.g. 2144 free space for building [...] <diskspace_bi 2145 2146 * When you reach ':ref:`Start preparing a ker 2147 <oldconfig_bissbs>`': before running ``make 2148 execute the following command to base your 2149 test machine's 'working' kernel:: 2150 2151 cp ~/test-machine-config-working ~/linux/ 2152 2153 * During the next step to ':ref:`disable any 2154 modules <localmodconfig_bissbs>`' use the f 2155 2156 yes '' | make localmodconfig LSMOD=~/lsmo 2157 2158 * Continue the guide, but ignore the instruct 2159 install, and reboot into a kernel every tim 2160 like this:: 2161 2162 cp ~/kernel-config-working .config 2163 make olddefconfig && 2164 make -j $(nproc --all) targz-pkg 2165 2166 This will generate a gzipped tar file whose 2167 line shown; for example, a kernel with the 2168 '6.0.0-rc1-local-g928a87efa423' built for x 2169 be stored as '~/linux/linux-6.0.0-rc1-local 2170 2171 Copy that file to your test machine's home 2172 2173 * Switch to the test machine to check if you 2174 kernel. Then extract the file you transferr 2175 2176 sudo tar -xvzf ~/linux-6.0.0-rc1-local-g9 2177 2178 Afterwards :ref:`generate the initramfs and 2179 loader's configuration <install_bisref>`; o 2180 command will take care of both these tasks: 2181 2182 sudo /sbin/installkernel 6.0.0-rc1-local- 2183 2184 Now reboot and ensure you started the inten 2185 2186 This approach even works when building for an 2187 cross-compilers and add the appropriate param 2188 (e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64 2189 2190 Additional reading material 2191 --------------------------- 2192 2193 * The `man page for 'git bisect' <https://git 2194 `fighting regressions with 'git bisect' <ht 2195 in the Git documentation. 2196 * `Working with git bisect <https://nathancha 2197 from kernel developer Nathan Chancellor. 2198 * `Using Git bisect to figure out when broken 2199 * `Fully automated bisecting with 'git bisect 2200 2201 .. 2202 end-of-content 2203 .. 2204 This document is maintained by Thorsten Le< 2205 you spot a typo or small mistake, feel fre 2206 he'll fix it. You are free to do the same 2207 want to contribute changes to the text -- 2208 linux-doc@vger.kernel.org and 'sign-off' y 2209 Documentation/process/submitting-patches.r 2210 your work - the Developer's Certificate of 2211 .. 2212 This text is available under GPL-2.0+ or C 2213 of the file. If you want to distribute thi 2214 please use 'The Linux kernel development c 2215 and link this as source: 2216 https://git.kernel.org/pub/scm/linux/kerne 2217 2218 .. 2219 Note: Only the content of this RST file as 2220 is available under CC-BY-4.0, as versions 2221 (for example by the kernel's build system) 2222 files which use a more restrictive license
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.