1 Bug hunting
2 ===========
3
4 Kernel bug reports often come with a stack dum
5
6 ------------[ cut here ]------------
7 WARNING: CPU: 1 PID: 28102 at kernel/m
8 Modules linked in: dvb_usb_gp8psk(-) d
9 CPU: 1 PID: 28102 Comm: rmmod Tainted:
10 Hardware name: MSI MS-7309/MS-7309, BI
11 00000000 c12ba080 00000000 00000000 c
12 c1615862 00000454 c109e8a7 c109e8a7 0
13 f5f5a600 c103ee33 00000009 00000000 0
14 Call Trace:
15 [<c12ba080>] ? dump_stack+0x44/0x64
16 [<c103ed6a>] ? __warn+0xfa/0x120
17 [<c109e8a7>] ? module_put+0x57/0x70
18 [<c109e8a7>] ? module_put+0x57/0x70
19 [<c103ee33>] ? warn_slowpath_null+0x2
20 [<c109e8a7>] ? module_put+0x57/0x70
21 [<f80ca4d0>] ? gp8psk_fe_set_frontend
22 [<c109f617>] ? symbol_put_addr+0x27/0
23 [<f80bc9ca>] ? dvb_usb_adapter_fronte
24 [<f80bb3bf>] ? dvb_usb_exit+0x2f/0xd0
25 [<c13d03bc>] ? usb_disable_endpoint+0
26 [<f80bb48a>] ? dvb_usb_device_exit+0x
27 [<c13d2882>] ? usb_unbind_interface+0
28 [<c136b514>] ? __pm_runtime_idle+0x44
29 [<c13620d8>] ? __device_release_drive
30 [<c1362907>] ? driver_detach+0x87/0x9
31 [<c1361c48>] ? bus_remove_driver+0x38
32 [<c13d1c18>] ? usb_deregister+0x58/0x
33 [<c109fbb0>] ? SyS_delete_module+0x13
34 [<c1055654>] ? task_work_run+0x64/0x8
35 [<c1000fa5>] ? exit_to_usermode_loop+
36 [<c10013f0>] ? do_fast_syscall_32+0x8
37 [<c1549f43>] ? sysenter_past_esp+0x40
38 ---[ end trace 6ebc60ef3981792f ]---
39
40 Such stack traces provide enough information t
41 Kernel's source code where the bug happened. D
42 the issue, it may also contain the word **Oops
43
44 BUG: unable to handle kernel NULL poin
45 IP: [<c06969d4>] iret_exc+0x7d0/0xa59
46 *pdpt = 000000002258a001 *pde = 000000
47 Oops: 0002 [#1] PREEMPT SMP
48 ...
49
50 Despite being an **Oops** or some other sort o
51 line is usually required to identify and handl
52 we'll refer to "Oops" for all kinds of stack t
53
54 If the kernel is compiled with ``CONFIG_DEBUG_
55 quality of the stack trace by using file:`scri
56
57 Modules linked in
58 -----------------
59
60 Modules that are tainted or are being loaded o
61 "(...)", where the taint flags are described i
62 file:`Documentation/admin-guide/tainted-kernel
63 annotated with "+", and "being unloaded" is an
64
65
66 Where is the Oops message is located?
67 -------------------------------------
68
69 Normally the Oops text is read from the kernel
70 handed to ``syslogd`` which writes it to a sys
71 ``/var/log/messages`` (depends on ``/etc/syslo
72 systemd, it may also be stored by the ``journa
73 by running ``journalctl`` command.
74
75 Sometimes ``klogd`` dies, in which case you ca
76 read the data from the kernel buffers and save
77 ``cat /proc/kmsg > file``, however you have to
78 since ``kmsg`` is a "never ending file".
79
80 If the machine has crashed so badly that you c
81 the disk is not available then you have three
82
83 (1) Hand copy the text from the screen and typ
84 has restarted. Messy but it is the only o
85 planned for a crash. Alternatively, you ca
86 the screen with a digital camera - not nic
87 nothing. If the messages scroll off the t
88 may find that booting with a higher resolu
89 will allow you to read more of the text. (
90 so won't help for 'early' oopses.)
91
92 (2) Boot with a serial console (see
93 :ref:`Documentation/admin-guide/serial-con
94 run a null modem to a second machine and c
95 using your favourite communication program
96
97 (3) Use Kdump (see Documentation/admin-guide/k
98 extract the kernel ring buffer from old me
99 gdbmacro in Documentation/admin-guide/kdum
100
101 Finding the bug's location
102 --------------------------
103
104 Reporting a bug works best if you point the lo
105 Kernel source file. There are two methods for
106 ``gdb`` is easier, but the Kernel should be pr
107
108 gdb
109 ^^^
110
111 The GNU debugger (``gdb``) is the best way to
112 number of the OOPS from the ``vmlinux`` file.
113
114 The usage of gdb works best on a kernel compil
115 This can be set by running::
116
117 $ ./scripts/config -d COMPILE_TEST -e DEBUG_
118
119 On a kernel compiled with ``CONFIG_DEBUG_INFO`
120 EIP value from the OOPS::
121
122 EIP: 0060:[<c021e50e>] Not tainted VLI
123
124 And use GDB to translate that to human-readabl
125
126 $ gdb vmlinux
127 (gdb) l *0xc021e50e
128
129 If you don't have ``CONFIG_DEBUG_INFO`` enable
130 offset from the OOPS::
131
132 EIP is at vt_ioctl+0xda8/0x1482
133
134 And recompile the kernel with ``CONFIG_DEBUG_I
135
136 $ ./scripts/config -d COMPILE_TEST -e DEBUG_
137 $ make vmlinux
138 $ gdb vmlinux
139 (gdb) l *vt_ioctl+0xda8
140 0x1888 is in vt_ioctl (drivers/tty/vt/vt_ioc
141 288 {
142 289 struct vc_data *vc = NULL;
143 290 int ret = 0;
144 291
145 292 console_lock();
146 293 if (VT_BUSY(vc_num))
147 294 ret = -EBUSY;
148 295 else if (vc_num)
149 296 vc = vc_deallocate(vc_
150 297 console_unlock();
151
152 or, if you want to be more verbose::
153
154 (gdb) p vt_ioctl
155 $1 = {int (struct tty_struct *, unsigned int
156 (gdb) l *0xae0+0xda8
157
158 You could, instead, use the object file::
159
160 $ make drivers/tty/
161 $ gdb drivers/tty/vt/vt_ioctl.o
162 (gdb) l *vt_ioctl+0xda8
163
164 If you have a call trace, such as::
165
166 Call Trace:
167 [<ffffffff8802c8e9>] :jbd:log_wait_commi
168 [<ffffffff810482d9>] autoremove_wake_fun
169 [<ffffffff8802770b>] :jbd:journal_stop+0
170 ...
171
172 this shows the problem likely is in the :jbd:
173 in gdb and list the relevant code::
174
175 $ gdb fs/jbd/jbd.ko
176 (gdb) l *log_wait_commit+0xa3
177
178 .. note::
179
180 You can also do the same for any function
181 like this one::
182
183 [<f80bc9ca>] ? dvb_usb_adapter_fronte
184
185 The position where the above call happene
186
187 $ gdb drivers/media/usb/dvb-usb/dvb-us
188 (gdb) l *dvb_usb_adapter_frontend_exit
189
190 objdump
191 ^^^^^^^
192
193 To debug a kernel, use objdump and look for th
194 output to find the valid line of code/assemble
195 will see the assembler code for the routine sh
196 debug symbols the C code will also be availabl
197 in the kernel hacking menu of the menu configu
198
199 $ objdump -r -S -l --disassemble net/dccp/
200
201 .. note::
202
203 You need to be at the top level of the kern
204 your C files.
205
206 If you don't have access to the source code yo
207 dumps using the following method (example cras
208 Dave Miller)::
209
210 EIP is at +0x14/0x4c0
211 ...
212 Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff
213 00 00 55 57 56 53 81 ec bc 00 00 00 8b a
214 <8b> 83 3c 01 00 00 89 44 24 14 8b 45 28
215
216 Put the bytes into a "foo.s" file like th
217
218 .text
219 .globl foo
220 foo:
221 .byte .... /* bytes from Code: pa
222
223 Compile it with "gcc -c -o foo.o foo.s" t
224 "objdump --disassemble foo.o".
225
226 Output:
227
228 ip_queue_xmit:
229 push %ebp
230 push %edi
231 push %esi
232 push %ebx
233 sub $0xbc, %esp
234 mov 0xd0(%esp), %ebp !
235 mov 0x8(%ebp), %ebx !
236 mov 0x13c(%ebx), %eax !
237
238 file:`scripts/decodecode` can be used to autom
239 on what CPU architecture is being debugged.
240
241 Reporting the bug
242 -----------------
243
244 Once you find where the bug happened, by inspe
245 you could either try to fix it yourself or rep
246
247 In order to report it upstream, you should ide
248 used for the development of the affected code.
249 the ``get_maintainer.pl`` script.
250
251 For example, if you find a bug at the gspca's
252 its maintainers with::
253
254 $ ./scripts/get_maintainer.pl -f drive
255 Hans Verkuil <hverkuil@xs4all.nl> (odd
256 Mauro Carvalho Chehab <mchehab@kernel.o
257 Tejun Heo <tj@kernel.org> (commit_signe
258 Bhaktipriya Shridhar <bhaktipriya96@gma
259 linux-media@vger.kernel.org (open list
260 linux-kernel@vger.kernel.org (open lis
261
262 Please notice that it will point to:
263
264 - The last developers that touched the source
265 a git tree). On the above example, Tejun and
266 specific case, none really involved on the d
267 - The driver maintainer (Hans Verkuil);
268 - The subsystem maintainer (Mauro Carvalho Che
269 - The driver and/or subsystem mailing list (li
270 - the Linux Kernel mailing list (linux-kernel@
271
272 Usually, the fastest way to have your bug fixe
273 list used for the development of the code (lin
274 driver maintainer (Hans).
275
276 If you are totally stumped as to whom to send
277 ``get_maintainer.pl`` didn't provide you anyth
278 linux-kernel@vger.kernel.org.
279
280 Thanks for your help in making Linux as stable
281
282 Fixing the bug
283 --------------
284
285 If you know programming, you could help us by
286 but also providing us with a solution. After a
287 sharing what you do and don't you want to be r
288
289 If you decide to take this way, once you have
290 it upstream.
291
292 Please do read
293 :ref:`Documentation/process/submitting-patches
294 to help your code get accepted.
295
296
297 ----------------------------------------------
298
299 Notes on Oops tracing with ``klogd``
300 ------------------------------------
301
302 In order to help Linus and the other kernel de
303 substantial support incorporated into ``klogd`
304 faults. In order to have full support for add
305 version 1.3-pl3 of the ``sysklogd`` package sh
306
307 When a protection fault occurs the ``klogd`` d
308 translates important addresses in the kernel l
309 symbolic equivalents. This translated kernel
310 forwarded through whatever reporting mechanism
311 protection fault message can be simply cut out
312 and forwarded to the kernel developers.
313
314 Two types of address resolution are performed
315 static translation and the second is dynamic t
316 Static translation uses the System.map file.
317 In order to do static translation the ``klogd`
318 must be able to find a system map file at daem
319 See the klogd man page for information on how
320 files.
321
322 Dynamic address translation is important when
323 are being used. Since memory for kernel modul
324 kernel's dynamic memory pools there are no fix
325 the start of the module or for functions and s
326
327 The kernel supports system calls which allow a
328 which modules are loaded and their location in
329 system calls the klogd daemon builds a symbol
330 to debug a protection fault which occurs in a
331
332 At the very minimum klogd will provide the nam
333 generated the protection fault. There may be
334 information available if the developer of the
335 export symbol information from the module.
336
337 Since the kernel module environment can be dyn
338 mechanism for notifying the ``klogd`` daemon w
339 environment occurs. There are command line op
340 allow klogd to signal the currently executing
341 information should be refreshed. See the ``kl
342 information.
343
344 A patch is included with the sysklogd distribu
345 ``modules-2.0.0`` package to automatically sig
346 is loaded or unloaded. Applying this patch pr
347 seamless support for debugging protection faul
348 kernel loadable modules.
349
350 The following is an example of a protection fa
351 processed by ``klogd``::
352
353 Aug 29 09:51:01 blizard kernel: Unable
354 Aug 29 09:51:01 blizard kernel: curren
355 Aug 29 09:51:01 blizard kernel: *pde =
356 Aug 29 09:51:01 blizard kernel: Oops:
357 Aug 29 09:51:01 blizard kernel: CPU:
358 Aug 29 09:51:01 blizard kernel: EIP:
359 Aug 29 09:51:01 blizard kernel: EFLAGS
360 Aug 29 09:51:01 blizard kernel: eax: 3
361 Aug 29 09:51:01 blizard kernel: esi: 0
362 Aug 29 09:51:01 blizard kernel: ds: 00
363 Aug 29 09:51:01 blizard kernel: Proces
364 Aug 29 09:51:01 blizard kernel: Stack:
365 Aug 29 09:51:01 blizard kernel:
366 Aug 29 09:51:01 blizard kernel:
367 Aug 29 09:51:01 blizard kernel: Call T
368 Aug 29 09:51:01 blizard kernel: Code:
369
370 ----------------------------------------------
371
372 ::
373
374 Dr. G.W. Wettstein Oncology Resear
375 Roger Maris Cancer Center INTERNET: greg@
376 820 4th St. N.
377 Fargo, ND 58122
378 Phone: 701-234-7556
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.