1 ==============================
2 Using the tracer for debugging
3 ==============================
4
5 Copyright 2024 Google LLC.
6
7 :Author: Steven Rostedt <rostedt@goodmis.org>
8 :License: The GNU Free Documentation License,
9 (dual licensed under the GPL v2)
10
11 - Written for: 6.12
12
13 Introduction
14 ------------
15 The tracing infrastructure can be very useful
16 kernel. This document is a place to add variou
17 for debugging.
18
19 First, make sure that the tracefs file system
20
21 $ sudo mount -t tracefs tracefs /sys/kernel/t
22
23
24 Using trace_printk()
25 --------------------
26
27 trace_printk() is a very lightweight utility t
28 inside the kernel, with the exception of "noin
29 in normal, softirq, interrupt and even NMI con
30 written to the tracing ring buffer in a lockle
31 lighter weight, when possible, it will only re
32 string, and save the raw arguments into the bu
33 arguments will be post processed when the ring
34 trace_printk() format conversions are not done
35 the trace is being recorded.
36
37 trace_printk() is meant only for debugging, an
38 a subsystem of the kernel. If you need debuggi
39 instead. If a trace_printk() is found in the k
40 appear in the dmesg::
41
42 ********************************************
43 ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTI
44 **
45 ** trace_printk() being used. Allocating ext
46 **
47 ** This means that this is a DEBUG kernel an
48 ** unsafe for production use.
49 **
50 ** If you see this message and you are not d
51 ** the kernel, report this immediately to yo
52 **
53 ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTI
54 ********************************************
55
56 Debugging kernel crashes
57 ------------------------
58 There is various methods of acquiring the stat
59 crash occurs. This could be from the oops mess
60 use kexec/kdump. But these just show what happ
61 It can be very useful in knowing what happened
62 The tracing ring buffer, by default, is a circ
63 overwrite older events with newer ones. When a
64 the ring buffer will be all the events that le
65
66 There are several kernel command line paramete
67 this. The first is "ftrace_dump_on_oops". This
68 buffer when a oops occurs to the console. This
69 is being logged somewhere. If a serial console
70 make sure the ring buffer is relatively small,
71 ring buffer may take several minutes to hours
72 of the kernel command line::
73
74 ftrace_dump_on_oops trace_buf_size=50K
75
76 Note, the tracing buffer is made up of per CPU
77 buffers is broken up into sub-buffers that are
78 above trace_buf_size option above sets each of
79 so, on a machine with 8 CPUs, that's actually
80
81 Persistent buffers across boots
82 -------------------------------
83 If the system memory allows it, the tracing ri
84 a specific location in memory. If the location
85 the memory is not modified, the tracing buffer
86 following boot. There's two ways to reserve me
87 buffer.
88
89 The more reliable way (on x86) is to reserve m
90 command line option and then use that memory f
91 requires a bit of knowledge of the physical me
92 advantage of using this method, is that the me
93 always be the same::
94
95 memmap==12M$0x284500000 trace_instance=boot_
96
97 The memmap above reserves 12 megabytes of memo
98 location 0x284500000. Then the trace_instance
99 instance "boot_map" at that same location with
100 reserved. As the ring buffer is broke up into
101 megabytes will be broken up evenly between tho
102 each per CPU ring buffer will be 1.5 megabytes
103 includes meta data, so the amount of memory ac
104 will be slightly smaller.
105
106 Another more generic but less robust way to al
107 at boot is with the "reserve_mem" option::
108
109 reserve_mem=12M:4096:trace trace_instance=bo
110
111 The reserve_mem option above will find 12 mega
112 boot up, and align it by 4096 bytes. It will l
113 that can be used by later command line options
114
115 The trace_instance option creates a "boot_map"
116 memory reserved by reserve_mem that was labele
117 more generic but may not be as reliable. Due t
118 by reserve_mem may not be located at the same
119 then the ring buffer will not be from the prev
120
121 Sometimes, by using a larger alignment, it can
122 around in such a way that it will move the loc
123 using a larger alignment, you may find better
124 consistent to where it is placed::
125
126 reserve_mem=12M:0x2000000:trace trace_instan
127
128 On boot up, the memory reserved for the ring b
129 through a series of tests to make sure that th
130 data. If it is, it will then set it up to be a
131 instance. If it fails any of the tests, it wil
132 and initialize it as new.
133
134 The layout of this mapped memory may not be co
135 kernel, so only the same kernel is guaranteed
136 preserved. Switching to a different kernel ver
137 layout and mark the buffer as invalid.
138
139 Using trace_printk() in the boot instance
140 -----------------------------------------
141 By default, the content of trace_printk() goes
142 instance. But this instance is never preserved
143 trace_printk() content, and some other interna
144 buffer (like dump stacks), either set the inst
145 destination from the kernel command line, or s
146 trace_printk_dest option.
147
148 After boot up::
149
150 echo 1 > /sys/kernel/tracing/instances/boot_
151
152 From the kernel command line::
153
154 reserve_mem=12M:4096:trace trace_instance=bo
155
156 If setting it from the kernel command line, it
157 disable tracing with the "traceoff" flag, and
158 Otherwise the trace from the most recent boot
159 from the previous boot, and may make it confus
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.