1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ================= 3 =================
4 Boot-time tracing 4 Boot-time tracing
5 ================= 5 =================
6 6
7 :Author: Masami Hiramatsu <mhiramat@kernel.org> 7 :Author: Masami Hiramatsu <mhiramat@kernel.org>
8 8
9 Overview 9 Overview
10 ======== 10 ========
11 11
12 Boot-time tracing allows users to trace boot-t 12 Boot-time tracing allows users to trace boot-time process including
13 device initialization with full features of ft 13 device initialization with full features of ftrace including per-event
14 filter and actions, histograms, kprobe-events 14 filter and actions, histograms, kprobe-events and synthetic-events,
15 and trace instances. 15 and trace instances.
16 Since kernel command line is not enough to con 16 Since kernel command line is not enough to control these complex features,
17 this uses bootconfig file to describe tracing 17 this uses bootconfig file to describe tracing feature programming.
18 18
19 Options in the Boot Config 19 Options in the Boot Config
20 ========================== 20 ==========================
21 21
22 Here is the list of available options list for 22 Here is the list of available options list for boot time tracing in
23 boot config file [1]_. All options are under " 23 boot config file [1]_. All options are under "ftrace." or "kernel."
24 prefix. See kernel parameters for the options 24 prefix. See kernel parameters for the options which starts
25 with "kernel." prefix [2]_. 25 with "kernel." prefix [2]_.
26 26
27 .. [1] See :ref:`Documentation/admin-guide/boo 27 .. [1] See :ref:`Documentation/admin-guide/bootconfig.rst <bootconfig>`
28 .. [2] See :ref:`Documentation/admin-guide/ker 28 .. [2] See :ref:`Documentation/admin-guide/kernel-parameters.rst <kernelparameters>`
29 29
30 Ftrace Global Options 30 Ftrace Global Options
31 --------------------- 31 ---------------------
32 32
33 Ftrace global options have "kernel." prefix in 33 Ftrace global options have "kernel." prefix in boot config, which means
34 these options are passed as a part of kernel l 34 these options are passed as a part of kernel legacy command line.
35 35
36 kernel.tp_printk 36 kernel.tp_printk
37 Output trace-event data on printk buffer to 37 Output trace-event data on printk buffer too.
38 38
39 kernel.dump_on_oops [= MODE] 39 kernel.dump_on_oops [= MODE]
40 Dump ftrace on Oops. If MODE = 1 or omitted 40 Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer
41 on all CPUs. If MODE = 2, dump a buffer on 41 on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops.
42 42
43 kernel.traceoff_on_warning 43 kernel.traceoff_on_warning
44 Stop tracing if WARN_ON() occurs. 44 Stop tracing if WARN_ON() occurs.
45 45
46 kernel.fgraph_max_depth = MAX_DEPTH 46 kernel.fgraph_max_depth = MAX_DEPTH
47 Set MAX_DEPTH to maximum depth of fgraph tr 47 Set MAX_DEPTH to maximum depth of fgraph tracer.
48 48
49 kernel.fgraph_filters = FILTER[, FILTER2...] 49 kernel.fgraph_filters = FILTER[, FILTER2...]
50 Add fgraph tracing function filters. 50 Add fgraph tracing function filters.
51 51
52 kernel.fgraph_notraces = FILTER[, FILTER2...] 52 kernel.fgraph_notraces = FILTER[, FILTER2...]
53 Add fgraph non-tracing function filters. 53 Add fgraph non-tracing function filters.
54 54
55 55
56 Ftrace Per-instance Options 56 Ftrace Per-instance Options
57 --------------------------- 57 ---------------------------
58 58
59 These options can be used for each instance in 59 These options can be used for each instance including global ftrace node.
60 60
61 ftrace.[instance.INSTANCE.]options = OPT1[, OP 61 ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]]
62 Enable given ftrace options. 62 Enable given ftrace options.
63 63
64 ftrace.[instance.INSTANCE.]tracing_on = 0|1 64 ftrace.[instance.INSTANCE.]tracing_on = 0|1
65 Enable/Disable tracing on this instance whe 65 Enable/Disable tracing on this instance when starting boot-time tracing.
66 (you can enable it by the "traceon" event t 66 (you can enable it by the "traceon" event trigger action)
67 67
68 ftrace.[instance.INSTANCE.]trace_clock = CLOCK 68 ftrace.[instance.INSTANCE.]trace_clock = CLOCK
69 Set given CLOCK to ftrace's trace_clock. 69 Set given CLOCK to ftrace's trace_clock.
70 70
71 ftrace.[instance.INSTANCE.]buffer_size = SIZE 71 ftrace.[instance.INSTANCE.]buffer_size = SIZE
72 Configure ftrace buffer size to SIZE. You c 72 Configure ftrace buffer size to SIZE. You can use "KB" or "MB"
73 for that SIZE. 73 for that SIZE.
74 74
75 ftrace.[instance.INSTANCE.]alloc_snapshot 75 ftrace.[instance.INSTANCE.]alloc_snapshot
76 Allocate snapshot buffer. 76 Allocate snapshot buffer.
77 77
78 ftrace.[instance.INSTANCE.]cpumask = CPUMASK 78 ftrace.[instance.INSTANCE.]cpumask = CPUMASK
79 Set CPUMASK as trace cpu-mask. 79 Set CPUMASK as trace cpu-mask.
80 80
81 ftrace.[instance.INSTANCE.]events = EVENT[, EV 81 ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2[...]]
82 Enable given events on boot. You can use a 82 Enable given events on boot. You can use a wild card in EVENT.
83 83
84 ftrace.[instance.INSTANCE.]tracer = TRACER 84 ftrace.[instance.INSTANCE.]tracer = TRACER
85 Set TRACER to current tracer on boot. (e.g. 85 Set TRACER to current tracer on boot. (e.g. function)
86 86
87 ftrace.[instance.INSTANCE.]ftrace.filters 87 ftrace.[instance.INSTANCE.]ftrace.filters
88 This will take an array of tracing function 88 This will take an array of tracing function filter rules.
89 89
90 ftrace.[instance.INSTANCE.]ftrace.notraces 90 ftrace.[instance.INSTANCE.]ftrace.notraces
91 This will take an array of NON-tracing func 91 This will take an array of NON-tracing function filter rules.
92 92
93 93
94 Ftrace Per-Event Options 94 Ftrace Per-Event Options
95 ------------------------ 95 ------------------------
96 96
97 These options are setting per-event options. 97 These options are setting per-event options.
98 98
99 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.e 99 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable
100 Enable GROUP:EVENT tracing. 100 Enable GROUP:EVENT tracing.
101 101
102 ftrace.[instance.INSTANCE.]event.GROUP.enable 102 ftrace.[instance.INSTANCE.]event.GROUP.enable
103 Enable all event tracing within GROUP. 103 Enable all event tracing within GROUP.
104 104
105 ftrace.[instance.INSTANCE.]event.enable 105 ftrace.[instance.INSTANCE.]event.enable
106 Enable all event tracing. 106 Enable all event tracing.
107 107
108 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.f 108 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER
109 Set FILTER rule to the GROUP:EVENT. 109 Set FILTER rule to the GROUP:EVENT.
110 110
111 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.a 111 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2[...]]
112 Set ACTIONs to the GROUP:EVENT. 112 Set ACTIONs to the GROUP:EVENT.
113 113
114 ftrace.[instance.INSTANCE.]event.kprobes.EVENT 114 ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2[...]]
115 Defines new kprobe event based on PROBEs. I 115 Defines new kprobe event based on PROBEs. It is able to define
116 multiple probes on one event, but those mus 116 multiple probes on one event, but those must have same type of
117 arguments. This option is available only fo 117 arguments. This option is available only for the event which
118 group name is "kprobes". 118 group name is "kprobes".
119 119
120 ftrace.[instance.INSTANCE.]event.synthetic.EVE 120 ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]]
121 Defines new synthetic event with FIELDs. Ea 121 Defines new synthetic event with FIELDs. Each field should be
122 "type varname". 122 "type varname".
123 123
124 Note that kprobe and synthetic event definitio 124 Note that kprobe and synthetic event definitions can be written under
125 instance node, but those are also visible from 125 instance node, but those are also visible from other instances. So please
126 take care for event name conflict. 126 take care for event name conflict.
127 127
128 Ftrace Histogram Options 128 Ftrace Histogram Options
129 ------------------------ 129 ------------------------
130 130
131 Since it is too long to write a histogram acti 131 Since it is too long to write a histogram action as a string for per-event
132 action option, there are tree-style options un 132 action option, there are tree-style options under per-event 'hist' subkey
133 for the histogram actions. For the detail of t 133 for the histogram actions. For the detail of the each parameter,
134 please read the event histogram document (Docu !! 134 please read the event histogram document [3]_.
>> 135
>> 136 .. [3] See :ref:`Documentation/trace/histogram.rst <histogram>`
135 137
136 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 138 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]keys = KEY1[, KEY2[...]]
137 Set histogram key parameters. (Mandatory) 139 Set histogram key parameters. (Mandatory)
138 The 'N' is a digit string for the multiple h 140 The 'N' is a digit string for the multiple histogram. You can omit it
139 if there is one histogram on the event. 141 if there is one histogram on the event.
140 142
141 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 143 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]values = VAL1[, VAL2[...]]
142 Set histogram value parameters. 144 Set histogram value parameters.
143 145
144 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 146 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]sort = SORT1[, SORT2[...]]
145 Set histogram sort parameter options. 147 Set histogram sort parameter options.
146 148
147 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 149 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]size = NR_ENTRIES
148 Set histogram size (number of entries). 150 Set histogram size (number of entries).
149 151
150 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 152 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]name = NAME
151 Set histogram name. 153 Set histogram name.
152 154
153 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 155 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]var.VARIABLE = EXPR
154 Define a new VARIABLE by EXPR expression. 156 Define a new VARIABLE by EXPR expression.
155 157
156 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 158 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<pause|continue|clear>
157 Set histogram control parameter. You can set 159 Set histogram control parameter. You can set one of them.
158 160
159 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 161 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]event = GROUP.EVENT
160 Set histogram 'onmatch' handler matching eve 162 Set histogram 'onmatch' handler matching event parameter.
161 The 'M' is a digit string for the multiple ' 163 The 'M' is a digit string for the multiple 'onmatch' handler. You can omit it
162 if there is one 'onmatch' handler on this hi 164 if there is one 'onmatch' handler on this histogram.
163 165
164 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 166 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]trace = EVENT[, ARG1[...]]
165 Set histogram 'trace' action for 'onmatch'. 167 Set histogram 'trace' action for 'onmatch'.
166 EVENT must be a synthetic event name, and AR 168 EVENT must be a synthetic event name, and ARG1... are parameters
167 for that event. Mandatory if 'onmatch.event' 169 for that event. Mandatory if 'onmatch.event' option is set.
168 170
169 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 171 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmax.[M.]var = VAR
170 Set histogram 'onmax' handler variable param 172 Set histogram 'onmax' handler variable parameter.
171 173
172 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 174 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onchange.[M.]var = VAR
173 Set histogram 'onchange' handler variable pa 175 Set histogram 'onchange' handler variable parameter.
174 176
175 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 177 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]save = ARG1[, ARG2[...]]
176 Set histogram 'save' action parameters for ' 178 Set histogram 'save' action parameters for 'onmax' or 'onchange' handler.
177 This option or below 'snapshot' option is ma 179 This option or below 'snapshot' option is mandatory if 'onmax.var' or
178 'onchange.var' option is set. 180 'onchange.var' option is set.
179 181
180 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 182 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]snapshot
181 Set histogram 'snapshot' action for 'onmax' 183 Set histogram 'snapshot' action for 'onmax' or 'onchange' handler.
182 This option or above 'save' option is mandat 184 This option or above 'save' option is mandatory if 'onmax.var' or
183 'onchange.var' option is set. 185 'onchange.var' option is set.
184 186
185 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.h 187 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.filter = FILTER_EXPR
186 Set histogram filter expression. You don't n 188 Set histogram filter expression. You don't need 'if' in the FILTER_EXPR.
187 189
188 Note that this 'hist' option can conflict with 190 Note that this 'hist' option can conflict with the per-event 'actions'
189 option if the 'actions' option has a histogram 191 option if the 'actions' option has a histogram action.
190 192
191 193
192 When to Start 194 When to Start
193 ============= 195 =============
194 196
195 All boot-time tracing options starting with `` 197 All boot-time tracing options starting with ``ftrace`` will be enabled at the
196 end of core_initcall. This means you can trace 198 end of core_initcall. This means you can trace the events from postcore_initcall.
197 Most of the subsystems and architecture depend 199 Most of the subsystems and architecture dependent drivers will be initialized
198 after that (arch_initcall or subsys_initcall). 200 after that (arch_initcall or subsys_initcall). Thus, you can trace those with
199 boot-time tracing. 201 boot-time tracing.
200 If you want to trace events before core_initca 202 If you want to trace events before core_initcall, you can use the options
201 starting with ``kernel``. Some of them will be 203 starting with ``kernel``. Some of them will be enabled eariler than the initcall
202 processing (for example,. ``kernel.ftrace=func 204 processing (for example,. ``kernel.ftrace=function`` and ``kernel.trace_event``
203 will start before the initcall.) 205 will start before the initcall.)
204 206
205 207
206 Examples 208 Examples
207 ======== 209 ========
208 210
209 For example, to add filter and actions for eac 211 For example, to add filter and actions for each event, define kprobe
210 events, and synthetic events with histogram, w 212 events, and synthetic events with histogram, write a boot config like
211 below:: 213 below::
212 214
213 ftrace.event { 215 ftrace.event {
214 task.task_newtask { 216 task.task_newtask {
215 filter = "pid < 128" 217 filter = "pid < 128"
216 enable 218 enable
217 } 219 }
218 kprobes.vfs_read { 220 kprobes.vfs_read {
219 probes = "vfs_read $arg1 $arg2 221 probes = "vfs_read $arg1 $arg2"
220 filter = "common_pid < 200" 222 filter = "common_pid < 200"
221 enable 223 enable
222 } 224 }
223 synthetic.initcall_latency { 225 synthetic.initcall_latency {
224 fields = "unsigned long func", 226 fields = "unsigned long func", "u64 lat"
225 hist { 227 hist {
226 keys = func.sym, lat 228 keys = func.sym, lat
227 values = lat 229 values = lat
228 sort = lat 230 sort = lat
229 } 231 }
230 } 232 }
231 initcall.initcall_start.hist { 233 initcall.initcall_start.hist {
232 keys = func 234 keys = func
233 var.ts0 = common_timestamp.use 235 var.ts0 = common_timestamp.usecs
234 } 236 }
235 initcall.initcall_finish.hist { 237 initcall.initcall_finish.hist {
236 keys = func 238 keys = func
237 var.lat = common_timestamp.use 239 var.lat = common_timestamp.usecs - $ts0
238 onmatch { 240 onmatch {
239 event = initcall.initc 241 event = initcall.initcall_start
240 trace = initcall_laten 242 trace = initcall_latency, func, $lat
241 } 243 }
242 } 244 }
243 } 245 }
244 246
245 Also, boot-time tracing supports "instance" no 247 Also, boot-time tracing supports "instance" node, which allows us to run
246 several tracers for different purpose at once. 248 several tracers for different purpose at once. For example, one tracer
247 is for tracing functions starting with "user\_ 249 is for tracing functions starting with "user\_", and others tracing
248 "kernel\_" functions, you can write boot confi 250 "kernel\_" functions, you can write boot config as below::
249 251
250 ftrace.instance { 252 ftrace.instance {
251 foo { 253 foo {
252 tracer = "function" 254 tracer = "function"
253 ftrace.filters = "user_*" 255 ftrace.filters = "user_*"
254 } 256 }
255 bar { 257 bar {
256 tracer = "function" 258 tracer = "function"
257 ftrace.filters = "kernel_*" 259 ftrace.filters = "kernel_*"
258 } 260 }
259 } 261 }
260 262
261 The instance node also accepts event nodes so 263 The instance node also accepts event nodes so that each instance
262 can customize its event tracing. 264 can customize its event tracing.
263 265
264 With the trigger action and kprobes, you can t 266 With the trigger action and kprobes, you can trace function-graph while
265 a function is called. For example, this will t 267 a function is called. For example, this will trace all function calls in
266 the pci_proc_init():: 268 the pci_proc_init()::
267 269
268 ftrace { 270 ftrace {
269 tracing_on = 0 271 tracing_on = 0
270 tracer = function_graph 272 tracer = function_graph
271 event.kprobes { 273 event.kprobes {
272 start_event { 274 start_event {
273 probes = "pci_proc_ini 275 probes = "pci_proc_init"
274 actions = "traceon" 276 actions = "traceon"
275 } 277 }
276 end_event { 278 end_event {
277 probes = "pci_proc_ini 279 probes = "pci_proc_init%return"
278 actions = "traceoff" 280 actions = "traceoff"
279 } 281 }
280 } 282 }
281 } 283 }
282 284
283 285
284 This boot-time tracing also supports ftrace ke 286 This boot-time tracing also supports ftrace kernel parameters via boot
285 config. 287 config.
286 For example, following kernel parameters:: 288 For example, following kernel parameters::
287 289
288 trace_options=sym-addr trace_event=initcall:* 290 trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"
289 291
290 This can be written in boot config like below: 292 This can be written in boot config like below::
291 293
292 kernel { 294 kernel {
293 trace_options = sym-addr 295 trace_options = sym-addr
294 trace_event = "initcall:*" 296 trace_event = "initcall:*"
295 tp_printk 297 tp_printk
296 trace_buf_size = 1M 298 trace_buf_size = 1M
297 ftrace = function 299 ftrace = function
298 ftrace_filter = "vfs*" 300 ftrace_filter = "vfs*"
299 } 301 }
300 302
301 Note that parameters start with "kernel" prefi 303 Note that parameters start with "kernel" prefix instead of "ftrace".
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.