1 =============
2 Event Tracing
3 =============
4
5 :Author: Theodore Ts'o
6 :Updated: Li Zefan and Tom Zanussi
7
8 1. Introduction
9 ===============
10
11 Tracepoints (see Documentation/trace/tracepoints.rst) can be used
12 without creating custom kernel modules to register probe functions
13 using the event tracing infrastructure.
14
15 Not all tracepoints can be traced using the event tracing system;
16 the kernel developer must provide code snippets which define how the
17 tracing information is saved into the tracing buffer, and how the
18 tracing information should be printed.
19
20 2. Using Event Tracing
21 ======================
22
23 2.1 Via the 'set_event' interface
24 ---------------------------------
25
26 The events which are available for tracing can be found in the file
27 /sys/kernel/tracing/available_events.
28
29 To enable a particular event, such as 'sched_wakeup', simply echo it
30 to /sys/kernel/tracing/set_event. For example::
31
32 # echo sched_wakeup >> /sys/kernel/tracing/set_event
33
34 .. Note:: '>>' is necessary, otherwise it will firstly disable all the events.
35
36 To disable an event, echo the event name to the set_event file prefixed
37 with an exclamation point::
38
39 # echo '!sched_wakeup' >> /sys/kernel/tracing/set_event
40
41 To disable all events, echo an empty line to the set_event file::
42
43 # echo > /sys/kernel/tracing/set_event
44
45 To enable all events, echo ``*:*`` or ``*:`` to the set_event file::
46
47 # echo *:* > /sys/kernel/tracing/set_event
48
49 The events are organized into subsystems, such as ext4, irq, sched,
50 etc., and a full event name looks like this: <subsystem>:<event>. The
51 subsystem name is optional, but it is displayed in the available_events
52 file. All of the events in a subsystem can be specified via the syntax
53 ``<subsystem>:*``; for example, to enable all irq events, you can use the
54 command::
55
56 # echo 'irq:*' > /sys/kernel/tracing/set_event
57
58 2.2 Via the 'enable' toggle
59 ---------------------------
60
61 The events available are also listed in /sys/kernel/tracing/events/ hierarchy
62 of directories.
63
64 To enable event 'sched_wakeup'::
65
66 # echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enable
67
68 To disable it::
69
70 # echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enable
71
72 To enable all events in sched subsystem::
73
74 # echo 1 > /sys/kernel/tracing/events/sched/enable
75
76 To enable all events::
77
78 # echo 1 > /sys/kernel/tracing/events/enable
79
80 When reading one of these enable files, there are four results:
81
82 - 0 - all events this file affects are disabled
83 - 1 - all events this file affects are enabled
84 - X - there is a mixture of events enabled and disabled
85 - ? - this file does not affect any event
86
87 2.3 Boot option
88 ---------------
89
90 In order to facilitate early boot debugging, use boot option::
91
92 trace_event=[event-list]
93
94 event-list is a comma separated list of events. See section 2.1 for event
95 format.
96
97 3. Defining an event-enabled tracepoint
98 =======================================
99
100 See The example provided in samples/trace_events
101
102 4. Event formats
103 ================
104
105 Each trace event has a 'format' file associated with it that contains
106 a description of each field in a logged event. This information can
107 be used to parse the binary trace stream, and is also the place to
108 find the field names that can be used in event filters (see section 5).
109
110 It also displays the format string that will be used to print the
111 event in text mode, along with the event name and ID used for
112 profiling.
113
114 Every event has a set of ``common`` fields associated with it; these are
115 the fields prefixed with ``common_``. The other fields vary between
116 events and correspond to the fields defined in the TRACE_EVENT
117 definition for that event.
118
119 Each field in the format has the form::
120
121 field:field-type field-name; offset:N; size:N;
122
123 where offset is the offset of the field in the trace record and size
124 is the size of the data item, in bytes.
125
126 For example, here's the information displayed for the 'sched_wakeup'
127 event::
128
129 # cat /sys/kernel/tracing/events/sched/sched_wakeup/format
130
131 name: sched_wakeup
132 ID: 60
133 format:
134 field:unsigned short common_type; offset:0; size:2;
135 field:unsigned char common_flags; offset:2; size:1;
136 field:unsigned char common_preempt_count; offset:3; size:1;
137 field:int common_pid; offset:4; size:4;
138 field:int common_tgid; offset:8; size:4;
139
140 field:char comm[TASK_COMM_LEN]; offset:12; size:16;
141 field:pid_t pid; offset:28; size:4;
142 field:int prio; offset:32; size:4;
143 field:int success; offset:36; size:4;
144 field:int cpu; offset:40; size:4;
145
146 print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,
147 REC->prio, REC->success, REC->cpu
148
149 This event contains 10 fields, the first 5 common and the remaining 5
150 event-specific. All the fields for this event are numeric, except for
151 'comm' which is a string, a distinction important for event filtering.
152
153 5. Event filtering
154 ==================
155
156 Trace events can be filtered in the kernel by associating boolean
157 'filter expressions' with them. As soon as an event is logged into
158 the trace buffer, its fields are checked against the filter expression
159 associated with that event type. An event with field values that
160 'match' the filter will appear in the trace output, and an event whose
161 values don't match will be discarded. An event with no filter
162 associated with it matches everything, and is the default when no
163 filter has been set for an event.
164
165 5.1 Expression syntax
166 ---------------------
167
168 A filter expression consists of one or more 'predicates' that can be
169 combined using the logical operators '&&' and '||'. A predicate is
170 simply a clause that compares the value of a field contained within a
171 logged event with a constant value and returns either 0 or 1 depending
172 on whether the field value matched (1) or didn't match (0)::
173
174 field-name relational-operator value
175
176 Parentheses can be used to provide arbitrary logical groupings and
177 double-quotes can be used to prevent the shell from interpreting
178 operators as shell metacharacters.
179
180 The field-names available for use in filters can be found in the
181 'format' files for trace events (see section 4).
182
183 The relational-operators depend on the type of the field being tested:
184
185 The operators available for numeric fields are:
186
187 ==, !=, <, <=, >, >=, &
188
189 And for string fields they are:
190
191 ==, !=, ~
192
193 The glob (~) accepts a wild card character (\*,?) and character classes
194 ([). For example::
195
196 prev_comm ~ "*sh"
197 prev_comm ~ "sh*"
198 prev_comm ~ "*sh*"
199 prev_comm ~ "ba*sh"
200
201 If the field is a pointer that points into user space (for example
202 "filename" from sys_enter_openat), then you have to append ".ustring" to the
203 field name::
204
205 filename.ustring ~ "password"
206
207 As the kernel will have to know how to retrieve the memory that the pointer
208 is at from user space.
209
210 You can convert any long type to a function address and search by function name::
211
212 call_site.function == security_prepare_creds
213
214 The above will filter when the field "call_site" falls on the address within
215 "security_prepare_creds". That is, it will compare the value of "call_site" and
216 the filter will return true if it is greater than or equal to the start of
217 the function "security_prepare_creds" and less than the end of that function.
218
219 The ".function" postfix can only be attached to values of size long, and can only
220 be compared with "==" or "!=".
221
222 Cpumask fields or scalar fields that encode a CPU number can be filtered using
223 a user-provided cpumask in cpulist format. The format is as follows::
224
225 CPUS{$cpulist}
226
227 Operators available to cpumask filtering are:
228
229 & (intersection), ==, !=
230
231 For example, this will filter events that have their .target_cpu field present
232 in the given cpumask::
233
234 target_cpu & CPUS{17-42}
235
236 5.2 Setting filters
237 -------------------
238
239 A filter for an individual event is set by writing a filter expression
240 to the 'filter' file for the given event.
241
242 For example::
243
244 # cd /sys/kernel/tracing/events/sched/sched_wakeup
245 # echo "common_preempt_count > 4" > filter
246
247 A slightly more involved example::
248
249 # cd /sys/kernel/tracing/events/signal/signal_generate
250 # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter
251
252 If there is an error in the expression, you'll get an 'Invalid
253 argument' error when setting it, and the erroneous string along with
254 an error message can be seen by looking at the filter e.g.::
255
256 # cd /sys/kernel/tracing/events/signal/signal_generate
257 # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter
258 -bash: echo: write error: Invalid argument
259 # cat filter
260 ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash
261 ^
262 parse_error: Field not found
263
264 Currently the caret ('^') for an error always appears at the beginning of
265 the filter string; the error message should still be useful though
266 even without more accurate position info.
267
268 5.2.1 Filter limitations
269 ------------------------
270
271 If a filter is placed on a string pointer ``(char *)`` that does not point
272 to a string on the ring buffer, but instead points to kernel or user space
273 memory, then, for safety reasons, at most 1024 bytes of the content is
274 copied onto a temporary buffer to do the compare. If the copy of the memory
275 faults (the pointer points to memory that should not be accessed), then the
276 string compare will be treated as not matching.
277
278 5.3 Clearing filters
279 --------------------
280
281 To clear the filter for an event, write a '0' to the event's filter
282 file.
283
284 To clear the filters for all events in a subsystem, write a '0' to the
285 subsystem's filter file.
286
287 5.4 Subsystem filters
288 ---------------------
289
290 For convenience, filters for every event in a subsystem can be set or
291 cleared as a group by writing a filter expression into the filter file
292 at the root of the subsystem. Note however, that if a filter for any
293 event within the subsystem lacks a field specified in the subsystem
294 filter, or if the filter can't be applied for any other reason, the
295 filter for that event will retain its previous setting. This can
296 result in an unintended mixture of filters which could lead to
297 confusing (to the user who might think different filters are in
298 effect) trace output. Only filters that reference just the common
299 fields can be guaranteed to propagate successfully to all events.
300
301 Here are a few subsystem filter examples that also illustrate the
302 above points:
303
304 Clear the filters on all events in the sched subsystem::
305
306 # cd /sys/kernel/tracing/events/sched
307 # echo 0 > filter
308 # cat sched_switch/filter
309 none
310 # cat sched_wakeup/filter
311 none
312
313 Set a filter using only common fields for all events in the sched
314 subsystem (all events end up with the same filter)::
315
316 # cd /sys/kernel/tracing/events/sched
317 # echo common_pid == 0 > filter
318 # cat sched_switch/filter
319 common_pid == 0
320 # cat sched_wakeup/filter
321 common_pid == 0
322
323 Attempt to set a filter using a non-common field for all events in the
324 sched subsystem (all events but those that have a prev_pid field retain
325 their old filters)::
326
327 # cd /sys/kernel/tracing/events/sched
328 # echo prev_pid == 0 > filter
329 # cat sched_switch/filter
330 prev_pid == 0
331 # cat sched_wakeup/filter
332 common_pid == 0
333
334 5.5 PID filtering
335 -----------------
336
337 The set_event_pid file in the same directory as the top events directory
338 exists, will filter all events from tracing any task that does not have the
339 PID listed in the set_event_pid file.
340 ::
341
342 # cd /sys/kernel/tracing
343 # echo $$ > set_event_pid
344 # echo 1 > events/enable
345
346 Will only trace events for the current task.
347
348 To add more PIDs without losing the PIDs already included, use '>>'.
349 ::
350
351 # echo 123 244 1 >> set_event_pid
352
353
354 6. Event triggers
355 =================
356
357 Trace events can be made to conditionally invoke trigger 'commands'
358 which can take various forms and are described in detail below;
359 examples would be enabling or disabling other trace events or invoking
360 a stack trace whenever the trace event is hit. Whenever a trace event
361 with attached triggers is invoked, the set of trigger commands
362 associated with that event is invoked. Any given trigger can
363 additionally have an event filter of the same form as described in
364 section 5 (Event filtering) associated with it - the command will only
365 be invoked if the event being invoked passes the associated filter.
366 If no filter is associated with the trigger, it always passes.
367
368 Triggers are added to and removed from a particular event by writing
369 trigger expressions to the 'trigger' file for the given event.
370
371 A given event can have any number of triggers associated with it,
372 subject to any restrictions that individual commands may have in that
373 regard.
374
375 Event triggers are implemented on top of "soft" mode, which means that
376 whenever a trace event has one or more triggers associated with it,
377 the event is activated even if it isn't actually enabled, but is
378 disabled in a "soft" mode. That is, the tracepoint will be called,
379 but just will not be traced, unless of course it's actually enabled.
380 This scheme allows triggers to be invoked even for events that aren't
381 enabled, and also allows the current event filter implementation to be
382 used for conditionally invoking triggers.
383
384 The syntax for event triggers is roughly based on the syntax for
385 set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands'
386 section of Documentation/trace/ftrace.rst), but there are major
387 differences and the implementation isn't currently tied to it in any
388 way, so beware about making generalizations between the two.
389
390 .. Note::
391 Writing into trace_marker (See Documentation/trace/ftrace.rst)
392 can also enable triggers that are written into
393 /sys/kernel/tracing/events/ftrace/print/trigger
394
395 6.1 Expression syntax
396 ---------------------
397
398 Triggers are added by echoing the command to the 'trigger' file::
399
400 # echo 'command[:count] [if filter]' > trigger
401
402 Triggers are removed by echoing the same command but starting with '!'
403 to the 'trigger' file::
404
405 # echo '!command[:count] [if filter]' > trigger
406
407 The [if filter] part isn't used in matching commands when removing, so
408 leaving that off in a '!' command will accomplish the same thing as
409 having it in.
410
411 The filter syntax is the same as that described in the 'Event
412 filtering' section above.
413
414 For ease of use, writing to the trigger file using '>' currently just
415 adds or removes a single trigger and there's no explicit '>>' support
416 ('>' actually behaves like '>>') or truncation support to remove all
417 triggers (you have to use '!' for each one added.)
418
419 6.2 Supported trigger commands
420 ------------------------------
421
422 The following commands are supported:
423
424 - enable_event/disable_event
425
426 These commands can enable or disable another trace event whenever
427 the triggering event is hit. When these commands are registered,
428 the other trace event is activated, but disabled in a "soft" mode.
429 That is, the tracepoint will be called, but just will not be traced.
430 The event tracepoint stays in this mode as long as there's a trigger
431 in effect that can trigger it.
432
433 For example, the following trigger causes kmalloc events to be
434 traced when a read system call is entered, and the :1 at the end
435 specifies that this enablement happens only once::
436
437 # echo 'enable_event:kmem:kmalloc:1' > \
438 /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger
439
440 The following trigger causes kmalloc events to stop being traced
441 when a read system call exits. This disablement happens on every
442 read system call exit::
443
444 # echo 'disable_event:kmem:kmalloc' > \
445 /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger
446
447 The format is::
448
449 enable_event:<system>:<event>[:count]
450 disable_event:<system>:<event>[:count]
451
452 To remove the above commands::
453
454 # echo '!enable_event:kmem:kmalloc:1' > \
455 /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger
456
457 # echo '!disable_event:kmem:kmalloc' > \
458 /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger
459
460 Note that there can be any number of enable/disable_event triggers
461 per triggering event, but there can only be one trigger per
462 triggered event. e.g. sys_enter_read can have triggers enabling both
463 kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc
464 versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if
465 bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they
466 could be combined into a single filter on kmem:kmalloc though).
467
468 - stacktrace
469
470 This command dumps a stacktrace in the trace buffer whenever the
471 triggering event occurs.
472
473 For example, the following trigger dumps a stacktrace every time the
474 kmalloc tracepoint is hit::
475
476 # echo 'stacktrace' > \
477 /sys/kernel/tracing/events/kmem/kmalloc/trigger
478
479 The following trigger dumps a stacktrace the first 5 times a kmalloc
480 request happens with a size >= 64K::
481
482 # echo 'stacktrace:5 if bytes_req >= 65536' > \
483 /sys/kernel/tracing/events/kmem/kmalloc/trigger
484
485 The format is::
486
487 stacktrace[:count]
488
489 To remove the above commands::
490
491 # echo '!stacktrace' > \
492 /sys/kernel/tracing/events/kmem/kmalloc/trigger
493
494 # echo '!stacktrace:5 if bytes_req >= 65536' > \
495 /sys/kernel/tracing/events/kmem/kmalloc/trigger
496
497 The latter can also be removed more simply by the following (without
498 the filter)::
499
500 # echo '!stacktrace:5' > \
501 /sys/kernel/tracing/events/kmem/kmalloc/trigger
502
503 Note that there can be only one stacktrace trigger per triggering
504 event.
505
506 - snapshot
507
508 This command causes a snapshot to be triggered whenever the
509 triggering event occurs.
510
511 The following command creates a snapshot every time a block request
512 queue is unplugged with a depth > 1. If you were tracing a set of
513 events or functions at the time, the snapshot trace buffer would
514 capture those events when the trigger event occurred::
515
516 # echo 'snapshot if nr_rq > 1' > \
517 /sys/kernel/tracing/events/block/block_unplug/trigger
518
519 To only snapshot once::
520
521 # echo 'snapshot:1 if nr_rq > 1' > \
522 /sys/kernel/tracing/events/block/block_unplug/trigger
523
524 To remove the above commands::
525
526 # echo '!snapshot if nr_rq > 1' > \
527 /sys/kernel/tracing/events/block/block_unplug/trigger
528
529 # echo '!snapshot:1 if nr_rq > 1' > \
530 /sys/kernel/tracing/events/block/block_unplug/trigger
531
532 Note that there can be only one snapshot trigger per triggering
533 event.
534
535 - traceon/traceoff
536
537 These commands turn tracing on and off when the specified events are
538 hit. The parameter determines how many times the tracing system is
539 turned on and off. If unspecified, there is no limit.
540
541 The following command turns tracing off the first time a block
542 request queue is unplugged with a depth > 1. If you were tracing a
543 set of events or functions at the time, you could then examine the
544 trace buffer to see the sequence of events that led up to the
545 trigger event::
546
547 # echo 'traceoff:1 if nr_rq > 1' > \
548 /sys/kernel/tracing/events/block/block_unplug/trigger
549
550 To always disable tracing when nr_rq > 1::
551
552 # echo 'traceoff if nr_rq > 1' > \
553 /sys/kernel/tracing/events/block/block_unplug/trigger
554
555 To remove the above commands::
556
557 # echo '!traceoff:1 if nr_rq > 1' > \
558 /sys/kernel/tracing/events/block/block_unplug/trigger
559
560 # echo '!traceoff if nr_rq > 1' > \
561 /sys/kernel/tracing/events/block/block_unplug/trigger
562
563 Note that there can be only one traceon or traceoff trigger per
564 triggering event.
565
566 - hist
567
568 This command aggregates event hits into a hash table keyed on one or
569 more trace event format fields (or stacktrace) and a set of running
570 totals derived from one or more trace event format fields and/or
571 event counts (hitcount).
572
573 See Documentation/trace/histogram.rst for details and examples.
574
575 7. In-kernel trace event API
576 ============================
577
578 In most cases, the command-line interface to trace events is more than
579 sufficient. Sometimes, however, applications might find the need for
580 more complex relationships than can be expressed through a simple
581 series of linked command-line expressions, or putting together sets of
582 commands may be simply too cumbersome. An example might be an
583 application that needs to 'listen' to the trace stream in order to
584 maintain an in-kernel state machine detecting, for instance, when an
585 illegal kernel state occurs in the scheduler.
586
587 The trace event subsystem provides an in-kernel API allowing modules
588 or other kernel code to generate user-defined 'synthetic' events at
589 will, which can be used to either augment the existing trace stream
590 and/or signal that a particular important state has occurred.
591
592 A similar in-kernel API is also available for creating kprobe and
593 kretprobe events.
594
595 Both the synthetic event and k/ret/probe event APIs are built on top
596 of a lower-level "dynevent_cmd" event command API, which is also
597 available for more specialized applications, or as the basis of other
598 higher-level trace event APIs.
599
600 The API provided for these purposes is describe below and allows the
601 following:
602
603 - dynamically creating synthetic event definitions
604 - dynamically creating kprobe and kretprobe event definitions
605 - tracing synthetic events from in-kernel code
606 - the low-level "dynevent_cmd" API
607
608 7.1 Dyamically creating synthetic event definitions
609 ---------------------------------------------------
610
611 There are a couple ways to create a new synthetic event from a kernel
612 module or other kernel code.
613
614 The first creates the event in one step, using synth_event_create().
615 In this method, the name of the event to create and an array defining
616 the fields is supplied to synth_event_create(). If successful, a
617 synthetic event with that name and fields will exist following that
618 call. For example, to create a new "schedtest" synthetic event::
619
620 ret = synth_event_create("schedtest", sched_fields,
621 ARRAY_SIZE(sched_fields), THIS_MODULE);
622
623 The sched_fields param in this example points to an array of struct
624 synth_field_desc, each of which describes an event field by type and
625 name::
626
627 static struct synth_field_desc sched_fields[] = {
628 { .type = "pid_t", .name = "next_pid_field" },
629 { .type = "char[16]", .name = "next_comm_field" },
630 { .type = "u64", .name = "ts_ns" },
631 { .type = "u64", .name = "ts_ms" },
632 { .type = "unsigned int", .name = "cpu" },
633 { .type = "char[64]", .name = "my_string_field" },
634 { .type = "int", .name = "my_int_field" },
635 };
636
637 See synth_field_size() for available types.
638
639 If field_name contains [n], the field is considered to be a static array.
640
641 If field_names contains[] (no subscript), the field is considered to
642 be a dynamic array, which will only take as much space in the event as
643 is required to hold the array.
644
645 Because space for an event is reserved before assigning field values
646 to the event, using dynamic arrays implies that the piecewise
647 in-kernel API described below can't be used with dynamic arrays. The
648 other non-piecewise in-kernel APIs can, however, be used with dynamic
649 arrays.
650
651 If the event is created from within a module, a pointer to the module
652 must be passed to synth_event_create(). This will ensure that the
653 trace buffer won't contain unreadable events when the module is
654 removed.
655
656 At this point, the event object is ready to be used for generating new
657 events.
658
659 In the second method, the event is created in several steps. This
660 allows events to be created dynamically and without the need to create
661 and populate an array of fields beforehand.
662
663 To use this method, an empty or partially empty synthetic event should
664 first be created using synth_event_gen_cmd_start() or
665 synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(),
666 the name of the event along with one or more pairs of args each pair
667 representing a 'type field_name;' field specification should be
668 supplied. For synth_event_gen_cmd_array_start(), the name of the
669 event along with an array of struct synth_field_desc should be
670 supplied. Before calling synth_event_gen_cmd_start() or
671 synth_event_gen_cmd_array_start(), the user should create and
672 initialize a dynevent_cmd object using synth_event_cmd_init().
673
674 For example, to create a new "schedtest" synthetic event with two
675 fields::
676
677 struct dynevent_cmd cmd;
678 char *buf;
679
680 /* Create a buffer to hold the generated command */
681 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
682
683 /* Before generating the command, initialize the cmd object */
684 synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
685
686 ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,
687 "pid_t", "next_pid_field",
688 "u64", "ts_ns");
689
690 Alternatively, using an array of struct synth_field_desc fields
691 containing the same information::
692
693 ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,
694 fields, n_fields);
695
696 Once the synthetic event object has been created, it can then be
697 populated with more fields. Fields are added one by one using
698 synth_event_add_field(), supplying the dynevent_cmd object, a field
699 type, and a field name. For example, to add a new int field named
700 "intfield", the following call should be made::
701
702 ret = synth_event_add_field(&cmd, "int", "intfield");
703
704 See synth_field_size() for available types. If field_name contains [n]
705 the field is considered to be an array.
706
707 A group of fields can also be added all at once using an array of
708 synth_field_desc with add_synth_fields(). For example, this would add
709 just the first four sched_fields::
710
711 ret = synth_event_add_fields(&cmd, sched_fields, 4);
712
713 If you already have a string of the form 'type field_name',
714 synth_event_add_field_str() can be used to add it as-is; it will
715 also automatically append a ';' to the string.
716
717 Once all the fields have been added, the event should be finalized and
718 registered by calling the synth_event_gen_cmd_end() function::
719
720 ret = synth_event_gen_cmd_end(&cmd);
721
722 At this point, the event object is ready to be used for tracing new
723 events.
724
725 7.2 Tracing synthetic events from in-kernel code
726 ------------------------------------------------
727
728 To trace a synthetic event, there are several options. The first
729 option is to trace the event in one call, using synth_event_trace()
730 with a variable number of values, or synth_event_trace_array() with an
731 array of values to be set. A second option can be used to avoid the
732 need for a pre-formed array of values or list of arguments, via
733 synth_event_trace_start() and synth_event_trace_end() along with
734 synth_event_add_next_val() or synth_event_add_val() to add the values
735 piecewise.
736
737 7.2.1 Tracing a synthetic event all at once
738 -------------------------------------------
739
740 To trace a synthetic event all at once, the synth_event_trace() or
741 synth_event_trace_array() functions can be used.
742
743 The synth_event_trace() function is passed the trace_event_file
744 representing the synthetic event (which can be retrieved using
745 trace_get_event_file() using the synthetic event name, "synthetic" as
746 the system name, and the trace instance name (NULL if using the global
747 trace array)), along with an variable number of u64 args, one for each
748 synthetic event field, and the number of values being passed.
749
750 So, to trace an event corresponding to the synthetic event definition
751 above, code like the following could be used::
752
753 ret = synth_event_trace(create_synth_test, 7, /* number of values */
754 444, /* next_pid_field */
755 (u64)"clackers", /* next_comm_field */
756 1000000, /* ts_ns */
757 1000, /* ts_ms */
758 smp_processor_id(),/* cpu */
759 (u64)"Thneed", /* my_string_field */
760 999); /* my_int_field */
761
762 All vals should be cast to u64, and string vals are just pointers to
763 strings, cast to u64. Strings will be copied into space reserved in
764 the event for the string, using these pointers.
765
766 Alternatively, the synth_event_trace_array() function can be used to
767 accomplish the same thing. It is passed the trace_event_file
768 representing the synthetic event (which can be retrieved using
769 trace_get_event_file() using the synthetic event name, "synthetic" as
770 the system name, and the trace instance name (NULL if using the global
771 trace array)), along with an array of u64, one for each synthetic
772 event field.
773
774 To trace an event corresponding to the synthetic event definition
775 above, code like the following could be used::
776
777 u64 vals[7];
778
779 vals[0] = 777; /* next_pid_field */
780 vals[1] = (u64)"tiddlywinks"; /* next_comm_field */
781 vals[2] = 1000000; /* ts_ns */
782 vals[3] = 1000; /* ts_ms */
783 vals[4] = smp_processor_id(); /* cpu */
784 vals[5] = (u64)"thneed"; /* my_string_field */
785 vals[6] = 398; /* my_int_field */
786
787 The 'vals' array is just an array of u64, the number of which must
788 match the number of field in the synthetic event, and which must be in
789 the same order as the synthetic event fields.
790
791 All vals should be cast to u64, and string vals are just pointers to
792 strings, cast to u64. Strings will be copied into space reserved in
793 the event for the string, using these pointers.
794
795 In order to trace a synthetic event, a pointer to the trace event file
796 is needed. The trace_get_event_file() function can be used to get
797 it - it will find the file in the given trace instance (in this case
798 NULL since the top trace array is being used) while at the same time
799 preventing the instance containing it from going away::
800
801 schedtest_event_file = trace_get_event_file(NULL, "synthetic",
802 "schedtest");
803
804 Before tracing the event, it should be enabled in some way, otherwise
805 the synthetic event won't actually show up in the trace buffer.
806
807 To enable a synthetic event from the kernel, trace_array_set_clr_event()
808 can be used (which is not specific to synthetic events, so does need
809 the "synthetic" system name to be specified explicitly).
810
811 To enable the event, pass 'true' to it::
812
813 trace_array_set_clr_event(schedtest_event_file->tr,
814 "synthetic", "schedtest", true);
815
816 To disable it pass false::
817
818 trace_array_set_clr_event(schedtest_event_file->tr,
819 "synthetic", "schedtest", false);
820
821 Finally, synth_event_trace_array() can be used to actually trace the
822 event, which should be visible in the trace buffer afterwards::
823
824 ret = synth_event_trace_array(schedtest_event_file, vals,
825 ARRAY_SIZE(vals));
826
827 To remove the synthetic event, the event should be disabled, and the
828 trace instance should be 'put' back using trace_put_event_file()::
829
830 trace_array_set_clr_event(schedtest_event_file->tr,
831 "synthetic", "schedtest", false);
832 trace_put_event_file(schedtest_event_file);
833
834 If those have been successful, synth_event_delete() can be called to
835 remove the event::
836
837 ret = synth_event_delete("schedtest");
838
839 7.2.2 Tracing a synthetic event piecewise
840 -----------------------------------------
841
842 To trace a synthetic using the piecewise method described above, the
843 synth_event_trace_start() function is used to 'open' the synthetic
844 event trace::
845
846 struct synth_event_trace_state trace_state;
847
848 ret = synth_event_trace_start(schedtest_event_file, &trace_state);
849
850 It's passed the trace_event_file representing the synthetic event
851 using the same methods as described above, along with a pointer to a
852 struct synth_event_trace_state object, which will be zeroed before use and
853 used to maintain state between this and following calls.
854
855 Once the event has been opened, which means space for it has been
856 reserved in the trace buffer, the individual fields can be set. There
857 are two ways to do that, either one after another for each field in
858 the event, which requires no lookups, or by name, which does. The
859 tradeoff is flexibility in doing the assignments vs the cost of a
860 lookup per field.
861
862 To assign the values one after the other without lookups,
863 synth_event_add_next_val() should be used. Each call is passed the
864 same synth_event_trace_state object used in the synth_event_trace_start(),
865 along with the value to set the next field in the event. After each
866 field is set, the 'cursor' points to the next field, which will be set
867 by the subsequent call, continuing until all the fields have been set
868 in order. The same sequence of calls as in the above examples using
869 this method would be (without error-handling code)::
870
871 /* next_pid_field */
872 ret = synth_event_add_next_val(777, &trace_state);
873
874 /* next_comm_field */
875 ret = synth_event_add_next_val((u64)"slinky", &trace_state);
876
877 /* ts_ns */
878 ret = synth_event_add_next_val(1000000, &trace_state);
879
880 /* ts_ms */
881 ret = synth_event_add_next_val(1000, &trace_state);
882
883 /* cpu */
884 ret = synth_event_add_next_val(smp_processor_id(), &trace_state);
885
886 /* my_string_field */
887 ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);
888
889 /* my_int_field */
890 ret = synth_event_add_next_val(395, &trace_state);
891
892 To assign the values in any order, synth_event_add_val() should be
893 used. Each call is passed the same synth_event_trace_state object used in
894 the synth_event_trace_start(), along with the field name of the field
895 to set and the value to set it to. The same sequence of calls as in
896 the above examples using this method would be (without error-handling
897 code)::
898
899 ret = synth_event_add_val("next_pid_field", 777, &trace_state);
900 ret = synth_event_add_val("next_comm_field", (u64)"silly putty",
901 &trace_state);
902 ret = synth_event_add_val("ts_ns", 1000000, &trace_state);
903 ret = synth_event_add_val("ts_ms", 1000, &trace_state);
904 ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);
905 ret = synth_event_add_val("my_string_field", (u64)"thneed_9",
906 &trace_state);
907 ret = synth_event_add_val("my_int_field", 3999, &trace_state);
908
909 Note that synth_event_add_next_val() and synth_event_add_val() are
910 incompatible if used within the same trace of an event - either one
911 can be used but not both at the same time.
912
913 Finally, the event won't be actually traced until it's 'closed',
914 which is done using synth_event_trace_end(), which takes only the
915 struct synth_event_trace_state object used in the previous calls::
916
917 ret = synth_event_trace_end(&trace_state);
918
919 Note that synth_event_trace_end() must be called at the end regardless
920 of whether any of the add calls failed (say due to a bad field name
921 being passed in).
922
923 7.3 Dyamically creating kprobe and kretprobe event definitions
924 --------------------------------------------------------------
925
926 To create a kprobe or kretprobe trace event from kernel code, the
927 kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start()
928 functions can be used.
929
930 To create a kprobe event, an empty or partially empty kprobe event
931 should first be created using kprobe_event_gen_cmd_start(). The name
932 of the event and the probe location should be specified along with one
933 or args each representing a probe field should be supplied to this
934 function. Before calling kprobe_event_gen_cmd_start(), the user
935 should create and initialize a dynevent_cmd object using
936 kprobe_event_cmd_init().
937
938 For example, to create a new "schedtest" kprobe event with two fields::
939
940 struct dynevent_cmd cmd;
941 char *buf;
942
943 /* Create a buffer to hold the generated command */
944 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
945
946 /* Before generating the command, initialize the cmd object */
947 kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
948
949 /*
950 * Define the gen_kprobe_test event with the first 2 kprobe
951 * fields.
952 */
953 ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",
954 "dfd=%ax", "filename=%dx");
955
956 Once the kprobe event object has been created, it can then be
957 populated with more fields. Fields can be added using
958 kprobe_event_add_fields(), supplying the dynevent_cmd object along
959 with a variable arg list of probe fields. For example, to add a
960 couple additional fields, the following call could be made::
961
962 ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");
963
964 Once all the fields have been added, the event should be finalized and
965 registered by calling the kprobe_event_gen_cmd_end() or
966 kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe
967 or kretprobe command was started::
968
969 ret = kprobe_event_gen_cmd_end(&cmd);
970
971 or::
972
973 ret = kretprobe_event_gen_cmd_end(&cmd);
974
975 At this point, the event object is ready to be used for tracing new
976 events.
977
978 Similarly, a kretprobe event can be created using
979 kretprobe_event_gen_cmd_start() with a probe name and location and
980 additional params such as $retval::
981
982 ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",
983 "do_sys_open", "$retval");
984
985 Similar to the synthetic event case, code like the following can be
986 used to enable the newly created kprobe event::
987
988 gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");
989
990 ret = trace_array_set_clr_event(gen_kprobe_test->tr,
991 "kprobes", "gen_kprobe_test", true);
992
993 Finally, also similar to synthetic events, the following code can be
994 used to give the kprobe event file back and delete the event::
995
996 trace_put_event_file(gen_kprobe_test);
997
998 ret = kprobe_event_delete("gen_kprobe_test");
999
1000 7.4 The "dynevent_cmd" low-level API
1001 ------------------------------------
1002
1003 Both the in-kernel synthetic event and kprobe interfaces are built on
1004 top of a lower-level "dynevent_cmd" interface. This interface is
1005 meant to provide the basis for higher-level interfaces such as the
1006 synthetic and kprobe interfaces, which can be used as examples.
1007
1008 The basic idea is simple and amounts to providing a general-purpose
1009 layer that can be used to generate trace event commands. The
1010 generated command strings can then be passed to the command-parsing
1011 and event creation code that already exists in the trace event
1012 subsystem for creating the corresponding trace events.
1013
1014 In a nutshell, the way it works is that the higher-level interface
1015 code creates a struct dynevent_cmd object, then uses a couple
1016 functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up
1017 a command string, which finally causes the command to be executed
1018 using the dynevent_create() function. The details of the interface
1019 are described below.
1020
1021 The first step in building a new command string is to create and
1022 initialize an instance of a dynevent_cmd. Here, for instance, we
1023 create a dynevent_cmd on the stack and initialize it::
1024
1025 struct dynevent_cmd cmd;
1026 char *buf;
1027 int ret;
1028
1029 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
1030
1031 dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,
1032 foo_event_run_command);
1033
1034 The dynevent_cmd initialization needs to be given a user-specified
1035 buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used
1036 for this purpose - at 2k it's generally too big to be comfortably put
1037 on the stack, so is dynamically allocated), a dynevent type id, which
1038 is meant to be used to check that further API calls are for the
1039 correct command type, and a pointer to an event-specific run_command()
1040 callback that will be called to actually execute the event-specific
1041 command function.
1042
1043 Once that's done, the command string can by built up by successive
1044 calls to argument-adding functions.
1045
1046 To add a single argument, define and initialize a struct dynevent_arg
1047 or struct dynevent_arg_pair object. Here's an example of the simplest
1048 possible arg addition, which is simply to append the given string as
1049 a whitespace-separated argument to the command::
1050
1051 struct dynevent_arg arg;
1052
1053 dynevent_arg_init(&arg, NULL, 0);
1054
1055 arg.str = name;
1056
1057 ret = dynevent_arg_add(cmd, &arg);
1058
1059 The arg object is first initialized using dynevent_arg_init() and in
1060 this case the parameters are NULL or 0, which means there's no
1061 optional sanity-checking function or separator appended to the end of
1062 the arg.
1063
1064 Here's another more complicated example using an 'arg pair', which is
1065 used to create an argument that consists of a couple components added
1066 together as a unit, for example, a 'type field_name;' arg or a simple
1067 expression arg e.g. 'flags=%cx'::
1068
1069 struct dynevent_arg_pair arg_pair;
1070
1071 dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');
1072
1073 arg_pair.lhs = type;
1074 arg_pair.rhs = name;
1075
1076 ret = dynevent_arg_pair_add(cmd, &arg_pair);
1077
1078 Again, the arg_pair is first initialized, in this case with a callback
1079 function used to check the sanity of the args (for example, that
1080 neither part of the pair is NULL), along with a character to be used
1081 to add an operator between the pair (here none) and a separator to be
1082 appended onto the end of the arg pair (here ';').
1083
1084 There's also a dynevent_str_add() function that can be used to simply
1085 add a string as-is, with no spaces, delimiters, or arg check.
1086
1087 Any number of dynevent_*_add() calls can be made to build up the string
1088 (until its length surpasses cmd->maxlen). When all the arguments have
1089 been added and the command string is complete, the only thing left to
1090 do is run the command, which happens by simply calling
1091 dynevent_create()::
1092
1093 ret = dynevent_create(&cmd);
1094
1095 At that point, if the return value is 0, the dynamic event has been
1096 created and is ready to use.
1097
1098 See the dynevent_cmd function definitions themselves for the details
1099 of the API.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.