1 perf-probe(1) 1 perf-probe(1)
2 ============= 2 =============
3 3
4 NAME 4 NAME
5 ---- 5 ----
6 perf-probe - Define new dynamic tracepoints 6 perf-probe - Define new dynamic tracepoints
7 7
8 SYNOPSIS 8 SYNOPSIS
9 -------- 9 --------
10 [verse] 10 [verse]
11 'perf probe' [options] --add='PROBE' [...] 11 'perf probe' [options] --add='PROBE' [...]
12 or 12 or
13 'perf probe' [options] PROBE 13 'perf probe' [options] PROBE
14 or 14 or
15 'perf probe' [options] --del='[GROUP:]EVENT' [ 15 'perf probe' [options] --del='[GROUP:]EVENT' [...]
16 or 16 or
17 'perf probe' --list[=[GROUP:]EVENT] 17 'perf probe' --list[=[GROUP:]EVENT]
18 or 18 or
19 'perf probe' [options] --line='LINE' 19 'perf probe' [options] --line='LINE'
20 or 20 or
21 'perf probe' [options] --vars='PROBEPOINT' 21 'perf probe' [options] --vars='PROBEPOINT'
22 or 22 or
23 'perf probe' [options] --funcs 23 'perf probe' [options] --funcs
24 or 24 or
25 'perf probe' [options] --definition='PROBE' [. 25 'perf probe' [options] --definition='PROBE' [...]
26 26
27 DESCRIPTION 27 DESCRIPTION
28 ----------- 28 -----------
29 This command defines dynamic tracepoint events 29 This command defines dynamic tracepoint events, by symbol and registers
30 without debuginfo, or by C expressions (C line 30 without debuginfo, or by C expressions (C line numbers, C function names,
31 and C local variables) with debuginfo. 31 and C local variables) with debuginfo.
32 32
33 33
34 OPTIONS 34 OPTIONS
35 ------- 35 -------
36 -k:: 36 -k::
37 --vmlinux=PATH:: 37 --vmlinux=PATH::
38 Specify vmlinux path which has debugin 38 Specify vmlinux path which has debuginfo (Dwarf binary).
39 Only when using this with --definition 39 Only when using this with --definition, you can give an offline
40 vmlinux file. 40 vmlinux file.
41 41
42 -m:: 42 -m::
43 --module=MODNAME|PATH:: 43 --module=MODNAME|PATH::
44 Specify module name in which perf-prob 44 Specify module name in which perf-probe searches probe points
45 or lines. If a path of module file is 45 or lines. If a path of module file is passed, perf-probe
46 treat it as an offline module (this me 46 treat it as an offline module (this means you can add a probe on
47 a module which has not been loaded yet 47 a module which has not been loaded yet).
48 48
49 -s:: 49 -s::
50 --source=PATH:: 50 --source=PATH::
51 Specify path to kernel source. 51 Specify path to kernel source.
52 52
53 -v:: 53 -v::
54 --verbose:: 54 --verbose::
55 Be more verbose (show parsed arguments 55 Be more verbose (show parsed arguments, etc).
56 Can not use with -q. 56 Can not use with -q.
57 57
58 -q:: 58 -q::
59 --quiet:: 59 --quiet::
60 Do not show any warnings or messages. 60 Do not show any warnings or messages.
61 Can not use with -v. 61 Can not use with -v.
62 62
63 -a:: 63 -a::
64 --add=:: 64 --add=::
65 Define a probe event (see PROBE SYNTAX 65 Define a probe event (see PROBE SYNTAX for detail).
66 66
67 -d:: 67 -d::
68 --del=:: 68 --del=::
69 Delete probe events. This accepts glob 69 Delete probe events. This accepts glob wildcards('*', '?') and character
70 classes(e.g. [a-z], [!A-Z]). 70 classes(e.g. [a-z], [!A-Z]).
71 71
72 -l:: 72 -l::
73 --list[=[GROUP:]EVENT]:: 73 --list[=[GROUP:]EVENT]::
74 List up current probe events. This can 74 List up current probe events. This can also accept filtering patterns of
75 event names. 75 event names.
76 When this is used with --cache, perf s 76 When this is used with --cache, perf shows all cached probes instead of
77 the live probes. 77 the live probes.
78 78
79 -L:: 79 -L::
80 --line=:: 80 --line=::
81 Show source code lines which can be pr 81 Show source code lines which can be probed. This needs an argument
82 which specifies a range of the source 82 which specifies a range of the source code. (see LINE SYNTAX for detail)
83 83
84 -V:: 84 -V::
85 --vars=:: 85 --vars=::
86 Show available local variables at give 86 Show available local variables at given probe point. The argument
87 syntax is same as PROBE SYNTAX, but NO 87 syntax is same as PROBE SYNTAX, but NO ARGs.
88 88
89 --externs:: 89 --externs::
90 (Only for --vars) Show external define 90 (Only for --vars) Show external defined variables in addition to local
91 variables. 91 variables.
92 92
93 --no-inlines:: 93 --no-inlines::
94 (Only for --add) Search only for non-i 94 (Only for --add) Search only for non-inlined functions. The functions
95 which do not have instances are ignore 95 which do not have instances are ignored.
96 96
97 -F:: 97 -F::
98 --funcs[=FILTER]:: 98 --funcs[=FILTER]::
99 Show available functions in given modu 99 Show available functions in given module or kernel. With -x/--exec,
100 can also list functions in a user spac 100 can also list functions in a user space executable / shared library.
101 This also can accept a FILTER rule arg 101 This also can accept a FILTER rule argument.
102 102
103 -D:: 103 -D::
104 --definition=:: 104 --definition=::
105 Show trace-event definition converted 105 Show trace-event definition converted from given probe-event instead
106 of write it into tracing/[k,u]probe_ev 106 of write it into tracing/[k,u]probe_events.
107 107
108 --filter=FILTER:: 108 --filter=FILTER::
109 (Only for --vars and --funcs) Set filt 109 (Only for --vars and --funcs) Set filter. FILTER is a combination of glob
110 pattern, see FILTER PATTERN for detail 110 pattern, see FILTER PATTERN for detail.
111 Default FILTER is "!__k???tab_* & !__c 111 Default FILTER is "!__k???tab_* & !__crc_*" for --vars, and "!_*"
112 for --funcs. 112 for --funcs.
113 If several filters are specified, only 113 If several filters are specified, only the last filter is used.
114 114
115 -f:: 115 -f::
116 --force:: 116 --force::
117 Forcibly add events with existing name 117 Forcibly add events with existing name.
118 118
119 -n:: 119 -n::
120 --dry-run:: 120 --dry-run::
121 Dry run. With this option, --add and - 121 Dry run. With this option, --add and --del doesn't execute actual
122 adding and removal operations. 122 adding and removal operations.
123 123
124 --cache:: 124 --cache::
125 (With --add) Cache the probes. Any eve 125 (With --add) Cache the probes. Any events which successfully added
126 are also stored in the cache file. 126 are also stored in the cache file.
127 (With --list) Show cached probes. 127 (With --list) Show cached probes.
128 (With --del) Remove cached probes. 128 (With --del) Remove cached probes.
129 129
130 --max-probes=NUM:: 130 --max-probes=NUM::
131 Set the maximum number of probe points 131 Set the maximum number of probe points for an event. Default is 128.
132 132
133 --target-ns=PID: 133 --target-ns=PID:
134 Obtain mount namespace information fro 134 Obtain mount namespace information from the target pid. This is
135 used when creating a uprobe for a proc 135 used when creating a uprobe for a process that resides in a
136 different mount namespace from the per 136 different mount namespace from the perf(1) utility.
137 137
138 -x:: 138 -x::
139 --exec=PATH:: 139 --exec=PATH::
140 Specify path to the executable or shar 140 Specify path to the executable or shared library file for user
141 space tracing. Can also be used with - 141 space tracing. Can also be used with --funcs option.
142 142
143 --demangle:: 143 --demangle::
144 Demangle application symbols. --no-dem 144 Demangle application symbols. --no-demangle is also available
145 for disabling demangling. 145 for disabling demangling.
146 146
147 --demangle-kernel:: 147 --demangle-kernel::
148 Demangle kernel symbols. --no-demangle 148 Demangle kernel symbols. --no-demangle-kernel is also available
149 for disabling kernel demangling. 149 for disabling kernel demangling.
150 150
151 In absence of -m/-x options, perf probe checks 151 In absence of -m/-x options, perf probe checks if the first argument after
152 the options is an absolute path name. If its a 152 the options is an absolute path name. If its an absolute path, perf probe
153 uses it as a target module/target user space b 153 uses it as a target module/target user space binary to probe.
154 154
155 PROBE SYNTAX 155 PROBE SYNTAX
156 ------------ 156 ------------
157 Probe points are defined by following syntax. 157 Probe points are defined by following syntax.
158 158
159 1) Define event based on function name 159 1) Define event based on function name
160 [[GROUP:]EVENT=]FUNC[@SRC][:RLN|+OFFS|%re 160 [[GROUP:]EVENT=]FUNC[@SRC][:RLN|+OFFS|%return|;PTN] [ARG ...]
161 161
162 2) Define event based on source file with 162 2) Define event based on source file with line number
163 [[GROUP:]EVENT=]SRC:ALN [ARG ...] 163 [[GROUP:]EVENT=]SRC:ALN [ARG ...]
164 164
165 3) Define event based on source file with 165 3) Define event based on source file with lazy pattern
166 [[GROUP:]EVENT=]SRC;PTN [ARG ...] 166 [[GROUP:]EVENT=]SRC;PTN [ARG ...]
167 167
168 4) Pre-defined SDT events or cached event 168 4) Pre-defined SDT events or cached event with name
169 %[sdt_PROVIDER:]SDTEVENT 169 %[sdt_PROVIDER:]SDTEVENT
170 or, 170 or,
171 sdt_PROVIDER:SDTEVENT 171 sdt_PROVIDER:SDTEVENT
172 172
173 'EVENT' specifies the name of new event, if om 173 'EVENT' specifies the name of new event, if omitted, it will be set the name of the probed function, and for return probes, a "\_\_return" suffix is automatically added to the function name. You can also specify a group name by 'GROUP', if omitted, set 'probe' is used for kprobe and 'probe_<bin>' is used for uprobe.
174 Note that using existing group name can confli 174 Note that using existing group name can conflict with other events. Especially, using the group name reserved for kernel modules can hide embedded events in the
175 modules. 175 modules.
176 'FUNC' specifies a probed function name, and i 176 'FUNC' specifies a probed function name, and it may have one of the following options; '+OFFS' is the offset from function entry address in bytes, ':RLN' is the relative-line number from function entry line, and '%return' means that it probes function return. And ';PTN' means lazy matching pattern (see LAZY MATCHING). Note that ';PTN' must be the end of the probe point definition. In addition, '@SRC' specifies a source file which has that function.
177 It is also possible to specify a probe point b 177 It is also possible to specify a probe point by the source line number or lazy matching by using 'SRC:ALN' or 'SRC;PTN' syntax, where 'SRC' is the source file path, ':ALN' is the line number and ';PTN' is the lazy matching pattern.
178 'ARG' specifies the arguments of this probe po 178 'ARG' specifies the arguments of this probe point, (see PROBE ARGUMENT).
179 'SDTEVENT' and 'PROVIDER' is the pre-defined e 179 'SDTEVENT' and 'PROVIDER' is the pre-defined event name which is defined by user SDT (Statically Defined Tracing) or the pre-cached probes with event name.
180 Note that before using the SDT event, the targ 180 Note that before using the SDT event, the target binary (on which SDT events are defined) must be scanned by linkperf:perf-buildid-cache[1] to make SDT events as cached events.
181 181
182 For details of the SDT, see below. 182 For details of the SDT, see below.
183 https://sourceware.org/gdb/onlinedocs/gdb/Stat 183 https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html
184 184
185 ESCAPED CHARACTER 185 ESCAPED CHARACTER
186 ----------------- 186 -----------------
187 187
188 In the probe syntax, '=', '@', '+', ':' and '; 188 In the probe syntax, '=', '@', '+', ':' and ';' are treated as a special character. You can use a backslash ('\') to escape the special characters.
189 This is useful if you need to probe on a speci 189 This is useful if you need to probe on a specific versioned symbols, like @GLIBC_... suffixes, or also you need to specify a source file which includes the special characters.
190 Note that usually single backslash is consumed 190 Note that usually single backslash is consumed by shell, so you might need to pass double backslash (\\) or wrapping with single quotes (\'AAA\@BBB').
191 See EXAMPLES how it is used. 191 See EXAMPLES how it is used.
192 192
193 PROBE ARGUMENT 193 PROBE ARGUMENT
194 -------------- 194 --------------
195 Each probe argument follows below syntax. 195 Each probe argument follows below syntax.
196 196
197 [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE][@ 197 [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE][@user]
198 198
199 'NAME' specifies the name of this argument (op 199 'NAME' specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var->field, var.field2), local array with fixed index (e.g. array[1], var->array[0], var->pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this argument will be set as the last member name if you specify a local data structure member (e.g. field2 for 'var->field1.field2'.)
200 '$vars' and '$params' special arguments are al 200 '$vars' and '$params' special arguments are also available for NAME, '$vars' is expanded to the local variables (including function parameters) which can access at given probe point. '$params' is expanded to only the function parameters.
201 'TYPE' casts the type of this argument (option 201 'TYPE' casts the type of this argument (optional). If omitted, perf probe automatically set the type based on debuginfo (*). Currently, basic types (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal integers (x/x8/x16/x32/x64), signedness casting (u/s), "string" and bitfield are supported. (see TYPES for detail)
202 On x86 systems %REG is always the short form o 202 On x86 systems %REG is always the short form of the register: for example %AX. %RAX or %EAX is not valid.
203 "@user" is a special attribute which means the 203 "@user" is a special attribute which means the LOCALVAR will be treated as a user-space memory. This is only valid for kprobe event.
204 204
205 TYPES 205 TYPES
206 ----- 206 -----
207 Basic types (u8/u16/u32/u64/s8/s16/s32/s64) an 207 Basic types (u8/u16/u32/u64/s8/s16/s32/s64) and hexadecimal integers (x8/x16/x32/x64) are integer types. Prefix 's' and 'u' means those types are signed and unsigned respectively, and 'x' means that is shown in hexadecimal format. Traced arguments are shown in decimal (sNN/uNN) or hex (xNN). You can also use 's' or 'u' to specify only signedness and leave its size auto-detected by perf probe. Moreover, you can use 'x' to explicitly specify to be shown in hexadecimal (the size is also auto-detected).
208 String type is a special type, which fetches a 208 String type is a special type, which fetches a "null-terminated" string from kernel space. This means it will fail and store NULL if the string container has been paged out. You can specify 'string' type only for the local variable or structure member which is an array of or a pointer to 'char' or 'unsigned char' type.
209 Bitfield is another special type, which takes 209 Bitfield is another special type, which takes 3 parameters, bit-width, bit-offset, and container-size (usually 32). The syntax is;
210 210
211 b<bit-width>@<bit-offset>/<container-size> 211 b<bit-width>@<bit-offset>/<container-size>
212 212
213 LINE SYNTAX 213 LINE SYNTAX
214 ----------- 214 -----------
215 Line range is described by following syntax. 215 Line range is described by following syntax.
216 216
217 "FUNC[@SRC][:RLN[+NUM|-RLN2]]|SRC[:ALN[+NUM|- 217 "FUNC[@SRC][:RLN[+NUM|-RLN2]]|SRC[:ALN[+NUM|-ALN2]]"
218 218
219 FUNC specifies the function name of showing li 219 FUNC specifies the function name of showing lines. 'RLN' is the start line
220 number from function entry line, and 'RLN2' is 220 number from function entry line, and 'RLN2' is the end line number. As same as
221 probe syntax, 'SRC' means the source file path 221 probe syntax, 'SRC' means the source file path, 'ALN' is start line number,
222 and 'ALN2' is end line number in the file. It 222 and 'ALN2' is end line number in the file. It is also possible to specify how
223 many lines to show by using 'NUM'. Moreover, ' 223 many lines to show by using 'NUM'. Moreover, 'FUNC@SRC' combination is good
224 for searching a specific function when several 224 for searching a specific function when several functions share same name.
225 So, "source.c:100-120" shows lines between 100 225 So, "source.c:100-120" shows lines between 100th to 120th in source.c file. And "func:10+20" shows 20 lines from 10th line of func function.
226 226
227 LAZY MATCHING 227 LAZY MATCHING
228 ------------- 228 -------------
229 The lazy line matching is similar to glob matc 229 The lazy line matching is similar to glob matching but ignoring spaces in both of pattern and target. So this accepts wildcards('*', '?') and character classes(e.g. [a-z], [!A-Z]).
230 230
231 e.g. 231 e.g.
232 'a=*' can matches 'a=b', 'a = b', 'a == b' an 232 'a=*' can matches 'a=b', 'a = b', 'a == b' and so on.
233 233
234 This provides some sort of flexibility and rob 234 This provides some sort of flexibility and robustness to probe point definitions against minor code changes. For example, actual 10th line of schedule() can be moved easily by modifying schedule(), but the same line matching 'rq=cpu_rq*' may still exist in the function.)
235 235
236 FILTER PATTERN 236 FILTER PATTERN
237 -------------- 237 --------------
238 The filter pattern is a glob matching pattern( 238 The filter pattern is a glob matching pattern(s) to filter variables.
239 In addition, you can use "!" for specifying fi 239 In addition, you can use "!" for specifying filter-out rule. You also can give several rules combined with "&" or "|", and fold those rules as one rule by using "(" ")".
240 240
241 e.g. 241 e.g.
242 With --filter "foo* | bar*", perf probe -V sh 242 With --filter "foo* | bar*", perf probe -V shows variables which start with "foo" or "bar".
243 With --filter "!foo* & *bar", perf probe -V s 243 With --filter "!foo* & *bar", perf probe -V shows variables which don't start with "foo" and end with "bar", like "fizzbar". But "foobar" is filtered out.
244 244
245 EXAMPLES 245 EXAMPLES
246 -------- 246 --------
247 Display which lines in schedule() can be probe 247 Display which lines in schedule() can be probed:
248 248
249 ./perf probe --line schedule 249 ./perf probe --line schedule
250 250
251 Add a probe on schedule() function 12th line w 251 Add a probe on schedule() function 12th line with recording cpu local variable:
252 252
253 ./perf probe schedule:12 cpu 253 ./perf probe schedule:12 cpu
254 or 254 or
255 ./perf probe --add='schedule:12 cpu' 255 ./perf probe --add='schedule:12 cpu'
256 256
257 Add one or more probes which has the name star 257 Add one or more probes which has the name start with "schedule".
258 258
259 ./perf probe schedule* 259 ./perf probe schedule*
260 or 260 or
261 ./perf probe --add='schedule*' 261 ./perf probe --add='schedule*'
262 262
263 Add probes on lines in schedule() function whi 263 Add probes on lines in schedule() function which calls update_rq_clock().
264 264
265 ./perf probe 'schedule;update_rq_clock*' 265 ./perf probe 'schedule;update_rq_clock*'
266 or 266 or
267 ./perf probe --add='schedule;update_rq_clock* 267 ./perf probe --add='schedule;update_rq_clock*'
268 268
269 Delete all probes on schedule(). 269 Delete all probes on schedule().
270 270
271 ./perf probe --del='schedule*' 271 ./perf probe --del='schedule*'
272 272
273 Add probes at zfree() function on /bin/zsh 273 Add probes at zfree() function on /bin/zsh
274 274
275 ./perf probe -x /bin/zsh zfree or ./perf prob 275 ./perf probe -x /bin/zsh zfree or ./perf probe /bin/zsh zfree
276 276
277 Add probes at malloc() function on libc 277 Add probes at malloc() function on libc
278 278
279 ./perf probe -x /lib/libc.so.6 malloc or ./pe 279 ./perf probe -x /lib/libc.so.6 malloc or ./perf probe /lib/libc.so.6 malloc
280 280
281 Add a uprobe to a target process running in a 281 Add a uprobe to a target process running in a different mount namespace
282 282
283 ./perf probe --target-ns <target pid> -x /lib 283 ./perf probe --target-ns <target pid> -x /lib64/libc.so.6 malloc
284 284
285 Add a USDT probe to a target process running i 285 Add a USDT probe to a target process running in a different mount namespace
286 286
287 ./perf probe --target-ns <target pid> -x /usr 287 ./perf probe --target-ns <target pid> -x /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/lib/amd64/server/libjvm.so %sdt_hotspot:thread__sleep__end
288 288
289 Add a probe on specific versioned symbol by ba 289 Add a probe on specific versioned symbol by backslash escape
290 290
291 ./perf probe -x /lib64/libc-2.25.so 'malloc_g 291 ./perf probe -x /lib64/libc-2.25.so 'malloc_get_state\@GLIBC_2.2.5'
292 292
293 Add a probe in a source file using special cha 293 Add a probe in a source file using special characters by backslash escape
294 294
295 ./perf probe -x /opt/test/a.out 'foo\+bar.c:4 295 ./perf probe -x /opt/test/a.out 'foo\+bar.c:4'
296 296
297 297
298 PERMISSIONS AND SYSCTL 298 PERMISSIONS AND SYSCTL
299 ---------------------- 299 ----------------------
300 Since perf probe depends on ftrace (tracefs) a 300 Since perf probe depends on ftrace (tracefs) and kallsyms (/proc/kallsyms), you have to care about the permission and some sysctl knobs.
301 301
302 - Since tracefs and kallsyms requires root or 302 - Since tracefs and kallsyms requires root or privileged user to access it, the following perf probe commands also require it; --add, --del, --list (except for --cache option)
303 303
304 - The system admin can remount the tracefs wi 304 - The system admin can remount the tracefs with 755 (`sudo mount -o remount,mode=755 /sys/kernel/tracing/`) to allow unprivileged user to run the perf probe --list command.
305 305
306 - /proc/sys/kernel/kptr_restrict = 2 (restric 306 - /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also prevents perf probe to retrieve the important information from kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users) for the above commands. Since the user-space probe doesn't need to access kallsyms, this is only for probing the kernel function (kprobes).
307 307
308 - Since the perf probe commands read the vmli 308 - Since the perf probe commands read the vmlinux (for kernel) and/or the debuginfo file (including user-space application), you need to ensure that you can read those files.
309 309
310 310
311 SEE ALSO 311 SEE ALSO
312 -------- 312 --------
313 linkperf:perf-trace[1], linkperf:perf-record[1 313 linkperf:perf-trace[1], linkperf:perf-record[1], linkperf:perf-buildid-cache[1]
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.