1 ========================================= 1 =========================================
2 Uprobe-tracer: Uprobe-based Event Tracing 2 Uprobe-tracer: Uprobe-based Event Tracing
3 ========================================= 3 =========================================
4 4
5 :Author: Srikar Dronamraju 5 :Author: Srikar Dronamraju
6 6
7 7
8 Overview 8 Overview
9 -------- 9 --------
10 Uprobe based trace events are similar to kprob 10 Uprobe based trace events are similar to kprobe based trace events.
11 To enable this feature, build your kernel with 11 To enable this feature, build your kernel with CONFIG_UPROBE_EVENTS=y.
12 12
13 Similar to the kprobe-event tracer, this doesn 13 Similar to the kprobe-event tracer, this doesn't need to be activated via
14 current_tracer. Instead of that, add probe poi 14 current_tracer. Instead of that, add probe points via
15 /sys/kernel/tracing/uprobe_events, and enable !! 15 /sys/kernel/debug/tracing/uprobe_events, and enable it via
16 /sys/kernel/tracing/events/uprobes/<EVENT>/ena !! 16 /sys/kernel/debug/tracing/events/uprobes/<EVENT>/enable.
17 17
18 However unlike kprobe-event tracer, the uprobe 18 However unlike kprobe-event tracer, the uprobe event interface expects the
19 user to calculate the offset of the probepoint 19 user to calculate the offset of the probepoint in the object.
20 20
21 You can also use /sys/kernel/tracing/dynamic_e !! 21 You can also use /sys/kernel/debug/tracing/dynamic_events instead of
22 uprobe_events. That interface will provide uni 22 uprobe_events. That interface will provide unified access to other
23 dynamic events too. 23 dynamic events too.
24 24
25 Synopsis of uprobe_tracer 25 Synopsis of uprobe_tracer
26 ------------------------- 26 -------------------------
27 :: 27 ::
28 28
29 p[:[GRP/][EVENT]] PATH:OFFSET [FETCHARGS] : !! 29 p[:[GRP/]EVENT] PATH:OFFSET [FETCHARGS] : Set a uprobe
30 r[:[GRP/][EVENT]] PATH:OFFSET [FETCHARGS] : !! 30 r[:[GRP/]EVENT] PATH:OFFSET [FETCHARGS] : Set a return uprobe (uretprobe)
31 p[:[GRP/][EVENT]] PATH:OFFSET%return [FETCHA !! 31 -:[GRP/]EVENT : Clear uprobe or uretprobe event
32 -:[GRP/][EVENT] : <<
33 32
34 GRP : Group name. If omitted, "upr 33 GRP : Group name. If omitted, "uprobes" is the default value.
35 EVENT : Event name. If omitted, the 34 EVENT : Event name. If omitted, the event name is generated based
36 on PATH+OFFSET. 35 on PATH+OFFSET.
37 PATH : Path to an executable or a l 36 PATH : Path to an executable or a library.
38 OFFSET : Offset where the probe is in 37 OFFSET : Offset where the probe is inserted.
39 OFFSET%return : Offset where the return prob <<
40 38
41 FETCHARGS : Arguments. Each probe can ha 39 FETCHARGS : Arguments. Each probe can have up to 128 args.
42 %REG : Fetch register REG 40 %REG : Fetch register REG
43 @ADDR : Fetch memory at ADDR (ADDR s 41 @ADDR : Fetch memory at ADDR (ADDR should be in userspace)
44 @+OFFSET : Fetch memory at OFFSET (OFFS 42 @+OFFSET : Fetch memory at OFFSET (OFFSET from same file as PATH)
45 $stackN : Fetch Nth entry of stack (N 43 $stackN : Fetch Nth entry of stack (N >= 0)
46 $stack : Fetch stack address. 44 $stack : Fetch stack address.
47 $retval : Fetch return value.(\*1) !! 45 $retval : Fetch return value.(*)
48 $comm : Fetch current task comm. 46 $comm : Fetch current task comm.
49 +|-[u]OFFS(FETCHARG) : Fetch memory at FETC !! 47 +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(**)
50 \IMM : Store an immediate value to <<
51 NAME=FETCHARG : Set NAME as the argumen 48 NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
52 FETCHARG:TYPE : Set TYPE as the type of 49 FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
53 (u8/u16/u32/u64/s8/s16/ 50 (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
54 (x8/x16/x32/x64), "stri 51 (x8/x16/x32/x64), "string" and bitfield are supported.
55 52
56 (\*1) only for return probe. !! 53 (*) only for return probe.
57 (\*2) this is useful for fetching a field of !! 54 (**) this is useful for fetching a field of data structures.
58 (\*3) Unlike kprobe event, "u" prefix will j <<
59 events can access only user-space memo <<
60 55
61 Types 56 Types
62 ----- 57 -----
63 Several types are supported for fetch-args. Up 58 Several types are supported for fetch-args. Uprobe tracer will access memory
64 by given type. Prefix 's' and 'u' means those 59 by given type. Prefix 's' and 'u' means those types are signed and unsigned
65 respectively. 'x' prefix implies it is unsigne 60 respectively. 'x' prefix implies it is unsigned. Traced arguments are shown
66 in decimal ('s' and 'u') or hexadecimal ('x'). 61 in decimal ('s' and 'u') or hexadecimal ('x'). Without type casting, 'x32'
67 or 'x64' is used depends on the architecture ( 62 or 'x64' is used depends on the architecture (e.g. x86-32 uses x32, and
68 x86-64 uses x64). 63 x86-64 uses x64).
69 String type is a special type, which fetches a 64 String type is a special type, which fetches a "null-terminated" string from
70 user space. 65 user space.
71 Bitfield is another special type, which takes 66 Bitfield is another special type, which takes 3 parameters, bit-width, bit-
72 offset, and container-size (usually 32). The s 67 offset, and container-size (usually 32). The syntax is::
73 68
74 b<bit-width>@<bit-offset>/<container-size> 69 b<bit-width>@<bit-offset>/<container-size>
75 70
76 For $comm, the default type is "string"; any o 71 For $comm, the default type is "string"; any other type is invalid.
77 72
78 73
79 Event Profiling 74 Event Profiling
80 --------------- 75 ---------------
81 You can check the total number of probe hits p 76 You can check the total number of probe hits per event via
82 /sys/kernel/tracing/uprobe_profile. The first !! 77 /sys/kernel/debug/tracing/uprobe_profile. The first column is the filename,
83 the second is the event name, the third is the 78 the second is the event name, the third is the number of probe hits.
84 79
85 Usage examples 80 Usage examples
86 -------------- 81 --------------
87 * Add a probe as a new uprobe event, write a 82 * Add a probe as a new uprobe event, write a new definition to uprobe_events
88 as below (sets a uprobe at an offset of 0x4 83 as below (sets a uprobe at an offset of 0x4245c0 in the executable /bin/bash)::
89 84
90 echo 'p /bin/bash:0x4245c0' > /sys/kernel/ !! 85 echo 'p /bin/bash:0x4245c0' > /sys/kernel/debug/tracing/uprobe_events
91 86
92 * Add a probe as a new uretprobe event:: 87 * Add a probe as a new uretprobe event::
93 88
94 echo 'r /bin/bash:0x4245c0' > /sys/kernel/ !! 89 echo 'r /bin/bash:0x4245c0' > /sys/kernel/debug/tracing/uprobe_events
95 90
96 * Unset registered event:: 91 * Unset registered event::
97 92
98 echo '-:p_bash_0x4245c0' >> /sys/kernel/tr !! 93 echo '-:p_bash_0x4245c0' >> /sys/kernel/debug/tracing/uprobe_events
99 94
100 * Print out the events that are registered:: 95 * Print out the events that are registered::
101 96
102 cat /sys/kernel/tracing/uprobe_events !! 97 cat /sys/kernel/debug/tracing/uprobe_events
103 98
104 * Clear all events:: 99 * Clear all events::
105 100
106 echo > /sys/kernel/tracing/uprobe_events !! 101 echo > /sys/kernel/debug/tracing/uprobe_events
107 102
108 Following example shows how to dump the instru 103 Following example shows how to dump the instruction pointer and %ax register
109 at the probed text address. Probe zfree functi 104 at the probed text address. Probe zfree function in /bin/zsh::
110 105
111 # cd /sys/kernel/tracing/ !! 106 # cd /sys/kernel/debug/tracing/
112 # cat /proc/`pgrep zsh`/maps | grep /bin/z 107 # cat /proc/`pgrep zsh`/maps | grep /bin/zsh | grep r-xp
113 00400000-0048a000 r-xp 00000000 08:03 1309 108 00400000-0048a000 r-xp 00000000 08:03 130904 /bin/zsh
114 # objdump -T /bin/zsh | grep -w zfree 109 # objdump -T /bin/zsh | grep -w zfree
115 0000000000446420 g DF .text 0000000000 110 0000000000446420 g DF .text 0000000000000012 Base zfree
116 111
117 0x46420 is the offset of zfree in object /bin/ 112 0x46420 is the offset of zfree in object /bin/zsh that is loaded at
118 0x00400000. Hence the command to uprobe would 113 0x00400000. Hence the command to uprobe would be::
119 114
120 # echo 'p:zfree_entry /bin/zsh:0x46420 %ip 115 # echo 'p:zfree_entry /bin/zsh:0x46420 %ip %ax' > uprobe_events
121 116
122 And the same for the uretprobe would be:: 117 And the same for the uretprobe would be::
123 118
124 # echo 'r:zfree_exit /bin/zsh:0x46420 %ip 119 # echo 'r:zfree_exit /bin/zsh:0x46420 %ip %ax' >> uprobe_events
125 120
126 .. note:: User has to explicitly calculate the 121 .. note:: User has to explicitly calculate the offset of the probe-point
127 in the object. 122 in the object.
128 123
129 We can see the events that are registered by l 124 We can see the events that are registered by looking at the uprobe_events file.
130 :: 125 ::
131 126
132 # cat uprobe_events 127 # cat uprobe_events
133 p:uprobes/zfree_entry /bin/zsh:0x00046420 128 p:uprobes/zfree_entry /bin/zsh:0x00046420 arg1=%ip arg2=%ax
134 r:uprobes/zfree_exit /bin/zsh:0x00046420 a 129 r:uprobes/zfree_exit /bin/zsh:0x00046420 arg1=%ip arg2=%ax
135 130
136 Format of events can be seen by viewing the fi 131 Format of events can be seen by viewing the file events/uprobes/zfree_entry/format.
137 :: 132 ::
138 133
139 # cat events/uprobes/zfree_entry/format 134 # cat events/uprobes/zfree_entry/format
140 name: zfree_entry 135 name: zfree_entry
141 ID: 922 136 ID: 922
142 format: 137 format:
143 field:unsigned short common_type; 138 field:unsigned short common_type; offset:0; size:2; signed:0;
144 field:unsigned char common_flags; 139 field:unsigned char common_flags; offset:2; size:1; signed:0;
145 field:unsigned char common_preempt_co 140 field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
146 field:int common_pid; 141 field:int common_pid; offset:4; size:4; signed:1;
147 field:int common_padding; 142 field:int common_padding; offset:8; size:4; signed:1;
148 143
149 field:unsigned long __probe_ip; 144 field:unsigned long __probe_ip; offset:12; size:4; signed:0;
150 field:u32 arg1; 145 field:u32 arg1; offset:16; size:4; signed:0;
151 field:u32 arg2; 146 field:u32 arg2; offset:20; size:4; signed:0;
152 147
153 print fmt: "(%lx) arg1=%lx arg2=%lx", REC- 148 print fmt: "(%lx) arg1=%lx arg2=%lx", REC->__probe_ip, REC->arg1, REC->arg2
154 149
155 Right after definition, each event is disabled 150 Right after definition, each event is disabled by default. For tracing these
156 events, you need to enable it by:: 151 events, you need to enable it by::
157 152
158 # echo 1 > events/uprobes/enable 153 # echo 1 > events/uprobes/enable
159 154
160 Lets start tracing, sleep for some time and st !! 155 Lets disable the event after sleeping for some time.
161 :: 156 ::
162 157
163 # echo 1 > tracing_on <<
164 # sleep 20 158 # sleep 20
165 # echo 0 > tracing_on <<
166 <<
167 Also, you can disable the event by:: <<
168 <<
169 # echo 0 > events/uprobes/enable 159 # echo 0 > events/uprobes/enable
170 160
171 And you can see the traced information via /sy !! 161 And you can see the traced information via /sys/kernel/debug/tracing/trace.
172 :: 162 ::
173 163
174 # cat trace 164 # cat trace
175 # tracer: nop 165 # tracer: nop
176 # 166 #
177 # TASK-PID CPU# TIMESTAMP 167 # TASK-PID CPU# TIMESTAMP FUNCTION
178 # | | | | 168 # | | | | |
179 zsh-24842 [006] 258544.995456 169 zsh-24842 [006] 258544.995456: zfree_entry: (0x446420) arg1=446420 arg2=79
180 zsh-24842 [007] 258545.000270 170 zsh-24842 [007] 258545.000270: zfree_exit: (0x446540 <- 0x446420) arg1=446540 arg2=0
181 zsh-24842 [002] 258545.043929 171 zsh-24842 [002] 258545.043929: zfree_entry: (0x446420) arg1=446420 arg2=79
182 zsh-24842 [004] 258547.046129 172 zsh-24842 [004] 258547.046129: zfree_exit: (0x446540 <- 0x446420) arg1=446540 arg2=0
183 173
184 Output shows us uprobe was triggered for a pid 174 Output shows us uprobe was triggered for a pid 24842 with ip being 0x446420
185 and contents of ax register being 79. And uret 175 and contents of ax register being 79. And uretprobe was triggered with ip at
186 0x446540 with counterpart function entry at 0x 176 0x446540 with counterpart function entry at 0x446420.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.