1 !! 1 p m - g r a p h
2 _ __ _ __ ___ __ _ _ __ __ _ _ __ <<
3 | '_ \| '_ ` _ \ _____ / _` | '__/ _` | '_ <<
4 | |_) | | | | | |_____| (_| | | | (_| | |_) <<
5 | .__/|_| |_| |_| \__, |_| \__,_| .__ <<
6 |_| |___/ |_| <<
7 2
8 pm-graph: suspend/resume/boot timing analys 3 pm-graph: suspend/resume/boot timing analysis tools
9 Version: 5.11 !! 4 Version: 5.4
10 Author: Todd Brandt <todd.e.brandt@intel.c 5 Author: Todd Brandt <todd.e.brandt@intel.com>
11 Home Page: https://www.intel.com/content/www !! 6 Home Page: https://01.org/pm-graph
12 7
13 Report bugs/issues at bugzilla.kernel.org Too 8 Report bugs/issues at bugzilla.kernel.org Tools/pm-graph
14 - https://bugzilla.kernel.org/buglist. 9 - https://bugzilla.kernel.org/buglist.cgi?component=pm-graph&product=Tools
15 10
16 Full documentation available online & in man 11 Full documentation available online & in man pages
17 - Getting Started: 12 - Getting Started:
18 https://www.intel.com/content/www/us !! 13 https://01.org/pm-graph/documentation/getting-started
19 14
20 - Feature Summary: !! 15 - Config File Format:
21 https://www.intel.com/content/www/us !! 16 https://01.org/pm-graph/documentation/3-config-file-format
22 17
23 - upstream version in git: 18 - upstream version in git:
24 git clone https://github.com/intel/p !! 19 https://github.com/intel/pm-graph/
25 20
26 Table of Contents 21 Table of Contents
27 - Overview 22 - Overview
28 - Setup 23 - Setup
29 - Usage 24 - Usage
30 - Basic Usage 25 - Basic Usage
31 - Dev Mode Usage 26 - Dev Mode Usage
32 - Proc Mode Usage 27 - Proc Mode Usage
33 - Endurance Testing <<
34 - Usage Examples <<
35 - Configuration Files 28 - Configuration Files
36 - Usage Examples 29 - Usage Examples
37 - Config File Options 30 - Config File Options
38 - Custom Timeline Entries 31 - Custom Timeline Entries
39 - Adding/Editing Timeline Func 32 - Adding/Editing Timeline Functions
40 - Adding/Editing Dev Timeline 33 - Adding/Editing Dev Timeline Source Functions
41 - Verifying your Custom Functi 34 - Verifying your Custom Functions
42 - Testing on consumer linux Operating 35 - Testing on consumer linux Operating Systems
43 - Android 36 - Android
44 37
45 ---------------------------------------------- 38 ------------------------------------------------------------------
46 | OVERVIEW 39 | OVERVIEW |
47 ---------------------------------------------- 40 ------------------------------------------------------------------
48 41
49 This tool suite is designed to assist kernel 42 This tool suite is designed to assist kernel and OS developers in optimizing
50 their linux stack's suspend/resume & boot tim 43 their linux stack's suspend/resume & boot time. Using a kernel image built
51 with a few extra options enabled, the tools w 44 with a few extra options enabled, the tools will execute a suspend or boot,
52 and will capture dmesg and ftrace data. This 45 and will capture dmesg and ftrace data. This data is transformed into a set of
53 timelines and a callgraph to give a quick and 46 timelines and a callgraph to give a quick and detailed view of which devices
54 and kernel processes are taking the most time 47 and kernel processes are taking the most time in suspend/resume & boot.
55 48
56 ---------------------------------------------- 49 ------------------------------------------------------------------
57 | SETUP 50 | SETUP |
58 ---------------------------------------------- 51 ------------------------------------------------------------------
59 52
60 Package Requirements !! 53 These packages are required to execute the scripts
61 - runs with python2 or python3, choice <<
62 - python 54 - python
63 - python-configparser (for python2 slee !! 55 - python-requests
64 - python-requests (for stresstester.py) <<
65 - linux-tools-common (for turbostat usa <<
66 56
67 Ubuntu: 57 Ubuntu:
68 sudo apt-get install python python-c !! 58 sudo apt-get install python python-requests
69 59
70 Fedora: 60 Fedora:
71 sudo dnf install python python-confi !! 61 sudo dnf install python python-requests
72 62
73 The tools can most easily be installed via 63 The tools can most easily be installed via git clone and make install
74 64
75 $> git clone http://github.com/intel/pm-gr 65 $> git clone http://github.com/intel/pm-graph.git
76 $> cd pm-graph 66 $> cd pm-graph
77 $> sudo make install 67 $> sudo make install
78 $> man sleepgraph ; man bootgraph 68 $> man sleepgraph ; man bootgraph
79 69
80 Setup involves some minor kernel configura 70 Setup involves some minor kernel configuration
81 71
82 The following kernel build options are req 72 The following kernel build options are required for all kernels:
83 CONFIG_DEVMEM=y 73 CONFIG_DEVMEM=y
84 CONFIG_PM_DEBUG=y 74 CONFIG_PM_DEBUG=y
85 CONFIG_PM_SLEEP_DEBUG=y 75 CONFIG_PM_SLEEP_DEBUG=y
86 CONFIG_FTRACE=y 76 CONFIG_FTRACE=y
87 CONFIG_FUNCTION_TRACER=y 77 CONFIG_FUNCTION_TRACER=y
88 CONFIG_FUNCTION_GRAPH_TRACER=y 78 CONFIG_FUNCTION_GRAPH_TRACER=y
89 CONFIG_KPROBES=y 79 CONFIG_KPROBES=y
90 CONFIG_KPROBES_ON_FTRACE=y 80 CONFIG_KPROBES_ON_FTRACE=y
91 81
92 In kernel 3.15.0, two patches were ups 82 In kernel 3.15.0, two patches were upstreamed which enable the
93 v3.0 behavior. These patches allow the 83 v3.0 behavior. These patches allow the tool to read all the
94 data from trace events instead of from 84 data from trace events instead of from dmesg. You can enable
95 this behavior on earlier kernels with 85 this behavior on earlier kernels with these patches:
96 86
97 (kernel/pre-3.15/enable_trace_events_s 87 (kernel/pre-3.15/enable_trace_events_suspend_resume.patch)
98 (kernel/pre-3.15/enable_trace_events_d 88 (kernel/pre-3.15/enable_trace_events_device_pm_callback.patch)
99 89
100 If you're using bootgraph, or sleepgra !! 90 If you're using a kernel older than 3.15.0, the following
101 the following additional kerne !! 91 additional kernel parameters are required:
102 (e.g. in file /etc/default/grub) 92 (e.g. in file /etc/default/grub)
103 GRUB_CMDLINE_LINUX_DEFAULT="... initca 93 GRUB_CMDLINE_LINUX_DEFAULT="... initcall_debug log_buf_len=32M ..."
104 94
105 If you're using a kernel older than 3. 95 If you're using a kernel older than 3.11-rc2, the following simple
106 patch must be applied to enabl 96 patch must be applied to enable ftrace data:
107 in file: kernel/power/suspend.c 97 in file: kernel/power/suspend.c
108 in function: int suspend_devices_and_e 98 in function: int suspend_devices_and_enter(suspend_state_t state)
109 remove call to "ftrace_stop();" 99 remove call to "ftrace_stop();"
110 remove call to "ftrace_start();" 100 remove call to "ftrace_start();"
111 101
112 There is a patch which does this for k 102 There is a patch which does this for kernel v3.8.0:
113 (kernel/pre-3.11-rc2/enable_ftrace_in_ 103 (kernel/pre-3.11-rc2/enable_ftrace_in_suspendresume.patch)
114 104
115 105
116 106
117 ---------------------------------------------- 107 ------------------------------------------------------------------
118 | USAGE 108 | USAGE |
119 ---------------------------------------------- 109 ------------------------------------------------------------------
120 110
121 Basic Usage 111 Basic Usage
122 ___________ 112 ___________
123 113
124 1) First configure a kernel using the instruc 114 1) First configure a kernel using the instructions from the previous sections.
125 Then build, install, and boot with it. 115 Then build, install, and boot with it.
126 2) Open up a terminal window and execute the 116 2) Open up a terminal window and execute the mode list command:
127 117
128 %> sudo ./sleepgraph.py -modes 118 %> sudo ./sleepgraph.py -modes
129 ['freeze', 'mem', 'disk'] 119 ['freeze', 'mem', 'disk']
130 120
131 Execute a test using one of the available pow 121 Execute a test using one of the available power modes, e.g. mem (S3):
132 122
133 %> sudo ./sleepgraph.py -m mem -rtcwak 123 %> sudo ./sleepgraph.py -m mem -rtcwake 15
134 124
135 or with a config file 125 or with a config file
136 126
137 %> sudo ./sleepgraph.py -config config 127 %> sudo ./sleepgraph.py -config config/suspend.cfg
138 128
139 When the system comes back you'll see the scr 129 When the system comes back you'll see the script finishing up and
140 creating the output files in the test subdir. 130 creating the output files in the test subdir. It generates output
141 files in subdirectory: suspend-mmddyy-HHMMSS. 131 files in subdirectory: suspend-mmddyy-HHMMSS. The ftrace file can
142 be used to regenerate the html timeline with 132 be used to regenerate the html timeline with different options
143 133
144 HTML output: <hostname 134 HTML output: <hostname>_<mode>.html
145 raw dmesg output: <hostname 135 raw dmesg output: <hostname>_<mode>_dmesg.txt
146 raw ftrace output: <hostname 136 raw ftrace output: <hostname>_<mode>_ftrace.txt
147 137
148 View the html in firefox or chrome. 138 View the html in firefox or chrome.
149 139
150 140
151 Dev Mode Usage 141 Dev Mode Usage
152 ______________ 142 ______________
153 143
154 Developer mode adds information on low level 144 Developer mode adds information on low level source calls to the timeline.
155 The tool sets kprobes on all delay and mutex 145 The tool sets kprobes on all delay and mutex calls to see which devices
156 are waiting for something and when. It also s 146 are waiting for something and when. It also sets a suite of kprobes on
157 subsystem dependent calls to better fill out 147 subsystem dependent calls to better fill out the timeline.
158 148
159 The tool will also expose kernel threads that 149 The tool will also expose kernel threads that don't normally show up in the
160 timeline. This is useful in discovering depen 150 timeline. This is useful in discovering dependent threads to get a better
161 idea of what each device is waiting for. For 151 idea of what each device is waiting for. For instance, the scsi_eh thread,
162 a.k.a. scsi resume error handler, is what eac 152 a.k.a. scsi resume error handler, is what each SATA disk device waits for
163 before it can continue resume. 153 before it can continue resume.
164 154
165 The timeline will be much larger if run with 155 The timeline will be much larger if run with dev mode, so it can be useful
166 to set the -mindev option to clip out any dev 156 to set the -mindev option to clip out any device blocks that are too small
167 to see easily. The following command will giv 157 to see easily. The following command will give a nice dev mode run:
168 158
169 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -m 159 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -mindev 1 -dev
170 160
171 or with a config file 161 or with a config file
172 162
173 %> sudo ./sleepgraph.py -config config/suspen 163 %> sudo ./sleepgraph.py -config config/suspend-dev.cfg
174 164
175 165
176 Proc Mode Usage 166 Proc Mode Usage
177 _______________ 167 _______________
178 168
179 Proc mode adds user process info to the timel 169 Proc mode adds user process info to the timeline. This is done in a manner
180 similar to the bootchart utility, which graph 170 similar to the bootchart utility, which graphs init processes and their
181 execution as the system boots. This tool opti 171 execution as the system boots. This tool option does the same thing but for
182 the period before and after suspend/resume. 172 the period before and after suspend/resume.
183 173
184 In order to see any process info, there needs 174 In order to see any process info, there needs to be some delay before or
185 after resume since processes are frozen in su 175 after resume since processes are frozen in suspend_prepare and thawed in
186 resume_complete. The predelay and postdelay a 176 resume_complete. The predelay and postdelay args allow you to do this. It
187 can also be useful to run in x2 mode with an 177 can also be useful to run in x2 mode with an x2 delay, this way you can
188 see process activity before and after resume, 178 see process activity before and after resume, and in between two
189 successive suspend/resumes. 179 successive suspend/resumes.
190 180
191 The command can be run like this: 181 The command can be run like this:
192 182
193 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -x 183 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -x2 -x2delay 1000 -predelay 1000 -postdelay 1000 -proc
194 184
195 or with a config file 185 or with a config file
196 186
197 %> sudo ./sleepgraph.py -config config/suspen 187 %> sudo ./sleepgraph.py -config config/suspend-proc.cfg
198 188
199 ---------------------------------------------- <<
200 | ENDURANCE TESTING <<
201 ---------------------------------------------- <<
202 <<
203 The best way to gauge the health of a system <<
204 suspend/resumes over an extended period and a <<
205 accomplished with sleepgraph's -multi argumen <<
206 number of tests to run OR the duration in day <<
207 delay in seconds between them. For instance, <<
208 a 5 second delay between each, or -multi 24h <<
209 period with no delay between tests. You can i <<
210 to generate the data you want. It's most usef <<
211 as the kprobes don't alter the performance mu <<
212 <<
213 On completion, the output folder contains a s <<
214 individual test data and a set of summary pag <<
215 file is a tabular list of the tests with rele <<
216 summary-issue.html and summary-devices.html f <<
217 all tests on kernel issues and device perform <<
218 <<
219 suspend-xN-{date}-{time}: <<
220 summary.html <<
221 summary-issues.html <<
222 summary-devices.html <<
223 suspend-{date}-{time} (1) <<
224 suspend-{date}-{time} (2) <<
225 ... <<
226 <<
227 These are the relevant arguments to use for t <<
228 <<
229 -m mode <<
230 Mode to initiate for suspend e.g. mem, <<
231 <<
232 -rtcwake t <<
233 Use rtcwake to autoresume after t seco <<
234 <<
235 -gzip (optional) <<
236 Gzip the trace and dmesg logs to save <<
237 gzipped logs for processing. This redu <<
238 <<
239 -dev (optional) <<
240 Add kernel source calls and threads to <<
241 <<
242 -multi n d <<
243 Execute n consecutive tests at d secon <<
244 created in a new subdirectory: suspend <<
245 run is done, the -summary command is c <<
246 html files for all the data (unless yo <<
247 speed up the testing by not creating t <<
248 can then run the tool again at a later <<
249 create the timelines. <<
250 <<
251 -skiphtml (optional) <<
252 Run the test and capture the trace log <<
253 html generation. This can greatly spee <<
254 copy the data to a faster host machine <<
255 generate the timelines and summary. <<
256 <<
257 These are the relevant commands to use after <<
258 <<
259 -summary indir <<
260 Generate or regenerate the summary for <<
261 files: summary.html, summary-issues.ht <<
262 current folder. summary.html is a tabl <<
263 by kernel/host/mode, and links to the <<
264 is a list of kernel issues found in dm <<
265 summary-devices.html is a list of devi <<
266 <<
267 -genhtml <<
268 Used with -summary to regenerate any <<
269 dmesg and ftrace logs. This will requi <<
270 there are thousands of tests. <<
271 <<
272 Usage Examples <<
273 _______________ <<
274 <<
275 A multitest is initiated like this: <<
276 <<
277 %> sudo ./sleepgraph.py -m mem -rtcwake 10 - <<
278 <<
279 or you can skip timeline generation in <<
280 <<
281 %> sudo ./sleepgraph.py -m mem -rtcwake 10 - <<
282 <<
283 The tool will produce an output folder with a <<
284 Each test subfolder contains the dmesg/ftrace <<
285 depending on whether you used the -skiphtml o <<
286 the summary.html files. <<
287 <<
288 The summary for an existing multitest is gene <<
289 <<
290 %> cd suspend-x2000-{date}-{time} <<
291 %> sleepgraph.py -summary . <<
292 <<
293 or if you need to generate the html ti <<
294 <<
295 %> cd suspend-xN-{date}-{time} <<
296 %> sleepgraph.py -summary . -genhtml <<
297 189
298 ---------------------------------------------- 190 ------------------------------------------------------------------
299 | CONFIGURATION FILES 191 | CONFIGURATION FILES |
300 ---------------------------------------------- 192 ------------------------------------------------------------------
301 193
302 Since 4.0 we've moved to using config files i 194 Since 4.0 we've moved to using config files in lieu of command line options.
303 The config folder contains a collection of ty 195 The config folder contains a collection of typical use cases.
304 There are corresponding configs for other pow 196 There are corresponding configs for other power modes:
305 197
306 Simple suspend/resume with basic timel 198 Simple suspend/resume with basic timeline (mem/freeze/standby)
307 config/suspend.cfg 199 config/suspend.cfg
308 config/freeze.cfg 200 config/freeze.cfg
309 config/standby.cfg 201 config/standby.cfg
310 202
311 Dev mode suspend/resume with dev timel 203 Dev mode suspend/resume with dev timeline (mem/freeze/standby)
312 config/suspend-dev.cfg 204 config/suspend-dev.cfg
313 config/freeze-dev.cfg 205 config/freeze-dev.cfg
314 config/standby-dev.cfg 206 config/standby-dev.cfg
315 207
316 Simple suspend/resume with timeline an 208 Simple suspend/resume with timeline and callgraph (mem/freeze/standby)
317 config/suspend-callgraph.cfg 209 config/suspend-callgraph.cfg
318 config/freeze-callgraph.cfg 210 config/freeze-callgraph.cfg
319 config/standby-callgraph.cfg 211 config/standby-callgraph.cfg
320 212
321 Sample proc mode x2 run using mem susp 213 Sample proc mode x2 run using mem suspend
322 config/suspend-x2-proc.cfg 214 config/suspend-x2-proc.cfg
323 215
324 Sample for editing timeline funcs (mov 216 Sample for editing timeline funcs (moves internal functions into config)
325 config/custom-timeline-functio 217 config/custom-timeline-functions.cfg
326 218
327 Sample debug config for serio subsyste 219 Sample debug config for serio subsystem
328 config/debug-serio-suspend.cfg 220 config/debug-serio-suspend.cfg
329 221
330 222
331 Usage Examples 223 Usage Examples
332 ______________ 224 ______________
333 225
334 Run a simple mem suspend: 226 Run a simple mem suspend:
335 %> sudo ./sleepgraph.py -config config/suspen 227 %> sudo ./sleepgraph.py -config config/suspend.cfg
336 228
337 Run a mem suspend with callgraph data: 229 Run a mem suspend with callgraph data:
338 %> sudo ./sleepgraph.py -config config/suspen 230 %> sudo ./sleepgraph.py -config config/suspend-callgraph.cfg
339 231
340 Run a mem suspend with dev mode detail: 232 Run a mem suspend with dev mode detail:
341 %> sudo ./sleepgraph.py -config config/suspen 233 %> sudo ./sleepgraph.py -config config/suspend-dev.cfg
342 234
343 235
344 Config File Options 236 Config File Options
345 ___________________ 237 ___________________
346 238
347 [Settings] 239 [Settings]
348 240
349 # Verbosity: print verbose messages (def: fal 241 # Verbosity: print verbose messages (def: false)
350 verbose: false 242 verbose: false
351 243
352 # Suspend Mode: e.g. standby, mem, freeze, di 244 # Suspend Mode: e.g. standby, mem, freeze, disk (def: mem)
353 mode: mem 245 mode: mem
354 246
355 # Output Directory Format: {hostname}, {date} 247 # Output Directory Format: {hostname}, {date}, {time} give current values
356 output-dir: suspend-{hostname}-{date}-{time} 248 output-dir: suspend-{hostname}-{date}-{time}
357 249
358 # Automatic Wakeup: use rtcwake to wakeup aft 250 # Automatic Wakeup: use rtcwake to wakeup after X seconds (def: infinity)
359 rtcwake: 15 251 rtcwake: 15
360 252
361 # Add Logs: add the dmesg and ftrace log to t 253 # Add Logs: add the dmesg and ftrace log to the html output (def: false)
362 addlogs: false 254 addlogs: false
363 255
364 # Sus/Res Gap: insert a gap between sus & res 256 # Sus/Res Gap: insert a gap between sus & res in the timeline (def: false)
365 srgap: false 257 srgap: false
366 258
367 # Custom Command: Command to execute in lieu 259 # Custom Command: Command to execute in lieu of suspend (def: "")
368 command: echo mem > /sys/power/state 260 command: echo mem > /sys/power/state
369 261
370 # Proc mode: graph user processes and cpu usa 262 # Proc mode: graph user processes and cpu usage in the timeline (def: false)
371 proc: false 263 proc: false
372 264
373 # Dev mode: graph source functions in the tim 265 # Dev mode: graph source functions in the timeline (def: false)
374 dev: false 266 dev: false
375 267
376 # Suspend/Resume x2: run 2 suspend/resumes ba 268 # Suspend/Resume x2: run 2 suspend/resumes back to back (def: false)
377 x2: false 269 x2: false
378 270
379 # x2 Suspend Delay: time delay between the tw 271 # x2 Suspend Delay: time delay between the two test runs in ms (def: 0 ms)
380 x2delay: 0 272 x2delay: 0
381 273
382 # Pre Suspend Delay: nclude an N ms delay bef 274 # Pre Suspend Delay: nclude an N ms delay before (1st) suspend (def: 0 ms)
383 predelay: 0 275 predelay: 0
384 276
385 # Post Resume Delay: include an N ms delay af 277 # Post Resume Delay: include an N ms delay after (last) resume (def: 0 ms)
386 postdelay: 0 278 postdelay: 0
387 279
388 # Min Device Length: graph only dev callbacks 280 # Min Device Length: graph only dev callbacks longer than min (def: 0.001 ms)
389 mindev: 0.001 281 mindev: 0.001
390 282
391 # Callgraph: gather ftrace callgraph data on 283 # Callgraph: gather ftrace callgraph data on all timeline events (def: false)
392 callgraph: false 284 callgraph: false
393 285
394 # Expand Callgraph: pre-expand the callgraph 286 # Expand Callgraph: pre-expand the callgraph treeviews in html (def: false)
395 expandcg: false 287 expandcg: false
396 288
397 # Min Callgraph Length: show callgraphs only 289 # Min Callgraph Length: show callgraphs only if longer than min (def: 1 ms)
398 mincg: 1 290 mincg: 1
399 291
400 # Timestamp Precision: number of sig digits i 292 # Timestamp Precision: number of sig digits in timestamps (0:S, [3:ms], 6:us)
401 timeprec: 3 293 timeprec: 3
402 294
403 # Device Filter: show only devs whose name/dr 295 # Device Filter: show only devs whose name/driver includes one of these strings
404 devicefilter: _cpu_up,_cpu_down,i915,usb 296 devicefilter: _cpu_up,_cpu_down,i915,usb
405 297
406 # Override default timeline entries: 298 # Override default timeline entries:
407 # Do not use the internal default functions f 299 # Do not use the internal default functions for timeline entries (def: false)
408 # Set this to true if you intend to only use 300 # Set this to true if you intend to only use the ones defined in the config
409 override-timeline-functions: true 301 override-timeline-functions: true
410 302
411 # Override default dev timeline entries: 303 # Override default dev timeline entries:
412 # Do not use the internal default functions f 304 # Do not use the internal default functions for dev timeline entries (def: false)
413 # Set this to true if you intend to only use 305 # Set this to true if you intend to only use the ones defined in the config
414 override-dev-timeline-functions: true 306 override-dev-timeline-functions: true
415 307
416 # Call Loop Max Gap (dev mode only) 308 # Call Loop Max Gap (dev mode only)
417 # merge loops of the same call if each is les 309 # merge loops of the same call if each is less than maxgap apart (def: 100us)
418 callloop-maxgap: 0.0001 310 callloop-maxgap: 0.0001
419 311
420 # Call Loop Max Length (dev mode only) 312 # Call Loop Max Length (dev mode only)
421 # merge loops of the same call if each is les 313 # merge loops of the same call if each is less than maxlen in length (def: 5ms)
422 callloop-maxlen: 0.005 314 callloop-maxlen: 0.005
423 315
424 ---------------------------------------------- 316 ------------------------------------------------------------------
425 | CUSTOM TIMELINE ENTRIES 317 | CUSTOM TIMELINE ENTRIES |
426 ---------------------------------------------- 318 ------------------------------------------------------------------
427 319
428 Adding or Editing Timeline Functions 320 Adding or Editing Timeline Functions
429 ____________________________________ 321 ____________________________________
430 322
431 The tool uses an array of function names to f 323 The tool uses an array of function names to fill out empty spaces in the
432 timeline where device callbacks don't appear. 324 timeline where device callbacks don't appear. For instance, in suspend_prepare
433 the tool adds the sys_sync and freeze_process 325 the tool adds the sys_sync and freeze_processes calls as virtual device blocks
434 in the timeline to show you where the time is 326 in the timeline to show you where the time is going. These calls should fill
435 the timeline with contiguous data so that mos 327 the timeline with contiguous data so that most kernel execution is covered.
436 328
437 It is possible to add new function calls to t 329 It is possible to add new function calls to the timeline by adding them to
438 the config. It's also possible to copy the in 330 the config. It's also possible to copy the internal timeline functions into
439 the config so that you can override and edit 331 the config so that you can override and edit them. Place them in the
440 timeline_functions_ARCH section with the name 332 timeline_functions_ARCH section with the name of your architecture appended.
441 i.e. for x86_64: [timeline_functions_x86_64] 333 i.e. for x86_64: [timeline_functions_x86_64]
442 334
443 Use the override-timeline-functions option if 335 Use the override-timeline-functions option if you only want to use your
444 custom calls, or leave it false to append the 336 custom calls, or leave it false to append them to the internal ones.
445 337
446 This section includes a list of functions (se 338 This section includes a list of functions (set using kprobes) which use both
447 symbol data and function arg data. The args a 339 symbol data and function arg data. The args are pulled directly from the
448 stack using this architecture's registers and 340 stack using this architecture's registers and stack formatting. Each entry
449 can include up to four pieces of info: The fu 341 can include up to four pieces of info: The function name, a format string,
450 an argument list, and a color. But only a fun 342 an argument list, and a color. But only a function name is required.
451 343
452 For a full example config, see config/custom- 344 For a full example config, see config/custom-timeline-functions.cfg. It pulls
453 all the internal timeline functions into the 345 all the internal timeline functions into the config and allows you to edit
454 them. 346 them.
455 347
456 Entry format: 348 Entry format:
457 349
458 function: format{fn_arg1}_{fn_arg2} fn_arg 350 function: format{fn_arg1}_{fn_arg2} fn_arg1 fn_arg2 ... [color=purple]
459 351
460 Required Arguments: 352 Required Arguments:
461 353
462 function: The symbol name for the function 354 function: The symbol name for the function you want probed, this is the
463 minimum required for an entry, i 355 minimum required for an entry, it will show up as the function
464 name with no arguments. 356 name with no arguments.
465 357
466 example: _cpu_up: 358 example: _cpu_up:
467 359
468 Optional Arguments: 360 Optional Arguments:
469 361
470 format: The format to display the data on 362 format: The format to display the data on the timeline in. Use braces to
471 enclose the arg names. 363 enclose the arg names.
472 364
473 example: CPU_ON[{cpu}] 365 example: CPU_ON[{cpu}]
474 366
475 color: The color of the entry block in the 367 color: The color of the entry block in the timeline. The default color is
476 transparent, so the entry shares th 368 transparent, so the entry shares the phase color. The color is an
477 html color string, either a word, o 369 html color string, either a word, or an RGB.
478 370
479 example: [color=#CC00CC] 371 example: [color=#CC00CC]
480 372
481 arglist: A list of arguments from register 373 arglist: A list of arguments from registers/stack addresses. See URL:
482 https://www.kernel.org/doc/Docume 374 https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt
483 375
484 example: cpu=%di:s32 376 example: cpu=%di:s32
485 377
486 Here is a full example entry. It displays cpu 378 Here is a full example entry. It displays cpu resume calls in the timeline
487 in orange. They will appear as CPU_ON[0], CPU 379 in orange. They will appear as CPU_ON[0], CPU_ON[1], etc.
488 380
489 [timeline_functions_x86_64] 381 [timeline_functions_x86_64]
490 _cpu_up: CPU_ON[{cpu}] cpu=%di:s32 [color=or 382 _cpu_up: CPU_ON[{cpu}] cpu=%di:s32 [color=orange]
491 383
492 384
493 Adding or Editing Dev Mode Timeline Source Fun 385 Adding or Editing Dev Mode Timeline Source Functions
494 ______________________________________________ 386 ____________________________________________________
495 387
496 In dev mode, the tool uses an array of functi 388 In dev mode, the tool uses an array of function names to monitor source
497 execution within the timeline entries. 389 execution within the timeline entries.
498 390
499 The function calls are displayed inside the m 391 The function calls are displayed inside the main device/call blocks in the
500 timeline. However, if a function call is not 392 timeline. However, if a function call is not within a main timeline event,
501 it will spawn an entirely new event named aft 393 it will spawn an entirely new event named after the caller's kernel thread.
502 These asynchronous kernel threads will popula 394 These asynchronous kernel threads will populate in a separate section
503 beneath the main device/call section. 395 beneath the main device/call section.
504 396
505 The tool has a set of hard coded calls which 397 The tool has a set of hard coded calls which focus on the most common use
506 cases: msleep, udelay, schedule_timeout, mute 398 cases: msleep, udelay, schedule_timeout, mutex_lock_slowpath, etc. These are
507 the functions that add a hardcoded time delay 399 the functions that add a hardcoded time delay to the suspend/resume path.
508 The tool also includes some common functions 400 The tool also includes some common functions native to important
509 subsystems: ata, i915, and ACPI, etc. 401 subsystems: ata, i915, and ACPI, etc.
510 402
511 It is possible to add new function calls to t 403 It is possible to add new function calls to the dev timeline by adding them
512 to the config. It's also possible to copy the 404 to the config. It's also possible to copy the internal dev timeline
513 functions into the config so that you can ove 405 functions into the config so that you can override and edit them. Place them
514 in the dev_timeline_functions_ARCH section wi 406 in the dev_timeline_functions_ARCH section with the name of your architecture
515 appended. i.e. for x86_64: [dev_timeline_func 407 appended. i.e. for x86_64: [dev_timeline_functions_x86_64]
516 408
517 Use the override-dev-timeline-functions optio 409 Use the override-dev-timeline-functions option if you only want to use your
518 custom calls, or leave it false to append the 410 custom calls, or leave it false to append them to the internal ones.
519 411
520 The format is the same as the timeline_functi 412 The format is the same as the timeline_functions_x86_64 section. It's a
521 list of functions (set using kprobes) which u 413 list of functions (set using kprobes) which use both symbol data and function
522 arg data. The args are pulled directly from t 414 arg data. The args are pulled directly from the stack using this
523 architecture's registers and stack formatting 415 architecture's registers and stack formatting. Each entry can include up
524 to four pieces of info: The function name, a 416 to four pieces of info: The function name, a format string, an argument list,
525 and a color. But only the function name is re 417 and a color. But only the function name is required.
526 418
527 For a full example config, see config/custom- 419 For a full example config, see config/custom-timeline-functions.cfg. It pulls
528 all the internal dev timeline functions into 420 all the internal dev timeline functions into the config and allows you to edit
529 them. 421 them.
530 422
531 Here is a full example entry. It displays the 423 Here is a full example entry. It displays the ATA port reset calls as
532 ataN_port_reset in the timeline. This is wher 424 ataN_port_reset in the timeline. This is where most of the SATA disk resume
533 time goes, so it can be helpful to see the lo 425 time goes, so it can be helpful to see the low level call.
534 426
535 [dev_timeline_functions_x86_64] 427 [dev_timeline_functions_x86_64]
536 ata_eh_recover: ata{port}_port_reset port=+3 428 ata_eh_recover: ata{port}_port_reset port=+36(%di):s32 [color=#CC00CC]
537 429
538 430
539 Verifying your custom functions 431 Verifying your custom functions
540 _______________________________ 432 _______________________________
541 433
542 Once you have a set of functions (kprobes) de 434 Once you have a set of functions (kprobes) defined, it can be useful to
543 perform a quick check to see if you formatted 435 perform a quick check to see if you formatted them correctly and if the system
544 actually supports them. To do this, run the t 436 actually supports them. To do this, run the tool with your config file
545 and the -status option. The tool will go thro 437 and the -status option. The tool will go through all the kprobes (both
546 custom and internal if you haven't overridden 438 custom and internal if you haven't overridden them) and actually attempts
547 to set them in ftrace. It will then print out 439 to set them in ftrace. It will then print out success or fail for you.
548 440
549 Note that kprobes which don't actually exist 441 Note that kprobes which don't actually exist in the kernel won't stop the
550 tool, they just wont show up. 442 tool, they just wont show up.
551 443
552 For example: 444 For example:
553 445
554 sudo ./sleepgraph.py -config config/custom-ti 446 sudo ./sleepgraph.py -config config/custom-timeline-functions.cfg -status
555 Checking this system (myhostname)... 447 Checking this system (myhostname)...
556 have root access: YES 448 have root access: YES
557 is sysfs mounted: YES 449 is sysfs mounted: YES
558 is "mem" a valid power mode: YES 450 is "mem" a valid power mode: YES
559 is ftrace supported: YES 451 is ftrace supported: YES
560 are kprobes supported: YES 452 are kprobes supported: YES
561 timeline data source: FTRACE (all trace ev 453 timeline data source: FTRACE (all trace events found)
562 is rtcwake supported: YES 454 is rtcwake supported: YES
563 verifying timeline kprobes work: 455 verifying timeline kprobes work:
564 _cpu_down: YES 456 _cpu_down: YES
565 _cpu_up: YES 457 _cpu_up: YES
566 acpi_pm_finish: YES 458 acpi_pm_finish: YES
567 acpi_pm_prepare: YES 459 acpi_pm_prepare: YES
568 freeze_kernel_threads: YES 460 freeze_kernel_threads: YES
569 freeze_processes: YES 461 freeze_processes: YES
570 sys_sync: YES 462 sys_sync: YES
571 thaw_processes: YES 463 thaw_processes: YES
572 verifying dev kprobes work: 464 verifying dev kprobes work:
573 __const_udelay: YES 465 __const_udelay: YES
574 __mutex_lock_slowpath: YES 466 __mutex_lock_slowpath: YES
575 acpi_os_stall: YES 467 acpi_os_stall: YES
576 acpi_ps_parse_aml: YES 468 acpi_ps_parse_aml: YES
577 intel_opregion_init: NO 469 intel_opregion_init: NO
578 intel_opregion_register: NO 470 intel_opregion_register: NO
579 intel_opregion_setup: NO 471 intel_opregion_setup: NO
580 msleep: YES 472 msleep: YES
581 schedule_timeout: YES 473 schedule_timeout: YES
582 schedule_timeout_uninterruptible: YES 474 schedule_timeout_uninterruptible: YES
583 usleep_range: YES 475 usleep_range: YES
584 476
585 477
586 ---------------------------------------------- 478 ------------------------------------------------------------------
587 | TESTING ON CONSUMER LINUX OPERATIN 479 | TESTING ON CONSUMER LINUX OPERATING SYSTEMS |
588 ---------------------------------------------- 480 ------------------------------------------------------------------
589 481
590 Android 482 Android
591 _______ 483 _______
592 484
593 The easiest way to execute on an android devi 485 The easiest way to execute on an android device is to run the android.sh
594 script on the device, then pull the ftrace lo 486 script on the device, then pull the ftrace log back to the host and run
595 sleepgraph.py on it. 487 sleepgraph.py on it.
596 488
597 Here are the steps: 489 Here are the steps:
598 490
599 [download and install the tool on the device] 491 [download and install the tool on the device]
600 492
601 host%> wget https://raw.githubusercont 493 host%> wget https://raw.githubusercontent.com/intel/pm-graph/master/tools/android.sh
602 host%> adb connect 192.168.1.6 494 host%> adb connect 192.168.1.6
603 host%> adb root 495 host%> adb root
604 # push the script to a writeable locat 496 # push the script to a writeable location
605 host%> adb push android.sh /sdcard/ 497 host%> adb push android.sh /sdcard/
606 498
607 [check whether the tool will run on your devi 499 [check whether the tool will run on your device]
608 500
609 host%> adb shell 501 host%> adb shell
610 dev%> cd /sdcard 502 dev%> cd /sdcard
611 dev%> sh android.sh status 503 dev%> sh android.sh status
612 host : asus_t100 504 host : asus_t100
613 kernel : 3.14.0-i386-dirty 505 kernel : 3.14.0-i386-dirty
614 modes : freeze mem 506 modes : freeze mem
615 rtcwake : supported 507 rtcwake : supported
616 ftrace : supported 508 ftrace : supported
617 trace events { 509 trace events {
618 suspend_resume: found 510 suspend_resume: found
619 device_pm_callback_end: fo 511 device_pm_callback_end: found
620 device_pm_callback_start: 512 device_pm_callback_start: found
621 } 513 }
622 # the above is what you see on a syste 514 # the above is what you see on a system that's properly patched
623 515
624 [execute the suspend] 516 [execute the suspend]
625 517
626 # NOTE: The suspend will only work if 518 # NOTE: The suspend will only work if the screen isn't timed out,
627 # so you have to press some keys first 519 # so you have to press some keys first to wake it up b4 suspend)
628 dev%> sh android.sh suspend mem 520 dev%> sh android.sh suspend mem
629 ------------------------------------ 521 ------------------------------------
630 Suspend/Resume timing test initiated 522 Suspend/Resume timing test initiated
631 ------------------------------------ 523 ------------------------------------
632 hostname : asus_t100 524 hostname : asus_t100
633 kernel : 3.14.0-i386-dirty 525 kernel : 3.14.0-i386-dirty
634 mode : mem 526 mode : mem
635 ftrace out : /mnt/shell/emulated/0/ftr 527 ftrace out : /mnt/shell/emulated/0/ftrace.txt
636 dmesg out : /mnt/shell/emulated/0/dme 528 dmesg out : /mnt/shell/emulated/0/dmesg.txt
637 log file : /mnt/shell/emulated/0/log 529 log file : /mnt/shell/emulated/0/log.txt
638 ------------------------------------ 530 ------------------------------------
639 INITIALIZING FTRACE........DONE 531 INITIALIZING FTRACE........DONE
640 STARTING FTRACE 532 STARTING FTRACE
641 SUSPEND START @ 21:24:02 (rtcwake in 1 533 SUSPEND START @ 21:24:02 (rtcwake in 10 seconds)
642 <adb connection will now terminate> 534 <adb connection will now terminate>
643 535
644 [retrieve the data from the device] 536 [retrieve the data from the device]
645 537
646 # I find that you have to actually kil 538 # I find that you have to actually kill the adb process and
647 # reconnect sometimes in order for the 539 # reconnect sometimes in order for the connection to work post-suspend
648 host%> adb connect 192.168.1.6 540 host%> adb connect 192.168.1.6
649 # (required) get the ftrace data, this 541 # (required) get the ftrace data, this is the most important piece
650 host%> adb pull /sdcard/ftrace.txt 542 host%> adb pull /sdcard/ftrace.txt
651 # (optional) get the dmesg data, this 543 # (optional) get the dmesg data, this is for debugging
652 host%> adb pull /sdcard/dmesg.txt 544 host%> adb pull /sdcard/dmesg.txt
653 # (optional) get the log, which just l 545 # (optional) get the log, which just lists some test times for comparison
654 host%> adb pull /sdcard/log.txt 546 host%> adb pull /sdcard/log.txt
655 547
656 [create an output html file using sleepgraph. 548 [create an output html file using sleepgraph.py]
657 549
658 host%> sleepgraph.py -ftrace ftrace.tx 550 host%> sleepgraph.py -ftrace ftrace.txt
659 551
660 You should now have an output.html with the a 552 You should now have an output.html with the android data, enjoy!
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.