1 1 _
2 _ __ _ __ ___ __ _ _ __ __ _ _ __ 2 _ __ _ __ ___ __ _ _ __ __ _ _ __ | |__
3 | '_ \| '_ ` _ \ _____ / _` | '__/ _` | '_ 3 | '_ \| '_ ` _ \ _____ / _` | '__/ _` | '_ \| '_ \
4 | |_) | | | | | |_____| (_| | | | (_| | |_) 4 | |_) | | | | | |_____| (_| | | | (_| | |_) | | | |
5 | .__/|_| |_| |_| \__, |_| \__,_| .__ 5 | .__/|_| |_| |_| \__, |_| \__,_| .__/|_| |_|
6 |_| |___/ |_| 6 |_| |___/ |_|
7 7
8 pm-graph: suspend/resume/boot timing analys 8 pm-graph: suspend/resume/boot timing analysis tools
9 Version: 5.11 !! 9 Version: 5.6
10 Author: Todd Brandt <todd.e.brandt@intel.c 10 Author: Todd Brandt <todd.e.brandt@intel.com>
11 Home Page: https://www.intel.com/content/www !! 11 Home Page: https://01.org/pm-graph
12 12
13 Report bugs/issues at bugzilla.kernel.org Too 13 Report bugs/issues at bugzilla.kernel.org Tools/pm-graph
14 - https://bugzilla.kernel.org/buglist. 14 - https://bugzilla.kernel.org/buglist.cgi?component=pm-graph&product=Tools
15 15
16 Full documentation available online & in man 16 Full documentation available online & in man pages
17 - Getting Started: 17 - Getting Started:
18 https://www.intel.com/content/www/us !! 18 https://01.org/pm-graph/documentation/getting-started
19 19
20 - Feature Summary: !! 20 - Config File Format:
21 https://www.intel.com/content/www/us !! 21 https://01.org/pm-graph/documentation/3-config-file-format
22 22
23 - upstream version in git: 23 - upstream version in git:
24 git clone https://github.com/intel/p !! 24 https://github.com/intel/pm-graph/
25 25
26 Table of Contents 26 Table of Contents
27 - Overview 27 - Overview
28 - Setup 28 - Setup
29 - Usage 29 - Usage
30 - Basic Usage 30 - Basic Usage
31 - Dev Mode Usage 31 - Dev Mode Usage
32 - Proc Mode Usage 32 - Proc Mode Usage
33 - Endurance Testing 33 - Endurance Testing
34 - Usage Examples 34 - Usage Examples
35 - Configuration Files 35 - Configuration Files
36 - Usage Examples 36 - Usage Examples
37 - Config File Options 37 - Config File Options
38 - Custom Timeline Entries 38 - Custom Timeline Entries
39 - Adding/Editing Timeline Func 39 - Adding/Editing Timeline Functions
40 - Adding/Editing Dev Timeline 40 - Adding/Editing Dev Timeline Source Functions
41 - Verifying your Custom Functi 41 - Verifying your Custom Functions
42 - Testing on consumer linux Operating 42 - Testing on consumer linux Operating Systems
43 - Android 43 - Android
44 44
45 ---------------------------------------------- 45 ------------------------------------------------------------------
46 | OVERVIEW 46 | OVERVIEW |
47 ---------------------------------------------- 47 ------------------------------------------------------------------
48 48
49 This tool suite is designed to assist kernel 49 This tool suite is designed to assist kernel and OS developers in optimizing
50 their linux stack's suspend/resume & boot tim 50 their linux stack's suspend/resume & boot time. Using a kernel image built
51 with a few extra options enabled, the tools w 51 with a few extra options enabled, the tools will execute a suspend or boot,
52 and will capture dmesg and ftrace data. This 52 and will capture dmesg and ftrace data. This data is transformed into a set of
53 timelines and a callgraph to give a quick and 53 timelines and a callgraph to give a quick and detailed view of which devices
54 and kernel processes are taking the most time 54 and kernel processes are taking the most time in suspend/resume & boot.
55 55
56 ---------------------------------------------- 56 ------------------------------------------------------------------
57 | SETUP 57 | SETUP |
58 ---------------------------------------------- 58 ------------------------------------------------------------------
59 59
60 Package Requirements 60 Package Requirements
61 - runs with python2 or python3, choice 61 - runs with python2 or python3, choice is made by /usr/bin/python link
62 - python 62 - python
63 - python-configparser (for python2 slee 63 - python-configparser (for python2 sleepgraph)
64 - python-requests (for stresstester.py) !! 64 - python-requests (for googlesheet.py)
65 - linux-tools-common (for turbostat usa 65 - linux-tools-common (for turbostat usage in sleepgraph)
66 66
67 Ubuntu: 67 Ubuntu:
68 sudo apt-get install python python-c 68 sudo apt-get install python python-configparser python-requests linux-tools-common
69 69
70 Fedora: 70 Fedora:
71 sudo dnf install python python-confi 71 sudo dnf install python python-configparser python-requests linux-tools-common
72 72
73 The tools can most easily be installed via 73 The tools can most easily be installed via git clone and make install
74 74
75 $> git clone http://github.com/intel/pm-gr 75 $> git clone http://github.com/intel/pm-graph.git
76 $> cd pm-graph 76 $> cd pm-graph
77 $> sudo make install 77 $> sudo make install
78 $> man sleepgraph ; man bootgraph 78 $> man sleepgraph ; man bootgraph
79 79
80 Setup involves some minor kernel configura 80 Setup involves some minor kernel configuration
81 81
82 The following kernel build options are req 82 The following kernel build options are required for all kernels:
83 CONFIG_DEVMEM=y 83 CONFIG_DEVMEM=y
84 CONFIG_PM_DEBUG=y 84 CONFIG_PM_DEBUG=y
85 CONFIG_PM_SLEEP_DEBUG=y 85 CONFIG_PM_SLEEP_DEBUG=y
86 CONFIG_FTRACE=y 86 CONFIG_FTRACE=y
87 CONFIG_FUNCTION_TRACER=y 87 CONFIG_FUNCTION_TRACER=y
88 CONFIG_FUNCTION_GRAPH_TRACER=y 88 CONFIG_FUNCTION_GRAPH_TRACER=y
89 CONFIG_KPROBES=y 89 CONFIG_KPROBES=y
90 CONFIG_KPROBES_ON_FTRACE=y 90 CONFIG_KPROBES_ON_FTRACE=y
91 91
92 In kernel 3.15.0, two patches were ups 92 In kernel 3.15.0, two patches were upstreamed which enable the
93 v3.0 behavior. These patches allow the 93 v3.0 behavior. These patches allow the tool to read all the
94 data from trace events instead of from 94 data from trace events instead of from dmesg. You can enable
95 this behavior on earlier kernels with 95 this behavior on earlier kernels with these patches:
96 96
97 (kernel/pre-3.15/enable_trace_events_s 97 (kernel/pre-3.15/enable_trace_events_suspend_resume.patch)
98 (kernel/pre-3.15/enable_trace_events_d 98 (kernel/pre-3.15/enable_trace_events_device_pm_callback.patch)
99 99
100 If you're using bootgraph, or sleepgra !! 100 If you're using a kernel older than 3.15.0, the following
101 the following additional kerne !! 101 additional kernel parameters are required:
102 (e.g. in file /etc/default/grub) 102 (e.g. in file /etc/default/grub)
103 GRUB_CMDLINE_LINUX_DEFAULT="... initca 103 GRUB_CMDLINE_LINUX_DEFAULT="... initcall_debug log_buf_len=32M ..."
104 104
105 If you're using a kernel older than 3. 105 If you're using a kernel older than 3.11-rc2, the following simple
106 patch must be applied to enabl 106 patch must be applied to enable ftrace data:
107 in file: kernel/power/suspend.c 107 in file: kernel/power/suspend.c
108 in function: int suspend_devices_and_e 108 in function: int suspend_devices_and_enter(suspend_state_t state)
109 remove call to "ftrace_stop();" 109 remove call to "ftrace_stop();"
110 remove call to "ftrace_start();" 110 remove call to "ftrace_start();"
111 111
112 There is a patch which does this for k 112 There is a patch which does this for kernel v3.8.0:
113 (kernel/pre-3.11-rc2/enable_ftrace_in_ 113 (kernel/pre-3.11-rc2/enable_ftrace_in_suspendresume.patch)
114 114
115 115
116 116
117 ---------------------------------------------- 117 ------------------------------------------------------------------
118 | USAGE 118 | USAGE |
119 ---------------------------------------------- 119 ------------------------------------------------------------------
120 120
121 Basic Usage 121 Basic Usage
122 ___________ 122 ___________
123 123
124 1) First configure a kernel using the instruc 124 1) First configure a kernel using the instructions from the previous sections.
125 Then build, install, and boot with it. 125 Then build, install, and boot with it.
126 2) Open up a terminal window and execute the 126 2) Open up a terminal window and execute the mode list command:
127 127
128 %> sudo ./sleepgraph.py -modes 128 %> sudo ./sleepgraph.py -modes
129 ['freeze', 'mem', 'disk'] 129 ['freeze', 'mem', 'disk']
130 130
131 Execute a test using one of the available pow 131 Execute a test using one of the available power modes, e.g. mem (S3):
132 132
133 %> sudo ./sleepgraph.py -m mem -rtcwak 133 %> sudo ./sleepgraph.py -m mem -rtcwake 15
134 134
135 or with a config file 135 or with a config file
136 136
137 %> sudo ./sleepgraph.py -config config 137 %> sudo ./sleepgraph.py -config config/suspend.cfg
138 138
139 When the system comes back you'll see the scr 139 When the system comes back you'll see the script finishing up and
140 creating the output files in the test subdir. 140 creating the output files in the test subdir. It generates output
141 files in subdirectory: suspend-mmddyy-HHMMSS. 141 files in subdirectory: suspend-mmddyy-HHMMSS. The ftrace file can
142 be used to regenerate the html timeline with 142 be used to regenerate the html timeline with different options
143 143
144 HTML output: <hostname 144 HTML output: <hostname>_<mode>.html
145 raw dmesg output: <hostname 145 raw dmesg output: <hostname>_<mode>_dmesg.txt
146 raw ftrace output: <hostname 146 raw ftrace output: <hostname>_<mode>_ftrace.txt
147 147
148 View the html in firefox or chrome. 148 View the html in firefox or chrome.
149 149
150 150
151 Dev Mode Usage 151 Dev Mode Usage
152 ______________ 152 ______________
153 153
154 Developer mode adds information on low level 154 Developer mode adds information on low level source calls to the timeline.
155 The tool sets kprobes on all delay and mutex 155 The tool sets kprobes on all delay and mutex calls to see which devices
156 are waiting for something and when. It also s 156 are waiting for something and when. It also sets a suite of kprobes on
157 subsystem dependent calls to better fill out 157 subsystem dependent calls to better fill out the timeline.
158 158
159 The tool will also expose kernel threads that 159 The tool will also expose kernel threads that don't normally show up in the
160 timeline. This is useful in discovering depen 160 timeline. This is useful in discovering dependent threads to get a better
161 idea of what each device is waiting for. For 161 idea of what each device is waiting for. For instance, the scsi_eh thread,
162 a.k.a. scsi resume error handler, is what eac 162 a.k.a. scsi resume error handler, is what each SATA disk device waits for
163 before it can continue resume. 163 before it can continue resume.
164 164
165 The timeline will be much larger if run with 165 The timeline will be much larger if run with dev mode, so it can be useful
166 to set the -mindev option to clip out any dev 166 to set the -mindev option to clip out any device blocks that are too small
167 to see easily. The following command will giv 167 to see easily. The following command will give a nice dev mode run:
168 168
169 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -m 169 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -mindev 1 -dev
170 170
171 or with a config file 171 or with a config file
172 172
173 %> sudo ./sleepgraph.py -config config/suspen 173 %> sudo ./sleepgraph.py -config config/suspend-dev.cfg
174 174
175 175
176 Proc Mode Usage 176 Proc Mode Usage
177 _______________ 177 _______________
178 178
179 Proc mode adds user process info to the timel 179 Proc mode adds user process info to the timeline. This is done in a manner
180 similar to the bootchart utility, which graph 180 similar to the bootchart utility, which graphs init processes and their
181 execution as the system boots. This tool opti 181 execution as the system boots. This tool option does the same thing but for
182 the period before and after suspend/resume. 182 the period before and after suspend/resume.
183 183
184 In order to see any process info, there needs 184 In order to see any process info, there needs to be some delay before or
185 after resume since processes are frozen in su 185 after resume since processes are frozen in suspend_prepare and thawed in
186 resume_complete. The predelay and postdelay a 186 resume_complete. The predelay and postdelay args allow you to do this. It
187 can also be useful to run in x2 mode with an 187 can also be useful to run in x2 mode with an x2 delay, this way you can
188 see process activity before and after resume, 188 see process activity before and after resume, and in between two
189 successive suspend/resumes. 189 successive suspend/resumes.
190 190
191 The command can be run like this: 191 The command can be run like this:
192 192
193 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -x 193 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -x2 -x2delay 1000 -predelay 1000 -postdelay 1000 -proc
194 194
195 or with a config file 195 or with a config file
196 196
197 %> sudo ./sleepgraph.py -config config/suspen 197 %> sudo ./sleepgraph.py -config config/suspend-proc.cfg
198 198
199 ---------------------------------------------- 199 ------------------------------------------------------------------
200 | ENDURANCE TESTING 200 | ENDURANCE TESTING |
201 ---------------------------------------------- 201 ------------------------------------------------------------------
202 202
203 The best way to gauge the health of a system 203 The best way to gauge the health of a system is to run a series of
204 suspend/resumes over an extended period and a 204 suspend/resumes over an extended period and analyze the behavior. This can be
205 accomplished with sleepgraph's -multi argumen 205 accomplished with sleepgraph's -multi argument. You specify two numbers: the
206 number of tests to run OR the duration in day 206 number of tests to run OR the duration in days, hours, or minutes, and the
207 delay in seconds between them. For instance, 207 delay in seconds between them. For instance, -multi 20 5: execute 20 tests with
208 a 5 second delay between each, or -multi 24h 208 a 5 second delay between each, or -multi 24h 0: execute tests over a 24 hour
209 period with no delay between tests. You can i 209 period with no delay between tests. You can include any other options you like
210 to generate the data you want. It's most usef 210 to generate the data you want. It's most useful to collect dev mode timelines
211 as the kprobes don't alter the performance mu 211 as the kprobes don't alter the performance much and you get more insight.
212 212
213 On completion, the output folder contains a s 213 On completion, the output folder contains a series of folders for the
214 individual test data and a set of summary pag 214 individual test data and a set of summary pages in the root. The summary.html
215 file is a tabular list of the tests with rele 215 file is a tabular list of the tests with relevant info and links. The
216 summary-issue.html and summary-devices.html f 216 summary-issue.html and summary-devices.html files include data taken from
217 all tests on kernel issues and device perform 217 all tests on kernel issues and device performance. The folder looks like this:
218 218
219 suspend-xN-{date}-{time}: 219 suspend-xN-{date}-{time}:
220 summary.html 220 summary.html
221 summary-issues.html 221 summary-issues.html
222 summary-devices.html 222 summary-devices.html
223 suspend-{date}-{time} (1) 223 suspend-{date}-{time} (1)
224 suspend-{date}-{time} (2) 224 suspend-{date}-{time} (2)
225 ... 225 ...
226 226
227 These are the relevant arguments to use for t 227 These are the relevant arguments to use for testing:
228 228
229 -m mode 229 -m mode
230 Mode to initiate for suspend e.g. mem, 230 Mode to initiate for suspend e.g. mem, freeze, standby (default: mem).
231 231
232 -rtcwake t 232 -rtcwake t
233 Use rtcwake to autoresume after t seco 233 Use rtcwake to autoresume after t seconds (default: 15).
234 234
235 -gzip (optional) 235 -gzip (optional)
236 Gzip the trace and dmesg logs to save 236 Gzip the trace and dmesg logs to save space. The tool can also read in
237 gzipped logs for processing. This redu 237 gzipped logs for processing. This reduces the multitest folder size.
238 238
239 -dev (optional) 239 -dev (optional)
240 Add kernel source calls and threads to 240 Add kernel source calls and threads to the timeline (default: disabled).
241 241
242 -multi n d 242 -multi n d
243 Execute n consecutive tests at d secon 243 Execute n consecutive tests at d seconds intervals. The outputs will be
244 created in a new subdirectory: suspend 244 created in a new subdirectory: suspend-xN-{date}-{time}. When the multitest
245 run is done, the -summary command is c 245 run is done, the -summary command is called automatically to create summary
246 html files for all the data (unless yo 246 html files for all the data (unless you use -skiphtml). -skiphtml will
247 speed up the testing by not creating t 247 speed up the testing by not creating timelines or summary html files. You
248 can then run the tool again at a later 248 can then run the tool again at a later time with -summary and -genhtml to
249 create the timelines. 249 create the timelines.
250 250
251 -skiphtml (optional) 251 -skiphtml (optional)
252 Run the test and capture the trace log 252 Run the test and capture the trace logs, but skip the timeline and summary
253 html generation. This can greatly spee 253 html generation. This can greatly speed up overall testing. You can then
254 copy the data to a faster host machine 254 copy the data to a faster host machine and run -summary -genhtml to
255 generate the timelines and summary. 255 generate the timelines and summary.
256 256
257 These are the relevant commands to use after 257 These are the relevant commands to use after testing is complete:
258 258
259 -summary indir 259 -summary indir
260 Generate or regenerate the summary for 260 Generate or regenerate the summary for a -multi test run. Creates three
261 files: summary.html, summary-issues.ht 261 files: summary.html, summary-issues.html, and summary-devices.html in the
262 current folder. summary.html is a tabl 262 current folder. summary.html is a table of tests with relevant info sorted
263 by kernel/host/mode, and links to the 263 by kernel/host/mode, and links to the test html files. summary-issues.html
264 is a list of kernel issues found in dm 264 is a list of kernel issues found in dmesg from all the tests.
265 summary-devices.html is a list of devi 265 summary-devices.html is a list of devices and times from all the tests.
266 266
267 -genhtml 267 -genhtml
268 Used with -summary to regenerate any 268 Used with -summary to regenerate any missing html timelines from their
269 dmesg and ftrace logs. This will requi 269 dmesg and ftrace logs. This will require a significant amount of time if
270 there are thousands of tests. 270 there are thousands of tests.
271 271
272 Usage Examples 272 Usage Examples
273 _______________ 273 _______________
274 274
275 A multitest is initiated like this: 275 A multitest is initiated like this:
276 276
277 %> sudo ./sleepgraph.py -m mem -rtcwake 10 - 277 %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0
278 278
279 or you can skip timeline generation in 279 or you can skip timeline generation in order to speed things up
280 280
281 %> sudo ./sleepgraph.py -m mem -rtcwake 10 - 281 %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0 -skiphtml
282 282
283 The tool will produce an output folder with a 283 The tool will produce an output folder with all the test subfolders inside.
284 Each test subfolder contains the dmesg/ftrace 284 Each test subfolder contains the dmesg/ftrace logs and/or the html timeline
285 depending on whether you used the -skiphtml o 285 depending on whether you used the -skiphtml option. The root folder contains
286 the summary.html files. 286 the summary.html files.
287 287
288 The summary for an existing multitest is gene 288 The summary for an existing multitest is generated like this:
289 289
290 %> cd suspend-x2000-{date}-{time} 290 %> cd suspend-x2000-{date}-{time}
291 %> sleepgraph.py -summary . 291 %> sleepgraph.py -summary .
292 292
293 or if you need to generate the html ti 293 or if you need to generate the html timelines you can use -genhtml
294 294
295 %> cd suspend-xN-{date}-{time} 295 %> cd suspend-xN-{date}-{time}
296 %> sleepgraph.py -summary . -genhtml 296 %> sleepgraph.py -summary . -genhtml
297 297
298 ---------------------------------------------- 298 ------------------------------------------------------------------
299 | CONFIGURATION FILES 299 | CONFIGURATION FILES |
300 ---------------------------------------------- 300 ------------------------------------------------------------------
301 301
302 Since 4.0 we've moved to using config files i 302 Since 4.0 we've moved to using config files in lieu of command line options.
303 The config folder contains a collection of ty 303 The config folder contains a collection of typical use cases.
304 There are corresponding configs for other pow 304 There are corresponding configs for other power modes:
305 305
306 Simple suspend/resume with basic timel 306 Simple suspend/resume with basic timeline (mem/freeze/standby)
307 config/suspend.cfg 307 config/suspend.cfg
308 config/freeze.cfg 308 config/freeze.cfg
309 config/standby.cfg 309 config/standby.cfg
310 310
311 Dev mode suspend/resume with dev timel 311 Dev mode suspend/resume with dev timeline (mem/freeze/standby)
312 config/suspend-dev.cfg 312 config/suspend-dev.cfg
313 config/freeze-dev.cfg 313 config/freeze-dev.cfg
314 config/standby-dev.cfg 314 config/standby-dev.cfg
315 315
316 Simple suspend/resume with timeline an 316 Simple suspend/resume with timeline and callgraph (mem/freeze/standby)
317 config/suspend-callgraph.cfg 317 config/suspend-callgraph.cfg
318 config/freeze-callgraph.cfg 318 config/freeze-callgraph.cfg
319 config/standby-callgraph.cfg 319 config/standby-callgraph.cfg
320 320
321 Sample proc mode x2 run using mem susp 321 Sample proc mode x2 run using mem suspend
322 config/suspend-x2-proc.cfg 322 config/suspend-x2-proc.cfg
323 323
324 Sample for editing timeline funcs (mov 324 Sample for editing timeline funcs (moves internal functions into config)
325 config/custom-timeline-functio 325 config/custom-timeline-functions.cfg
326 326
327 Sample debug config for serio subsyste 327 Sample debug config for serio subsystem
328 config/debug-serio-suspend.cfg 328 config/debug-serio-suspend.cfg
329 329
330 330
331 Usage Examples 331 Usage Examples
332 ______________ 332 ______________
333 333
334 Run a simple mem suspend: 334 Run a simple mem suspend:
335 %> sudo ./sleepgraph.py -config config/suspen 335 %> sudo ./sleepgraph.py -config config/suspend.cfg
336 336
337 Run a mem suspend with callgraph data: 337 Run a mem suspend with callgraph data:
338 %> sudo ./sleepgraph.py -config config/suspen 338 %> sudo ./sleepgraph.py -config config/suspend-callgraph.cfg
339 339
340 Run a mem suspend with dev mode detail: 340 Run a mem suspend with dev mode detail:
341 %> sudo ./sleepgraph.py -config config/suspen 341 %> sudo ./sleepgraph.py -config config/suspend-dev.cfg
342 342
343 343
344 Config File Options 344 Config File Options
345 ___________________ 345 ___________________
346 346
347 [Settings] 347 [Settings]
348 348
349 # Verbosity: print verbose messages (def: fal 349 # Verbosity: print verbose messages (def: false)
350 verbose: false 350 verbose: false
351 351
352 # Suspend Mode: e.g. standby, mem, freeze, di 352 # Suspend Mode: e.g. standby, mem, freeze, disk (def: mem)
353 mode: mem 353 mode: mem
354 354
355 # Output Directory Format: {hostname}, {date} 355 # Output Directory Format: {hostname}, {date}, {time} give current values
356 output-dir: suspend-{hostname}-{date}-{time} 356 output-dir: suspend-{hostname}-{date}-{time}
357 357
358 # Automatic Wakeup: use rtcwake to wakeup aft 358 # Automatic Wakeup: use rtcwake to wakeup after X seconds (def: infinity)
359 rtcwake: 15 359 rtcwake: 15
360 360
361 # Add Logs: add the dmesg and ftrace log to t 361 # Add Logs: add the dmesg and ftrace log to the html output (def: false)
362 addlogs: false 362 addlogs: false
363 363
364 # Sus/Res Gap: insert a gap between sus & res 364 # Sus/Res Gap: insert a gap between sus & res in the timeline (def: false)
365 srgap: false 365 srgap: false
366 366
367 # Custom Command: Command to execute in lieu 367 # Custom Command: Command to execute in lieu of suspend (def: "")
368 command: echo mem > /sys/power/state 368 command: echo mem > /sys/power/state
369 369
370 # Proc mode: graph user processes and cpu usa 370 # Proc mode: graph user processes and cpu usage in the timeline (def: false)
371 proc: false 371 proc: false
372 372
373 # Dev mode: graph source functions in the tim 373 # Dev mode: graph source functions in the timeline (def: false)
374 dev: false 374 dev: false
375 375
376 # Suspend/Resume x2: run 2 suspend/resumes ba 376 # Suspend/Resume x2: run 2 suspend/resumes back to back (def: false)
377 x2: false 377 x2: false
378 378
379 # x2 Suspend Delay: time delay between the tw 379 # x2 Suspend Delay: time delay between the two test runs in ms (def: 0 ms)
380 x2delay: 0 380 x2delay: 0
381 381
382 # Pre Suspend Delay: nclude an N ms delay bef 382 # Pre Suspend Delay: nclude an N ms delay before (1st) suspend (def: 0 ms)
383 predelay: 0 383 predelay: 0
384 384
385 # Post Resume Delay: include an N ms delay af 385 # Post Resume Delay: include an N ms delay after (last) resume (def: 0 ms)
386 postdelay: 0 386 postdelay: 0
387 387
388 # Min Device Length: graph only dev callbacks 388 # Min Device Length: graph only dev callbacks longer than min (def: 0.001 ms)
389 mindev: 0.001 389 mindev: 0.001
390 390
391 # Callgraph: gather ftrace callgraph data on 391 # Callgraph: gather ftrace callgraph data on all timeline events (def: false)
392 callgraph: false 392 callgraph: false
393 393
394 # Expand Callgraph: pre-expand the callgraph 394 # Expand Callgraph: pre-expand the callgraph treeviews in html (def: false)
395 expandcg: false 395 expandcg: false
396 396
397 # Min Callgraph Length: show callgraphs only 397 # Min Callgraph Length: show callgraphs only if longer than min (def: 1 ms)
398 mincg: 1 398 mincg: 1
399 399
400 # Timestamp Precision: number of sig digits i 400 # Timestamp Precision: number of sig digits in timestamps (0:S, [3:ms], 6:us)
401 timeprec: 3 401 timeprec: 3
402 402
403 # Device Filter: show only devs whose name/dr 403 # Device Filter: show only devs whose name/driver includes one of these strings
404 devicefilter: _cpu_up,_cpu_down,i915,usb 404 devicefilter: _cpu_up,_cpu_down,i915,usb
405 405
406 # Override default timeline entries: 406 # Override default timeline entries:
407 # Do not use the internal default functions f 407 # Do not use the internal default functions for timeline entries (def: false)
408 # Set this to true if you intend to only use 408 # Set this to true if you intend to only use the ones defined in the config
409 override-timeline-functions: true 409 override-timeline-functions: true
410 410
411 # Override default dev timeline entries: 411 # Override default dev timeline entries:
412 # Do not use the internal default functions f 412 # Do not use the internal default functions for dev timeline entries (def: false)
413 # Set this to true if you intend to only use 413 # Set this to true if you intend to only use the ones defined in the config
414 override-dev-timeline-functions: true 414 override-dev-timeline-functions: true
415 415
416 # Call Loop Max Gap (dev mode only) 416 # Call Loop Max Gap (dev mode only)
417 # merge loops of the same call if each is les 417 # merge loops of the same call if each is less than maxgap apart (def: 100us)
418 callloop-maxgap: 0.0001 418 callloop-maxgap: 0.0001
419 419
420 # Call Loop Max Length (dev mode only) 420 # Call Loop Max Length (dev mode only)
421 # merge loops of the same call if each is les 421 # merge loops of the same call if each is less than maxlen in length (def: 5ms)
422 callloop-maxlen: 0.005 422 callloop-maxlen: 0.005
423 423
424 ---------------------------------------------- 424 ------------------------------------------------------------------
425 | CUSTOM TIMELINE ENTRIES 425 | CUSTOM TIMELINE ENTRIES |
426 ---------------------------------------------- 426 ------------------------------------------------------------------
427 427
428 Adding or Editing Timeline Functions 428 Adding or Editing Timeline Functions
429 ____________________________________ 429 ____________________________________
430 430
431 The tool uses an array of function names to f 431 The tool uses an array of function names to fill out empty spaces in the
432 timeline where device callbacks don't appear. 432 timeline where device callbacks don't appear. For instance, in suspend_prepare
433 the tool adds the sys_sync and freeze_process 433 the tool adds the sys_sync and freeze_processes calls as virtual device blocks
434 in the timeline to show you where the time is 434 in the timeline to show you where the time is going. These calls should fill
435 the timeline with contiguous data so that mos 435 the timeline with contiguous data so that most kernel execution is covered.
436 436
437 It is possible to add new function calls to t 437 It is possible to add new function calls to the timeline by adding them to
438 the config. It's also possible to copy the in 438 the config. It's also possible to copy the internal timeline functions into
439 the config so that you can override and edit 439 the config so that you can override and edit them. Place them in the
440 timeline_functions_ARCH section with the name 440 timeline_functions_ARCH section with the name of your architecture appended.
441 i.e. for x86_64: [timeline_functions_x86_64] 441 i.e. for x86_64: [timeline_functions_x86_64]
442 442
443 Use the override-timeline-functions option if 443 Use the override-timeline-functions option if you only want to use your
444 custom calls, or leave it false to append the 444 custom calls, or leave it false to append them to the internal ones.
445 445
446 This section includes a list of functions (se 446 This section includes a list of functions (set using kprobes) which use both
447 symbol data and function arg data. The args a 447 symbol data and function arg data. The args are pulled directly from the
448 stack using this architecture's registers and 448 stack using this architecture's registers and stack formatting. Each entry
449 can include up to four pieces of info: The fu 449 can include up to four pieces of info: The function name, a format string,
450 an argument list, and a color. But only a fun 450 an argument list, and a color. But only a function name is required.
451 451
452 For a full example config, see config/custom- 452 For a full example config, see config/custom-timeline-functions.cfg. It pulls
453 all the internal timeline functions into the 453 all the internal timeline functions into the config and allows you to edit
454 them. 454 them.
455 455
456 Entry format: 456 Entry format:
457 457
458 function: format{fn_arg1}_{fn_arg2} fn_arg 458 function: format{fn_arg1}_{fn_arg2} fn_arg1 fn_arg2 ... [color=purple]
459 459
460 Required Arguments: 460 Required Arguments:
461 461
462 function: The symbol name for the function 462 function: The symbol name for the function you want probed, this is the
463 minimum required for an entry, i 463 minimum required for an entry, it will show up as the function
464 name with no arguments. 464 name with no arguments.
465 465
466 example: _cpu_up: 466 example: _cpu_up:
467 467
468 Optional Arguments: 468 Optional Arguments:
469 469
470 format: The format to display the data on 470 format: The format to display the data on the timeline in. Use braces to
471 enclose the arg names. 471 enclose the arg names.
472 472
473 example: CPU_ON[{cpu}] 473 example: CPU_ON[{cpu}]
474 474
475 color: The color of the entry block in the 475 color: The color of the entry block in the timeline. The default color is
476 transparent, so the entry shares th 476 transparent, so the entry shares the phase color. The color is an
477 html color string, either a word, o 477 html color string, either a word, or an RGB.
478 478
479 example: [color=#CC00CC] 479 example: [color=#CC00CC]
480 480
481 arglist: A list of arguments from register 481 arglist: A list of arguments from registers/stack addresses. See URL:
482 https://www.kernel.org/doc/Docume 482 https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt
483 483
484 example: cpu=%di:s32 484 example: cpu=%di:s32
485 485
486 Here is a full example entry. It displays cpu 486 Here is a full example entry. It displays cpu resume calls in the timeline
487 in orange. They will appear as CPU_ON[0], CPU 487 in orange. They will appear as CPU_ON[0], CPU_ON[1], etc.
488 488
489 [timeline_functions_x86_64] 489 [timeline_functions_x86_64]
490 _cpu_up: CPU_ON[{cpu}] cpu=%di:s32 [color=or 490 _cpu_up: CPU_ON[{cpu}] cpu=%di:s32 [color=orange]
491 491
492 492
493 Adding or Editing Dev Mode Timeline Source Fun 493 Adding or Editing Dev Mode Timeline Source Functions
494 ______________________________________________ 494 ____________________________________________________
495 495
496 In dev mode, the tool uses an array of functi 496 In dev mode, the tool uses an array of function names to monitor source
497 execution within the timeline entries. 497 execution within the timeline entries.
498 498
499 The function calls are displayed inside the m 499 The function calls are displayed inside the main device/call blocks in the
500 timeline. However, if a function call is not 500 timeline. However, if a function call is not within a main timeline event,
501 it will spawn an entirely new event named aft 501 it will spawn an entirely new event named after the caller's kernel thread.
502 These asynchronous kernel threads will popula 502 These asynchronous kernel threads will populate in a separate section
503 beneath the main device/call section. 503 beneath the main device/call section.
504 504
505 The tool has a set of hard coded calls which 505 The tool has a set of hard coded calls which focus on the most common use
506 cases: msleep, udelay, schedule_timeout, mute 506 cases: msleep, udelay, schedule_timeout, mutex_lock_slowpath, etc. These are
507 the functions that add a hardcoded time delay 507 the functions that add a hardcoded time delay to the suspend/resume path.
508 The tool also includes some common functions 508 The tool also includes some common functions native to important
509 subsystems: ata, i915, and ACPI, etc. 509 subsystems: ata, i915, and ACPI, etc.
510 510
511 It is possible to add new function calls to t 511 It is possible to add new function calls to the dev timeline by adding them
512 to the config. It's also possible to copy the 512 to the config. It's also possible to copy the internal dev timeline
513 functions into the config so that you can ove 513 functions into the config so that you can override and edit them. Place them
514 in the dev_timeline_functions_ARCH section wi 514 in the dev_timeline_functions_ARCH section with the name of your architecture
515 appended. i.e. for x86_64: [dev_timeline_func 515 appended. i.e. for x86_64: [dev_timeline_functions_x86_64]
516 516
517 Use the override-dev-timeline-functions optio 517 Use the override-dev-timeline-functions option if you only want to use your
518 custom calls, or leave it false to append the 518 custom calls, or leave it false to append them to the internal ones.
519 519
520 The format is the same as the timeline_functi 520 The format is the same as the timeline_functions_x86_64 section. It's a
521 list of functions (set using kprobes) which u 521 list of functions (set using kprobes) which use both symbol data and function
522 arg data. The args are pulled directly from t 522 arg data. The args are pulled directly from the stack using this
523 architecture's registers and stack formatting 523 architecture's registers and stack formatting. Each entry can include up
524 to four pieces of info: The function name, a 524 to four pieces of info: The function name, a format string, an argument list,
525 and a color. But only the function name is re 525 and a color. But only the function name is required.
526 526
527 For a full example config, see config/custom- 527 For a full example config, see config/custom-timeline-functions.cfg. It pulls
528 all the internal dev timeline functions into 528 all the internal dev timeline functions into the config and allows you to edit
529 them. 529 them.
530 530
531 Here is a full example entry. It displays the 531 Here is a full example entry. It displays the ATA port reset calls as
532 ataN_port_reset in the timeline. This is wher 532 ataN_port_reset in the timeline. This is where most of the SATA disk resume
533 time goes, so it can be helpful to see the lo 533 time goes, so it can be helpful to see the low level call.
534 534
535 [dev_timeline_functions_x86_64] 535 [dev_timeline_functions_x86_64]
536 ata_eh_recover: ata{port}_port_reset port=+3 536 ata_eh_recover: ata{port}_port_reset port=+36(%di):s32 [color=#CC00CC]
537 537
538 538
539 Verifying your custom functions 539 Verifying your custom functions
540 _______________________________ 540 _______________________________
541 541
542 Once you have a set of functions (kprobes) de 542 Once you have a set of functions (kprobes) defined, it can be useful to
543 perform a quick check to see if you formatted 543 perform a quick check to see if you formatted them correctly and if the system
544 actually supports them. To do this, run the t 544 actually supports them. To do this, run the tool with your config file
545 and the -status option. The tool will go thro 545 and the -status option. The tool will go through all the kprobes (both
546 custom and internal if you haven't overridden 546 custom and internal if you haven't overridden them) and actually attempts
547 to set them in ftrace. It will then print out 547 to set them in ftrace. It will then print out success or fail for you.
548 548
549 Note that kprobes which don't actually exist 549 Note that kprobes which don't actually exist in the kernel won't stop the
550 tool, they just wont show up. 550 tool, they just wont show up.
551 551
552 For example: 552 For example:
553 553
554 sudo ./sleepgraph.py -config config/custom-ti 554 sudo ./sleepgraph.py -config config/custom-timeline-functions.cfg -status
555 Checking this system (myhostname)... 555 Checking this system (myhostname)...
556 have root access: YES 556 have root access: YES
557 is sysfs mounted: YES 557 is sysfs mounted: YES
558 is "mem" a valid power mode: YES 558 is "mem" a valid power mode: YES
559 is ftrace supported: YES 559 is ftrace supported: YES
560 are kprobes supported: YES 560 are kprobes supported: YES
561 timeline data source: FTRACE (all trace ev 561 timeline data source: FTRACE (all trace events found)
562 is rtcwake supported: YES 562 is rtcwake supported: YES
563 verifying timeline kprobes work: 563 verifying timeline kprobes work:
564 _cpu_down: YES 564 _cpu_down: YES
565 _cpu_up: YES 565 _cpu_up: YES
566 acpi_pm_finish: YES 566 acpi_pm_finish: YES
567 acpi_pm_prepare: YES 567 acpi_pm_prepare: YES
568 freeze_kernel_threads: YES 568 freeze_kernel_threads: YES
569 freeze_processes: YES 569 freeze_processes: YES
570 sys_sync: YES 570 sys_sync: YES
571 thaw_processes: YES 571 thaw_processes: YES
572 verifying dev kprobes work: 572 verifying dev kprobes work:
573 __const_udelay: YES 573 __const_udelay: YES
574 __mutex_lock_slowpath: YES 574 __mutex_lock_slowpath: YES
575 acpi_os_stall: YES 575 acpi_os_stall: YES
576 acpi_ps_parse_aml: YES 576 acpi_ps_parse_aml: YES
577 intel_opregion_init: NO 577 intel_opregion_init: NO
578 intel_opregion_register: NO 578 intel_opregion_register: NO
579 intel_opregion_setup: NO 579 intel_opregion_setup: NO
580 msleep: YES 580 msleep: YES
581 schedule_timeout: YES 581 schedule_timeout: YES
582 schedule_timeout_uninterruptible: YES 582 schedule_timeout_uninterruptible: YES
583 usleep_range: YES 583 usleep_range: YES
584 584
585 585
586 ---------------------------------------------- 586 ------------------------------------------------------------------
587 | TESTING ON CONSUMER LINUX OPERATIN 587 | TESTING ON CONSUMER LINUX OPERATING SYSTEMS |
588 ---------------------------------------------- 588 ------------------------------------------------------------------
589 589
590 Android 590 Android
591 _______ 591 _______
592 592
593 The easiest way to execute on an android devi 593 The easiest way to execute on an android device is to run the android.sh
594 script on the device, then pull the ftrace lo 594 script on the device, then pull the ftrace log back to the host and run
595 sleepgraph.py on it. 595 sleepgraph.py on it.
596 596
597 Here are the steps: 597 Here are the steps:
598 598
599 [download and install the tool on the device] 599 [download and install the tool on the device]
600 600
601 host%> wget https://raw.githubusercont 601 host%> wget https://raw.githubusercontent.com/intel/pm-graph/master/tools/android.sh
602 host%> adb connect 192.168.1.6 602 host%> adb connect 192.168.1.6
603 host%> adb root 603 host%> adb root
604 # push the script to a writeable locat 604 # push the script to a writeable location
605 host%> adb push android.sh /sdcard/ 605 host%> adb push android.sh /sdcard/
606 606
607 [check whether the tool will run on your devi 607 [check whether the tool will run on your device]
608 608
609 host%> adb shell 609 host%> adb shell
610 dev%> cd /sdcard 610 dev%> cd /sdcard
611 dev%> sh android.sh status 611 dev%> sh android.sh status
612 host : asus_t100 612 host : asus_t100
613 kernel : 3.14.0-i386-dirty 613 kernel : 3.14.0-i386-dirty
614 modes : freeze mem 614 modes : freeze mem
615 rtcwake : supported 615 rtcwake : supported
616 ftrace : supported 616 ftrace : supported
617 trace events { 617 trace events {
618 suspend_resume: found 618 suspend_resume: found
619 device_pm_callback_end: fo 619 device_pm_callback_end: found
620 device_pm_callback_start: 620 device_pm_callback_start: found
621 } 621 }
622 # the above is what you see on a syste 622 # the above is what you see on a system that's properly patched
623 623
624 [execute the suspend] 624 [execute the suspend]
625 625
626 # NOTE: The suspend will only work if 626 # NOTE: The suspend will only work if the screen isn't timed out,
627 # so you have to press some keys first 627 # so you have to press some keys first to wake it up b4 suspend)
628 dev%> sh android.sh suspend mem 628 dev%> sh android.sh suspend mem
629 ------------------------------------ 629 ------------------------------------
630 Suspend/Resume timing test initiated 630 Suspend/Resume timing test initiated
631 ------------------------------------ 631 ------------------------------------
632 hostname : asus_t100 632 hostname : asus_t100
633 kernel : 3.14.0-i386-dirty 633 kernel : 3.14.0-i386-dirty
634 mode : mem 634 mode : mem
635 ftrace out : /mnt/shell/emulated/0/ftr 635 ftrace out : /mnt/shell/emulated/0/ftrace.txt
636 dmesg out : /mnt/shell/emulated/0/dme 636 dmesg out : /mnt/shell/emulated/0/dmesg.txt
637 log file : /mnt/shell/emulated/0/log 637 log file : /mnt/shell/emulated/0/log.txt
638 ------------------------------------ 638 ------------------------------------
639 INITIALIZING FTRACE........DONE 639 INITIALIZING FTRACE........DONE
640 STARTING FTRACE 640 STARTING FTRACE
641 SUSPEND START @ 21:24:02 (rtcwake in 1 641 SUSPEND START @ 21:24:02 (rtcwake in 10 seconds)
642 <adb connection will now terminate> 642 <adb connection will now terminate>
643 643
644 [retrieve the data from the device] 644 [retrieve the data from the device]
645 645
646 # I find that you have to actually kil 646 # I find that you have to actually kill the adb process and
647 # reconnect sometimes in order for the 647 # reconnect sometimes in order for the connection to work post-suspend
648 host%> adb connect 192.168.1.6 648 host%> adb connect 192.168.1.6
649 # (required) get the ftrace data, this 649 # (required) get the ftrace data, this is the most important piece
650 host%> adb pull /sdcard/ftrace.txt 650 host%> adb pull /sdcard/ftrace.txt
651 # (optional) get the dmesg data, this 651 # (optional) get the dmesg data, this is for debugging
652 host%> adb pull /sdcard/dmesg.txt 652 host%> adb pull /sdcard/dmesg.txt
653 # (optional) get the log, which just l 653 # (optional) get the log, which just lists some test times for comparison
654 host%> adb pull /sdcard/log.txt 654 host%> adb pull /sdcard/log.txt
655 655
656 [create an output html file using sleepgraph. 656 [create an output html file using sleepgraph.py]
657 657
658 host%> sleepgraph.py -ftrace ftrace.tx 658 host%> sleepgraph.py -ftrace ftrace.txt
659 659
660 You should now have an output.html with the a 660 You should now have an output.html with the android data, enjoy!
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.