1 perf-intel-pt(1)
2 ================
3
4 NAME
5 ----
6 perf-intel-pt - Support for Intel Processor Tr
7
8 SYNOPSIS
9 --------
10 [verse]
11 'perf record' -e intel_pt//
12
13 DESCRIPTION
14 -----------
15
16 Intel Processor Trace (Intel PT) is an extensi
17 collects information about software execution
18 modes and timings and formats it into highly c
19 Technical details are documented in the Intel
20 Software Developer Manuals, Chapter 36 Intel P
21
22 Intel PT is first supported in Intel Core M an
23 processors that are based on the Intel micro-a
24
25 Trace data is collected by 'perf record' and s
26 See below for options to 'perf record'.
27
28 Trace data must be 'decoded' which involves wa
29 the trace data packets. For example a TNT pack
30 conditional branch was taken or not taken, so
31 decoder must know precisely which instruction
32
33 Decoding is done on-the-fly. The decoder outp
34 samples output by perf hardware events, for ex
35 or "branches" events had been recorded. Prese
36 'perf script', 'perf report' and 'perf inject'
37 on using those tools.
38
39 The main distinguishing feature of Intel PT is
40 the exact flow of software execution. Intel P
41 and how did software get to a certain point, o
42 software does not have to be recompiled, so In
43 builds, however the executed images are needed
44 environments, or with self-modified code, a ch
45 provided to make sense of addresses.
46
47 A limitation of Intel PT is that it produces h
48 (hundreds of megabytes per second per core) wh
49 for example two or three orders of magnitude l
50 Another limitation is the performance impact o
51 vary depending on the use-case and architectur
52
53
54 Quickstart
55 ----------
56
57 It is important to start small. That is becau
58 more data than can possibly be processed.
59
60 The simplest thing to do with Intel PT is user
61 Data is captured with 'perf record' e.g. to tr
62
63 perf record -e intel_pt//u ls
64
65 And profiled with 'perf report' e.g.
66
67 perf report
68
69 To also trace kernel space presents a problem,
70 code. A fairly good kernel image is available
71 accurate image a copy of /proc/kcore needs to
72 as the data capture. 'perf record' can make a
73 --kcore is used, but access to /proc/kcore is
74
75 sudo perf record -o pt_ls --kcore -e i
76
77 which will create a directory named 'pt_ls' an
78 simply 'data') and copies of /proc/kcore, /pro
79 it. The other tools understand the directory
80 becomes:
81
82 sudo perf report -i pt_ls
83
84 Because samples are synthesized after-the-fact
85 selected for reporting. e.g. sample every micr
86
87 sudo perf report pt_ls --itrace=i1usge
88
89 See the sections below for more information ab
90
91 Beware the smaller the period, the more sample
92 longer it takes to process them.
93
94 Also note that the coarseness of Intel PT timi
95 distort the statistical value of the sampling
96 smaller.
97
98 To represent software control flow, "branches"
99 a branch sample is synthesized for every singl
100 data is available you can use the 'perf script
101 options, which will list all the samples.
102
103 perf record -e intel_pt//u ls
104 perf script --itrace=iybxwpe
105
106 An interesting field that is not printed by de
107 displayed as follows:
108
109 perf script --itrace=iybxwpe -F+flags
110
111 The flags are "bcrosyiABExghDt" which stand fo
112 system, asynchronous, interrupt, transaction a
113 in transaction, VM-entry, VM-exit, interrupt d
114 toggle respectively.
115
116 perf script also supports higher level ways to
117
118 perf script --insn-trace=disasm
119
120 or to use the xed disassembler, which requires
121 (see XED below):
122
123 perf script --insn-trace --xed
124
125 Dumping all instructions in a long trace can b
126 to start with higher level decoding, like
127
128 perf script --call-trace
129
130 or
131
132 perf script --call-ret-trace
133
134 and then select a time range of interest. The
135 in detail with
136
137 perf script --time starttime,stoptime
138
139 While examining the trace it's also useful to
140 the -C option
141
142 perf script --time starttime,stoptime
143
144 Dump all instructions in time range on CPU 1.
145
146 Another interesting field that is not printed
147 displayed as follows:
148
149 perf script --itrace=be -F+ipc
150
151 There are two ways that instructions-per-cycle
152 on the recording.
153
154 If the 'cyc' config term (see config terms sec
155 and cycle events are calculated using the cycl
156 MTC packets are used - refer to the 'mtc' conf
157 the values are less accurate because the timin
158
159 Because Intel PT does not update the cycle cou
160 the values will often be zero. When there are
161 of instructions and number of cycles since the
162 the average IPC cycle count since the last IPC
163 Note IPC for "branches" events is calculated s
164 events.
165
166 Even with the 'cyc' config term, it is possibl
167 every change of timestamp, but at the expense
168 specifying the itrace 'A' option. Due to the
169 actual number of cycles increases even though
170 The number of instructions is known, but if IP
171 low and so IPC is too high. Note that inaccur
172 sampling increases i.e. if the number of cycle
173 that becomes less significant if the number of
174 useful to use the 'A' option in conjunction wi
175 provide higher granularity cycle information.
176
177 Also note that the IPC instruction count may o
178 instruction. If the cycle count is associated
179 (e.g. page fault or interrupt), then the instr
180 current instruction, otherwise it does. That
181 that instruction has retired when the cycle co
182
183 Another note, in the case of "branches" events
184 presently sampled, so IPC values for them do n
185 TNT packet that starts with a non-taken branch
186 value, "instructions" events can be used e.g.
187
188 While it is possible to create scripts to anal
189 approach is available to export the data to a
190 Refer to script export-to-sqlite.py or export-
191 and to script exported-sql-viewer.py for an ex
192
193 There is also script intel-pt-events.py which
194 unpack the raw data for power events and PTWRI
195 branches, and supports 2 additional modes sele
196
197 - --insn-trace - instruction trace
198 - --src-trace - source trace
199
200 The intel-pt-events.py script also has options
201
202 - --all-switch-events - display all switch ev
203 - --interleave [<n>] - interleave sample outp
204 no more than n samples for a CPU are displaye
205 Note this only affects the order of output, a
206 same.
207
208 As mentioned above, it is easy to capture too
209 data captured is to use 'snapshot' mode which
210 Refer to 'new snapshot option' and 'Intel PT m
211
212 Another problem that will be experienced is de
213 by inability to access the executed image, sel
214 inability to match side-band information (such
215 which results in the decoder not knowing what
216
217 There is also the problem of perf not being ab
218 resulting in data lost because the buffer was
219 for more details.
220
221
222 perf record
223 -----------
224
225 new event
226 ~~~~~~~~~
227
228 The Intel PT kernel driver creates a new PMU f
229 selected by providing the PMU name followed by
230 An enhancement has been made to allow default
231
232 -e intel_pt//
233
234 will use a default config value. Currently th
235
236 -e intel_pt/tsc,noretcomp=0/
237
238 which is the same as
239
240 -e intel_pt/tsc=1,noretcomp=0/
241
242 Note there are now new config terms - see sect
243
244 The config terms are listed in /sys/devices/in
245 fields within the config member of the struct
246 passed to the kernel by the perf_event_open sy
247 fields in the IA32_RTIT_CTL MSR. Here is a li
248
249 $ grep -H . /sys/bus/event_source/devi
250 /sys/bus/event_source/devices/intel_pt
251 /sys/bus/event_source/devices/intel_pt
252 /sys/bus/event_source/devices/intel_pt
253 /sys/bus/event_source/devices/intel_pt
254 /sys/bus/event_source/devices/intel_pt
255 /sys/bus/event_source/devices/intel_pt
256 /sys/bus/event_source/devices/intel_pt
257
258 Note that the default config must be overridde
259
260 -e intel_pt/noretcomp=0/
261
262 is the same as:
263
264 -e intel_pt/tsc=1,noretcomp=0/
265
266 So, to disable TSC packets use:
267
268 -e intel_pt/tsc=0/
269
270 It is also possible to specify the config valu
271
272 -e intel_pt/config=0x400/
273
274 Note that, as with all events, the event is su
275
276 u userspace
277 k kernel
278 h hypervisor
279 G guest
280 H host
281 p precise ip
282
283 'h', 'G' and 'H' are for virtualization which
284 'p' is also not relevant to Intel PT. So only
285 meaningful for Intel PT.
286
287 perf_event_attr is displayed if the -vv option
288
289 --------------------------------------
290 perf_event_attr:
291 type 6
292 size 112
293 config 0x400
294 { sample_period, sample_freq } 1
295 sample_type IP|TI
296 read_format ID
297 disabled 1
298 inherit 1
299 exclude_kernel 1
300 exclude_hv 1
301 enable_on_exec 1
302 sample_id_all 1
303 --------------------------------------
304 sys_perf_event_open: pid 31104 cpu 0
305 sys_perf_event_open: pid 31104 cpu 1
306 sys_perf_event_open: pid 31104 cpu 2
307 sys_perf_event_open: pid 31104 cpu 3
308 --------------------------------------
309
310
311 config terms
312 ~~~~~~~~~~~~
313
314 The June 2015 version of Intel 64 and IA-32 Ar
315 Manuals, Chapter 36 Intel Processor Trace, def
316 Some of the features are reflect in new config
317 described below.
318
319 tsc Always supported. Produces TS
320 timing information. In some c
321 without timing information, fo
322 that does not overlap executab
323
324 The default config selects tsc
325
326 noretcomp Always supported. Disables "r
327 is produced when a function re
328 produced but might make decodi
329
330 The default config does not se
331
332 psb_period Allows the frequency of PSB pa
333
334 The PSB packet is a synchroniz
335 starting point for decoding or
336
337 Support for psb_period is indi
338
339 /sys/bus/event_source/
340
341 which contains "1" if the feat
342 otherwise.
343
344 Valid values are given by:
345
346 /sys/bus/event_source/
347
348 which contains a hexadecimal v
349 valid values e.g. bit 2 set me
350
351 The psb_period value is conver
352 trace bytes between PSB packet
353
354 2 ^ (value + 11)
355
356 e.g. value 3 means 16KiB bytes
357
358 If an invalid value is entered
359 will give a list of valid valu
360
361 $ perf record -e intel
362 Invalid psb_period for
363
364 If MTC packets are selected, t
365 of 3 (i.e. psb_period=3) or th
366 supported (0 is always support
367
368 If decoding is expected to be
369 then a large PSB period can be
370
371 Because a TSC packet is produc
372 also affect the granularity to
373 of MTC or CYC.
374
375 mtc Produces MTC timing packets.
376
377 MTC packets provide finer grai
378 packets. MTC packets record t
379 clock (CTC) which is related t
380
381 Support for this feature is in
382
383 /sys/bus/event_source/
384
385 which contains "1" if the feat
386 "0" otherwise.
387
388 The frequency of MTC packets c
389 mtc_period below.
390
391 mtc_period Specifies how frequently MTC p
392 above for how to determine if
393
394 Valid values are given by:
395
396 /sys/bus/event_source/
397
398 which contains a hexadecimal v
399 valid values e.g. bit 2 set me
400
401 The mtc_period value is conver
402
403 CTC-frequency / (2 ^ v
404
405 e.g. value 3 means one eighth
406
407 Where CTC is the hardware crys
408 can be related to TSC via valu
409
410 If an invalid value is entered
411 will give a list of valid valu
412
413 $ perf record -e intel
414 Invalid mtc_period for
415
416 The default value is 3 or the
417 that is supported (0 is always
418
419 cyc Produces CYC timing packets.
420
421 CYC packets provide even finer
422 MTC and TSC packets. A CYC pa
423 cycles since the last CYC pack
424 CYC packets are only sent when
425
426 Support for this feature is in
427
428 /sys/bus/event_source/
429
430 which contains "1" if the feat
431 "0" otherwise.
432
433 The number of CYC packets prod
434 a threshold - see cyc_thresh b
435
436 cyc_thresh Specifies how frequently CYC p
437 above for how to determine if
438
439 Valid cyc_thresh values are gi
440
441 /sys/bus/event_source/
442
443 which contains a hexadecimal v
444 valid values e.g. bit 2 set me
445
446 The cyc_thresh value represent
447 that must have passed before a
448 number of CPU cycles is:
449
450 2 ^ (value - 1)
451
452 e.g. value 4 means 8 CPU cycle
453 can be sent. Note a CYC packe
454 packet is sent, not at, e.g. e
455
456 If an invalid value is entered
457 will give a list of valid valu
458
459 $ perf record -e intel
460 Invalid cyc_thresh for
461
462 CYC packets are not requested
463
464 pt Specifies pass-through which e
465
466 The default config selects 'pt
467 never need to specify this ter
468
469 branch Enable branch tracing. Branch
470 disable branch tracing use 'br
471
472 The default config selects 'br
473
474 ptw Enable PTWRITE packets which a
475 is executed.
476
477 Support for this feature is in
478
479 /sys/bus/event_source/
480
481 which contains "1" if the feat
482 "0" otherwise.
483
484 As an alternative, refer to "E
485
486 fup_on_ptw Enable a FUP packet to follow
487 provides the address of the pt
488 fup_on_ptw, the decoder will u
489 if branch tracing is enabled,
490 Note that fup_on_ptw will work
491
492 pwr_evt Enable power events. The powe
493 changes to the CPU C-state.
494
495 Support for this feature is in
496
497 /sys/bus/event_source/
498
499 which contains "1" if the feat
500 "0" otherwise.
501
502 event Enable Event Trace. The event
503 events.
504
505 Support for this feature is in
506
507 /sys/bus/event_source/
508
509 which contains "1" if the feat
510 "0" otherwise.
511
512 notnt Disable TNT packets. Without
513 executable code to reconstruct
514 and TIP.PGD packets still indi
515 return compression is disabled
516 The advantage of eliminating T
517 trace and corresponding tracin
518
519 Support for this feature is in
520
521 /sys/bus/event_source/
522
523 which contains "1" if the feat
524 "0" otherwise.
525
526
527 AUX area sampling option
528 ~~~~~~~~~~~~~~~~~~~~~~~~
529
530 To select Intel PT "sampling" the AUX area sam
531
532 --aux-sample
533
534 Optionally it can be followed by the sample si
535
536 --aux-sample=8192
537
538 In addition, the Intel PT event to sample must
539
540 -e intel_pt//u
541
542 Samples on other events will be created contai
543 following will create Intel PT samples on the
544 events must be grouped using {}:
545
546 perf record --aux-sample -e '{intel_pt
547
548 An alternative to '--aux-sample' is to add the
549 events. In this case, the grouping is implied
550
551 perf record -e intel_pt//u -e branch-m
552
553 is the same as:
554
555 perf record -e '{intel_pt//u,branch-mi
556
557 but allows for also using an address filter e.
558
559 perf record -e intel_pt//u --filter 'f
560
561 It is important to select a sample size that i
562 one PSB packet. If not a warning will be disp
563
564 Intel PT sample size (%zu) may be too
565
566 The calculation used for that is: if sample_si
567 warning. When sampling is used, psb_period de
568
569 The default sample size is 4KiB.
570
571 The sample size is passed in aux_sample_size i
572 sample size is limited by the maximum event si
573 difficult to know how big the event might be w
574 but the tool validates that the sample size is
575
576
577 new snapshot option
578 ~~~~~~~~~~~~~~~~~~~
579
580 The difference between full trace and snapshot
581 that in full trace we don't overwrite trace da
582 yet (and indicated that by advancing aux_tail)
583 the trace run and overwrite older data in the
584 interesting happens, we can stop it and grab a
585 around that interesting moment.
586
587 To select snapshot mode a new option has been
588
589 -S
590
591 Optionally it can be followed by the snapshot
592
593 -S0x100000
594
595 The default snapshot size is the auxtrace mmap
596 nor snapshot size is specified, then the defau
597 (or if /proc/sys/kernel/perf_event_paranoid <
598 If an unprivileged user does not specify mmap
599 reduced as described in the 'new auxtrace mmap
600
601 The snapshot size is displayed if the option -
602
603 Intel PT snapshot size: %zu
604
605
606 new auxtrace mmap size option
607 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
608
609 Intel PT buffer size is specified by an additi
610
611 -m,16
612
613 selects a buffer size of 16 pages i.e. 64KiB.
614
615 Note that the existing functionality of -m is
616 is specified by the optional addition of a com
617
618 The default auxtrace mmap size for Intel PT is
619 (or if /proc/sys/kernel/perf_event_paranoid <
620 If an unprivileged user does not specify mmap
621 reduced from the default 512KiB/page_size to 2
622 user is likely to get an error as they exceed
623 memory as shown in /proc/self/limits). Note t
624 512KiB (actually /proc/sys/kernel/perf_event_m
625 against the mlock limit so an unprivileged use
626 their mlock limit (which defaults to 64KiB but
627 of cpus).
628
629 In full-trace mode, powers of two are allowed
630 size of 2 pages. In snapshot mode or sampling
631 minimum size is 1 page.
632
633 The mmap size and auxtrace mmap size are displ
634
635 mmap length 528384
636 auxtrace mmap length 4198400
637
638
639 Intel PT modes of operation
640 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
641
642 Intel PT can be used in 3 modes:
643 full-trace mode
644 sample mode
645 snapshot mode
646
647 Full-trace mode traces continuously e.g.
648
649 perf record -e intel_pt//u uname
650
651 Sample mode attaches a Intel PT sample to othe
652
653 perf record --aux-sample -e intel_pt//
654
655 Snapshot mode captures the available data when
656 control command is issued. e.g. using a signal
657
658 perf record -v -e intel_pt//u -S ./loo
659 [1] 11435
660 kill -USR2 11435
661 Recording AUX area tracing snapshot
662
663 Note that the signal sent is SIGUSR2.
664 Note that "Recording AUX area tracing snapshot
665 option is used.
666
667 The advantage of using "snapshot" control comm
668 controlled by access to a FIFO e.g.
669
670 $ mkfifo perf.control
671 $ mkfifo perf.ack
672 $ cat perf.ack &
673 [1] 15235
674 $ sudo ~/bin/perf record --control fif
675 [2] 15243
676 $ ps -e | grep perf
677 15244 pts/1 00:00:00 perf
678 $ kill -USR2 15244
679 bash: kill: (15244) - Operation not pe
680 $ echo snapshot > perf.control
681 ack
682
683 The 3 Intel PT modes of operation cannot be us
684
685
686 Buffer handling
687 ~~~~~~~~~~~~~~~
688
689 There may be buffer limitations (i.e. single T
690 buffer sizes are limited to powers of 2 up to
691 provide other sizes, and in particular an arbi
692 buffers are logically concatenated. However a
693 between buffers. That has two potential probl
694 a) the interrupt may not be handled in
695 becomes full and some trace data is lo
696 b) the interrupts may slow the system
697 results.
698
699 If trace data is lost, the driver sets 'trunca
700 which the tools report as an error.
701
702 In full-trace mode, the driver waits for data
703 the (logical) buffer to wrap-around. If data
704 again 'truncated' is set in the PERF_RECORD_AU
705 wait, the intel_pt event gets disabled. Becau
706 that happens, perf tools always re-enable the
707 data.
708
709
710 Intel PT and build ids
711 ~~~~~~~~~~~~~~~~~~~~~~
712
713 By default "perf record" post-processes the ev
714 for executables for all addresses sampled. De
715 decoded for that purpose (it would take too lo
716 all executables encountered (due to mmap, comm
717 in the perf.data file.
718
719 To see buildids included in the perf.data file
720
721 perf buildid-list
722
723 If the perf.data file contains Intel PT data,
724
725 perf buildid-list --with-hits
726
727
728 Snapshot mode and event disabling
729 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
730
731 In order to make a snapshot, the intel_pt even
732 namely PERF_EVENT_IOC_DISABLE. However doing
733 collection of side-band information. In order
734 software event has been introduced that permit
735 continue to be recorded while intel_pt is disa
736 there is complete side-band information to all
737 snapshots.
738
739 A test has been created for that. To find the
740
741 perf test list
742 ...
743 23: Test using a dummy software event
744
745 To run the test:
746
747 perf test 23
748 23: Test using a dummy software event
749
750
751 perf record modes (nothing new here)
752 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
753
754 perf record essentially operates in one of thr
755 per thread
756 per cpu
757 workload only
758
759 "per thread" mode is selected by -t or by --pe
760 workload).
761 "per cpu" is selected by -C or -a.
762 "workload only" mode is selected by not using
763 command to run (i.e. the workload).
764
765 In per-thread mode an exact list of threads is
766 Each thread has its own event buffer.
767
768 In per-cpu mode all processes (or processes fr
769 option, or processes selected with -p or -u) a
770 buffer. Inheritance is allowed.
771
772 In workload-only mode, the workload is traced
773 Inheritance is allowed. Note that you can now
774 mode by using the --per-thread option.
775
776
777 Privileged vs non-privileged users
778 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
779
780 Unless /proc/sys/kernel/perf_event_paranoid is
781 have memory limits imposed upon them. That af
782 have as outlined above.
783
784 The v4.2 kernel introduced support for a conte
785 PERF_RECORD_SWITCH, which allows unprivileged
786 are scheduled out and in, just not by whom, wh
787 PERF_RECORD_SWITCH_CPU_WIDE, that is only acce
788 which in turn requires CAP_PERFMON or CAP_SYS_
789
790 Please see the 45ac1403f564 ("perf: Add PERF_R
791 switches") commit, that introduces these metad
792
793 When working with kernels < v4.2, the followin
794 as the sched:sched_switch tracepoints will be
795
796 Unless /proc/sys/kernel/perf_event_paranoid is
797 not permitted to use tracepoints which means t
798 information to decode Intel PT in per-cpu mode
799 mode too if the workload creates new processes
800
801 Note also, that to use tracepoints, read-acces
802 debugfs is not mounted or the user does not ha
803 be possible to decode Intel PT in per-cpu mode
804
805
806 sched_switch tracepoint
807 ~~~~~~~~~~~~~~~~~~~~~~~
808
809 The sched_switch tracepoint is used to provide
810 decoding in kernels where the PERF_RECORD_SWIT
811 available.
812
813 The sched_switch events are automatically adde
814 below:
815
816 $ perf record -vv -e intel_pt//u uname
817 --------------------------------------
818 perf_event_attr:
819 type 6
820 size 112
821 config 0x400
822 { sample_period, sample_freq } 1
823 sample_type IP|TI
824 read_format ID
825 disabled 1
826 inherit 1
827 exclude_kernel 1
828 exclude_hv 1
829 enable_on_exec 1
830 sample_id_all 1
831 --------------------------------------
832 sys_perf_event_open: pid 31104 cpu 0
833 sys_perf_event_open: pid 31104 cpu 1
834 sys_perf_event_open: pid 31104 cpu 2
835 sys_perf_event_open: pid 31104 cpu 3
836 --------------------------------------
837 perf_event_attr:
838 type 2
839 size 112
840 config 0x108
841 { sample_period, sample_freq } 1
842 sample_type IP|TI
843 read_format ID
844 inherit 1
845 sample_id_all 1
846 exclude_guest 1
847 --------------------------------------
848 sys_perf_event_open: pid -1 cpu 0 gr
849 sys_perf_event_open: pid -1 cpu 1 gr
850 sys_perf_event_open: pid -1 cpu 2 gr
851 sys_perf_event_open: pid -1 cpu 3 gr
852 --------------------------------------
853 perf_event_attr:
854 type 1
855 size 112
856 config 0x9
857 { sample_period, sample_freq } 1
858 sample_type IP|TI
859 read_format ID
860 disabled 1
861 inherit 1
862 exclude_kernel 1
863 exclude_hv 1
864 mmap 1
865 comm 1
866 enable_on_exec 1
867 task 1
868 sample_id_all 1
869 mmap2 1
870 comm_exec 1
871 --------------------------------------
872 sys_perf_event_open: pid 31104 cpu 0
873 sys_perf_event_open: pid 31104 cpu 1
874 sys_perf_event_open: pid 31104 cpu 2
875 sys_perf_event_open: pid 31104 cpu 3
876 mmap size 528384B
877 AUX area mmap length 4194304
878 perf event ring buffer mmapped per cpu
879 Synthesizing auxtrace information
880 Linux
881 [ perf record: Woken up 1 times to wri
882 [ perf record: Captured and wrote 0.04
883
884 Note, the sched_switch event is only added if
885 and only in per-cpu mode.
886
887 Note also, the sched_switch event is only adde
888 That is because, in the absence of timing info
889 cannot be matched against the Intel PT trace.
890
891
892 perf script
893 -----------
894
895 By default, perf script will decode trace data
896 This can be further controlled by new option -
897
898
899 New --itrace option
900 ~~~~~~~~~~~~~~~~~~~
901
902 Having no option is the same as
903
904 --itrace
905
906 which, in turn, is the same as
907
908 --itrace=cepwxy
909
910 The letters are:
911
912 i synthesize "instructions" even
913 y synthesize "cycles" events
914 b synthesize "branches" events
915 x synthesize "transactions" even
916 w synthesize "ptwrite" events
917 p synthesize "power" events (inc
918 c synthesize branches events (ca
919 r synthesize branches events (re
920 o synthesize PEBS-via-PT events
921 I synthesize Event Trace events
922 e synthesize tracing error event
923 d create a debug log
924 g synthesize a call chain (use w
925 G synthesize a call chain on exi
926 l synthesize last branch entries
927 L synthesize last branch entries
928 s skip initial number of events
929 q quicker (less detailed) decodi
930 A approximate IPC
931 Z prefer to ignore timestamps (s
932
933 "Instructions" events look like they were reco
934 instructions".
935
936 "Cycles" events look like they were recorded b
937 (ie., the default). Note that even with CYC pa
938 these are not fully accurate, since CYC packet
939 instruction, only when some other event (like
940 TNT packet representing multiple branches) hap
941 be emitted. Thus, it is more effective for att
942 (and possibly basic blocks) than to individual
943 is not even perfect for functions (although it
944 option is active).
945
946 "Branches" events look like they were recorded
947 and "r" can be combined to get calls and retur
948
949 "Transactions" events correspond to the start
950 'flags' field can be used in perf script to de
951 transaction start, commit or abort.
952
953 Note that "instructions", "cycles", "branches"
954 depend on code flow packets which can be disab
955 "branch=0". Refer to the config terms section
956
957 "ptwrite" events record the payload of the ptw
958 "fup_on_ptw" was used. "ptwrite" events depen
959 recorded only if the "ptw" config term was use
960 section above. perf script "synth" field disp
961 this: "ip: 0 payload: 0x123456789abcdef0" whe
962 used.
963
964 "Power" events correspond to power event packe
965 packets. While CBR packets are always recorde
966 event packets are recorded only if the "pwr_ev
967 the config terms section above. The power eve
968 C-state changes, whereas CBR is indicative of
969 "event,synth" fields display information like
970
971 cbr: cbr: 22 freq: 2189 MHz (200%)
972 mwait: hints: 0x60 extensions: 0x1
973 pwre: hw: 0 cstate: 2 sub-cstate: 0
974 exstop: ip: 1
975 pwrx: deepest cstate: 2 last cstate:
976
977 Where:
978
979 "cbr" includes the frequency and the p
980 "mwait" shows mwait hints and extensio
981 "pwre" shows C-state transitions (to a
982 whether initiated by hardware
983 "exstop" indicates execution stopped a
984 exactly,
985 "pwrx" indicates return to C0
986
987 For more details refer to the Intel 64 and IA-
988 Developer Manuals.
989
990 PSB events show when a PSB+ occurred and also
991 Emitting a PSB+ can cause a CPU a slight delay
992 of code with Intel PT, it is useful to know if
993 by Intel PT or not.
994
995 Error events show where the decoder lost the t
996 are quite important. Users must know if what
997 picture or not. The "e" option may be followed
998 will or will not be reported. Each flag must
999 The flags supported by Intel PT are:
1000
1001 -o Suppress overflow err
1002 -l Suppress trace data l
1003
1004 For example, for errors but not overflow or d
1005
1006 --itrace=e-o-l
1007
1008 The "d" option will cause the creation of a f
1009 decoded packets and instructions. Note that
1010 and that the resulting file may be very large
1011 by flags which affect what debug messages wil
1012 must be preceded by either '+' or '-'. The fl
1013
1014 -a Suppress logging of p
1015 +a Log all perf events
1016 +e Output only on decodi
1017 +o Output to stdout inst
1018
1019 By default, logged perf events are filtered b
1020 flag +a overrides that. The +e flag can be u
1021 default, the log size in that case is 16384 b
1022 linkperf:perf-config[1] e.g. perf config itra
1023
1024 In addition, the period of the "instructions"
1025
1026 --itrace=i10us
1027
1028 sets the period to 10us i.e. one instruction
1029 microseconds of trace. Alternatives to "us"
1030 "ns" (nanoseconds), "t" (TSC ticks) or "i" (i
1031
1032 "ms", "us" and "ns" are converted to TSC tick
1033
1034 The timing information included with Intel PT
1035 instruction. Consequently, for the purpose o
1036 the time since the last timing packet based o
1037 on the sample is *not* adjusted and reflects
1038
1039 For Intel PT, the default period is 100us.
1040
1041 Setting it to a zero period means "as often a
1042
1043 In the case of Intel PT that is the same as a
1044 'instructions' (i.e. --itrace=i1i).
1045
1046 Also the call chain size (default 16, max. 10
1047 transactions events can be specified. e.g.
1048
1049 --itrace=ig32
1050 --itrace=xg32
1051
1052 Also the number of last branch entries (defau
1053 transactions events can be specified. e.g.
1054
1055 --itrace=il10
1056 --itrace=xl10
1057
1058 Note that last branch entries are cleared for
1059 from one sample to the next.
1060
1061 The G and L options are designed in particula
1062 like g and l but add call chain and branch st
1063 instead of synthesized events. For example, t
1064 'ls' and then add a call chain derived from t
1065
1066 perf record --aux-sample -e '{intel_p
1067 perf report --itrace=Ge
1068
1069 Although in fact G is a default for perf repo
1070
1071 perf report
1072
1073 One caveat with the G and L options is that t
1074 Large PEBS means PEBS records will be accumul
1075 into the event buffer in one go. That reduce
1076 late timestamps. Because the Intel PT trace
1077 the PEBS events do not match the trace. Curr
1078 certain circumstances:
1079 - hardware supports it
1080 - PEBS is used
1081 - event period is specified, instead
1082 - the sample type is limited to the f
1083 PERF_SAMPLE_IP | PERF_SAMPLE_
1084 PERF_SAMPLE_ID | PERF_SAMPLE_
1085 PERF_SAMPLE_DATA_SRC | PERF_S
1086 PERF_SAMPLE_TRANSACTION | PER
1087 PERF_SAMPLE_REGS_INTR | PERF_
1088 PERF_SAMPLE_PERIOD (and somet
1089 Because Intel PT sample mode uses a different
1090 Large PEBS is not used with Intel PT sample m
1091 cases, avoid specifying the event period i.e.
1092 --count option, or 'period' config term.
1093
1094 To disable trace decoding entirely, use the o
1095
1096 It is also possible to skip events generated
1097 at the beginning. This is useful to ignore in
1098
1099 --itrace=i0nss1000000
1100
1101 skips the first million instructions.
1102
1103 The q option changes the way the trace is dec
1104 but much less detailed. Specifically, with t
1105 decode TNT packets, and does not walk object
1106 TIP packets. The q option can be used with t
1107 is not used. The q option decodes more quick
1108 control flow of interest is represented or in
1109 TIP.PGD packets (refer below). However the q
1110 ranges that could then be decoded fully using
1111
1112 What will *not* be decoded with the (single)
1113
1114 - direct calls and jmps
1115 - conditional branches
1116 - non-branch instructions
1117
1118 What *will* be decoded with the (single) q op
1119
1120 - asynchronous branches such as inter
1121 - indirect branches
1122 - function return target address *if*
1123 config terms section) was used
1124 - start of (control-flow) tracing
1125 - end of (control-flow) tracing, if i
1126 - power events, ptwrite, transaction
1127 - instruction pointer associated with
1128
1129 Note the q option does not specify what event
1130 option must be used also to show power events
1131
1132 Repeating the q option (double-q i.e. qq) res
1133 less detail. The decoder decodes only extend
1134 instruction pointer if there is a FUP packet
1135 PSBEND). Note PSB packets occur regularly in
1136 config term (refer config terms section). Th
1137 PSB+ occurs while control flow is being trace
1138
1139 What will *not* be decoded with the qq option
1140
1141 - everything except instruction point
1142
1143 What *will* be decoded with the qq option:
1144
1145 - instruction pointer associated with
1146
1147 The Z option is equivalent to having recorded
1148 (i.e. config term tsc=0). It can be useful to
1149 decoding a trace of a virtual machine.
1150
1151
1152 dlfilter-show-cycles.so
1153 ~~~~~~~~~~~~~~~~~~~~~~~
1154
1155 Cycles can be displayed using dlfilter-show-c
1156 option can be useful to provide higher granul
1157
1158 perf script --itrace=A --call-trace -
1159
1160 To see a list of dlfilters:
1161
1162 perf script -v --list-dlfilters
1163
1164 See also linkperf:perf-dlfilters[1]
1165
1166
1167 dump option
1168 ~~~~~~~~~~~
1169
1170 perf script has an option (-D) to "dump" the
1171 data.
1172
1173 When -D is used, Intel PT packets are display
1174 pay attention to PSB packets, but just decode
1175 by the actual decoder may not be identical in
1176 One example of that would be when the buffer-
1177 slow, and the buffer has been filled complete
1178 in the buffer might be truncated and immediat
1179 continues in the next buffer.
1180
1181 To disable the display of Intel PT packets, c
1182 --no-itrace.
1183
1184
1185 perf report
1186 -----------
1187
1188 By default, perf report will decode trace dat
1189 This can be further controlled by new option
1190 perf script, with the exception that the defa
1191
1192
1193 perf inject
1194 -----------
1195
1196 perf inject also accepts the --itrace option
1197 removed and replaced with the synthesized eve
1198
1199 perf inject --itrace -i perf.data -o
1200
1201 Below is an example of using Intel PT with au
1202 (https://github.com/google/autofdo) and gcc v
1203 sort example is from the AutoFDO tutorial (ht
1204 amended to take the number of elements as a p
1205
1206 $ gcc-5 -O3 sort.c -o sort_optimized
1207 $ ./sort_optimized 30000
1208 Bubble sorting array of 30000 element
1209 2254 ms
1210
1211 $ cat ~/.perfconfig
1212 [intel-pt]
1213 mispred-all = on
1214
1215 $ perf record -e intel_pt//u ./sort 3
1216 Bubble sorting array of 3000 elements
1217 58 ms
1218 [ perf record: Woken up 2 times to wr
1219 [ perf record: Captured and wrote 3.9
1220 $ perf inject -i perf.data -o inj --i
1221 $ ./create_gcov --binary=./sort --pro
1222 $ gcc-5 -O3 -fauto-profile=sort.gcov
1223 $ ./sort_autofdo 30000
1224 Bubble sorting array of 30000 element
1225 2155 ms
1226
1227 Note there is currently no advantage to using
1228 that may change in the future if greater use
1229
1230
1231 PEBS via Intel PT
1232 -----------------
1233
1234 Some hardware has the feature to redirect PEB
1235 Recording is selected by using the aux-output
1236
1237 perf record -c 10000 -e '{intel_pt/br
1238
1239 Originally, software only supported redirecti
1240 was not able to differentiate one event from
1241 kernels and perf tools add support for the PE
1242 To check for the presence of that event in a
1243
1244 perf script -D --no-itrace | grep PER
1245
1246 To display PEBS events from the Intel PT trac
1247
1248 perf script --itrace=oe
1249
1250 XED
1251 ---
1252
1253 include::build-xed.txt[]
1254
1255
1256 Tracing Virtual Machines (kernel only)
1257 --------------------------------------
1258
1259 Currently, kernel tracing is supported with e
1260 (i.e. no TSC timestamps) or VM Time Correlati
1261 using 'perf inject' and requires unchanging V
1262
1263 Other limitations and caveats
1264
1265 VMX controls may suppress packets needed for
1266 VMX controls may block the perf NMI to the h
1267 Guest kernel self-modifying code (e.g. jump
1268 Guest thread information is unknown
1269 Guest VCPU is unknown but may be able to be
1270 Callchains are not supported
1271
1272 Example using "timeless" decoding
1273
1274 Start VM
1275
1276 $ sudo virsh start kubuntu20.04
1277 Domain kubuntu20.04 started
1278
1279 Mount the guest file system. Note sshfs need
1280
1281 $ mkdir vm0
1282 $ sshfs -o direct_io root@vm0:/ vm0
1283
1284 Copy the guest /proc/kallsyms, /proc/modules
1285
1286 $ perf buildid-cache -v --kcore vm0/proc/kco
1287 kcore added to build-id cache directory /hom
1288 $ KALLSYMS=/home/user/.debug/[kernel.kcore]/
1289
1290 Find the VM process
1291
1292 $ ps -eLl | grep 'KVM\|PID'
1293 F S UID PID PPID LWP C PRI NI
1294 3 S 64055 1430 1 1440 1 80 0
1295 3 S 64055 1430 1 1441 1 80 0
1296 3 S 64055 1430 1 1442 1 80 0
1297 3 S 64055 1430 1 1443 2 80 0
1298
1299 Start an open-ended perf record, tracing the
1300 TSC is not supported and tsc=0 must be specif
1301 However, IPC can still be determined, hence c
1302 Only kernel decoding is supported, so 'k' mus
1303 Intel PT traces both the host and the guest s
1304 Without timestamps, --per-thread must be spec
1305
1306 $ sudo perf kvm --guest --host --guestkallsy
1307 ^C
1308 [ perf record: Woken up 1 times to write dat
1309 [ perf record: Captured and wrote 5.829 MB ]
1310
1311 perf script can be used to provide an instruc
1312
1313 $ perf script --guestkallsyms $KALLSYMS --in
1314 CPU 0/KVM 1440 ffffffff82133cdd __vm
1315 CPU 0/KVM 1440 ffffffff82133ce1 __vm
1316 CPU 0/KVM 1440 ffffffff82133ce5 __vm
1317 CPU 0/KVM 1440 ffffffff82133ce9 __vm
1318 CPU 0/KVM 1440 ffffffff82133ced __vm
1319 CPU 0/KVM 1440 ffffffff82133cf1 __vm
1320 CPU 0/KVM 1440 ffffffff82133cf5 __vm
1321 CPU 0/KVM 1440 ffffffff82133cf9 __vm
1322 CPU 0/KVM 1440 ffffffff82133cfc __vm
1323 CPU 0/KVM 1440 ffffffff82133c40 vmx_
1324 CPU 0/KVM 1440 ffffffff82133c42 vmx_
1325 :1440 1440 ffffffffbb678b06 nati
1326 :1440 1440 ffffffffbb678b0b nati
1327 :1440 1440 ffffffffbb666646 lapi
1328 :1440 1440 ffffffffbb666648 lapi
1329 :1440 1440 ffffffffbb66664a lapi
1330 :1440 1440 ffffffffbb66664b lapi
1331 :1440 1440 ffffffffbb74607f cloc
1332 :1440 1440 ffffffffbb746081 cloc
1333 :1440 1440 ffffffffbb74603c cloc
1334 :1440 1440 ffffffffbb74603d cloc
1335
1336 Example using VM Time Correlation
1337
1338 Start VM
1339
1340 $ sudo virsh start kubuntu20.04
1341 Domain kubuntu20.04 started
1342
1343 Mount the guest file system. Note sshfs need
1344
1345 $ mkdir -p vm0
1346 $ sshfs -o direct_io root@vm0:/ vm0
1347
1348 Copy the guest /proc/kallsyms, /proc/modules
1349
1350 $ perf buildid-cache -v --kcore vm0/proc/kco
1351 same kcore found in /home/user/.debug/[kerne
1352 $ KALLSYMS=/home/user/.debug/\[kernel.kcore\
1353
1354 Find the VM process
1355
1356 $ ps -eLl | grep 'KVM\|PID'
1357 F S UID PID PPID LWP C PRI NI
1358 3 S 64055 16998 1 17005 13 80 0
1359 3 S 64055 16998 1 17006 4 80 0
1360 3 S 64055 16998 1 17007 3 80 0
1361 3 S 64055 16998 1 17008 4 80 0
1362
1363 Start an open-ended perf record, tracing the
1364 IPC can be determined, hence cyc=1 can be add
1365 Only kernel decoding is supported, so 'k' mus
1366 Intel PT traces both the host and the guest s
1367
1368 $ sudo perf kvm --guest --host --guestkallsy
1369 ^C[ perf record: Woken up 1 times to write d
1370 [ perf record: Captured and wrote 9.041 MB p
1371
1372 Now 'perf inject' can be used to determine th
1373 only 7-bytes, so the TSC Offset might differ
1374 have no effect i.e. the resulting timestamps
1375
1376 $ perf inject -i perf.data.kvm --vm-time-cor
1377 ERROR: Unknown TSC Offset for VMCS 0x1bff6a
1378 VMCS: 0x1bff6a TSC Offset 0xffffe42722c64c4
1379 ERROR: Unknown TSC Offset for VMCS 0x1cbc08
1380 VMCS: 0x1cbc08 TSC Offset 0xffffe42722c64c4
1381 ERROR: Unknown TSC Offset for VMCS 0x1c3ce8
1382 VMCS: 0x1c3ce8 TSC Offset 0xffffe42722c64c4
1383 ERROR: Unknown TSC Offset for VMCS 0x1cbce9
1384 VMCS: 0x1cbce9 TSC Offset 0xffffe42722c64c4
1385
1386 Each virtual CPU has a different Virtual Mach
1387 shown above with the calculated TSC Offset. F
1388 they should all be the same for the same virt
1389
1390 Now that the TSC Offset is known, it can be p
1391
1392 $ perf inject -i perf.data.kvm --vm-time-cor
1393
1394 Note the options for 'perf inject' --vm-time-
1395
1396 [ dry-run ] [ <TSC Offset> [ : <VMCS> [ , <V
1397
1398 So it is possible to specify different TSC Of
1399 The option "dry-run" will cause the file to b
1400 Note it is also possible to get a intel_pt.lo
1401
1402 There were no errors so, do it for real
1403
1404 $ perf inject -i perf.data.kvm --vm-time-cor
1405
1406 'perf script' can be used to see if there are
1407
1408 $ perf script -i perf.data.kvm --guestkallsy
1409
1410 There were none.
1411
1412 'perf script' can be used to provide an instr
1413
1414 $ perf script -i perf.data.kvm --guestkallsy
1415 CPU 1/KVM 17006 [001] 11500.262865593:
1416 CPU 1/KVM 17006 [001] 11500.262865593:
1417 CPU 1/KVM 17006 [001] 11500.262865593:
1418 CPU 1/KVM 17006 [001] 11500.262865593:
1419 CPU 1/KVM 17006 [001] 11500.262865593:
1420 CPU 1/KVM 17006 [001] 11500.262865593:
1421 CPU 1/KVM 17006 [001] 11500.262865593:
1422 CPU 1/KVM 17006 [001] 11500.262865593:
1423 CPU 1/KVM 17006 [001] 11500.262865593:
1424 CPU 1/KVM 17006 [001] 11500.262865593:
1425 CPU 1/KVM 17006 [001] 11500.262866075:
1426 :17006 17006 [001] 11500.262869216:
1427 :17006 17006 [001] 11500.262869216:
1428 :17006 17006 [001] 11500.262869216:
1429 :17006 17006 [001] 11500.262869216:
1430 :17006 17006 [001] 11500.262869216:
1431 :17006 17006 [001] 11500.262869216:
1432 :17006 17006 [001] 11500.262869216:
1433 :17006 17006 [001] 11500.262869216:
1434 :17006 17006 [001] 11500.262869216:
1435 :17006 17006 [001] 11500.262869216:
1436
1437
1438 Tracing Virtual Machines (including user spac
1439 ---------------------------------------------
1440
1441 It is possible to use perf record to record s
1442 Sideband events from the guest perf.data file
1443
1444 Here is an example of the steps needed:
1445
1446 On the guest machine:
1447
1448 Check that no-kvmclock kernel command line op
1449
1450 Note, this is essential to enable time correl
1451
1452 $ cat /proc/cmdline
1453 BOOT_IMAGE=/boot/vmlinuz-5.10.0-16-amd64 roo
1454
1455 There is no BPF support at present so, if pos
1456
1457 $ echo 0 | sudo tee /proc/sys/net/core/bpf_j
1458 0
1459
1460 Start perf record to collect sideband events:
1461
1462 $ sudo perf record -o guest-sideband-testing
1463
1464 On the host machine:
1465
1466 Start perf record to collect Intel PT trace:
1467
1468 Note, the host trace will get very big, very
1469
1470 $ sudo perf record -o guest-sideband-testing
1471
1472 On the guest machine:
1473
1474 Run a small test case, just 'uname' in this e
1475
1476 $ uname
1477 Linux
1478
1479 On the host machine:
1480
1481 Stop the Intel PT trace:
1482
1483 ^C
1484 [ perf record: Woken up 1 times to write dat
1485 [ perf record: Captured and wrote 76.122 MB
1486
1487 On the guest machine:
1488
1489 Stop the Intel PT trace:
1490
1491 ^C
1492 [ perf record: Woken up 1 times to write dat
1493 [ perf record: Captured and wrote 1.247 MB g
1494
1495 And then copy guest-sideband-testing-guest-pe
1496
1497 On the host machine:
1498
1499 With the 2 perf.data recordings, and with the
1500
1501 Identify the TSC Offset:
1502
1503 $ perf inject -i guest-sideband-testing-host
1504 VMCS: 0x103fc6 TSC Offset 0xfffffa6ae070cb2
1505 VMCS: 0x103ff2 TSC Offset 0xfffffa6ae070cb2
1506 VMCS: 0x10fdaa TSC Offset 0xfffffa6ae070cb2
1507 VMCS: 0x24d57c TSC Offset 0xfffffa6ae070cb2
1508
1509 Correct Intel PT TSC timestamps for the guest
1510
1511 $ perf inject -i guest-sideband-testing-host
1512
1513 Identify the guest machine PID:
1514
1515 $ perf script -i guest-sideband-testing-host
1516 CPU 0/KVM 0 [000] 0.000000: PE
1517 CPU 1/KVM 0 [000] 0.000000: PE
1518 CPU 2/KVM 0 [000] 0.000000: PE
1519 CPU 3/KVM 0 [000] 0.000000: PE
1520
1521 Note, the QEMU option -name debug-threads=on
1522 can be used to determine which thread is runn
1523
1524 Create a guestmount, assuming the guest machi
1525
1526 $ mkdir -p ~/guestmount/13376
1527 $ sshfs -o direct_io vm_to_test:/ ~/guestmou
1528
1529 Inject the guest perf.data file into the host
1530
1531 Note, due to the guestmount option, guest obj
1532 If needed, VDSO can be copied manually in a f
1533
1534 $ perf inject -i guest-sideband-testing-host
1535
1536 Show an excerpt from the result. In this cas
1537
1538 Notes:
1539
1540 - the CPU displayed, [002] in this ca
1541 - events happening in the virtual mac
1542 - only calls and errors are displayed
1543 - branches entering and exiting the v
1544
1545 $ perf script -i inj --itrace=ce -F+machine_
1546 CPU 3/KVM 13376/13384 [002] 7919.4088
1547 CPU 3/KVM 13376/13384 [002] 7919.4088
1548 CPU 3/KVM 13376/13384 [002] 7919.4088
1549 CPU 3/KVM 13376/13384 [002] 7919.4088
1550 VM:13376 VCPU:003 uname 3404/340
1551 VM:13376 VCPU:003 uname 3404/340
1552 CPU 3/KVM 13376/13384 [002] 7919.4088
1553 CPU 3/KVM 13376/13384 [002] 7919.4088
1554 CPU 3/KVM 13376/13384 [002] 7919.4088
1555 CPU 3/KVM 13376/13384 [002] 7919.4088
1556 CPU 3/KVM 13376/13384 [002] 7919.4088
1557 CPU 3/KVM 13376/13384 [002] 7919.4088
1558 CPU 3/KVM 13376/13384 [002] 7919.4088
1559 CPU 3/KVM 13376/13384 [002] 7919.4088
1560 CPU 3/KVM 13376/13384 [002] 7919.4088
1561 CPU 3/KVM 13376/13384 [002] 7919.4088
1562 CPU 3/KVM 13376/13384 [002] 7919.4088
1563 CPU 3/KVM 13376/13384 [002] 7919.4088
1564 CPU 3/KVM 13376/13384 [002] 7919.4088
1565 CPU 3/KVM 13376/13384 [002] 7919.4088
1566 CPU 3/KVM 13376/13384 [002] 7919.4088
1567 CPU 3/KVM 13376/13384 [002] 7919.4088
1568 CPU 3/KVM 13376/13384 [002] 7919.4088
1569 CPU 3/KVM 13376/13384 [002] 7919.4088
1570 CPU 3/KVM 13376/13384 [002] 7919.4088
1571 CPU 3/KVM 13376/13384 [002] 7919.4088
1572 CPU 3/KVM 13376/13384 [002] 7919.4088
1573 CPU 3/KVM 13376/13384 [002] 7919.4088
1574 CPU 3/KVM 13376/13384 [002] 7919.4088
1575 CPU 3/KVM 13376/13384 [002] 7919.4088
1576 CPU 3/KVM 13376/13384 [002] 7919.4088
1577 CPU 3/KVM 13376/13384 [002] 7919.4088
1578 CPU 3/KVM 13376/13384 [002] 7919.4088
1579 CPU 3/KVM 13376/13384 [002] 7919.4088
1580 CPU 3/KVM 13376/13384 [002] 7919.4088
1581 CPU 3/KVM 13376/13384 [002] 7919.4088
1582 CPU 3/KVM 13376/13384 [002] 7919.4088
1583 CPU 3/KVM 13376/13384 [002] 7919.4088
1584 CPU 3/KVM 13376/13384 [002] 7919.4088
1585 CPU 3/KVM 13376/13384 [002] 7919.4088
1586 CPU 3/KVM 13376/13384 [002] 7919.4088
1587 CPU 3/KVM 13376/13384 [002] 7919.4088
1588 CPU 3/KVM 13376/13384 [002] 7919.4088
1589 CPU 3/KVM 13376/13384 [002] 7919.4088
1590 CPU 3/KVM 13376/13384 [002] 7919.4088
1591 CPU 3/KVM 13376/13384 [002] 7919.4088
1592 CPU 3/KVM 13376/13384 [002] 7919.4088
1593 CPU 3/KVM 13376/13384 [002] 7919.4088
1594 CPU 3/KVM 13376/13384 [002] 7919.4088
1595 CPU 3/KVM 13376/13384 [002] 7919.4088
1596 CPU 3/KVM 13376/13384 [002] 7919.4088
1597 CPU 3/KVM 13376/13384 [002] 7919.4088
1598 CPU 3/KVM 13376/13384 [002] 7919.4088
1599 CPU 3/KVM 13376/13384 [002] 7919.4088
1600 CPU 3/KVM 13376/13384 [002] 7919.4088
1601 VM:13376 VCPU:003 uname 3404/340
1602 VM:13376 VCPU:003 uname 3404/340
1603 VM:13376 VCPU:003 uname 3404/340
1604 VM:13376 VCPU:003 uname 3404/340
1605 VM:13376 VCPU:003 uname 3404/340
1606 VM:13376 VCPU:003 uname 3404/340
1607 VM:13376 VCPU:003 uname 3404/340
1608 VM:13376 VCPU:003 uname 3404/340
1609
1610
1611 Tracing Virtual Machines - Guest Code
1612 -------------------------------------
1613
1614 A common case for KVM test programs is that t
1615 hypervisor, creating, running and destroying
1616 providing the guest object code from its own
1617 the VM is not running an OS, but only the fun
1618 hypervisor test program, and conveniently, lo
1619 addresses. To support that, option "--guest-c
1620 and perf kvm report.
1621
1622 Here is an example tracing a test program fro
1623
1624 # perf record --kcore -e intel_pt/cyc/ -- to
1625 [ perf record: Woken up 1 times to write dat
1626 [ perf record: Captured and wrote 0.280 MB p
1627 # perf script --guest-code --itrace=bep --ns
1628 [SNIP]
1629 tsc_msrs_test 18436 [007] 10897.962087733:
1630 tsc_msrs_test 18436 [007] 10897.962087733:
1631 tsc_msrs_test 18436 [007] 10897.962087733:
1632 tsc_msrs_test 18436 [007] 10897.962087836:
1633 [guest/18436] 18436 [007] 10897.962087836:
1634 [guest/18436] 18436 [007] 10897.962087836:
1635 [guest/18436] 18436 [007] 10897.962088248:
1636 tsc_msrs_test 18436 [007] 10897.962088248:
1637 tsc_msrs_test 18436 [007] 10897.962088248:
1638 tsc_msrs_test 18436 [007] 10897.962088256:
1639 tsc_msrs_test 18436 [007] 10897.962088270:
1640 [SNIP]
1641 tsc_msrs_test 18436 [007] 10897.962089321:
1642 tsc_msrs_test 18436 [007] 10897.962089321:
1643 tsc_msrs_test 18436 [007] 10897.962089321:
1644 tsc_msrs_test 18436 [007] 10897.962089424:
1645 [guest/18436] 18436 [007] 10897.962089424:
1646 [guest/18436] 18436 [007] 10897.962089701:
1647 [guest/18436] 18436 [007] 10897.962089701:
1648 [guest/18436] 18436 [007] 10897.962089701:
1649 [guest/18436] 18436 [007] 10897.962089701:
1650 [guest/18436] 18436 [007] 10897.962089878:
1651 tsc_msrs_test 18436 [007] 10897.962089878:
1652 tsc_msrs_test 18436 [007] 10897.962089878:
1653 tsc_msrs_test 18436 [007] 10897.962089887:
1654 tsc_msrs_test 18436 [007] 10897.962089901:
1655 [SNIP]
1656
1657 # perf kvm --guest-code --guest --host repor
1658
1659 # To display the perf.data header info, plea
1660 #
1661 #
1662 # Total Lost Samples: 0
1663 #
1664 # Samples: 12 of event 'instructions'
1665 # Event count (approx.): 2274583
1666 #
1667 # Children Self Command Shared
1668 # ........ ........ ............. .......
1669 #
1670 54.70% 0.00% tsc_msrs_test [kernel.
1671 |
1672 ---entry_SYSCALL_64_after_hwframe
1673 do_syscall_64
1674 |
1675 |--29.44%--syscall_exit_to_use
1676 | exit_to_user_mode_p
1677 | task_work_run
1678 | __fput
1679
1680
1681 Event Trace
1682 -----------
1683
1684 Event Trace records information about asynchr
1685 faults, VM exits and entries. The informatio
1686 and also the Interrupt Flag is recorded on th
1687 contains a type field to identify one of the
1688
1689 1 INTR interrupt, fa
1690 2 IRET interrupt ret
1691 3 SMI system manage
1692 4 RSM resume from s
1693 5 SIPI startup inter
1694 6 INIT INIT signal
1695 7 VMENTRY VM-Entry
1696 8 VMEXIT VM-Entry
1697 9 VMEXIT_INTR VM-Exit due t
1698 10 SHUTDOWN Shutdown
1699
1700 For more details, refer to the Intel 64 and I
1701 Developer Manuals (version 076 or later).
1702
1703 The capability to do Event Trace is indicated
1704 /sys/bus/event_source/devices/intel_pt/caps/e
1705
1706 Event trace is selected for recording using t
1707
1708 perf record -e intel_pt/event/u uname
1709
1710 Event trace events are output using the --itr
1711
1712 perf script --itrace=Ie
1713
1714 perf script displays events containing CFE ty
1715 in the form:
1716
1717 evt: hw int (t) cfe:
1718
1719 The IP flag indicates if the event binds to a
1720 flow control packet generation is enabled, as
1721 set.
1722
1723 perf script displays events containing change
1724
1725 iflag: t IFLAG
1726
1727 where "via branch" indicates a branch (interr
1728 "non branch" indicates an instruction such as
1729
1730 In addition, the current state of the interru
1731 or absence of the "D" (interrupt disabled) pe
1732 flag is changed, then the "t" flag is also in
1733
1734 no flag, interrupts enabled I
1735 t interrupts become disabled IF
1736 D interrupts are disabled IF=0
1737 Dt interrupts become enabled IF
1738
1739 The intel-pt-events.py script illustrates how
1740 using a Python script.
1741
1742
1743 TNT Disable
1744 -----------
1745
1746 TNT packets are disabled using the "notnt" co
1747
1748 perf record -e intel_pt/notnt/u uname
1749
1750 In that case the --itrace q option is forced
1751 to reconstruct the control flow is not possib
1752
1753
1754 Emulated PTWRITE
1755 ----------------
1756
1757 Later perf tools support a method to emulate
1758 can be useful if hardware does not support th
1759
1760 Instead of using the ptwrite instruction, a f
1761 a trace that encodes the payload data into TN
1762 of the function:
1763
1764 #include <stdint.h>
1765
1766 void perf_emulate_ptwrite(uint64_t x)
1767 __attribute__((externally_visible, noipa, no
1768
1769 #define PERF_EMULATE_PTWRITE_8_BITS \
1770 "1: shl %rax\n" \
1771 " jc 1f\n" \
1772 "1: shl %rax\n" \
1773 " jc 1f\n" \
1774 "1: shl %rax\n" \
1775 " jc 1f\n" \
1776 "1: shl %rax\n" \
1777 " jc 1f\n" \
1778 "1: shl %rax\n" \
1779 " jc 1f\n" \
1780 "1: shl %rax\n" \
1781 " jc 1f\n" \
1782 "1: shl %rax\n" \
1783 " jc 1f\n" \
1784 "1: shl %rax\n" \
1785 " jc 1f\n"
1786
1787 /* Undefined instruction */
1788 #define PERF_EMULATE_PTWRITE_UD2 ".by
1789
1790 #define PERF_EMULATE_PTWRITE_MAGIC PE
1791
1792 void perf_emulate_ptwrite(uint64_t x __attri
1793 {
1794 /* Assumes SysV ABI : x passed in r
1795 __asm__ volatile (
1796 "jmp 1f\n"
1797 PERF_EMULATE_PTWRITE_MAGIC
1798 "1: mov %rdi, %rax\n"
1799 PERF_EMULATE_PTWRITE_8_BITS
1800 PERF_EMULATE_PTWRITE_8_BITS
1801 PERF_EMULATE_PTWRITE_8_BITS
1802 PERF_EMULATE_PTWRITE_8_BITS
1803 PERF_EMULATE_PTWRITE_8_BITS
1804 PERF_EMULATE_PTWRITE_8_BITS
1805 PERF_EMULATE_PTWRITE_8_BITS
1806 PERF_EMULATE_PTWRITE_8_BITS
1807 "1: ret\n"
1808 );
1809 }
1810
1811 For example, a test program with the function
1812
1813 #include <stdio.h>
1814 #include <stdint.h>
1815 #include <stdlib.h>
1816
1817 #include "perf_emulate_ptwrite.h"
1818
1819 int main(int argc, char *argv[])
1820 {
1821 uint64_t x = 0;
1822
1823 if (argc > 1)
1824 x = strtoull(argv[1], NULL,
1825 perf_emulate_ptwrite(x);
1826 return 0;
1827 }
1828
1829 Can be compiled and traced:
1830
1831 $ gcc -Wall -Wextra -O3 -g -o eg_ptw eg_ptw.
1832 $ perf record -e intel_pt//u ./eg_ptw 0x1234
1833 [ perf record: Woken up 1 times to write dat
1834 [ perf record: Captured and wrote 0.017 MB p
1835 $ perf script --itrace=ew
1836 eg_ptw 19875 [007] 8061.235912:
1837 $
1838
1839
1840 Pipe mode
1841 ---------
1842 Pipe mode is a problem for Intel PT and possi
1843 It's not recommended to use a pipe as data ou
1844 of the following reason.
1845
1846 Essentially the auxtrace buffers do not behav
1847 event buffers. That is because the head and
1848 software, but in the auxtrace case the data i
1849 So the head and tail do not get updated as da
1850
1851 In the Intel PT case, the head and tail are u
1852 is disabled by software, for example:
1853 - full-trace, system wide : when buffer p
1854 - full-trace, not system-wide : when buff
1855 context s
1856 - snapshot mode : as above but also when
1857 - sample mode : as above but also when a
1858
1859 That means finished-round ordering doesn't wo
1860 can turn up that has data that extends back i
1861 very beginning of tracing.
1862
1863 For a perf.data file, that problem is solved
1864 and queuing up the auxtrace buffers in advanc
1865
1866 For pipe mode, the order of events and timest
1867 be messed up.
1868
1869
1870 EXAMPLE
1871 -------
1872
1873 Examples can be found on perf wiki page "Perf
1874
1875 https://perf.wiki.kernel.org/index.php/Perf_t
1876
1877
1878 SEE ALSO
1879 --------
1880
1881 linkperf:perf-record[1], linkperf:perf-script
1882 linkperf:perf-inject[1]
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.