1 .. SPDX-License-Identifier: GPL-2.0
2
3 ======================================
4 HiSilicon PCIe Tune and Trace device
5 ======================================
6
7 Introduction
8 ============
9
10 HiSilicon PCIe tune and trace device (PTT) is
11 integrated Endpoint (RCiEP) device, providing
12 to dynamically monitor and tune the PCIe link'
13 and trace the TLP headers (trace). The two fun
14 but is recommended to use them together to ana
15 PCIe link's performance.
16
17 On Kunpeng 930 SoC, the PCIe Root Complex is c
18 PCIe cores. Each PCIe core includes several Ro
19 RCiEP, like below. The PTT device is capable o
20 tracing the links of the PCIe core.
21 ::
22
23 +--------------Core 0-------+
24 | | [ PTT ] |
25 | | [Root Port]---[Endpo
26 | | [Root Port]---[Endpo
27 | | [Root Port]---[Endpo
28 Root Complex |------Core 1-------+
29 | | [ PTT ] |
30 | | [Root Port]---[ Swit
31 | | [Root Port]---[Endpo
32 | | [Root Port]---[Endpo
33 +---------------------------+
34
35 The PTT device driver registers one PMU device
36 The name of each PTT device is composed of 'hi
37 the id of the SICL and the Core where it locat
38 SoC encapsulates multiple CPU dies (SCCL, Supe
39 IO dies (SICL, Super I/O Cluster), where there
40 Complex for each SICL.
41 ::
42
43 /sys/bus/event_source/devices/hisi_ptt<sic
44
45 Tune
46 ====
47
48 PTT tune is designed for monitoring and adjust
49 Currently we support events in 2 classes. The
50 covers the PCIe core to which the PTT device b
51
52 Each event is presented as a file under $(PTT
53 a simple open/read/write/close cycle will be u
54 ::
55
56 $ cd /sys/bus/event_source/devices/hisi_pt
57 $ ls
58 qos_tx_cpl qos_tx_np qos_tx_p
59 tx_path_rx_req_alloc_buf_level
60 tx_path_tx_req_alloc_buf_level
61 $ cat qos_tx_dp
62 1
63 $ echo 2 > qos_tx_dp
64 $ cat qos_tx_dp
65 2
66
67 Current value (numerical value) of the event c
68 from the file, and the desired value written t
69
70 1. Tx Path QoS Control
71 ------------------------
72
73 The following files are provided to tune the Q
74 the PCIe core.
75
76 - qos_tx_cpl: weight of Tx completion TLPs
77 - qos_tx_np: weight of Tx non-posted TLPs
78 - qos_tx_p: weight of Tx posted TLPs
79
80 The weight influences the proportion of certai
81 For example, for the storage scenario, increas
82 of the completion packets on the link to enhan
83 more completions are consumed.
84
85 The available tune data of these events is [0,
86 Writing a negative value will return an error,
87 values will be converted to 2. Note that the e
88 indicates a probable level, but is not precise
89
90 2. Tx Path Buffer Control
91 -------------------------
92
93 Following files are provided to tune the buffe
94
95 - rx_alloc_buf_level: watermark of Rx requeste
96 - tx_alloc_buf_level: watermark of Tx requeste
97
98 These events influence the watermark of the bu
99 type. Rx means the inbound while Tx means outb
100 be stored in the buffer first and then transmi
101 watermark reached or when timed out. For a bus
102 increase the related buffer watermark to avoid
103 thus enhance the performance. In most cases ju
104
105 The available tune data of above events is [0,
106 Writing a negative value will return an error,
107 values will be converted to 2. Note that the e
108 indicates a probable level, but is not precise
109
110 Trace
111 =====
112
113 PTT trace is designed for dumping the TLP head
114 can be used to analyze the transactions and us
115 Link. You can choose to filter the traced head
116 or those downstream of a set of Root Ports on
117 device. It's also supported to trace the heade
118 certain direction.
119
120 You can use the perf command `perf record` to
121 trace and get the data. It's also supported to
122 data with `perf report`. The control parameter
123 as event code for each events, which will be f
124 An example usage is like
125 ::
126
127 $ perf record -e hisi_ptt0_2/filter=0x8000
128 format=1/ -- sleep 5
129
130 This will trace the TLP headers downstream roo
131 code for event 'filter' is 0x80001) with type
132 direction of inbound and traced data format of
133
134 1. Filter
135 ---------
136
137 The TLP headers to trace can be filtered by th
138 of the Endpoint, which are located on the same
139 set the filter by specifying the `filter` para
140 the trace. The parameter value is 20 bit. Bit
141 1 for Root Port filter and 0 for Requester fil
142 filter value. The value for a Root Port is a m
143 calculated from its PCI Slot ID as (slotid & 7
144 is the Requester ID (Device ID of the PCIe fun
145 reserved for extension.
146
147 For example, if the desired filter is Endpoint
148 value will be 0x00101. If the desired filter i
149 then filter value is calculated as 0x80001.
150
151 The driver also presents every supported Root
152 sysfs. Each filter will be an individual file
153 device name (domain:bus:device.function). The
154 under $(PTT PMU dir)/root_port_filters and fil
155 are under $(PTT PMU dir)/requester_filters.
156
157 Note that multiple Root Ports can be specified
158 Endpoint function can be specified in one trac
159 and function at the same time is not supported
160 available filters and will check the invalid i
161
162 The available filters will be dynamically upda
163 get correct filter information when hotplug ev
164 remove/rescan the devices.
165
166 2. Type
167 -------
168
169 You can trace the TLP headers of certain types
170 parameter, which is required to start the trac
171 8 bit. Current supported types and related val
172
173 - 8'b00000001: posted requests (P)
174 - 8'b00000010: non-posted requests (NP)
175 - 8'b00000100: completions (CPL)
176
177 You can specify multiple types when tracing in
178 specify one when tracing outbound TLP headers.
179
180 3. Direction
181 ------------
182
183 You can trace the TLP headers from certain dir
184 to the Root Port or the PCIe core, by specifyi
185 This is optional and the default parameter is
186 is 4 bit. When the desired format is 4DW, dire
187 supported are shown below:
188
189 - 4'b0000: inbound TLPs (P, NP, CPL)
190 - 4'b0001: outbound TLPs (P, NP, CPL)
191 - 4'b0010: outbound TLPs (P, NP, CPL) and inbo
192 - 4'b0011: outbound TLPs (P, NP, CPL) and inbo
193
194 When the desired format is 8DW, directions and
195 shown below:
196
197 - 4'b0000: reserved
198 - 4'b0001: outbound TLPs (P, NP, CPL)
199 - 4'b0010: inbound TLPs (P, NP, CPL B)
200 - 4'b0011: inbound TLPs (CPL A)
201
202 Inbound completions are classified into two ty
203
204 - completion A (CPL A): completion of CHI/DMA/
205 - completion B (CPL B): completion of DMA remo
206
207 4. Format
208 --------------
209
210 You can change the format of the traced TLP he
211 `format` parameter. The default format is 4DW.
212 Current supported formats and related values a
213
214 - 4'b0000: 4DW length per TLP header
215 - 4'b0001: 8DW length per TLP header
216
217 The traced TLP header format is different from
218
219 When using the 8DW data format, the entire TLP
220 (Header DW0-3 shown below). For example, the T
221 Reads with 64-bit addresses is shown in PCIe r
222 the header for Configuration Requests is shown
223
224 In addition, 8DW trace buffer entries contain
225 possibly a prefix for a PASID TLP prefix (see
226 Otherwise this field will be all 0.
227
228 The bit[31:11] of DW0 is always 0x1fffff, whic
229 used to distinguish the data format. 8DW forma
230 ::
231
232 bits [ 31:11
233 |------------------------------------
234 DW0 [ 0x1fffff
235 DW1 [ Prefix
236 DW2 [ Header DW0
237 DW3 [ Header DW1
238 DW4 [ Header DW2
239 DW5 [ Header DW3
240 DW6 [ Reserved (0x0)
241 DW7 [ Time
242
243 When using the 4DW data format, DW0 of the tra
244 contains selected fields of DW0 of the TLP, to
245 timestamp. DW1-DW3 of the trace buffer entry
246 directly from the TLP header.
247
248 4DW format is like
249 ::
250
251 bits [31:30] [ 29:25 ][24][23][22][21][
252 |-----|---------|---|---|---|---|----
253 DW0 [ Fmt ][ Type ][T9][T8][TH][SO][
254 DW1 [ Header DW1
255 DW2 [ Header DW2
256 DW3 [ Header DW3
257
258 5. Memory Management
259 --------------------
260
261 The traced TLP headers will be written to the
262 by the driver. The hardware accepts 4 DMA addr
263 and writes the buffer sequentially like below.
264 finished and the trace is still on, it will re
265 ::
266
267 +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2
268 +-----------------------------------------
269
270 Driver will allocate each DMA buffer of 4MiB.
271 will be copied to the perf AUX buffer allocate
272 Once the AUX buffer is full while the trace is
273 will commit the AUX buffer first and then appl
274 the same size. The size of AUX buffer is defau
275 adjust the size by specifying the `-m` paramet
276
277 6. Decoding
278 -----------
279
280 You can decode the traced data with `perf repo
281 only support to dump the raw trace data). The
282 according to the format described previously (
283 ::
284
285 [...perf headers and other information]
286 . ... HISI PTT data: size 4194304 bytes
287 . 00000000: 00 00 00 00
288 . 00000004: 01 00 00 60
289 . 00000008: 0f 1e 00 01
290 . 0000000c: 04 00 00 00
291 . 00000010: 40 00 81 02
292 . 00000014: 33 c0 04 00
293 . 00000020: 00 00 00 00
294 . 00000024: 01 00 00 60
295 . 00000028: 0f 1e 00 01
296 . 0000002c: 04 00 00 00
297 . 00000030: 40 00 81 02
298 . 00000034: 02 00 00 00
299 . 00000040: 00 00 00 00
300 . 00000044: 01 00 00 60
301 . 00000048: 0f 1e 00 01
302 . 0000004c: 04 00 00 00
303 . 00000050: 40 00 81 02
304 [...]
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.